I haven’t run into this issue and have the same use case. When you setup the lookup filter, you pick the object type. I haven’t been able to find any custom logic for polymorphic lookup filters, however, so I believe you’re limited to AND conditions. Here’s a screenshot on the case owner field where there are different filters for the group and the user, filtering out only active internal users:
