What happens if I create a dynamic membership group without enough P1 licenses?

Microsoft doesn't enforce licensing requirements in code, thus many of the features that "require" a specific license will work just fine even without one. In your scenario, the group be created with all members.

Also, there's no way Microsoft will ever auto-purchase/charge you for any additional licenses, that will create all sorts of problems for them. It is your responsibility as the admin/owner to make sure you comply with the license requirements.