How to stop end user creating public groups in office 365

To answer the additional questions,

1) You cannot limit users to creating just Private groups. It's all or nothing.

2) No, only the default limits apply, we cannot define any additional ones.

3) DG creation settings are not related to the O365 Group creation ones. The former are indeed controlled by the OWA policies. Removing those two checkboxes will however disallow users to manage existing groups they own, which might not be the desired behavior. If you only want to block creation of DGs, a better approach is to create a new OWA policy and remove the New-DistributionGroup from it.