Hide “Add event” button in calendar web part for read-only users

On our modern sharepoint online site, we have a calendar web part. For visitors, i.e. read-only users, the "Add event" button is grayed out. For our visitors, we'd rather remove the "Add event" button instead of just disabling it.

Is there a way to remove this button for users who don't have access to actually create new events?