However, whenever there is an active user with the same display name like the orphaned one PowerShell throws an error:
Remove-MailboxFolderPermission: Ex9E65A2|Microsoft.Exchange.Configuration.Tasks.ManagementObjectAmbiguousException|The operation couldn't be performed because '<display name>' matches multiple entries.
Is there any way to get rid of these orphaned ACEs?
Is this for Exchange Online or? I'm afraid you don't have any alternative options therein, as you likely don't have any other identifier you can provide for the orphaned user. Well, you can always use MFCMAPI or similar tools, or edit the folder permissions via EWS.