How do you work around the client restrictions for opening encrypted documents?

Hi, the reply of Ajeeth_Muthu give you an overview of feasable clients. Purview Custom Encrypted documents represent the highest wall to prevent access to content. In most cases the number of custom enrcypted documents, in organisations i was in contact with, represent a low one-digit percentage over all. 
We solve this isue with following cornerstones:

• custom protection only really highly confidential documents
• external consumers without MSFT ID use Guest Accounts in our tenant
• Guest account governance management and MS Teams Lifecycle Management using EasyLife365
• Guest Accounts do not have a mailbox and do not require a Office License to use the webclient
• Access to documents from unmanaged devices, mean all devices not managed by your tenant, are restricted to webb-only access, no download.
• Collaboration possible for this highly protected content.

There are restrictions, yes - and with such rigid restrictions there are always exceptions. Exception handling in our case underlies a strict governace and audit trail. To keep labeling on documents active you could introduce a dedicated label for sensitive content without encryption and only allow specific users to use this label or a technical user alone to force all decryption through downlabeling with justification in an application logic (eg. Powerautomate) to manage your auditing. This way documents remain labeled and tracable though metadata.

Kind regards