Encrypted labelled documents saved in SharePoint not working across organisations

I am working with a client to set up Sensitivity labels among other things for their organisation. I have created and published their required Labels: Public, Confidential and Restricted. With Restricted we have enabled the encryption and chose the option to let the document owner grant permission per document. I have also set up the Labels for SharePoints. If a user in the Organisation is creating a restricted document, granting my external email address read and change access, and then saving it in a SharePoint site. When I try to access the document it gives me the error message below: 

The same issue occurs if I create a document with encryption within my Org and upload it to their SharePoint, they receive the error. We set up the SharePoint settings to allow guests to be invited into sites and I am able to access a Restricted site without any issues but documents always cause this error.

Has anyone else encountered this or know of this issue? How do we resolve this?