Sensitivity Label error for PDFs

We have enabled Sensitivity Label PDF support for SharePoint Online in our test environment. In my testing I have found that I can apply a sensitivity label to most PDF files in SharePoint, but some PDF files throw an error when I try to apply the label: "Error: Sensitivity label cannot be applied at this time due to an internal error."

I have worked with Microsoft and they have narrowed it down to it being an issue with the files themselves, not a tenant level issue, not a policy issue or a sensitivity label issue.

Has anyone experienced this? What is it about a given PDF file that could cause the error with Sensitivity Labels?