Custom theme randomly not applying

I am having an issue where, sometimes, my custom theme colors will not apply to my collection site.

I have several custom themes uploaded via SharePoint Online Powershell, and have always used them without any major concern. However, it seems like ~10% of the time I will open a new page and my custom colors will not be applied to the site. The first thing I can notice is the navigation header, which will have a blue color, but also I have an extension that uses the custom theme and when this error happens my custom colors won't be applied there, either. It's like SharePoint couldn't get my color pallete and then my site gets a fallback one.

Anyone also having this issue?

Thanks