Error: No certificates were found that met all the given criteria.

This has me stumped for several days. I keep getting this error. My pfx has a password on it and when I type it in it installs it to the certificate store. It does this saying it has to because it has a password. Great. So I click OK on the prompt but when I build it says the following. I have tried to fix this 

Severity Code Description Project File Line Suppression State
Error Failed to sign 'E:\data\development\NET-ScheduFlow\WebsiteSideLoadedInstaller\bin\AnyCPU\Release\WebsiteSideLoadedInstaller_1.0.1.0_AnyCPU.msix'. SignTool Error: No certificates were found that met all the given criteria. WebsiteSideLoadedInstaller C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets 4746

When I play around with removing and adding I sometimes get an error that the 

<Identity
Name="..."
Publisher="CN=[my stuff]

Does not match the certificate. I can't get that error to happen because the one described above is happening. Should I attempt not to use from certificate store and just use the file without a password. Anyone have a clue what to do.