Home
%3CLINGO-SUB%20id%3D%22lingo-sub-867171%22%20slang%3D%22en-US%22%3EService%20Fabric%20Node%20Down%20being%20unable%20to%20read%20private%20key%20from%20certificate%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-867171%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSTRONG%3ESymptom%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3ESometime%20in%20your%20SF%20clusters%2C%20you%20may%20observe%20that%20nodes%20being%20down%20with%20%E2%80%9Cunable%20to%20read%20private%20key%20from%20certificate%E2%80%9D%20and%20checking%20the%20Service%20Fabric%20Admin%20logs%2C%20you%20will%20see%20error%20like%20below%3A%3C%2FP%3E%0A%3CP%3E%3CEM%3ESecurityUtility%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EFailed%20to%20get%20the%20Certificate's%20private%20key.%20Thumbprint%3AXXXXXXXXXXXXXXXXX%E2%80%A6..XXXXXXXX.%20Error%3A%20E_FAIL%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ECryptoUtility%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ECryptAcquireCertificatePrivateKey%20failed.%20Error%3A%200x80090014%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EError%20code%3A%200x80090014%20meaning%20%E2%80%9CInvalid%20provider%20type%20specified.%E2%80%9D%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%26nbsp%3B%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EMitigation%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EHence%20to%20confirm%20if%20you%E2%80%99re%20hitting%20into%20the%20issue%20where%20the%20certificate%20can%E2%80%99t%20be%20ACLed%20by%20the%20SF%20runtime%20due%20to%20the%20fact%20that%20the%20certificate%20being%20generated%20with%20an%20unsupported%20provider%2C%20then%20please%20try%20the%20following%20command%20in%20PowerShell%20by%20logging%20into%20the%20node%2C%20from%20which%20the%20error%20is%20thrown.%3C%2FP%3E%0A%3CP%3E%3CEM%3Ecd%20Cert%3A%5CLocalMachine%5CMy%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3Ecertutil%20-v%20-store%20my%20%7C%20findstr%20-i%20provider%3C%2FEM%3E%3C%2FP%3E%0A%3CDIV%20id%3D%22tinyMceEditorclipboard_image_0%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20558px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F132769iCAD5C935803256C1%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22SF-image.png%22%20title%3D%22SF-image.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EPlease%20see%20if%20the%20output%20contains%20something%20like%20this%3A%3CBR%20%2F%3E%3CSTRONG%3EProvider%20%3D%20Microsoft%20Software%20Key%20Storage%20Provider%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20we%20see%20the%20provider%20mentioned%20above%2C%20then%20this%20is%20indeed%20a%20CNG%20certificate%20issued%20with%20a%20Key%20Storage%20Provider.%3C%2FP%3E%0A%3CP%3EAs%20of%20now%20SF%20runtime%20supports%20certificates%20with%20providers%20as%20mentioned%20here%20-%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fwindows%2Fwin32%2Fseccrypto%2Fmicrosoft-cryptographic-service-providers%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fwindows%2Fwin32%2Fseccrypto%2Fmicrosoft-cryptographic-service-providers%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EHence%2C%20you%20might%20be%20using%20a%20self-signed%20certificate%20which%20was%20generated%20without%20any%20providers%20specified%2C%20will%20use%20a%20CNG%20provider.%20If%20this%20is%20the%20case%2C%20then%20you%20may%20need%20to%20create%20another%20certificate%20with%20a%20supported%20provider%20that%20you%20can%20associate%20with%20this%20cluster%20using%20following%20command%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENew-SelfSignedCertificate%20-NotBefore%20'%3CVALUES%3E'%20-NotAfter%20'%3CVALUES%3E'%20-DnsName%20'%3CDNSNAME%3E'%20-CertStoreLocation%20Cert%3A%5CLocalMachine%5CMy%20-Provider%20%22Microsoft%20Enhanced%20RSA%20and%20AES%20Cryptographic%20Provider%22%20-KeyExportPolicy%20ExportableEncrypted%20-Subject%20%22%3CENTER%20subject%3D%22%22%3E%22%3C%2FENTER%3E%3C%2FDNSNAME%3E%3C%2FVALUES%3E%3C%2FVALUES%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAfter%20creating%20this%20certificate%2C%20you%20can%20add%20this%20new%20certificate%20as%20the%20secondary%20certificate%20to%20the%20cluster%20and%20then%20swap%20this%20with%20the%20primary%20to%20avoid%20any%20down%20time%20-%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fservice-fabric%2Fservice-fabric-cluster-security-update-certs-azure%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fservice-fabric%2Fservice-fabric-cluster-security-update-certs-azure%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-867171%22%20slang%3D%22en-US%22%3E%3CP%3EService%20Fabric%20Node%20Down%20being%20unable%20to%20read%20private%20key%20from%20certificate%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-867171%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Service%20Fabric%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EService%20Fabric%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EService%20Fabric%20node%20down%20due%20to%20self-signed%20Certificate%20issue%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Microsoft

Symptom

Sometime in your SF clusters, you may observe that nodes being down with “unable to read private key from certificate” and checking the Service Fabric Admin logs, you will see error like below:

SecurityUtility

Failed to get the Certificate's private key. Thumbprint:XXXXXXXXXXXXXXXXX…..XXXXXXXX. Error: E_FAIL

CryptoUtility

CryptAcquireCertificatePrivateKey failed. Error: 0x80090014

Error code: 0x80090014 meaning “Invalid provider type specified.”

 

Mitigation

Hence to confirm if you’re hitting into the issue where the certificate can’t be ACLed by the SF runtime due to the fact that the certificate being generated with an unsupported provider, then please try the following command in PowerShell by logging into the node, from which the error is thrown.

cd Cert:\LocalMachine\My

certutil -v -store my | findstr -i provider

 

SF-image.png

Please see if the output contains something like this:
Provider = Microsoft Software Key Storage Provider

 

If we see the provider mentioned above, then this is indeed a CNG certificate issued with a Key Storage Provider.

As of now SF runtime supports certificates with providers as mentioned here - https://docs.microsoft.com/en-us/windows/win32/seccrypto/microsoft-cryptographic-service-providers

Hence, you might be using a self-signed certificate which was generated without any providers specified, will use a CNG provider. If this is the case, then you may need to create another certificate with a supported provider that you can associate with this cluster using following command:

 

New-SelfSignedCertificate -NotBefore '<Values>' -NotAfter '<Values>' -DnsName '<DnsName>' -CertStoreLocation Cert:\LocalMachine\My -Provider "Microsoft Enhanced RSA and AES Cryptographic Provider" -KeyExportPolicy ExportableEncrypted -Subject "<Enter Subject>"

 

After creating this certificate, you can add this new certificate as the secondary certificate to the cluster and then swap this with the primary to avoid any down time - https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-security-update-certs-a....