%3CLINGO-SUB%20id%3D%22lingo-sub-867171%22%20slang%3D%22en-US%22%3EService%20Fabric%20Node%20Down%20being%20unable%20to%20read%20private%20key%20from%20certificate%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-867171%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSTRONG%3ESymptom%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3ESometime%20in%20your%20SF%20clusters%2C%20you%20may%20observe%20that%20nodes%20being%20down%20with%20%E2%80%9Cunable%20to%20read%20private%20key%20from%20certificate%E2%80%9D%20and%20checking%20the%20Service%20Fabric%20Admin%20logs%2C%20you%20will%20see%20error%20like%20below%3A%3C%2FP%3E%0A%3CP%3E%3CEM%3ESecurityUtility%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EFailed%20to%20get%20the%20Certificate's%20private%20key.%20Thumbprint%3AXXXXXXXXXXXXXXXXX%E2%80%A6..XXXXXXXX.%20Error%3A%20E_FAIL%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ECryptoUtility%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ECryptAcquireCertificatePrivateKey%20failed.%20Error%3A%200x80090014%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EError%20code%3A%200x80090014%20meaning%20%E2%80%9CInvalid%20provider%20type%20specified.%E2%80%9D%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%26nbsp%3B%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EMitigation%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EHence%20to%20confirm%20if%20you%E2%80%99re%20hitting%20into%20the%20issue%20where%20the%20certificate%20can%E2%80%99t%20be%20ACLed%20by%20the%20SF%20runtime%20due%20to%20the%20fact%20that%20the%20certificate%20being%20generated%20with%20an%20unsupported%20provider%2C%20then%20please%20try%20the%20following%20command%20in%20PowerShell%20by%20logging%20into%20the%20node%2C%20from%20which%20the%20error%20is%20thrown.%3C%2FP%3E%0A%3CP%3E%3CEM%3Ecd%20Cert%3A%5CLocalMachine%5CMy%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3Ecertutil%20-store%20my%3C%2FEM%3E%3C%2FP%3E%0A%3CDIV%20id%3D%22tinyMceEditorclipboard_image_0%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CP%3EThis%20will%20dump%20all%20the%20certificates%20with%20its%20details%20from%20%E2%80%98my%E2%80%99%20store%2C%20you%20can%20now%20look%20for%20your%20concerned%20certificate%20with%20the%20help%20of%20the%20thumbprint%20and%20check%20what%20provider%20the%20certificate%20has%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20see%20the%20provider%20of%20the%20certificate%20contains%20something%20like%20below%2C%20then%20this%20is%20indeed%20a%20CNG%20certificate%20issued%20with%20a%20Key%20Storage%20Provider.%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EProvider%20%3D%20Microsoft%20Software%20Key%20Storage%20Provider%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CDIV%20id%3D%22tinyMceEditorBraja_0%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorBraja_1%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22SF-CNGCheck.png%22%20style%3D%22width%3A%20704px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F191952iCA816B95A47AA05F%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22SF-CNGCheck.png%22%20alt%3D%22SF-CNGCheck.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAs%20of%20now%20SF%20runtime%20supports%20certificates%20with%20providers%20as%20mentioned%20here%20-%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fwindows%2Fwin32%2Fseccrypto%2Fmicrosoft-cryptographic-service-providers%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fwindows%2Fwin32%2Fseccrypto%2Fmicrosoft-cryptographic-service-providers%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EHence%2C%20you%20might%20be%20using%20a%20self-signed%20certificate%20which%20was%20generated%20without%20any%20providers%20specified%2C%20had%20used%20a%20CNG%20provider%20instead.%20If%20this%20is%20the%20case%2C%20then%20you%20may%20need%20to%20create%20another%20certificate%20with%20a%20supported%20provider%20that%20you%20can%20associate%20with%20this%20cluster%20using%20following%20command%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENew-SelfSignedCertificate%20-NotBefore%20'%3CVALUES%3E'%20-NotAfter%20'%3CVALUES%3E'%20-DnsName%20'%3CDNSNAME%3E'%20-CertStoreLocation%20Cert%3A%5CLocalMachine%5CMy%20-Provider%20%22Microsoft%20Enhanced%20RSA%20and%20AES%20Cryptographic%20Provider%22%20-KeyExportPolicy%20ExportableEncrypted%20-Subject%20%22%3CENTER%20subject%3D%22%22%3E%22%3C%2FENTER%3E%3C%2FDNSNAME%3E%3C%2FVALUES%3E%3C%2FVALUES%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAfter%20creating%20this%20certificate%2C%20you%20can%20add%20this%20new%20certificate%20as%20the%20secondary%20certificate%20to%20the%20cluster%20and%20then%20swap%20this%20with%20the%20primary%20to%20avoid%20any%20down%20time%20-%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fservice-fabric%2Fservice-fabric-cluster-security-update-certs-azure%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fservice-fabric%2Fservice-fabric-cluster-security-update-certs-azure%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-867171%22%20slang%3D%22en-US%22%3E%3CP%3EService%20Fabric%20Node%20Down%20being%20unable%20to%20read%20private%20key%20from%20certificate%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-867171%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Service%20Fabric%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EService%20Fabric%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EService%20Fabric%20node%20down%20due%20to%20self-signed%20Certificate%20issue%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Microsoft

Symptom

Sometime in your SF clusters, you may observe that nodes being down with “unable to read private key from certificate” and checking the Service Fabric Admin logs, you will see error like below:

SecurityUtility

Failed to get the Certificate's private key. Thumbprint:XXXXXXXXXXXXXXXXX…..XXXXXXXX. Error: E_FAIL

CryptoUtility

CryptAcquireCertificatePrivateKey failed. Error: 0x80090014

Error code: 0x80090014 meaning “Invalid provider type specified.”

 

Mitigation

Hence to confirm if you’re hitting into the issue where the certificate can’t be ACLed by the SF runtime due to the fact that the certificate being generated with an unsupported provider, then please try the following command in PowerShell by logging into the node, from which the error is thrown.

cd Cert:\LocalMachine\My

certutil -store my

 

This will dump all the certificates with its details from ‘my’ store, you can now look for your concerned certificate with the help of the thumbprint and check what provider the certificate has:

 

If you see the provider of the certificate contains something like below, then this is indeed a CNG certificate issued with a Key Storage Provider.

Provider = Microsoft Software Key Storage Provider

 
 

SF-CNGCheck.png

 

As of now SF runtime supports certificates with providers as mentioned here - https://docs.microsoft.com/en-us/windows/win32/seccrypto/microsoft-cryptographic-service-providers

Hence, you might be using a self-signed certificate which was generated without any providers specified, had used a CNG provider instead. If this is the case, then you may need to create another certificate with a supported provider that you can associate with this cluster using following command:

 

New-SelfSignedCertificate -NotBefore '<Values>' -NotAfter '<Values>' -DnsName '<DnsName>' -CertStoreLocation Cert:\LocalMachine\My -Provider "Microsoft Enhanced RSA and AES Cryptographic Provider" -KeyExportPolicy ExportableEncrypted -Subject "<Enter Subject>"

 

After creating this certificate, you can add this new certificate as the secondary certificate to the cluster and then swap this with the primary to avoid any down time - https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-security-update-certs-a....