Service Fabric Clusters secured with certificates issued by DigiCert - at risk of undergoing outage

Published 11-24-2020 04:07 PM 2,335 Views
Microsoft

What is the Certificate Validation Issue? 

DigiCert introduced a new CA which reuses the signing key of an existing and still-valid CA. This means there are 2 different CA certificates in circulation, and either can be included in the chain built for a certificate signed by this shared key. Existing certificates declared in Service Fabric clusters by subject with issuer pinning are at risk of spontaneously failing validation.  

 

How to identify if your cluster is susceptible to the Certificate Validation Issue? 

This issue affects any SF cluster that uses a Cluster certificate that is a DigiCert-issued X509 certificate(s), and which meets both of the following conditions:

 

a) Cluster certificate is declared by common name with issuer pinning, and the list of issuer thumbprints specifies one - but not both - of the following thumbprints: 1fb86b1168ec743154062e8c9cc5b171a4b7ccb4, 626d44e704d1ceabe3bf0d53397464ac8080142c.
 

b) The cluster certificate is signed by one of the 2 conflicting CAs; you can determine if that is the case either by examining the certificate extensions, or its chain, as follows:

  • The certificate’s Authority Key Identifier extension (AKI, OId: 2.5.29.35) matches KeyID=0f80611c823161d52f28e78d4638b42ce1c6d9e2, or
  • The certificate's issuer is either of the following DigiCert SHA2 Secure Server CAs:
    • SHA1 thumbprint 1f:b8:6b:11:68:ec:74:31:54:06:2e:8c:9c:c5:b1:71:a4:b7:cc:b4
      valid until 08/Mar/2023
      serial #01:fd:a3:eb:6e:ca:75:c8:88:43:8b:72:4b:cf:bc:91
    • SHA1 thumbprint 62:6d:44:e7:04:d1:ce:ab:e3:bf:0d:53:39:74:64:ac:80:80:14:2c
      valid until 22/Sep/2030
      serial #02:74:2e:aa:17:ca:8e:21:c7:17:bb:1f:fc:fd:0c:a0

The cluster certificate configuration can be found in the ARM resource of your Service Fabric cluster. If your cluster is not configured using the above properties, you may disregard the rest of this post.  

 

Symptoms in impacted environments 

  • One or more cluster nodes appear down/unhealthy. 
  • Cluster is unreachable, whether from the Azure portal or directly (SFX/other clients). 
  • Event logs show errors like: “authorization failure: CertificateNotMatched”. 
  • Pending upgrades are not progressing/appear to be stuck.

Required Action 

  • Follow the Trouble Shooting guide with Mitigation steps: Troubleshooting Guide  
  • Mitigation specified in the TSG must be applied by you.  

 

If you have any questions or concerns, please contact us by opening a support request. In addition, here are your general support options for Service Fabric: Learn about Azure Service Fabric Support options - Azure Service Fabric | Microsoft Docs

%3CLINGO-SUB%20id%3D%22lingo-sub-1928992%22%20slang%3D%22en-US%22%3EService%20Fabric%20Clusters%20secured%20with%20certificates%20issued%20by%20DigiCert%20-%20at%20risk%20of%20undergoing%20outage%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1928992%22%20slang%3D%22en-US%22%3E%3CH2%20id%3D%22toc-hId--1185317759%22%20id%3D%22toc-hId--1185338876%22%3E%3CSPAN%3EWhat%20is%20the%20Certificate%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3EValidation%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3BIssue%3F%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FH2%3E%0A%3CP%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B134233117%26quot%3B%3Atrue%2C%26quot%3B134233118%26quot%3B%3Atrue%2C%26quot%3B201341983%26quot%3B%3A2%2C%26quot%3B335559740%26quot%3B%3A300%7D%22%3EDigiCert%20introduced%20a%20new%20CA%20which%20reuses%20the%20signing%20key%20of%20an%20existing%20and%20still-valid%20CA.%20This%20means%20there%20are%202%20different%20CA%20certificates%20in%20circulation%2C%20and%20either%20can%20be%20included%20in%20the%20chain%20built%20for%20a%20certificate%20signed%20by%20this%20shared%20key.%20Existing%20certificates%20declared%20in%20Service%20Fabric%20clusters%20by%20subject%20with%20issuer%20pinning%20are%20at%20risk%20of%20spontaneously%20failing%20validation.%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1302195074%22%20id%3D%22toc-hId-1302173957%22%3E%3CSPAN%3EHow%20to%20identify%20if%20your%20cluster%26nbsp%3Bis%26nbsp%3Bsusceptible%26nbsp%3Bto%26nbsp%3Bthe%20Certificate%20Validation%20Issue%3F%26nbsp%3B%3C%2FSPAN%3E%3C%2FH2%3E%0A%3CP%3E%3CSPAN%20data-contrast%3D%22none%22%3EThis%20issue%20affects%20any%26nbsp%3BSF%26nbsp%3Bcluster%20that%26nbsp%3Buses%20a%26nbsp%3BCluster%20certificate%26nbsp%3Bthat%20is%20a%26nbsp%3B%3CSTRONG%3EDigiCert-issued%26nbsp%3BX509%3C%2FSTRONG%3E%26nbsp%3Bcertificate(s)%2C%26nbsp%3B%3C%2FSPAN%3Eand%20which%20meets%20both%20of%20the%20following%20conditions%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CDIV%3E%0A%3CDIV%3E%3CSTRONG%3Ea)%3C%2FSTRONG%3E%26nbsp%3B%3CSPAN%20style%3D%22font-family%3A%20inherit%3B%22%3ECluster%20certificate%20is%20declared%20by%20common%20name%20with%20issuer%20pinning%2C%20and%20the%20list%20of%20issuer%20thumbprints%20specifies%20one%20-%20but%20not%20both%20-%20of%20the%20following%20thumbprints%3A%201fb86b1168ec743154062e8c9cc5b171a4b7ccb4%2C%20626d44e704d1ceabe3bf0d53397464ac8080142c%3CSTRONG%3E.%3C%2FSTRONG%3E%3C%2FSPAN%3E%3C%2FDIV%3E%0A%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CP%3E%3CSPAN%3E%3CSTRONG%3Eb)%3C%2FSTRONG%3E%20T%3C%2FSPAN%3Ehe%20cluster%20certificate%20is%20signed%20by%20one%20of%20the%202%20conflicting%20CAs%3B%20you%20can%20determine%20if%20that%20is%20the%20case%20either%20by%20examining%20the%20certificate%20extensions%2C%20or%20its%20chain%2C%20as%20follows%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EThe%20certificate%E2%80%99s%20Authority%20Key%20Identifier%20extension%20(AKI%2C%20OId%3A%202.5.29.35)%20matches%20KeyID%3D0f80611c823161d52f28e78d4638b42ce1c6d9e2%2C%20or%3C%2FLI%3E%0A%3CLI%3EThe%20certificate's%20issuer%20is%20either%20of%20the%20following%20%3CA%20href%3D%22https%3A%2F%2Fwww.digicert.com%2Fkb%2Fdigicert-root-certificates.htm%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3EDigiCert%20SHA2%20Secure%20Server%20CAs%3C%2FA%3E%3A%3CUL%3E%0A%3CLI%3ESHA1%20thumbprint%201f%3Ab8%3A6b%3A11%3A68%3Aec%3A74%3A31%3A54%3A06%3A2e%3A8c%3A9c%3Ac5%3Ab1%3A71%3Aa4%3Ab7%3Acc%3Ab4%3CBR%20%2F%3Evalid%20until%2008%2FMar%2F2023%3CBR%20%2F%3Eserial%20%2301%3Afd%3Aa3%3Aeb%3A6e%3Aca%3A75%3Ac8%3A88%3A43%3A8b%3A72%3A4b%3Acf%3Abc%3A91%3C%2FLI%3E%0A%3CLI%3ESHA1%20thumbprint%2062%3A6d%3A44%3Ae7%3A04%3Ad1%3Ace%3Aab%3Ae3%3Abf%3A0d%3A53%3A39%3A74%3A64%3Aac%3A80%3A80%3A14%3A2c%3CBR%20%2F%3Evalid%20until%2022%2FSep%2F2030%3CBR%20%2F%3Eserial%20%2302%3A74%3A2e%3Aaa%3A17%3Aca%3A8e%3A21%3Ac7%3A17%3Abb%3A1f%3Afc%3Afd%3A0c%3Aa0%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3C%2FDIV%3E%0A%3CP%3E%3CSPAN%20data-contrast%3D%22none%22%3EThe%20cluster%20certificate%20configuration%20can%20be%20found%20in%20the%20ARM%20resource%20of%20your%20Service%20Fabric%20cluster.%20If%20your%20cluster%20is%20not%20configured%20using%20the%20above%20properties%2C%20you%20may%20disregard%20the%20rest%20of%20this%20post.%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--505259389%22%20id%3D%22toc-hId--505280506%22%3E%3CSPAN%3ESymptoms%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Ein%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Eimpacted%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Benvironments%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FH2%3E%0A%3CUL%3E%0A%3CLI%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%20One%20or%26nbsp%3Bmore%26nbsp%3Bcluster%20nodes%20appear%20down%2Funhealthy.%26nbsp%3B%3C%2FSPAN%3E%3C%2FLI%3E%0A%3CLI%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%20Cluster%20is%20unreachable%2C%20whether%20from%20the%20Azure%20portal%20or%20directly%20(SFX%2Fother%20clients).%26nbsp%3B%3C%2FSPAN%3E%3C%2FLI%3E%0A%3CLI%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%20Event%20logs%20show%20errors%20like%3A%20%E2%80%9Cauthorization%20failure%3A%26nbsp%3BCertificateNotMatched%E2%80%9D.%26nbsp%3B%3C%2FSPAN%3E%3C%2FLI%3E%0A%3CLI%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%20Pending%20upgrades%20are%20not%20progressing%2Fappear%20to%20be%26nbsp%3Bstuck.%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FSPAN%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CH2%20id%3D%22toc-hId-1982253444%22%20id%3D%22toc-hId-1982232327%22%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A0%2C%26quot%3B335559740%26quot%3B%3A240%7D%22%3ERequired%20Action%26nbsp%3B%3C%2FSPAN%3E%3C%2FH2%3E%0A%3CUL%3E%0A%3CLI%20data-leveltext%3D%22%C2%B7%22%20data-font%3D%22Symbol%22%20data-listid%3D%221%22%20aria-setsize%3D%22-1%22%20data-aria-posinset%3D%225%22%20data-aria-level%3D%221%22%3E%3CSPAN%20data-contrast%3D%22none%22%3EFollow%20the%20Trouble%20Shooting%20guide%20with%20Mitigation%20steps%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2FService-Fabric-Troubleshooting-Guides%2Fblob%2Fmaster%2FKnown_Issues%2FService%2520Fabric%2520Common%2520Name%2520Digicert%2520Multiple%2520Issuer%2520Thumbprints.md%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ETroubleshooting%20Guide%3C%2FA%3E%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3C%2FLI%3E%0A%3CLI%20data-leveltext%3D%22%C2%B7%22%20data-font%3D%22Symbol%22%20data-listid%3D%221%22%20aria-setsize%3D%22-1%22%20data-aria-posinset%3D%226%22%20data-aria-level%3D%221%22%3E%3CSPAN%20data-contrast%3D%22none%22%3EMitigation%26nbsp%3Bspecified%20in%20the%26nbsp%3BTSG%26nbsp%3Bmust%26nbsp%3Bbe%26nbsp%3Bapplied%20by%20you.%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-contrast%3D%22none%22%3EIf%20you%20have%20any%20questions%20or%20concerns%2C%20please%E2%80%AF%3CA%20href%3D%22https%3A%2F%2Fportal.azure.com%2F%23blade%2FMicrosoft_Azure_Support%2FHelpAndSupportBlade%2Foverview%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Econtact%20us%3C%2FA%3E%E2%80%AFby%20opening%20a%20support%20request.%26nbsp%3BIn%20addition%2C%20here%20are%20your%20general%20support%20options%20for%20Service%20Fabric%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fservice-fabric%2Fservice-fabric-support%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ELearn%20about%20Azure%20Service%20Fabric%20Support%20options%20-%20Azure%20Service%20Fabric%20%7C%20Microsoft%20Docs%3C%2FA%3E.%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1928992%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20class%3D%22TextRun%20SCXW195768668%20BCX8%22%20data-contrast%3D%22none%22%3E%3CSPAN%20class%3D%22NormalTextRun%20SCXW195768668%20BCX8%22%3EService%20Fabric%20clusters%20may%20be%20impacted%20by%20the%20introduction%20of%20a%20new%20issuer%20thumbprint%20change%20by%20DigiCert%20CA.%20Symptoms%20include%20the%20cluster%20appearing%20down%20or%20unhealthy%2C%20cluster%20becoming%20unreachable%20(SFX%2FPortal%2FCLI)%2C%20or%20the%20event%20log%20showing%20authentication%20errors.%3C%2FSPAN%3E%3C%2FSPAN%3E%3CSPAN%20class%3D%22TextRun%20SCXW195768668%20BCX8%22%20data-contrast%3D%22auto%22%3E%3CSPAN%20class%3D%22NormalTextRun%20SCXW195768668%20BCX8%22%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FSPAN%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-TEASER%3E
Version history
Last update:
‎Nov 24 2020 06:54 PM
Updated by: