Home
%3CLINGO-SUB%20id%3D%22lingo-sub-1034225%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Access%20Issue%20in%20Azure%20portal%20for%20Log%20Analytics%20-%2011%2F26%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1034225%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Wednesday%2C%2027%20November%202019%2016%3A44%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2011%2F27%2C%2015%3A45%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2011%2F19%2C%2021%3A00%20UTC%20and%20that%20during%20the%207%20Days%2C%2018%20hours%20and%2045%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20customers%20would%20have%20seen%20error%20ID%206000%20in%20their%20event%20log%20with%20a%20source%20of%20Service%20Connector%20or%20error%201209%20with%20a%20source%20of%20HealthService.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20SHA2%20authentication%20configuration%20issue.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%207%20Days%2C%2018%20Hours%20%26amp%3B%2045%20minutes%20-%2011%2F19%2C%2021%3A00%20UTC%20through%2011%2F27%2C%2015%3A45%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Rama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2027%20November%202019%2011%3A17%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20an%20SHA2%20authentication%20configuration%20issue%20impacting%20agent%20intelligence%20packs%20and%20management%20packs%20on%20Windows%207%20and%20Windows%20Server%202008R2.%20Impacted%20customers%20would%20see%20error%20ID%206000%20in%20their%20event%20log%20with%20a%20source%20of%20Service%20Connector%20or%20error%201209%20with%20a%20source%20of%20HealthService.%20We%20have%20re-signed%20the%20patch%20with%20the%20SHA-1%20Hashing%20algorithm%20which%20is%20supported%20by%20Windows%207%20or%20Windows%20Server%202008%20and%20earlier.%20We%20are%20updating%20the%20management%20pack%20configuration%20to%20mitigate%20the%20problem%20without%20customer%20action.%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2011%2F27%2017%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Rama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2027%20November%202019%2006%3A27%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3EWe%20are%20still%20working%20hard%20to%20deploy%20the%20solution.%20There%20is%20currently%20no%20work%20around%20for%20this%20issue.%20We%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2011%2F27%2010%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Ian%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2027%20November%202019%2004%3A43%20UTC%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3EWe%20are%20currently%20deploying%20updates%20to%20mitigate%20the%20issue.%20%26nbsp%3B%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2011%2F27%2007%3A00%20UTC%26nbsp%3B%3C%2FLI%3E%3C%2FUL%3E%3CP%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20Helvetica%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3C%2FSPAN%3E%3CB%3E%3C%2FB%3E%3CI%3E%3C%2FI%3E%3CU%3E%3C%2FU%3E%3CSUB%3E%3C%2FSUB%3E%3CSUP%3E%3C%2FSUP%3E%3CSTRIKE%3E%3C%2FSTRIKE%3E%3CBR%20%2F%3E%3C%2FP%3E-Ian%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2026%20November%202019%2021%3A53%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20an%20SHA2%20authentication%20configuration%20issue%20impacting%20agent%20intelligence%20packs%20and%20management%20packs%20on%20Windows%207%20and%20Windows%20Server%202008R2.%20Impacted%20customers%20would%20see%20error%20ID%20%3CI%3E6000%3C%2FI%3E%20in%20their%20event%20log%20with%20a%20source%20of%20%3CI%3EService%20Connector%3C%2FI%3E%20or%20error%20%3CI%3E1209%3C%2FI%3E%20with%20a%20source%20of%20%3CI%3EHealthService%3C%2FI%3E.%26nbsp%3B%20We%20are%20currently%20working%20on%20updating%20the%20management%20pack%20configuration%20to%20mitigate%20the%20problem%20without%20customer%20action.%20However%20there%20is%20a%20workaround%20that%20can%20be%20applied%20if%20needed.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20%3CA%20href%3D%22https%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Fhelp%2F4472027%2F2019-sha-2-code-signing-support-requirement-for-windows-and-wsus%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Fhelp%2F4472027%2F2019-sha-2-code-signing-support-requirement-for-windows-and-wsus%3C%2FA%3E%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2011%2F27%2002%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jeff%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22Helvetica%22%3EInitial%20Update%3A%20Tuesday%2C%2026%20November%202019%2018%3A10%20UTCWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20and%20are%20actively%20investigating.%20Some%20customers%20running%20Windows%20Server%202008%20R2%20may%20see%20errors%20in%20event%20logs%20and%20clients%20agents%20may%20not%20be%20able%20to%20communicate%20back%20to%20the%20service%20properly.%3C%2FFONT%3E%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CFONT%20face%3D%22Helvetica%22%3EWork%20Around%3A%20none%3C%2FFONT%3E%3C%2FLI%3E%3CLI%3E%3CFONT%20face%3D%22Helvetica%22%3ENext%20Update%3A%20Before%2011%2F26%2020%3A30%20UTC%20%3C%2FFONT%3E%3C%2FLI%3E%3C%2FUL%3E%3CFONT%20face%3D%22Helvetica%22%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.-Ian%3C%2FFONT%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1034225%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Wednesday, 27 November 2019 16:44 UTC

We've confirmed that all systems are back to normal with no customer impact as of 11/27, 15:45 UTC. Our logs show the incident started on 11/19, 21:00 UTC and that during the 7 Days, 18 hours and 45 minutes that it took to resolve the issue customers would have seen error ID 6000 in their event log with a source of Service Connector or error 1209 with a source of HealthService.
  • Root Cause: The failure was due to SHA2 authentication configuration issue.
  • Incident Timeline: 7 Days, 18 Hours & 45 minutes - 11/19, 21:00 UTC through 11/27, 15:45 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Rama

Update: Wednesday, 27 November 2019 11:17 UTC

Root cause has been isolated to an SHA2 authentication configuration issue impacting agent intelligence packs and management packs on Windows 7 and Windows Server 2008R2. Impacted customers would see error ID 6000 in their event log with a source of Service Connector or error 1209 with a source of HealthService. We have re-signed the patch with the SHA-1 Hashing algorithm which is supported by Windows 7 or Windows Server 2008 and earlier. We are updating the management pack configuration to mitigate the problem without customer action.

  • Work Around: None
  • Next Update: Before 11/27 17:30 UTC
-Rama

Update: Wednesday, 27 November 2019 06:27 UTC

We are still working hard to deploy the solution. There is currently no work around for this issue. We apologize for any inconvenience.
  • Next Update: Before 11/27 10:30 UTC
-Ian

Update: Wednesday, 27 November 2019 04:43 UTC

We are currently deploying updates to mitigate the issue.  
  • Next Update: Before 11/27 07:00 UTC 

We are working hard to resolve this issue and apologize for any inconvenience.

-Ian

Update: Tuesday, 26 November 2019 21:53 UTC

Root cause has been isolated to an SHA2 authentication configuration issue impacting agent intelligence packs and management packs on Windows 7 and Windows Server 2008R2. Impacted customers would see error ID 6000 in their event log with a source of Service Connector or error 1209 with a source of HealthService.  We are currently working on updating the management pack configuration to mitigate the problem without customer action. However there is a workaround that can be applied if needed.
-Jeff

Initial Update: Tuesday, 26 November 2019 18:10 UTC

We are aware of issues within Log Analytics and are actively investigating. Some customers running Windows Server 2008 R2 may see errors in event logs and clients agents may not be able to communicate back to the service properly.

  • Work Around: none
  • Next Update: Before 11/26 20:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Ian