%3CLINGO-SUB%20id%3D%22lingo-sub-1045567%22%20slang%3D%22en-US%22%3EExperiencing%20Issues%20in%20Azure%20portal%20for%20Log%20Analytics%20-%2012%2F04%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1045567%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Wednesday%2C%2004%20December%202019%2009%3A47%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2012%2F04%2C%2007%3A30%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2011%2F21%2C%2019%3A34%20UTC%20and%20that%20during%20the%2013%20days%20and%2012%20hours%20that%20it%20took%20to%20resolve%20the%20issue%20some%20of%20customers%20experienced%20exception%20while%20accessing%20the%20free%20assessments.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20an%20issue%20with%20one%20of%20our%20dependent%20service.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%2013%20days%20%26amp%3B%2012%20hours-%2011%2F21%2C%2019%3A34%20UTC%20through%2012%2F04%2C%2007%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Monish%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Wednesday%2C%2004%20December%202019%2007%3A04%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20and%20are%20actively%20working%20on%20mitigation.%20Some%20customers%20may%20continue%20to%20experience%20exceptions%20while%20accessing%20the%20free%20assessments.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2012%2F04%2011%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Monish%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2004%20December%202019%2004%3A07%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20the%20issues%20within%20Log%20Analytics.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20continue%20to%20experience%20exceptions%20while%20accessing%20the%20free%20assessments.%26nbsp%3B%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2012%2F04%2007%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Monish%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Wednesday%2C%2004%20December%202019%2001%3A20%20UTC%3CBR%20%2F%3E%3CP%3E%3CFONT%20face%3D%22%26quot%3B%26quot%3B%26quot%3B%2Cserif%22%3E%3C%2FFONT%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3B%0Aline-height%3A107%25%3Bfont-family%3A%26quot%3B%26quot%3B%26quot%3B%2Cserif%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0Amso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bcolor%3Ablack%22%3E%3CBR%20%2F%3E%20%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bline-height%3A107%25%3Bfont-family%3A%26quot%3BHelvetica%26quot%3B%2Csans-serif%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bcolor%3Ablack%3Bbackground%3Awhite%22%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20and%20are%20actively%20investigating.%20Customer%20will%20be%20facing%20exceptions%20while%20accessing%20the%20free%20assessments%20-%20Active%20Directory%20Health%20Check%2C%20SQL%20Health%20Check%2C%20AD%20Replication%20Status%2C%20System%20Center%20Operations%20Manager%26nbsp%3BHealth%20Check%2C%20through%20Microsoft%20Monitoring%20Agent%20and%20SCOM%20Agent.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bline-height%3A107%25%3Bfont-family%3A%26quot%3BHelvetica%26quot%3B%2Csans-serif%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bcolor%3Ablack%3Bbackground%3Awhite%22%3EException%3A-%3C%2FSPAN%3E%3C%2FP%3E%0A%0A%3CP%3E%3CSPAN%20style%3D%22font-size%3A10.0pt%3Bline-height%3A107%25%3Bfont-family%3A%0AConsolas%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3ACalibri%3B%0Acolor%3Ablack%3Bmso-color-alt%3Awindowtext%3Bbackground%3A%23F0F2F4%22%3E'Microsoft.EnterpriseManagement.Mom.Modules.AdvisorAssessmentExecuter.ProbeAction.AdvisorAssessmentExecuterProbe'%20reported%20an%20exception%26nbsp%3BSystem.Exception%3A%20%5BADAssessment%5D%5B2905%5DFailed%20to%20run%20AdvisorAssessment.exe%20or%20generate%20output%26nbsp%3BDataItem(s).%20---%26gt%3B%26nbsp%3BSystem.IO.DirectoryNotFoundException%3A%20Could%20not%20find%20a%20part%20of%20the%20path%20'C%3A%5C%5CProgram%20Files%5C%5CMicrosoft%20Monitoring%20Agent%5C%5CAgent%5C%5CHealth%20Service%20State%5C%5CMonitoring%20Host%20Temporary%20Files%201%5C%5C4%5C%5C3311548'.%3C%2FSPAN%3E%3CSPAN%20style%3D%22mso-ascii-font-family%3ACalibri%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0Amso-hansi-font-family%3ACalibri%3Bmso-bidi-font-family%3ACalibri%3Bcolor%3Ablack%3B%0Amso-color-alt%3Awindowtext%3Bbackground%3A%23F0F2F4%22%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bline-height%3A107%25%3Bfont-family%3A%26quot%3BSegoe%20UI%26quot%3B%2Csans-serif%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%22%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%0A%3CP%20style%3D%22mso-margin-top-alt%3Aauto%3Bmso-margin-bottom-alt%3Aauto%3B%0Aline-height%3Anormal%22%3E%3CSPAN%20style%3D%22font-size%3A10.0pt%3Bfont-family%3AConsolas%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3ACalibri%3B%0Acolor%3Ablack%3Bmso-color-alt%3Awindowtext%3Bbackground%3A%23F0F2F4%22%3E%26nbsp%3B%26nbsp%3B%20at%26nbsp%3BSystem.IO.__Error.WinIOError(Int32%26nbsp%3BerrorCode%2C%20String%26nbsp%3BmaybeFullPath)%3C%2FSPAN%3E%3CSPAN%20style%3D%22mso-ascii-font-family%3ACalibri%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-hansi-font-family%3ACalibri%3B%0Amso-bidi-font-family%3ACalibri%3Bcolor%3Ablack%3Bmso-color-alt%3Awindowtext%3Bbackground%3A%0A%23F0F2F4%22%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3A%26quot%3BSegoe%20UI%26quot%3B%2Csans-serif%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%22%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%0A%3CP%20style%3D%22mso-margin-top-alt%3Aauto%3Bmso-margin-bottom-alt%3Aauto%3B%0Aline-height%3Anormal%22%3E%3CSPAN%20style%3D%22font-size%3A10.0pt%3Bfont-family%3AConsolas%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3ACalibri%3B%0Acolor%3Ablack%3Bmso-color-alt%3Awindowtext%3Bbackground%3A%23F0F2F4%22%3E%26nbsp%3B%26nbsp%3B%20at%20System.IO.FileSystemEnumerableIterator%601.CommonInit()%3C%2FSPAN%3E%3CSPAN%20style%3D%22mso-ascii-font-family%3ACalibri%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0Amso-hansi-font-family%3ACalibri%3Bmso-bidi-font-family%3ACalibri%3Bcolor%3Ablack%3B%0Amso-color-alt%3Awindowtext%3Bbackground%3A%23F0F2F4%22%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3A%26quot%3BSegoe%20UI%26quot%3B%2Csans-serif%3Bmso-fareast-font-family%3A%0A%26quot%3BTimes%20New%20Roman%26quot%3B%22%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%0A%3CP%20style%3D%22mso-margin-top-alt%3Aauto%3Bmso-margin-bottom-alt%3Aauto%3B%0Aline-height%3Anormal%22%3E%26nbsp%3B%26nbsp%3B%20%3CSPAN%20style%3D%22font-size%3A10.0pt%3Bfont-family%3AConsolas%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3ACalibri%3B%0Acolor%3Ablack%3Bmso-color-alt%3Awindowtext%3Bbackground%3A%23F0F2F4%22%3Eat%20System.IO.FileSystemEnumerableIterator%601..ctor(String%20path%2C%20String%26nbsp%3BoriginalUserPath%2C%20String%26nbsp%3BsearchPattern%2C%26nbsp%3BSearchOption%3C%2FSPAN%3E%3CSPAN%20style%3D%22mso-ascii-font-family%3ACalibri%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0Amso-hansi-font-family%3ACalibri%3Bmso-bidi-font-family%3ACalibri%3Bcolor%3Ablack%3B%0Amso-color-alt%3Awindowtext%3Bbackground%3A%23F0F2F4%22%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.0pt%3Bfont-family%3AConsolas%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0Amso-bidi-font-family%3ACalibri%3Bcolor%3Ablack%3Bmso-color-alt%3Awindowtext%3Bbackground%3A%0A%23F0F2F4%22%3EsearchOption%2C%20SearchResultHandler%601%26nbsp%3BresultHandler%2C%20Boolean%26nbsp%3BcheckHost)%3C%2FSPAN%3E%3CSPAN%20style%3D%22mso-ascii-font-family%3ACalibri%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0Amso-hansi-font-family%3ACalibri%3Bmso-bidi-font-family%3ACalibri%3Bcolor%3Ablack%3B%0Amso-color-alt%3Awindowtext%3Bbackground%3A%23F0F2F4%22%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3A%26quot%3BSegoe%20UI%26quot%3B%2Csans-serif%3Bmso-fareast-font-family%3A%0A%26quot%3BTimes%20New%20Roman%26quot%3B%22%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%0A%3CP%20style%3D%22mso-margin-top-alt%3Aauto%3Bmso-margin-bottom-alt%3Aauto%3B%0Aline-height%3Anormal%22%3E%3CSPAN%20style%3D%22font-size%3A10.0pt%3Bfont-family%3AConsolas%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3ACalibri%3B%0Acolor%3Ablack%3Bmso-color-alt%3Awindowtext%3Bbackground%3A%23F0F2F4%22%3E%26nbsp%3B%26nbsp%3B%20at%26nbsp%3BSystem.IO.Directory.GetDirectories(String%20path)%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3A%26quot%3BSegoe%20UI%26quot%3B%2Csans-serif%3Bmso-fareast-font-family%3A%0A%26quot%3BTimes%20New%20Roman%26quot%3B%22%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%0A%3CP%20style%3D%22margin-bottom%3A0in%3Bmargin-bottom%3A.0001pt%3Bline-height%3A%0Anormal%22%3E%3CSPAN%20style%3D%22font-size%3A12.0pt%3Bfont-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%2Cserif%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%0A%3CP%20style%3D%22margin-bottom%3A0in%3Bmargin-bottom%3A.0001pt%3Bline-height%3A%0Anormal%22%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3A%26quot%3BHelvetica%26quot%3B%2Csans-serif%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bcolor%3Ablack%3Bbackground%3Awhite%22%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3C%2FSPAN%3E%3C%2FP%3E%3CP%20style%3D%22margin-bottom%3A0in%3Bmargin-bottom%3A.0001pt%3Bline-height%3A%0Anormal%22%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3A%26quot%3BHelvetica%26quot%3B%2Csans-serif%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bcolor%3Ablack%3Bbackground%3Awhite%22%3E%3CBR%20%2F%3E%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3A%26quot%3B%26quot%3B%26quot%3B%2Cserif%3Bmso-fareast-font-family%3A%0A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bcolor%3Ablack%22%3E%3CBR%20%2F%3E%20%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3A%26quot%3BHelvetica%26quot%3B%2Csans-serif%3B%0Amso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bcolor%3Ablack%3Bbackground%3Awhite%22%3E-Saika%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3A%26quot%3BHelvetica%26quot%3B%2Csans-serif%3Bcolor%3Ablack%3B%0Abackground%3Awhite%22%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%0A%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1045567%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Wednesday, 04 December 2019 09:47 UTC

We've confirmed that all systems are back to normal with no customer impact as of 12/04, 07:30 UTC. Our logs show the incident started on 11/21, 19:34 UTC and that during the 13 days and 12 hours that it took to resolve the issue some of customers experienced exception while accessing the free assessments.
  • Root Cause: The failure was due to an issue with one of our dependent service.
  • Incident Timeline: 13 days & 12 hours- 11/21, 19:34 UTC through 12/04, 07:30 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Monish

Initial Update: Wednesday, 04 December 2019 07:04 UTC

We are aware of issues within Log Analytics and are actively working on mitigation. Some customers may continue to experience exceptions while accessing the free assessments.
  • Next Update: Before 12/04 11:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Monish

Update: Wednesday, 04 December 2019 04:07 UTC

We continue to investigate the issues within Log Analytics. Root cause is not fully understood at this time. Some customers continue to experience exceptions while accessing the free assessments. 
  • Next Update: Before 12/04 07:30 UTC
-Monish

Initial Update: Wednesday, 04 December 2019 01:20 UTC


We are aware of issues within Log Analytics and are actively investigating. Customer will be facing exceptions while accessing the free assessments - Active Directory Health Check, SQL Health Check, AD Replication Status, System Center Operations Manager Health Check, through Microsoft Monitoring Agent and SCOM Agent.

Exception:-

'Microsoft.EnterpriseManagement.Mom.Modules.AdvisorAssessmentExecuter.ProbeAction.AdvisorAssessmentExecuterProbe' reported an exception System.Exception: [ADAssessment][2905]Failed to run AdvisorAssessment.exe or generate output DataItem(s). ---> System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\Monitoring Host Temporary Files 1\\4\\3311548'. 

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) 

   at System.IO.FileSystemEnumerableIterator`1.CommonInit() 

   at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost) 

   at System.IO.Directory.GetDirectories(String path)

 

We are working hard to resolve this issue and apologize for any inconvenience.



-Saika