Final Update: Sunday, 21 February 2021 21:04 UTC
We've confirmed that all systems are back to normal with no customer impact as of 02/21, 20:40 UTC. Our logs show the incident started on 02/21, 14:50 UTC and that during the 5 hours and 50 minutes that it took to resolve the issue 95% of customers in Southeast Australia region experienced latent or missed log alerts.
-Jeff
We've confirmed that all systems are back to normal with no customer impact as of 02/21, 20:40 UTC. Our logs show the incident started on 02/21, 14:50 UTC and that during the 5 hours and 50 minutes that it took to resolve the issue 95% of customers in Southeast Australia region experienced latent or missed log alerts.
- Root Cause: The failure was due to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts.
- Incident Timeline: 5 Hours & 50 minutes - 02/21, 14:50 UTC through 02/21, 20:40 UTC
-Jeff
Update: Sunday, 21 February 2021 19:46 UTC
Root cause has been isolated to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts. To address this issue we have found the alerts generating a large quantity of noise and have removed them and scaled up the backend service to handle the backlog. The backlog is currently draining, alerts will continue to be impacted until we can complete this process.
Root cause has been isolated to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts. To address this issue we have found the alerts generating a large quantity of noise and have removed them and scaled up the backend service to handle the backlog. The backlog is currently draining, alerts will continue to be impacted until we can complete this process.
- Work Around: None
- Next Update: Before 02/21 21:00 UTC
Update: Sunday, 21 February 2021 18:49 UTC
Root cause has been isolated to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts. To address this issue we have found the alerts generating a large quantity of noise and are working to reduce them as well as scale up the backend service to handle the backlog. Alerts will continue to be impacted until we can complete this process.
Root cause has been isolated to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts. To address this issue we have found the alerts generating a large quantity of noise and are working to reduce them as well as scale up the backend service to handle the backlog. Alerts will continue to be impacted until we can complete this process.
- Work Around: None
- Next Update: Before 02/21 20:00 UTC
Update: Sunday, 21 February 2021 17:30 UTC
Root cause has been isolated to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts. To address this issue we have found the alerts generating a large quantity of noise and are working to reduce them. Alerts will continue to be impacted until we can complete this process.
Root cause has been isolated to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts. To address this issue we have found the alerts generating a large quantity of noise and are working to reduce them. Alerts will continue to be impacted until we can complete this process.
- Work Around: None
- Next Update: Before 02/21 18:30 UTC
Initial Update: Sunday, 21 February 2021 16:32 UTC
We are aware of issues within Log Search Alerts in Australia Southeast region and are actively investigating. Some customers may experience delayed or missed Log Search Alerts.
-Jeff
We are aware of issues within Log Search Alerts in Australia Southeast region and are actively investigating. Some customers may experience delayed or missed Log Search Alerts.
- Work Around: None
- Next Update: Before 02/21 19:00 UTC
-Jeff
Updated Feb 21, 2021
Version 5.0