[Final Update] 3/31 20:00 UTC
Hello everyone,
We would like to share a final update on the outage in Azure Lab Services. The outage has now been resolved. Thank you so much for your patience throughout the outage. Please read about the details of what happened and what we’re doing as preventative steps for the future in our blog post.
The labs that were affected by the outage are now in following two states:
- The lab was recovered
All labs that were recovered should now operate as normal and will no longer be marked as affected by the outage. Please note that if you need to increase the lab capacity (i.e. increase the number of virtual machines in the VM pool) of these labs, you will need to republish the labs first. If you can’t republish but still need to increase lab capacity, please contact us at AzLabsCOVIDSupport@microsoft.com for help.
- The lab was not recoverable, and needs to be deleted and recreated
Unfortunately, some labs affected by the outage were not recoverable. These labs will be marked as not recoverable (see screenshot below), and your users will no longer see the labs’ virtual machines in their 'My virtual machines' view. As much as it pains us to say this, it means loss of data for the users.
To recreate these labs, please follow the steps below:
- Click on a lab that could not be recovered
- Go to the Users tab of the lab and click the “Export CSV” button to download a CSV file of the users list.
- The other tabs (Template, Schedule, Settings, etc.) will be read-only in case you need to copy over the info to the new lab
- In the My labs view, click on the “+ New lab” button, and set up the lab as before
- Once the lab is created, go to the Users tab. Click on the “Add users” > “Upload CSV” to import the CSV file of users into the new lab. Please note that you need to update the user quota (of hours) in this new lab.
- Click on the “Invite all” button to send out an email to your lab users with the registration link to your lab.
- Users will be able to click on the registration link to get access to virtual machines in your new lab.
- Delete the old lab.
We understand that this can potentially be a lot of effort, and we are deeply sorry for the inconvenience this has caused. Please do not hesitate to contact us at AzLabsCOVIDSupport@microsoft.com if there’s anything we can do on our end to help.
-Labs Team
------------------------------------------------------------------------------------------------
[Update] 3/26 17:00 UTC
We would like to inform you that we are actively working on resolving the issue.
How do I know if my lab is impacted in the outage?
One or many of your labs would be impacted if you see the following banner when you navigate to labs.azure.com
To identify specific impacted labs, navigate to a lab; if you see the following banner on your lab, that means it is impacted.
The fastest way to mitigate the issue is to create a new lab.
Please feel free to add a comment below if you have problems creating a new lab and/or any other questions.
We apologize for the inconvenience and thank you for your patience.
------------------------------------------------------------------------------------------------
[Update] 3/26 02:11 UTC
We would like to inform you that we are actively working on resolving the issue.
The current mitigation is to create a new lab.
Please feel free to add a comment below if you have problems creating a new lab and/or any other questions.
We apologize for the inconvenience and thank you for your patience.
------------------------------------------------------------------------------------------------
[Update] 3/25 21:00 UTC
As we investigate the issue further, we would like to share that the current mitigation to resolve these errors is to create a new lab.
Here is a link to learn more on creating a new lab in lab services.
We apologize for the inconvenience and thank you for your patience.
------------------------------------------------------------------------------------------------
Hey everyone,
We are currently experiencing issues with certain labs where users are not able to perform specific actions such as start/stop lab virtual machines and/or publish a lab.
We would like to inform you that we are actively looking into the issue and will continue providing updates on this post until the problem is resolved.
We apologize for the inconvenience and thank you for your patience.
- Lab Services team
