With the recent service update, we are delivering on our promise to be data-driven by delivering regression analysis results based on intelligence about the crashes and hangs of specific processes. Developed in response to your feedback, this new feature provides you with clear insights into the compatibility of your application by comparing reliability signals.
Read on for more details or watch this video for a quick walkthrough of the latest improvements. Want to onboard your application to the Update Staging Lab and accelerate validation against Windows security updates? Visit https://aka.ms/JoinUSL to sign up or recommend your ISV for the service.
Regression status: process crashes and hangs
Since it was first introduced, the Update Staging Lab dashboard has provided you with crash and hang details of each process running on the test machine on which your application is tested. As the application gets exercised multiple times during the test pass, we provided crash or hang percentages to help you get a better idea of potential issues that might arise for an end user. With this latest update; however, we now perform a full regression analysis of crash and hang percentage by comparing the performance of your application on a new pre-released Windows update with its performance against an update from the previous month. This direct signal offers a clear validation result, saving your time and effort rather than requiring you to make the comparison on your own for each test run.
Regression signals are captured in the Reliability tab for all test runs. As you can see in the example below, a recent test run with the onboarded application “USL Smoke Test AppCrash” against the pre-release May security update is displaying a failure in reliability. This is caused by a 27% regression in crash percentage for “USLTestCrash.exe” on the pre-release May update compared to the released April update. Regressions for key processes can lead to issues for the end user experience, but can now be caught and isolated easily with Update Staging Lab.
Figure 1. Regression comparison for each process is captured in the Result column. There has also been a status icon update.
The failure signal caused by this regression is then also shown on the Test results page under Reliability.
Figure 2. Reliability displays failure because of crash regression in one of the processes.
Figure 3. Test summary page displays a compatibility failure because of the crash regression.
If your application experiences a failure caused by a higher degree of process crashes, you will a also see Fail displayed on the Test summary page. By providing these failure signals upfront, our goal is to ensure we flag potential issues that can disrupt and impact the end user experience. By catching these failures early with Update Staging Lab, together we can continue to investigate and remedy the issues behind them and help you prevent issues impacting the customers who use your applications.
Private preview participants can start using these enhancements today in the Update Staging Lab dashboard within the Azure portal. We are working on additional UI and service-level improvements and are excited to share details on those soon.
Please try out Update Staging Lab and these latest features and let us know how we can continue to make improvements that matter to you. Also, we would love your feedback so feel free to leave comments below or email us at uslpreview@microsoft.com.