With July 2021's update, we have added new features and user interface (UI) enhancements to make analyzing unexpected issues and test failures easier. You can expect to see a difference in how your test execution statuses appear throughout the portal, along with new information regarding the status itself. In addition, we are introducing a new Test Analysis tab that can be used in conjunction with your performance analytics to investigate your test results in greater detail.
Customer feedback has helped us to prioritize additional failure analysis tools and information. As we expand upon the issues and failures Test Base for Microsoft 365 can detect, resolving application issues and failures will require these additional resources as well.
New status column
Previously, a test execution that has completed would show a “Fail” or “Pass” value in the Result column, indicating if a failure or regression had been detected. With the help of feedback from our current customers, we have collapsed the Execution Status and Result columns into a single column that is labeled Status. Now, you will notice statuses will have informative bubble messaging to explain exactly what the status of your test execution means, and if there are results that may need your attention.
A current list of possible values you may encounter on your Test Base portal:
- Run in progress: We are currently executing your tests and results will be available to view soon.
- Data processing: The test execution is now complete, and we are processing your results.
- Test failure: There is a failure in your package. Your full test result details are available for you to review.
- Completed: Results of your test execution are available to view and there are no regressions or failures present at this time.
- Infrastructure failure: There is a failure in our infrastructure that prevented the test execution. The issue is being investigated and we will retry the execution.
Our goal is to provide clear understanding as to the type of failures you are experiencing upfront. By providing new status types with a greater amount of detail, it is less ambiguous as to what is causing issues in your test executions. With improved communication, we aim to help you investigate the service critical failures and minimize the end user impact as quickly as possible.
Test Analysis: A deeper look
When a failure is detected on Test Base for Microsoft 365, performance analytics alone are sometimes not enough for deep failure analysis and remediation.
Our first step to a series of enhancements in the failure analysis process is providing the Test Analysis tab in the details of your results. Here is where an itemized view of each failure affecting your execution will be listed. There will also be greater details regarding the test execution noted under the Test Details section, along with the general guidance to further investigate these issues with the resources provided. Future work will continue to improve the accuracy and detail of the data we provide in this Test Analysis tab.
We will continue to expand on the tools and resources we provide to help you isolate and resolve the issues detected on Test Base for Microsoft 365. In the upcoming months, feature work is planned to expand the failure analysis resources we offer. New test statuses will also be added as new issues are able to be detected. Discovering failures and issues is only the first step, and we aim to make the remediation process as streamlined as possible through Test Base for Microsoft 365.