App confidence: Optimize app validation with Test Base

Expect your apps to work on Windows 11. If you still need extra reassurance, test only your most critical apps with Test Base for Microsoft 365. This efficient and cost-effective testing service is available today to all IT pros, along with developers and independent software vendors.

We hope you're already experiencing the safer, smoother, and more collaborative work environment with the new Windows 11 features. We're also here to help you ensure business continuity by maintaining high application compatibility rates and supporting you in testing the mission-critical applications that you manage. Many of you have traditional (read legacy) testing systems that are provisioned in on-premises lab infrastructure. This setup has a myriad of challenges: limited capacity, manual setup and configuration, cost associated with third-party software, etc. That's why we created Test Base for Microsoft 365. It's a cloud-based testing solution that allows IT pros and developers to test your applications against pre-release and in-market versions of Windows and Office in a Microsoft-managed environment. Scale your testing, use data-driven insights, and ensure fewer deployment issues and helpdesk escalations with targeted testing of critical apps.

Let's take a closer look at how you can leverage Test Base to optimize your testing with:

• Proactive issue discovery with insights
• Automated testing against a large matrix
• Worry-free interoperability assurance
• Support throughout the entire validation lifecycle

Proactive issue discovery with insights

If you're on a mission to be more proactive with issue detection, give Test Base a try. Do you wish to have more time to test your apps before the monthly Windows updates go out? We know reactive issue detection is stressful after your end users have already been impacted. So why not leverage prebuilt test scripts in our smooth onboarding experience! With that, let Test Base perform the install, launch, close, and uninstall actions on your apps 30 times for extra assurance. Get helpful reports and know that your apps won't have any foundational issues on the new version of Windows.

Use prebuilt or personalized scripts to test updates

Here's how you can use the prebuilt test scripts and get actionable insights:

1. Access the Test Base portal.
2. Select New Package from the left-hand side menu.
3. Click on the Edit Package tab and select a pre-generated test script from the left-hand side pane.
4. Follow the step-by-step onboarding experience from there!

The following image shows the Package Preview step of the onboarding experience, illustrating a custom script to test the installation of an application. You can also choose to write your own functional tests to validate the specific functionality of your applications.

Pre-generated template scripts available on the New Package page inside Test Base

After you configure your test matrix on the following screen, we automatically test your applications against Windows updates. You'll also get an execution report and summary of any failures. Use this report to see data on your applications' reliability, memory, and CPU utilization across all your configured tests. At the end, see Event Trace Logs (ETLs) and a video recording of the test execution to easily debug any issues that you find.

Script execution report in the test results summary in Test Base

Compare CPU usage for feature updates

As part of the test results, view CPU utilization insights for your applications during Windows feature updates. Within the same onboarding experience in Test Base > New Package > Create New Package Online > Test Matrix tab, you can view a side-by-side analysis between your configured run and a baseline run. Select any of the familiar OSs as the baseline, and it will appear below your test graph. In the following example, the tested December monthly security build (2022.12 B) shows up on the top, while the baseline November monthly security build (2022.11 B) is listed on the bottom. The results are summarized above both charts with a regression status and details.

Side-by-side CPU utilization analysis of a test run against the baseline inside Test Base

How do we determine that? We check for a statistically significant difference in CPU usage for processes between the baseline and target run. A run is regressed if one or more relevant processes experiences a statistically significant increase at one of our observed percentiles. Use these insights for faster regression detection and to ensure enough lead time to fix any issues you might find. Learn more about the CPU regression analysis in our official documentation.

Automated testing against a large matrix

Can you identify with managing scores of devices that run multiple different versions of Windows? Then you might worry about eventually facing an unexpected compatibility issue between Windows and another Microsoft first-party app update. Test Base is integrated with the Windows build system, so we can help you test your apps against builds as soon as they are available and before they are released to Windows devices. Use our comprehensive OS coverage and automated testing to detect potential issues as early as possible, improve your SLA, and reduce labor costs.

1. Select New Package from the left-hand side menu.
2. Within the onboarding guide, click on the Test Matrix page.
3. Select the OS update type from among security update, feature update, or both.
4. Use the drop-down menu to select all applicable OS versions (either in-market or pre-release) that you want to test.
5. Optional: Select Inside Channel if you want to include tests on the latest available features.
6. Choose your OS baseline for insight.

The Test Base testing matrix of in-market and pre-release security and feature updates

That's how you can easily automate testing against in-market and pre-release Windows security and feature updates. Start today with these considerations:

• To become a Full Access customer and start testing on pre-release updates, see Request to change access level.
• To always get the latest features available, select a Windows Insider channel to run your tests on.
• In case you still don't find the OS you want to use as a baseline, let us know!

Worry-free interoperability assurance

We know your users have multiple applications running on their devices at the same time: mail app, browser, collaboration tools, etc. Sometimes, updates affect the interaction between applications and the OS, which could then affect the application's performance.

Do you find yourself wishing for better predictability of regressions from Microsoft product updates or more lead time to fix occasional resulting issues? Use our interoperability tests with detailed reliability signals just for that. Find them within the same Test Base onboarding experience:

1. Select the New Package flow.
2. Navigate to the Configure test tab.
3. Toggle the Pre-install Microsoft apps to turn it on under Test type.

The pre-install Microsoft apps is toggled on in the New package configuration in Test Base

For interoperability assurance, you'd want to look for any signals of regressions after running the test. From the test summary, select the run you want to examine closer, and navigate to the Reliability tab. Here's what a successful test would look like, comparing the number of crashes between the target test run and the baseline.

Detailed reliability signals for feature update test results in Test Base

With Test Base, you can conduct pre-release testing against monthly Microsoft 365 and Office 365 updates for extra confidence that you are covered across a broad range of Microsoft product updates. Leverage the automation in Test Base to schedule monthly tests whenever the monthly updates become available! For that, learn how to pick security testing options in the Set test matrix step in our official documentation.

Support throughout the entire validation lifecycle

The best part about Test Base is that you can use it throughout the entire Windows and Microsoft 365 apps validation lifecycle, supported by people and tools you trust. Use our support not only for the annual Windows updates, but also for the monthly security updates.

• If you're a developer, take advantage of integrations with familiar development tools like Azure DevOps and GitHub to test as you develop. That way, you can catch regressions before they are introduced to end users. Check out our documentation on how you can integrate Test Base into your Continuous Integration/Continuous Delivery pipeline.
• Whether you're an app developer or an IT pro, work with an App Assure engineer to get support with remediating any issues you find.

The table below outlines some of the key benefits that Test Base provides across the end-to-end validation lifecycle: from testing environment, to testing tools and software, and, finally, to testing services.

Get started today

Sign up for a free trial of Test Base to try out these features and optimize your testing process! To get started, simply sign up for an account via Azure or visit our website for more information.

Interested in learning more? Watch this contextualized walk-through of the Test Base service: How to build app confidence with Test Base. You can also leverage our series of how-to videos.

Check out more information and best practices on how to integrate Test Base into your Continuous Integration/Continuous Delivery pipeline in our GitHub documentation. We've also put together some sample packages that you can use to test out the onboarding process:

• Calculator-Appium-Charp-Sample
• Calculator-Appium-Java-Sample
• Calculator-Appium-Python-pytest-Sample
• Calculator-Appium-Python-unittest-Sample
• Calculator-MSTest-Sample
• Calculator-OOB-Sample
• StoreAPP-OOB-Sample

Finally, join our Test Base for Microsoft 365 Tech Community or GitHub Discussions Community to share your experiences and connect with other users.

Continue the conversation. Find best practices. Bookmark the Windows Tech Community and follow us @MSWindowsITPro on Twitter. Looking for support? Visit Windows on Microsoft Q&A.