There are several teams within Microsoft which use Azure DevTest Labs (DTL). SharePoint Online (SPO) team at Microsoft has created a solution, built on DTL, to solve the problem of making SPO testing environments readily available to SharePoint engineers – securely, at scale and with high availability – leveraging the strengths of DevTest Labs and Azure.
This blog post is the first in a series of posts that highlights key parts of this solution. The series enables anyone to build, configure and run a similar system on DTL. Each post will cover one core aspect of the SharePoint Online solution, with a brief description of the scenarios involved, and the strategies applied to solve them. Each comes equipped with architecture diagrams, walk-throughs and code samples, presented in a generic and reusable way. It is hoped that customers in the Azure community, desiring to solve similar problems for their users, can benefit from the content herein to create their own DTL solutions.
About SharePoint Online
For background, SharePoint Online is Microsoft’s document storage service for Microsoft365. It is a massive enterprise business. In addition to providing enterprise-class document management, it has team collaboration, portal sites and content management. It forms the document storage backbone for Exchange Online, Teams and Office applications such as Word and Excel. As one example, when you save a document in Teams, it is stored in SPO under the hood. Fortune 500 companies, schools, governments and non-profits critically depend on SharePoint Online for their business, storing over 480 exabytes of their data in it.
If you would like to learn more about SharePoint Online and its potential application to your business, start here. There are Microsoft365 subscriptions tailored for home, business, and institutional use.
SharePoint Online Scenario
With over 1000 engineers working on an ever-evolving 140 MLOC codebase in SharePoint Online, integration testing is a critical part of the engineering workflow. At its core, SharePoint is a three-tier web application, featuring a web tier, a business logic tier and a data layer, each working in conjunction as a single service in data centers across the globe. Engineers working on it require safe, isolated simulation environments that function like scaled-down three-tier data center deployments so that they can deeply test their code before it goes worldwide.
The SharePoint Online team has shrunk the core SPO components and services so that they all fit on a single, interactive virtual machine (VM) that simulates the full data center, but is stripped down to its essence. It is therefore much easier to deploy. These single-VM environments are well-suited to testing most SPO scenarios – of course core document storage and team collaboration, but notably scenarios involving interaction with Office or Teams. Using these VMs, code changes can be deployed, poked and prodded, and debugged as it interacts with these components and services – all well before the code is ultimately deployed for enterprise customers in production data centers.
Use of DevTest Labs
DevTest Labs provides “labs” of machines, intended to host pre-configured testing environments at scale. The SPO team has built a solution in DevTest Labs to host these single-VM SPO environments, giving engineers a private, isolated, safe, simulated data center in which to test their code.
Breaking it down step-by-step, the scenario for SharePoint Online engineers is as follows:
- An engineer wishes to test their code. They fetch a virtual machine (VM) from the DevTest Lab.
- The VM hosts a pre-configured single-VM SPO environment with a stable build.
- They log in, interact with the environment, and patch a new code change into place to do testing.
- When they are finished, they throw away their VM – they can always fetch a fresh one for their next iteration.
The complexity of building these single-VM SPO environments, with all services and components wired up correctly, is hidden from the lab user. The work is done before the VM is assigned to the engineer, saving them the time and effort. Environments are ready to use as they are. Engineers have the option to simply use them as-is, or optionally patch their private changes and onto the VM to see how it behaves. These single-VM environments are the heart of their inner loop debugging and integration testing.
Today, about 500 engineers use the SPO integration lab on a monthly basis, with plans to grow this to 750 or more. Typically about 2,500 VMs are allocated at any one point in time, running in three different Azure regions around the world. Azure allows the solution to easily scale to meet growing demand.
Why DevTest Labs?
Certainly a solution of this nature can be created without DevTest Labs, using Azure VMs directly. However, DevTest Labs provides out-of-box features that save time and money in implementation and maintenance. These include:
- Standard service grouping and administration
- Flexible and extensible networking and user connectivity
- Ability for lab users to claim pre-created VMs without going through the full creation process
- Policies and thresholds to manage costs by shutting unused machines down
- DTL Artifacts to reduce the time required to set up and configure virtual machines once created
The full feature set of Azure is also available to extend DevTest Labs where needed, but leveraging the native capabilities of DevTest Labs reduces the cost to build, maintain and extend the solution. It also sets the solution up for future improvements planned by the Azure team at Microsoft. In short, DevTest Labs is a quite natural fit for this use case, rather than reinventing the wheel.
If you haven’t tried DevTest Labs yet, click here to create your first lab. DevTest Labs is a free service!
Solution Architecture
The solution builds on DevTest Labs as its basis. It is an event-driven architecture, using Azure EventGrid and an Azure Function app. It uses Shared Image Gallery to host VM images and employs a Point-to-Site VPN for secure access to the network hosting the VMs. It has an easy-to-use interface for checkin/checkout that is hosted outside the Azure portal. It extends DevTest Labs policy and allows users to manage when machines are shut down, and even postpone shutting machines down, above the times assigned by Azure DevTest Labs policy.
Below is a diagram covering the architecture. Starting from the left, we have an Image Service which is responsible for creating Azure Compute images with SPO pre-configured. Image Service pulls the latest build available, triggers the deployment, waits for success, and publishes the result to Azure Shared Image Gallery for general consumption by DevTest Labs. It then fires an event that, by way of Event Grid, triggers an Azure Function App.
The Function App contains the business logic for the SPO solution, which extends the core functionality DevTest Labs natively provides. The Function App responds to the event by triggering VM creation in DevTest Labs, in accordance with desired VM pool targets. A VPN Gateway is configured to provide point-to-site VPN connectivity for end users, who use Azure VPN Client to ensure the communication between the lab user and the VMs is authenticated and secure.
The solution created has potential application to a broad set of scenarios. The designs and architecture described should be beneficial to Azure customers wanting to build similar solutions on DevTest Labs, including what code to build and what components to configure for their applications.
Introducing This Blog Series
In several upcoming blog posts, we’ll cover the following topics:
- Overview: How the SharePoint team leverages DevTest Labs (Current Blog Post)
- Enabling an Event-Driven Architecture with DevTest Labs
- Leveraging Point-to-Site VPN with DevTest Labs
- Maintaining a Pool of Claimable VMs
- Building a Simple User Interface for DevTest Labs
- Enabling Shutdown notifications via Teams
Each blog post borrows the concepts and architecture from the SPO solution, boiled down and made generally accessible for re-use by the Azure community. The content of this blog series forms a set of templates for others to build the same types of solutions for their applications as the SharePoint Online team did for their engineers.
What's Next?
In forthcoming blog posts, the rest of the topics will be covered, starting with how to enable an event-driven architecture with DevTest Labs similar to the one described above. Please stay tuned!
If you run into any problems with the content, or have any questions, please feel free to drop a comment below this post and we will respond. Your feedback is always welcome!
- Pete Harwood, Principal Engineering Manager, OneDrive and SharePoint Engineering Fundamentals at Microsoft
Microsoft
