How Pulp powers Microsoft's Linux Software Repositories

PMC, the backbone of Microsoft’s Linux Software Delivery 

Microsoft's mission is "to empower every person and organization on the planet to achieve more" - across all platforms. To support this, Microsoft provides PMC (packages.microsoft.com), a key service for distributing Microsoft software products for Linux; and a critical service for Microsoft's Linux Distributions and secure software.  

PMC is a critical part of Azure Core’s Linux infrastructure. It handles the ingestion and distribution of Linux packages from Microsoft, serving both internal Microsoft teams and external customers. For example, PMC hosts packages from Microsoft that support Azure Linux, Azure Security Pack, .NET, VSCode, and Edge Browser, to name a few. 

PMC delivers packages via standard Linux package managers, apt (Debian/Ubuntu),  dnf/yum (RHEL, Fedora, CentOS), and zypper (SUSE), along with necessary config files and keys. 

PMC’s Evolution with Pulp 

The PMC team is responsible for operating and maintaining the infrastructure which replicates and serves published packages, manages repositories, and controls publishing access. 

At the heart of PMC’s infrastructure is Pulp, an enterprise-grade, open-source platform for managing software repositories. Before adopting Pulp, PMC relied on a custom-built, in-house application deployed across virtual machines (VM’s) in multiple regions. Due to the entirely custom and aging nature of the infrastructure it was becoming increasingly difficult to maintain and prone to outages. 

Transitioning to Pulp enabled PMC to also modernize its architecture by leveraging best-in-class Azure Services: Azure Blob Storage for scalable package and metadata storage, Azure Kubernetes Service for improved application scalability and availability, and Azure Front Door (AFD) for global content delivery. This shift has significantly enhanced service reliability and operational efficiency. 

Pulp was initially developed by Red Hat to manage RPM packages - handling tasks like fetching, uploading, organizing, and distributing them. Over time, it evolved into a plugin-based system that now supports additional package types, including Debian packages. By integrating Pulp with Azure’s capabilities, PMC achieved greater service stability, reduced dependency on legacy tooling, minimized rate-limiting issues, and gained a more scalable, maintainable solution. Its flexibility and robustness made Pulp the clear choice, allowing the team to focus on delivering packages to users in hundreds of countries, rather than maintaining a bespoke system. 

Better Together: How PMC and Pulp Serve Microsoft Customers 

Pulp manages uploaded artifacts, repositories of content, and metadata for the apt/dnf clients. PMC is best thought of as two tightly integrated components working with a shared database and blob store. 

• Content Ingress: API which handles CRUD operations of repos and packages 

• Content Egress: Delivering packages and associated metadata to customers 

The PMC API manages user authentication and security enforcement in front of the Pulp backend while AFD caches and serves content stored by Pulp in blob storage. Everything in between is handled by Pulp.  

Together, these components form a secure, scalable, and globally accessible platform for distributing Debian and RPM packages to both internal Microsoft teams and external users. 

Figure 1: Content Ingress Architectural Diagram	Figure 2: Content Egress Architectural Diagram

(Click to enlarge)

Package Quality Assurance at Scale 

Before any package is published, PMC performs a signature check. PMC also provides publishers with a package quality container to scan their packages. This scanning check is currently being integrated directly into PMC to streamline quality assurance. 

While Pulp handles core metadata curation and repository management, PMC extends this with additional quality gates and validations tailored to Microsoft’s ecosystem. 

High Availability 

Pulp, as the origin, provides a container-based solution that we run on Azure Kubernetes Service (AKS). PMC then leverages AFD to replicate content across a global mirror fleet, ensuring fast, reliable access to packages regardless of customer location.

Monitoring and Observability 

Pulp provides a tasking system that PMC monitors for failures. PMC also supplements this tasking system by monitoring our own logs and synthetics to ensure service reliability. We monitor the edge as it reflects the customer experience whereas origin monitoring focuses on the internal health of PMC components. As a result, PMC has comprehensive monitoring, both at the edge and internally, to gain deeper insights into the overall health and performance of its service. 

Secure and Reliable Package Delivery 

PMC ensures that packages are delivered with integrity. If a package fails to download or has a checksum mismatch, PMC helps diagnose and resolve the issue. Using the Pulp feature, Checkpoints, PMC enables snapshotting for limited distros, allowing customers to lock into known-good states for safe deployments.  

Partnering with the Pulp Community 

In addition to leveraging Pulp, the PMC team actively contributes to and engages with the broader Pulp community. We regularly contribute to upstream Pulp by submitting bug fixes, reporting issues to Pulp's GitHub issue tracker, and developing new features to benefit both PMC and the entire Pulp ecosystem. 

Below are a few features and smaller improvements the PMC team has contributed back to Pulp: 

1. Checkpoint support: Checkpoints allow you to manage and access historical versions of repositories, enhancing your content management capabilities. Checkpoint Support - A Journey Towards Predictable and Consistent Deployments - Pulp Project 
2. Pulp-deb apt-by-hash support: Implemented apt-by-hash (also known as Acquire-By-Hash) support in pulp-deb, eliminating errors where clients retrieve inconsistent Debian metadata during repository updates (AptByHash - Ubuntu Wiki). 
3. Source Package Support for Debian Plug-in: Collaborated with community contributors to add source package support to the pulp_deb plugin, expanding its capabilities for Debian-based workflows (add source package support to pulp_deb). 
4. Redis Caching Improvements: Enhanced Redis caching mechanisms in Pulp to improve performance and reduce latency (Ensure the Redis cache is actually respecting the TTL). 
5. Signing Service Enhancements: Delivered improvements to Pulp’s signing service, including better key management and signing workflows (Pass correlation id to signing script through ENV variable). 

You can learn more about the PMC service (packages.microsoft.com), file issues, pull requests, or report a security vulnerability on the affiliated GitHub repo: Microsoft Linux Package Repositories. 

Access the most recent packages on https://packages.microsoft.com.