Linux and Open Source Blog articles

Dissecting LLM Container Cold-Start: Where the Time Actually Goes

robcronin — Wed, 15 Apr 2026 22:38:11 GMT

Cold-start latency determines whether GPU clusters can scale to zero, how fast they can autoscale, and whether bursty or low-QPS workloads are economically viable. Most optimization effort targets the container pull path – faster registries, lazy-pull snapshotters, different compression formats. But “cold-start” is actually a composite of pull, runtime startup, and model initialization, and the dominant phase varies dramatically by inference engine. An optimization that cuts time-to-first-token for one engine can be irrelevant for another, even on identical infrastructure.

What we measured

We decomposed cold-start for two architecturally different engines – vLLM (Python/CUDA, heavy JIT compilation) and llama.cpp (C++, minimal runtime) – running Llama 3.1 8B on A100 GPUs. Every run starts from a completely clean slate: containerd stopped, all state wiped, kernel page caches dropped. No warm starts, no pre-pulling, no caching.

We break TTFT into three phases: pull (download + decompression + snapshot creation), startup (container start → server ready), and first inference (first API response, including model weight loading for engines that defer it). We tested across three snapshotters (overlayfs, EROFS, Nydus) with gzip and uncompressed images, pulling from same-region Azure Container Registry.

Setup

All experiments ran on an NVIDIA A100 80GB (Azure NC24ads_A100_v4), pulling from same-region Azure Container Registry. Images were built with AIKit, which produces ModelPack-compliant OCI artifacts with uncompressed model weight layers, Cosign signatures, SBOMs, and provenance attestations. These are supply chain properties you lose when model weights live on a shared drive.

vLLM: startup dominates, pull barely matters

vLLM loads model weights, runs torch.compile, captures CUDA graphs for multiple batch shapes, allocates KV cache, and warms up, all before serving the first request. This takes ~176 seconds regardless of how fast the image arrived.

The breakdown makes the bottleneck obvious: the green bar (startup) is nearly constant across all four variants, swamping any pull-time differences.

Figure 1: vLLM cold-start breakdown. Startup (green, ~176s) dominates regardless of snapshotter.

Method	Pull	Startup	1st Inference	TTFT
overlayfs (gzip)	140.8s ±5.5	176.0s ±3.2	0.16s	317.2s ±2.2
overlayfs (uncomp.)	129.9s ±3.3	180.8s ±12.2	0.16s	310.9s ±8.9
EROFS (gzip)	158.9s ±8.8	175.3s ±0.8	0.16s	334.4s ±8.7
EROFS (uncomp.)	166.3s ±21.1	177.3s ±12.8	0.16s	343.8s ±8.2

Llama 3.1 8B Q4_K_M, ~14 GB image, n=2–3 per variant. ± = sample standard deviation. Three of twelve runs hit intermittent NVIDIA container runtime crashes (exit code 120, unrelated to snapshotters) and were excluded. We excluded Nydus because FUSE-streaming the 14 GB Python/CUDA stack caused startup to exceed 900s. Steady-state inference: ~0.134s across all snapshotters.

44% pull, 56% startup. Dropping gzip saves 11 seconds on a 317-second cold start (1.02x). If your engine is vLLM, optimizing the pull pipeline is the wrong lever.

llama.cpp: pull dominates, compression is the bottleneck

llama.cpp has the opposite profile. Its C++ runtime starts in 2–5 seconds, so the pull becomes the majority of cold-start. This is where filesystem and compression choices actually matter.

Here the picture flips. Pull (blue) is the widest bar, and the gzip-to-uncompressed difference is visible at a glance:

Figure 2: llama.cpp cold-start breakdown. Pull time (blue) dominates for gzip variants.

Method	Pull	Startup	1st Inference	TTFT
overlayfs (gzip)	88.3s ±0.2	5.3s ±0.5	45.1s ±1.4	138.8s ±0.8
overlayfs (uncomp.)	56.3s ±3.1	2.0s ±0.0	44.2s ±0.1	102.4s ±3.1
EROFS (gzip)	92.0s ±2.3	6.1s ±0.5	44.0s ±0.2	142.3s ±1.9
EROFS (uncomp.)	58.8s ±0.6	2.0s ±0.0	44.0s ±0.1	104.8s ±0.5

Llama 3.1 8B Q4_K_M, ~8 GB image, n=3 per variant, 12/12 runs succeeded. First inference includes model weight loading into GPU VRAM (~43s) plus token generation (~1.5s). Steady-state inference: ~1.5s across all snapshotters.

64% pull, 4% startup, 33% model loading. Dropping gzip saves 32 seconds (1.35x) with zero infrastructure changes.

Engine comparison

Placed side by side, the two engines tell opposite stories about the same infrastructure:

Figure 3: Where cold-start time goes. vLLM is compute-bound; llama.cpp is pull-bound.

	vLLM	llama.cpp
Time saved by dropping gzip	11s (3% of TTFT)	32s (23% of TTFT)
Startup time	176–181s	2–5s
Speedup from dropping gzip	1.02x	1.35x

Same optimization, completely different impact. Before investing in pull optimization (compression changes, lazy-pull infrastructure, registry tuning), profile your engine’s startup. If startup dominates, the pull isn’t where the time goes.

Why gzip hurts: model weights are incompressible

The AIKit image is 8.7 GB uncompressed, 6.6 GB with gzip (a modest 0.76x ratio). But this ratio hides what’s really happening:

Layer type	Size	% of image	Gzip ratio
Model weights (GGUF)	4.9 GB	56%	~1.00x (quantized binary, no redundancy)
CUDA + system layers	~3.8 GB	44%	~0.46x (compresses well)

The GGUF file is already quantized to 4-bit precision. Gzip reads every byte, burns CPU, and produces output the same size as the input. You’re paying full decompression cost on 56% of the image for zero size reduction.

Bottom line: gzip is doing real work on less than half your image and producing zero savings on the rest. Dropping it costs nothing and removes a bottleneck from every cold start.

The Nydus prefetch finding

If decompression is the bottleneck, what about skipping the full pull entirely?

Nydus lazy-pull takes a fundamentally different approach: it fetches only manifest metadata during “pull” (~0.7s), then streams model data on-demand via FUSE as the container reads it. Nydus TTFT isn’t directly comparable to the full-pull methods above because the download cost shifts from the pull column to the inference column.

With prefetch enabled, Nydus achieved 77.8s TTFT for llama.cpp vs 139.1s for overlayfs gzip. The critical detail is the prefetch_all flag:

Figure 4: Nydus prefetch ON vs OFF. One config flag, 2.87x difference. Overlayfs gzip shown as baseline.

Configuration	1st Inference	TTFT
Nydus, prefetch ON	72.4s ±0.6	77.8s ±0.5
Nydus, prefetch OFF	218.6s ±2.9	223.4s ±2.9
overlayfs gzip (baseline)	44.0s ±0.4	139.1s ±1.9

n=3 per config, 9/9 runs succeeded. Data: 03-prefetch-config-20260401-030725.csv

One flag in nydusd-config.json, 2.87x difference. Without prefetch, every model weight page fault fires an individual HTTP range request to the registry. With prefetch_all=true, Nydus streams the full blob in the background while the container starts, so chunks arrive ahead of the GPU’s read pattern.

Even with prefetch, Nydus first inference is ~28s slower than overlayfs (72s vs 44s) due to FUSE kernel-user roundtrips during model mmap. Nydus wins on total TTFT because it eliminates the blocking pull, but this overhead means its advantage shrinks on faster networks.

Bottom line: Nydus lazy-pull can halve cold-start for pull-bound engines, but only if prefetch is on. Treat prefetch_all=true as a hard requirement, not a tuning knob.

How to apply these findings

Pick your optimization by engine type

The right optimization depends on where your engine spends its cold-start time. This table summarizes the tradeoffs:

Engine type	Dominant phase	Speedup from dropping gzip	Nydus viable?	Best optimization	What NOT to optimize
vLLM / TensorRT-LLM	Startup (56%)	1.02x — negligible	No — FUSE + Python/CUDA stack exceeded 900s in our tests	Cache torch.compile artifacts and CUDA graphs	Pull pipeline (it’s <44% of TTFT and already fast enough)
llama.cpp / ONNX Runtime	Pull (64%)	1.35x — 32s saved	Yes, with prefetch_all=true (77.8s TTFT vs 139s gzip baseline)	Drop gzip on weight layers; consider lazy-pull on slow links	Startup (already 2–5s; no room to improve)
Large dense models (70B+)	Pull (projected)	>1.35x — scales with image size	Yes, strongest case for lazy-pull	Uncompressed or zstd; Nydus prefetch on bandwidth-constrained links	—

Recommendations

Profile your engine’s startup before touching the pull pipeline. If CUDA compilation dominates (vLLM, TensorRT-LLM), no amount of pull optimization will help. Cache torch.compile artifacts and CUDA graphs instead — production clusters that do this reduce vLLM restarts to ~45–60s.
Drop gzip on model weight layers. For pull-bound engines (llama.cpp, ONNX Runtime), this is the single highest-ROI change: build with --output=type=image,compression=uncompressed, or use AIKit, which defaults to uncompressed weight layers. Quantized model weights (GGUF, safetensors) are already dense binary — gzip burns CPU for a ~1.00x compression ratio on 56% of the image.
If using Nydus, set prefetch_all=true. Without it, every weight page fault triggers an individual HTTP range request and cold-start is 2.87x slower. This is a single flag in nydusd-config.json.
Package models as signed OCI artifacts, not volume mounts. Three CNCF projects implement this pipeline end-to-end: ModelPack defines the OCI artifact spec (model metadata, architecture, quantization format). AIKit builds ModelPack-compliant images with Cosign signatures, SBOMs, and provenance attestations — supply chain guarantees you lose when weights live on a shared drive. KAITO handles the Kubernetes deployment: GPU node provisioning, inference engine setup, and API exposure. Together they cover packaging → build → deploy, and they produce the exact image layout these benchmarks measured.

Why this matters: the cost of cold-start

On an A100 node (~$3–4/hr on major clouds), a 5-minute vLLM cold start burns ~$0.30 in idle GPU time per pod. That sounds small until you multiply it: a cluster that scales 50 pods to zero overnight and restarts them each morning wastes ~$15/day — over $5,000/year — on GPUs sitting idle during pull and CUDA compilation. More critically, cold-start latency determines whether scale-to-zero is feasible at all. If cold-start exceeds your SLO (say, 30s for an interactive app), you’re forced to keep warm replicas running 24/7, which can 2–3x your GPU spend. Cutting llama.cpp cold-start from 139s to 103s by dropping gzip doesn’t just save 36 seconds — it moves the needle on whether autoscaling is viable for your workload.

What this doesn’t cover

zstd compression: decompresses 5–10x faster than gzip; containerd supports it natively. The most obvious gap in this analysis.
Pre-pulling and caching: production clusters pre-pull images and cache CUDA graphs, reducing vLLM restarts to ~45–60s. We measure the cold case: scale-from-zero events and first-time deployments.
Volume-mounted weights: skips the pull entirely, but loses supply chain properties (signing, scanning, provenance).
Larger models (70B+): pull would dominate more, increasing the gzip penalty.
Sample size: n=3 per AIKit variant, n=2–3 per vLLM variant. The gzip finding for llama.cpp is statistically significant (Welch’s t-test, p=0.0014, Cohen’s d=16.3; verification script). Other comparisons are directional.

Reproduce it

Scripts and raw data: erofs-repro-repo. Data for this post: 02-aikit-five-way-20260401-004716.csv and 01-vllm-four-way-20260331-113848.csv. Full analysis: technical report.

Agent Governance Toolkit: Architecture Deep Dive, Policy Engines, Trust, and SRE for AI Agents

mosiddi — Fri, 10 Apr 2026 04:55:22 GMT

Last week we announced the Agent Governance Toolkit on the Microsoft Open Source Blog, an open-source project that brings runtime security governance to autonomous AI agents. In that announcement, we covered the why: AI agents are making autonomous decisions in production, and the security patterns that kept systems safe for decades need to be applied to this new class of workload.

In this post, we'll go deeper into the how: the architecture, the implementation details, and what it takes to run governed agents in production.

The Problem: Production Infrastructure Meets Autonomous Agents

If you manage production infrastructure, you already know the playbook: least privilege, mandatory access controls, process isolation, audit logging, and circuit breakers for cascading failures. These patterns have kept production systems safe for decades.

Now imagine a new class of workload arriving on your infrastructure, AI agents that autonomously execute code, call APIs, read databases, and spawn sub-processes. They reason about what to do, select tools, and act in loops. And in many current deployments, they do all of this without the security controls you'd demand of any other production workload.

That gap is what led us to build the Agent Governance Toolkit: an open-source project, that applies proven security concepts from operating systems, service meshes, and SRE to the emerging world of autonomous AI agents.

To frame this in familiar terms: most AI agent frameworks today are like running every process as root, no access controls, no isolation, no audit trail. The Agent Governance Toolkit is the kernel, the service mesh, and the SRE platform for AI agents.

When an agent calls a tool, say, `DELETE FROM users WHERE created_at < NOW()`, there is typically no policy layer checking whether that action is within scope. There is no identity verification when one agent communicates with another. There is no resource limit preventing an agent from making 10,000 API calls in a minute. And there is no circuit breaker to contain cascading failures when things go wrong.

OWASP Agentic Security Initiative

In December 2025, OWASP published the Agentic AI Top 10: the first formal taxonomy of risks specific to autonomous AI agents. The list reads like a security engineer's nightmare: goal hijacking, tool misuse, identity abuse, memory poisoning, cascading failures, rogue agents, and more.

If you've ever hardened a production server, these risks will feel both familiar and urgent. The Agent Governance Toolkit is designed to help address all 10 of these risks through deterministic policy enforcement, cryptographic identity, execution isolation, and reliability engineering patterns.

Note: The OWASP Agentic Security Initiative has since adopted the ASI 2026 taxonomy (ASI01–ASI10). The toolkit's copilot-governance package now uses these identifiers with backward compatibility for the original AT numbering.

Architecture: Nine Packages, One Governance Stack

The toolkit is structured as a v3.0.0 Public Preview monorepo with nine independently installable packages:

Package	What It Does
Agent OS	Stateless policy engine, intercepts agent actions before execution with configurable pattern matching and semantic intent classification
Agent Mesh	Cryptographic identity (DIDs with Ed25519), Inter-Agent Trust Protocol (IATP), and trust-gated communication between agents
Agent Hypervisor	Execution rings inspired by CPU privilege levels, saga orchestration for multi-step transactions, and shared session management
Agent Runtime	Runtime supervision with kill switches, dynamic resource allocation, and execution lifecycle management
Agent SRE	SLOs, error budgets, circuit breakers, chaos engineering, and progressive delivery, production reliability practices adapted for AI agents
Agent Compliance	Automated governance verification with compliance grading and regulatory framework mapping (EU AI Act, NIST AI RMF, HIPAA, SOC 2)
Agent Lightning	Reinforcement learning training governance with policy-enforced runners and reward shaping
Agent Marketplace	Plugin lifecycle management with Ed25519 signing, trust-tiered capability gating, and SBOM generation
Integrations	20+ framework adapters for LangChain, CrewAI, AutoGen, Semantic Kernel, Google ADK, Microsoft Agent Framework, OpenAI Agents SDK, and more

Agent OS: The Policy Engine

Agent OS intercepts agent tool calls before they execute:

from agent_os import StatelessKernel, ExecutionContext, Policy

kernel = StatelessKernel()
ctx = ExecutionContext(
    agent_id="analyst-1",
    policies=[
        Policy.read_only(),                    # No write operations
        Policy.rate_limit(100, "1m"),          # Max 100 calls/minute
        Policy.require_approval(
            actions=["delete_*", "write_production_*"],
            min_approvals=2,
            approval_timeout_minutes=30,
        ),
    ],
)

result = await kernel.execute(
    action="delete_user_record",
    params={"user_id": 12345},
    context=ctx,
)

The policy engine works in two layers: configurable pattern matching (with sample rule sets for SQL injection, privilege escalation, and prompt injection that users customize for their environment) and a semantic intent classifier that helps detect dangerous goals regardless of phrasing. When an action is classified as `DESTRUCTIVE_DATA`, `DATA_EXFILTRATION`, or `PRIVILEGE_ESCALATION`, the engine blocks it, routes it for human approval, or downgrades the agent's trust level, depending on the configured policy.

Important: All policy rules, detection patterns, and sensitivity thresholds are externalized to YAML configuration files. The toolkit ships with sample configurations in `examples/policies/` that must be reviewed and customized before production deployment. No built-in rule set should be considered exhaustive. Policy languages supported: YAML, OPA Rego, and Cedar.

The kernel is stateless by design, each request carries its own context. This means you can deploy it behind a load balancer, as a sidecar container in Kubernetes, or in a serverless function, with no shared state to manage. On AKS or any Kubernetes cluster, it fits naturally into existing deployment patterns. Helm charts are available for agent-os, agent-mesh, and agent-sre.

Agent Mesh: Zero-Trust Identity for Agents

In service mesh architectures, services prove their identity via mTLS certificates before communicating. AgentMesh applies the same principle to AI agents using decentralized identifiers (DIDs) with Ed25519 cryptography and the Inter-Agent Trust Protocol (IATP):

from agentmesh import AgentIdentity, TrustBridge

identity = AgentIdentity.create(
    name="data-analyst",
    sponsor="alice@company.com",          # Human accountability
    capabilities=["read:data", "write:reports"],
)
# identity.did -> "did:mesh:data-analyst:a7f3b2..."

bridge = TrustBridge()
verification = await bridge.verify_peer(
    peer_id="did:mesh:other-agent",
    required_trust_score=700, # Must score >= 700/1000
)

A critical feature is trust decay: an agent's trust score decreases over time without positive signals. An agent trusted last week but silent since then gradually becomes untrusted, modeling the reality that trust requires ongoing demonstration, not a one-time grant.

Delegation chains enforce scope narrowing: a parent agent with read+write permissions can delegate only read access to a child agent, never escalate.

Agent Hypervisor: Execution Rings

CPU architectures use privilege rings (Ring 0 for kernel, Ring 3 for userspace) to isolate workloads. The Agent Hypervisor applies this model to AI agents:

Ring	Trust Level	Capabilities
Ring 0 (Kernel)	Score ≥ 900	Full system access, can modify policies
Ring 1 (Supervisor)	Score ≥ 700	Cross-agent coordination, elevated tool access
Ring 2 (User)	Score ≥ 400	Standard tool access within assigned scope
Ring 3 (Untrusted)	Score < 400	Read-only, sandboxed execution only

New and untrusted agents start in Ring 3 and earn their way up, exactly the principle of least privilege that production engineers apply to every other workload.

Each ring enforces per-agent resource limits: maximum execution time, memory caps, CPU throttling, and request rate limits. If a Ring 2 agent attempts a Ring 1 operation, it gets blocked, just like a userspace process trying to access kernel memory.

These ring definitions and their associated trust score thresholds are fully configurable via policy. Organizations can define custom ring structures, adjust the number of rings, set different trust score thresholds for transitions, and configure per-ring resource limits to match their security requirements.

The hypervisor also provides saga orchestration for multi-step operations. When an agent executes a sequence, draft email → send → update CRM, and the final step fails, compensating actions fire in reverse. Borrowed from distributed transaction patterns, this ensures multi-agent workflows maintain consistency even when individual steps fail.

Agent SRE: SLOs and Circuit Breakers for Agents

If you practice SRE, you measure services by SLOs and manage risk through error budgets. Agent SRE extends this to AI agents:

When an agent's safety SLI drops below 99 percent, meaning more than 1 percent of its actions violate policy, the system automatically restricts the agent's capabilities until it recovers. This is the same error-budget model that SRE teams use for production services, applied to agent behavior.

We also built nine chaos engineering fault injection templates: network delays, LLM provider failures, tool timeouts, trust score manipulation, memory corruption, and concurrent access races. Because the only way to know if your agent system is resilient is to break it intentionally.

Agent SRE integrates with your existing observability stack through adapters for Datadog, PagerDuty, Prometheus, OpenTelemetry, Langfuse, LangSmith, Arize, MLflow, and more. Message broker adapters support Kafka, Redis, NATS, Azure Service Bus, AWS SQS, and RabbitMQ.

Compliance and Observability

If your organization already maps to CIS Benchmarks, NIST AI RMF, or other frameworks for infrastructure compliance, the OWASP Agentic Top 10 is the equivalent standard for AI agent workloads. The toolkit's agent-compliance package provides automated governance grading against these frameworks.

The toolkit is framework-agnostic, with 20+ adapters that hook into each framework's native extension points, so adding governance to an existing agent is typically a few lines of configuration, not a rewrite.

The toolkit exports metrics to any OpenTelemetry-compatible platform, Prometheus, Grafana, Datadog, Arize, or Langfuse. If you're already running an observability stack for your infrastructure, agent governance metrics flow through the same pipeline.

Key metrics include: policy decisions per second, trust score distributions, ring transitions, SLO burn rates, circuit breaker state, and governance workflow latency.

Getting Started

# Install all packages
pip install agent-governance-toolkit[full]

# Or individual packages
pip install agent-os-kernel agent-mesh agent-sre

The toolkit is available across language ecosystems: Python, TypeScript (`@microsoft/agentmesh-sdk` on npm), Rust, Go, and .NET (`Microsoft.AgentGovernance` on NuGet).

Azure Integrations

While the toolkit is platform-agnostic, we've included integrations that help enable the fastest path to production, on Azure:

Azure Kubernetes Service (AKS): Deploy the policy engine as a sidecar container alongside your agents. Helm charts provide production-ready manifests for agent-os, agent-mesh, and agent-sre.

Azure AI Foundry Agent Service: Use the built-in middleware integration for agents deployed through Azure AI Foundry.

OpenClaw Sidecar: One compelling deployment scenario is running OpenClaw, the open-source autonomous agent, inside a container with the Agent Governance Toolkit deployed as a sidecar. This gives you policy enforcement, identity verification, and SLO monitoring over OpenClaw's autonomous operations. On Azure Kubernetes Service (AKS), the deployment is a standard pod with two containers: OpenClaw as the primary workload and the governance toolkit as the sidecar, communicating over localhost. We have a reference architecture and Helm chart available in the repository.

The same sidecar pattern works with any containerized agent, OpenClaw is a particularly compelling example because of the interest in autonomous agent safety.

Tutorials and Resources

34+ step-by-step tutorials covering policy engines, trust, compliance, MCP security, observability, and cross-platform SDK usage are available in the repository.

git clone https://github.com/microsoft/agent-governance-toolkit
cd agent-governance-toolkit
pip install -e "packages/agent-os[dev]" -e "packages/agent-mesh[dev]" -e "packages/agent-sre[dev]"

# Run the demo
python -m agent_os.demo

What's Next

AI agents are becoming autonomous decision-makers in production infrastructure, executing code, managing databases, and orchestrating services. The security patterns that kept production systems safe for decades, least privilege, mandatory access controls, process isolation, audit logging, are exactly what these new workloads need. We built them. They're open source.

We're building this in the open because agent security is too important for any single organization to solve alone:

Security research: Adversarial testing, red-team results, and vulnerability reports strengthen the toolkit for everyone.
Community contributions: Framework adapters, detection rules, and compliance mappings from the community expand coverage across ecosystems.

We are committed to open governance. We're releasing this project under Microsoft today, and we aspire to move it into a foundation home, such as the AI and Data Foundation (AAIF), where it can benefit from cross-industry stewardship. We're actively engaging with foundation partners on this path.

The Agent Governance Toolkit is open source under the MIT license. Contributions welcome at github.com/microsoft/agent-governance-toolkit.

DPDK 25.11 Performance on Azure for High-Speed Packet Workloads

KashanK — Wed, 01 Apr 2026 18:37:04 GMT

At Microsoft Azure, performance is treated as an ongoing discipline grounded in careful engineering and real-world validation. As cloud workloads grow in scale and variety, customers depend on consistent, high-throughput networking. Technologies such as the Data Plane Development Kit (DPDK) play a key role in meeting these expectations

To support customers running advanced network functions, we’ve released our latest performance report based on DPDK 25.11. It is now available in the DPDK performance catalog (Microsoft Azure DPDK Performance Report). The report provides a clear view of how DPDK performs on Microsoft-developed Azure Boost within Azure infrastructure, with detailed insights into packet processing across a range of scenarios, from small packet sizes to multi-core scaling.

Why We Test DPDK on Azure

DPDK is widely used for high-performance packet processing in virtualized environments. It powers a range of workloads from customer-deployed virtual network functions to internal Azure network appliances.

But simply enabling DPDK is not enough. To ensure optimal performance, we validate it under realistic conditions, including:

Azure VM configurations with Accelerated Networking
NUMA-aware memory and CPU alignment
Hugepage-backed memory allocation
Multi-core PMD thread scaling
Packet forwarding using real traffic generators

This helps us understand how DPDK performs in actual cloud environments, not just idealized lab setups.

What the Report Covers

The DPDK 25.11 report includes performance benchmarks across different frame sizes, ranging from 64 bytes to 1518 bytes. It also evaluates CPU usage, queue configuration, and latency stability across various test conditions.

Key Report Highlights:

Line-rate throughput is achievable at common frame sizes when vCPUs are pinned correctly and memory is properly configured
Low jitter and consistent latency are observed across multi-queue and multi-core tests
Performance scales nearly linearly with additional cores, especially for smaller packet sizes
Queue and PMD thread alignment with the NUMA layout plays a critical role in maximizing efficiency

All tests were performed using Azure VM SKUs equipped with Microsoft NICs and configured for optimal isolation and performance.

Why We Shared This with the Community

Publishing this report reflects our commitment to open engineering and ecosystem collaboration. We believe performance transparency benefits everyone in the ecosystem, including developers, operators, and customers.

Here are a few reasons why we share:

It helps customers plan and tune their workloads using validated performance envelopes
It enables vendors and contributors to optimize drivers, firmware, and applications based on real-world data
It encourages reproducibility and standardization in cloud DPDK benchmarking
It creates a feedback loop between Azure, the DPDK community, and our partners

Our goal is not just to test internally but to foster open dialogue and measurable improvement across platforms.

Recommendations for Running DPDK on Azure

Based on the test results, we offer the following best practices for customers deploying DPDK-based applications:

Area	Recommendation
VM Selection	Choose Accelerated Networking-enabled SKUs like D, Fsv2, or Eav4
CPU Pinning	Use dedicated cores for PMD threads and align with NUMA topology
Memory	Configure hugepages and allocate memory from the local NUMA node
Queue Mapping	Match RX and TX queues to available vCPUs to avoid contention
Packet Generator	Use pktgen-dpdk or testpmd with controlled traffic profiles

These settings can significantly improve consistency and peak throughput across many DPDK scenarios.

Get Involved and Reproduce the Results

We invite you to read the full report and try the configurations in your own environment. Whether you are running a firewall, a router, or a telemetry appliance, DPDK on Azure offers scalable performance with the right tuning.

You can:

Download the report at Microsoft Azure DPDK Performance Report
Replicate the test setup using Azure VMs and your preferred packet generator github.com/mcgov/dpdk-perf
Share your feedback with us through GitHub or community channels or send feedback dpdk@microsoft.com
Suggest improvements or contribute new scenarios to future performance reports

Conclusion

DPDK is a powerful enabler of high-performance networking in the cloud. With this report, we aim to make Azure performance data open, useful, and actionable. It reflects our ongoing investment in validating and improving the underlying infrastructure that supports mission-critical workloads.

We thank the DPDK community for ongoing collaboration. We look forward to continued engagement as we scale performance transparency in cloud-native environments.

Run OpenClaw Agents on Azure Linux VMs (with Secure Defaults)

johnsonshi_msft — Sun, 22 Mar 2026 16:34:46 GMT

Many teams want an enterprise-ready personal AI assistant, but they need it on infrastructure they control, with security boundaries they can explain to IT. That is exactly where OpenClaw fits on Azure.

OpenClaw is a self-hosted, always-on personal agent runtime you run in your enterprise environment and Azure infrastructure. Instead of relying only on a hosted chat app from a third-party provider, you can deploy, operate, and experiment with an agent on an Azure Linux VM you control — using your existing GitHub Copilot licenses, Azure OpenAI deployments, or API plans from OpenAI, Anthropic Claude, Google Gemini, and other model providers you already subscribe to. Once deployed on Azure, you can interact with an OpenClaw agent through familiar channels like Microsoft Teams, Slack, Telegram, WhatsApp, and many more!

For Azure users, this gives you a practical middle ground: modern personal-agent workflows on familiar Azure infrastructure.

What is OpenClaw, and how is it different from ChatGPT/Claude/chat apps?

OpenClaw is a self-hosted personal agent runtime that can be hosted on Azure compute infrastructure.

How it differs:

ChatGPT/Claude apps are primarily hosted chat experiences tied to one provider's models
OpenClaw is an always-on runtime you operate yourself, backed by your choice of model provider — GitHub Copilot, Azure OpenAI, OpenAI, Anthropic Claude, Google Gemini, and others
OpenClaw lets you keep the runtime boundary in your own Azure VM environment within your Azure enterprise subscription

In practice, OpenClaw is useful when you want a persistent assistant for operational and workflow tasks, with your own infrastructure as the control point. You bring whatever model provider and API plan you already have — OpenClaw connects to it.

Why Azure Linux VMs?

Azure Linux VMs are a strong fit because they provide:

A suitable host machine for the OpenClaw agent to run on
Enterprise-friendly infrastructure and identity workflows
Repeatable provisioning via the Azure CLI
Network hardening with NSG rules
Managed SSH access through Azure Bastion instead of public SSH exposure

How to Set Up OpenClaw on an Azure Linux VM

This guide sets up an Azure Linux VM, applies NSG (Network Security Group) hardening, configures Azure Bastion for managed SSH access, and installs an always-on OpenClaw agent within the VM that you can interact with through various messaging channels.

What you'll do

Create Azure networking (VNet, subnets, NSG) and compute resources with the Azure CLI
Apply Network Security Group rules so VM SSH is allowed only from Azure Bastion
Use Azure Bastion for SSH access (no public IP on the VM)
Install OpenClaw on the Azure VM
Verify OpenClaw installation and configuration on the VM

What you need

An Azure subscription with permission to create compute and network resources
Azure CLI installed (install steps)
An SSH key pair (the guide covers generating one if needed)
~20–30 minutes

Configure deployment

Step 1: Sign in to Azure CLI

az login # Select a suitable Azure subscription during Azure login az extension add -n ssh # SSH extension is required for Azure Bastion SSH

The ssh extension is required for Azure Bastion native SSH tunneling.

Step 2: Register required resource providers (one-time)

az provider register --namespace Microsoft.Compute az provider register --namespace Microsoft.Network

Verify registration. Wait until both show Registered.

az provider show --namespace Microsoft.Compute --query registrationState -o tsv az provider show --namespace Microsoft.Network --query registrationState -o tsv

Step 3: Set deployment variables

Set the deployment environment variables that will be needed throughout this guide.

RG="rg-openclaw" LOCATION="westus2" VNET_NAME="vnet-openclaw" VNET_PREFIX="10.40.0.0/16" VM_SUBNET_NAME="snet-openclaw-vm" VM_SUBNET_PREFIX="10.40.2.0/24" BASTION_SUBNET_PREFIX="10.40.1.0/26" NSG_NAME="nsg-openclaw-vm" VM_NAME="vm-openclaw" ADMIN_USERNAME="openclaw" BASTION_NAME="bas-openclaw" BASTION_PIP_NAME="pip-openclaw-bastion"

Adjust names and CIDR ranges to fit your environment. The Bastion subnet must be at least /26.

Step 4: Select SSH key

Use your existing public key if you have one:

SSH_PUB_KEY="$(cat ~/.ssh/id_ed25519.pub)"

If you don't have an SSH key yet, generate one:

ssh-keygen -t ed25519 -a 100 -f ~/.ssh/id_ed25519 -C "you@example.com" SSH_PUB_KEY="$(cat ~/.ssh/id_ed25519.pub)"

Step 5: Select VM size and OS disk size

VM_SIZE="Standard_B2as_v2" OS_DISK_SIZE_GB=64

Choose a VM size and OS disk size available in your subscription and region:

Start smaller for light usage and scale up later
Use more vCPU/RAM/disk for heavier automation, more channels, or larger model/tool workloads
If a VM size is unavailable in your region or subscription quota, pick the closest available SKU

List VM sizes available in your target region:

az vm list-skus --location "${LOCATION}" --resource-type virtualMachines -o table

Check your current vCPU and disk usage/quota:

az vm list-usage --location "${LOCATION}" -o table

Deploy Azure resources

Step 1: Create the resource group

The Azure resource group will contain all of the Azure resources that the OpenClaw agent needs.

az group create -n "${RG}" -l "${LOCATION}"

Step 2: Create the network security group

Create the NSG and add rules so only the Bastion subnet can SSH into the VM.

az network nsg create \ -g "${RG}" -n "${NSG_NAME}" -l "${LOCATION}" # Allow SSH from the Bastion subnet only az network nsg rule create \ -g "${RG}" --nsg-name "${NSG_NAME}" \ -n AllowSshFromBastionSubnet --priority 100 \ --access Allow --direction Inbound --protocol Tcp \ --source-address-prefixes "${BASTION_SUBNET_PREFIX}" \ --destination-port-ranges 22 # Deny SSH from the public internet az network nsg rule create \ -g "${RG}" --nsg-name "${NSG_NAME}" \ -n DenyInternetSsh --priority 110 \ --access Deny --direction Inbound --protocol Tcp \ --source-address-prefixes Internet \ --destination-port-ranges 22 # Deny SSH from other VNet sources az network nsg rule create \ -g "${RG}" --nsg-name "${NSG_NAME}" \ -n DenyVnetSsh --priority 120 \ --access Deny --direction Inbound --protocol Tcp \ --source-address-prefixes VirtualNetwork \ --destination-port-ranges 22

The rules are evaluated by priority (lowest number first): Bastion traffic is allowed at 100, then all other SSH is blocked at 110 and 120.

Step 3: Create the virtual network and subnets

Create the VNet with the VM subnet (NSG attached), then add the Bastion subnet.

az network vnet create \ -g "${RG}" -n "${VNET_NAME}" -l "${LOCATION}" \ --address-prefixes "${VNET_PREFIX}" \ --subnet-name "${VM_SUBNET_NAME}" \ --subnet-prefixes "${VM_SUBNET_PREFIX}" # Attach the NSG to the VM subnet az network vnet subnet update \ -g "${RG}" --vnet-name "${VNET_NAME}" \ -n "${VM_SUBNET_NAME}" --nsg "${NSG_NAME}" # AzureBastionSubnet — name is required by Azure az network vnet subnet create \ -g "${RG}" --vnet-name "${VNET_NAME}" \ -n AzureBastionSubnet \ --address-prefixes "${BASTION_SUBNET_PREFIX}"

Step 4: Create the Virtual Machine

Create the VM with no public IP. SSH access for OpenClaw configuration will be exclusively through Azure Bastion.

az vm create \ -g "${RG}" -n "${VM_NAME}" -l "${LOCATION}" \ --image "Canonical:ubuntu-24_04-lts:server:latest" \ --size "${VM_SIZE}" \ --os-disk-size-gb "${OS_DISK_SIZE_GB}" \ --storage-sku StandardSSD_LRS \ --admin-username "${ADMIN_USERNAME}" \ --ssh-key-values "${SSH_PUB_KEY}" \ --vnet-name "${VNET_NAME}" \ --subnet "${VM_SUBNET_NAME}" \ --public-ip-address "" \ --nsg ""

--public-ip-address "" prevents a public IP from being assigned.

--nsg "" skips creating a per-NIC NSG (the subnet-level NSG created earlier handles security).

Reproducibility: The command above uses latest for the Ubuntu image. To pin a specific version, list available versions and replace latest:

az vm image list \ --publisher Canonical --offer ubuntu-24_04-lts \ --sku server --all -o table

Step 5: Create Azure Bastion

Azure Bastion provides secure-managed SSH access to the VM without exposing a public IP.

Bastion Standard SKU with tunneling is required for CLI-based "az network bastion ssh" command.

az network public-ip create \ -g "${RG}" -n "${BASTION_PIP_NAME}" -l "${LOCATION}" \ --sku Standard --allocation-method Static az network bastion create \ -g "${RG}" -n "${BASTION_NAME}" -l "${LOCATION}" \ --vnet-name "${VNET_NAME}" \ --public-ip-address "${BASTION_PIP_NAME}" \ --sku Standard --enable-tunneling true

Bastion provisioning typically takes 5–10 minutes but can take up to 15–30 minutes in some regions.

Step 6: Verify Deployments

After all resources are deployed, your resource group should look like the following:

Install OpenClaw

Step 1: SSH into the VM through Azure Bastion

VM_ID="$(az vm show -g "${RG}" -n "${VM_NAME}" --query id -o tsv)" az network bastion ssh \ --name "${BASTION_NAME}" \ --resource-group "${RG}" \ --target-resource-id "${VM_ID}" \ --auth-type ssh-key \ --username "${ADMIN_USERNAME}" \ --ssh-key ~/.ssh/id_ed25519

Step 2: Install OpenClaw (in the Bastion SSH shell)

curl -fsSL https://openclaw.ai/install.sh | bash

The installer installs Node LTS and dependencies if not already present, installs OpenClaw, and launches the OpenClaw onboarding wizard. For more information, see the open source OpenClaw install docs.

OpenClaw Onboarding: Choosing an AI Model Provider

During OpenClaw onboarding, you'll choose the AI model provider for the OpenClaw agent. This can be GitHub Copilot, Azure OpenAI, OpenAI, Anthropic Claude, Google Gemini, or another supported provider. See the open source OpenClaw install docs for details on choosing an AI model provider when going through the onboarding wizard.

Most enterprise Azure teams already have GitHub Copilot licenses. If that is your case, we recommend choosing the GitHub Copilot provider in the OpenClaw onboarding wizard. See the open source OpenClaw docs on configuring GitHub Copilot as the AI model provider.

OpenClaw Onboarding: Setting up Messaging Channels

During OpenClaw onboarding, there will be an optional step where you can set up various messaging channels to interact with your OpenClaw agent.

For first time users, we recommend setting up Telegram due to ease of setup. Other messaging channels such as Microsoft Teams, Slack, WhatsApp, and others can also be set up.

To configure OpenClaw for messaging through chat channels, see the open source OpenClaw chat channels docs.

Step 3: Verify OpenClaw Configuration

To validate that everything was set up correctly, run the following commands within the same Bastion SSH session:

openclaw status openclaw gateway status

If there are any issues reported, you can run the onboarding wizard again with the steps above. Alternatively, you can run the following command:

openclaw doctor

Message OpenClaw

Once you have configured the OpenClaw agent to be reachable via various messaging channels, you can verify that it is responsive by messaging it.

Enhancing OpenClaw for Use Cases

There you go! You now have a 24/7, always-on personal AI agent, living on its own Azure VM environment.

For awesome OpenClaw use cases, check out the awesome-openclaw-usecases repository.
To enhance your OpenClaw agent with additional AI skills so that it can autonomously perform multi-step operations on any domain, check out the awesome-openclaw-skills repository.
You can also check out ClawHub and ClawSkills, two popular open source skills directories that can enhance your OpenClaw agent.

Cleanup

To delete all resources created by this guide:

az group delete -n "${RG}" --yes --no-wait

This removes the resource group and everything inside it (VM, VNet, NSG, Bastion, public IP). This also deletes the OpenClaw agent running within the VM.

If you'd like to dive deeper about deploying OpenClaw on Azure, please check out the open source OpenClaw on Azure docs.

How Netstar Streamlined Fleet Monitoring and Reduced Custom Integrations with Drasi

CollinBrian — Thu, 05 Mar 2026 19:23:46 GMT

When a high-value container goes silent between waystations, logistics teams lose critical visibility, risking delays that can cascade into port congestion and missed connections. Netstar, a connected fleet solutions provider supporting customers like Maersk, faced this challenge as its operations scaled. Timely notifications of delays, arrivals, and status changes became critical to keeping cargo moving efficiently through port systems.

To address growing integration complexity and the need for real-time responsiveness, Netstar adopted Drasi. Drasi, built for change-driven solutions, provides continuously updated query results and automated reactions to data changes, enabling systems to detect and respond to critical changes as they happen. This shift to Drasi became foundational to how Netstar unified its fleet data, reduced engineering overhead, and improved monitoring workflows.

The Fragmentation Challenge

Growing operational complexity made an underlying challenge increasingly apparent. Tracking a container's journey from pickup to port terminal required reconciling data such as vehicle identifiers, waypoints, GPS location feeds, and IoT telemetry signals from siloed systems. With each new operational or business requirement, whether monitoring vehicle health or detecting route deviations, development teams found themselves repeatedly rebuilding similar patterns.

"We were essentially rebuilding the same integration architecture for every use case," explains Daniel Joubert, General Manager and technical lead at Netstar. "One week we'd build a dashboard for location tracking. The next week, we'd build another one for breakdown detection. The engineering overhead was unsustainable."

Batch-based processing compounded the issue. Critical signals such as missed health reports or route delays can surface long after they occur, potentially limiting Netstar’s ability to take timely action.

Introducing Drasi for Change-driven Architecture

Rather than continue building point solutions, Netstar adopted Drasi as the backbone of its real-time data architecture. Drasi simplifies systems that must detect, evaluate, and react to data changes quickly and efficiently at scale, aligning directly with Netstar’s needs.

A Unified, Continuously Updated View

Drasi connected directly to Netstar's existing data sources- Azure SQL databases for information such as vehicle identifiers and waypoints, and Azure EventHub for GPS location data and IoT telemetry. Drasi Continuous Queries joined this information into a single, always-current operational picture. Instead of multiple custom-built pipelines, Netstar gained a single source of truth for its fleet.

Using Drasi Reactions, Netstar defined actions that trigger when specific events occur. When a truck fails to send a health signal within its expected window, or when a delay notification indicates potential supply chain disruption, the system responds immediately without human intervention, reducing the likelihood of missed events.

Improvements Enabled by Drasi

Using the Drasi plugin for Grafana, Netstar consolidated results from Continuous Queries into one monitoring interface. Operators no longer reconciled conflicting views across separate tools; they now track vehicle health, location, alerts, and route deviations in real time from a single dashboard.

"The transformation was remarkable," says Dustyn Lightfoot, Solution Architect. "We were able to use a single Drasi instance to support multiple business use cases without building new infrastructure or writing additional code, for example, to stand up Blazor websites. More importantly, it eliminated the ongoing maintenance burden of managing dozens of custom pipelines."

Drasi’s flexibility also extended beyond fleet tracking. By attaching an additional data source and defining new Continuous Queries, the same Drasi instance now surfaces changes in customer billing status and the legal contracts. This work required no new infrastructure, just connecting the source and writing queries (leveraging Drasi’s custom Delta functions), providing business teams with up-to-date information without a separate integration effort.

Measurable Impact

Netstar reports tangible improvements across engineering operations and real-time responsiveness:

Faster incident response: Missing health signals now trigger alerts immediately rather than being discovered later through manual checks, improving the speed and reliability of operational response.

Improved logistics coordination: Real time visibility into container movement through waystations and toward port terminals has enabled Netstar and partners like Maersk to coordinate shipments more efficiently, with automated alerts keeping all stakeholders informed as conditions change.

Reduced development overhead: Using Drasi has reduced the amount of custom development previously needed to support fleet monitoring capabilities. The same Drasi-driven architecture now supports multiple business cases, from tracking and health monitoring to route optimization.

Streamlined operator experience: Teams moved from several monitoring tools to a single Drasi-powered Grafana interface, simplifying daily operations and eliminating time spent reconciling conflicting data from different systems.

Industry Context and What’s Next

Demand for real-time supply chain visibility has intensified as global logistics disruptions highlight the risks of delayed reporting.

"Our customers don't just want historical reports anymore. They need to know what's happening right now and be alerted the moment something changes," Daniel Joubert explains. "That shift from batch processing to continuous monitoring is becoming table stakes in fleet management."

Building on this foundation, Netstar is now investigating how Drasi can support predictive maintenance- spotting patterns in vehicle health data early enough to prevent failures altogether. The same change-driven architecture could also streamline coordination across broader supply chain workflows.

The Broader Architectural Shift

Netstar’s implementation reflects a wider architectural move emerging across operational solutions: from systems that store and query data to platforms that detect and react to changes as they happen. In fleet logistics, financial systems, and industrial operations, the competitive advantage increasingly lies in eliminating the lag between event and response.

"Building custom integrations for every use case was slowing us down and limiting what we could deliver to customers," Dustyn Lightfoot reflects. "Drasi gave us a reusable foundation that handles the hard parts, integrating disparate data sources and detecting meaningful changes, so we can focus on solving business problems rather than rebuilding infrastructure."

The collaboration between Drasi and Netstar demonstrates how open source change-driven platforms can simplify complex operational challenges whilst providing actionable insights across distributed systems. As logistics operations evolve, architectures like Drasi’s may define the next era of competitive advantage- one where actionable insight arrives the moment conditions change.

To learn more about Drasi visit Drasi.io.

Retina 1.0 Is Now Available

kamilp — Tue, 03 Feb 2026 14:36:06 GMT

We are excited to announce the first major release of Retina - a significant milestone for the project. This version brings along many new features, enhancements and bug fixes.

The Retina maintainer team would like to thank all contributors, community members, and early adopters who helped make this 1.0 release possible.

What is Retina?

Retina is an open-source, Kubernetes network observability platform. It enables you to continuously observe and measure network health, and investigate network issues on-demand with integrated Kubernetes-native workflows.

Why Retina?

Kubernetes networking failures are rarely isolated or easy to reproduce. Pods are ephemeral, services span multiple nodes, and network traffic crosses multiple layers (CNI, kube-proxy, node networking, policies), making crucial evidence difficult to capture. Manually connecting to nodes and stitching together logs or packet captures simply does not scale as clusters grow in size and complexity.

A modern approach to observability must automate and centralize data collection while exposing rich, actionable insights.

Retina represents a major step forward in solving the complexities of Kubernetes observability by leveraging the power of eBPF. Its cloud-agnostic design, deep integration with Hubble, and support for both real-time metrics and on-demand packet captures make it an invaluable tool for DevOps, SecOps, and compliance teams across diverse environments.

What Does It Do?

Retina can collect two types of telemetry: metrics and packet captures.

The Retina shell enables ad-hoc troubleshooting via pre-installed networking tools.

Metrics

Metrics provide continuous observability. They can be exported to multiple storage options such as Prometheus or Azure Monitor, and visualized in a variety of ways, including Grafana or Azure Log Analytics.

Retina supports two control planes: Hubble and Standard. Both are supported regardless of the underlying CNI. The choice of control plane affects the metrics which are collected.

You can customize which metrics are collected by enabling/disabling their corresponding plugins. Some examples of metrics may include:

Incoming/outcoming traffic
Dropped packets
TCP/UDP
DNS
API Server latency
Node/interface statistics

Grafana dashboard visualizing metrics from Retina - showing packets being dropped on the cluster.

Packet Captures

Captures provide on-demand observability. They allow users to perform distributed packet captures across the cluster, based on specified Nodes/Pods and other supported filters. They can be triggered via the CLI or through the capture CRD, and may be output to persistent storage options such as the host filesystem, a PVC, or a storage blob.

The result of the capture contains more than just a .pcap file. Retina also captures a number of networking metadata such as iptables rules, socket statistics, kernel network information from /proc/net, and more.

Retina packet capture performed through the CLI.

Shell

The Retina shell enables deep ad-hoc troubleshooting by providing a suite of networking tools. The CLI command starts an interactive shell on a Kubernetes node that runs a container image which includes standard tools such as ping or curl, as well as specialized tools like bpftool, pwru, Inspektor Gadget and more.

The Retina shell is currently only available on Linux. Note that some tools require particular capabilities to execute. These can be passed as parameters through the CLI.

Retina shell CLI - showcasing some of the available tools, including ping, dig, bpftool and pwru.

Use Cases

Debugging Pod Connectivity Issues: When services can’t communicate, Retina enables rapid, automated distributed packet capture and drop metrics, drastically reducing troubleshooting time. The Retina shell also brings specialized tools for deep manual investigations.
Continuous Monitoring of Network Health: Operators can set up alerts and dashboards for DNS failures, API server latency, or packet drops, gaining ongoing visibility into cluster networking.
Security Auditing and Compliance: Flow logs (in Hubble mode) and metrics support security investigations and compliance reporting, enabling quick identification of unexpected connections or data transfers.
Multi-Cluster / Multi-Cloud Visibility: Retina standardizes network observability across clouds, supporting unified dashboards and processes for SRE teams.

Where Does It Run?

Retina is designed for broad compatibility across Kubernetes distributions, cloud providers, and operating systems. There are no Azure-specific dependencies - Retina runs anywhere Kubernetes does.

Operating Systems: Both Linux and Windows nodes are supported.
Kubernetes Distributions: Retina is distribution-agnostic, deployable on managed services (AKS, EKS, GKE) or self-managed clusters.
CNI / Network Stack: Retina works with any CNI, focusing on kernel-level events rather than CNI-specific logs.
Cloud Integration: Retina exports metrics to Azure Monitor and Log Analytics, with pre-built Grafana dashboards for AKS. Integration with AWS CloudWatch or GCP Stackdriver is possible via Prometheus.
Observability Stacks: Retina integrates with Prometheus & Grafana, Cilium Hubble (for flow logs and UI), and can be extended to other exporters.

High-level overview of where Retina runs.

Design Overview

Retina’s architecture consists of two layers: a data collection layer in the kernel-space, and processing layer that converts low-level signals into Kubernetes-aware telemetry in the user-space.

When Retina is installed, each node in the cluster runs a Retina agent which collects raw network telemetry from the host kernel - backed by eBPF on Linux, and HNS/VFP on Windows. The agent processes the raw network data and enriches it with Kubernetes metadata, which is then exported for consumption by monitoring tools such as Prometheus, Grafana, or Hubble UI.

Modularity and extensibility are central to the design philosophy. Retina's plugin model lets you enable only the telemetry you need, and add new sources by implementing a common plugin interface. Built-in plugins include Drop Reason, DNS, Packet Forward, and more.

Check out our architecture docs for a deeper dive into Retina's design.

Get Started

Thanks to Helm charts deploying Retina is streamlined across all environments, and can be done with one configurable command. For complete documentation, visit our installation docs.

To install Retina with the Standard control plane and Basic metrics mode:

VERSION=$( curl -sL https://api.github.com/repos/microsoft/retina/releases/latest | jq -r .name) helm upgrade --install retina oci://ghcr.io/microsoft/retina/charts/retina \ --version $VERSION \ --namespace kube-system \ --set image.tag=$VERSION \ --set operator.tag=$VERSION \ --set logLevel=info \ --set operator.enabled=true \ --set enabledPlugin_linux="\[dropreason\,packetforward\,linuxutil\,dns\]"

Once Retina is running in your cluster, you can then configure Prometheus and Grafana to scrape and visualize your metrics.

Install the Retina CLI with Krew:

kubectl krew install retina

Get Involved

Retina is open-source under the MIT License and welcomes community contributions. Since its announcement in early 2024, the project has gained significant traction, with contributors from multiple organizations helping to expand its capabilities.

The project is hosted on GitHub · microsoft/retina and documentation is available at retina.sh.

If you would like to contribute to Retina you can follow our contributor guide.

What's Next?

Retina 1.1 of course!

We are also discussing the future roadmap, and exploring the possibility of moving the project to community ownership. Stay tuned!

In the meantime, we welcome you to raise an issue if you find any bugs, or start a discussion if you have any questions or suggestions.

You can also reach out to the Retina team via email, we would love to hear from you!

References

Scaling DNS on AKS with Cilium: NodeLocal DNSCache, LRP, and FQDN Policies

Simone_Rodigari — Tue, 10 Mar 2026 09:23:51 GMT

Why Adopt NodeLocal DNSCache?

The primary drivers for adoption are usually:

Eliminating Conntrack Pressure: In high-QPS UDP DNS scenarios, conntrack contention and UDP tracking can cause intermittent DNS response loss and retries; depending on resolver retry/timeouts, this can appear as multi-second lookup delays and sometimes much longer tails.
Reducing Latency: By placing a cache on every node, you remove the network hop to the CoreDNS service. Responses are practically instantaneous for cached records.
Offloading CoreDNS: A DaemonSet architecture effectively shards the DNS query load across the entire cluster, preventing the central CoreDNS deployment from becoming a single point of congestion during bursty scaling events.

Who needs this?

You should prioritize this architecture if you run:

Large-scale clusters large clusters (hundreds of nodes or thousands of pods), where CoreDNS scaling becomes difficult to manage.
High-churn endpoints, such as spot instances or frequent auto-scaling jobs that trigger massive waves of DNS queries.
Real-time applications where multi-second (and occasionally longer) DNS lookup delays are unacceptable.

The Challenge with Cilium

Deploying NodeLocal DNSCache on a cluster managed by Cilium (CNI) requires a specific approach. Standard NodeLocal DNSCache relies on node-level interface/iptables setup. In Cilium environments, you can instead implement the interception via Cilium Local Redirect Policy (LRP), which redirects traffic destined to the kube-dns ClusterIP service to a node-local backend pod.

This post details a production-ready deployment strategy aligned with Cilium’s Local Redirect Policy model. It covers necessary configuration tweaks to avoid conflicts and explains how to maintain security filtering.

Architecture Overview

In a standard Kubernetes deployment, NodeLocal DNSCache creates a dummy network interface and uses extensive iptables rules to hijack traffic destined for the Cluster DNS IP.

When using Cilium, we can achieve this more elegantly and efficiently using Local Redirect Policies.

DaemonSet: Runs node-local-dns on every node.
Configuration: Configured to skip interface creation and iptables manipulation.
Redirection: Cilium LRP intercepts traffic to the kube-dns Service IP and redirects it to the local pod on the same node.

1. The NodeLocal DNSCache DaemonSet

The critical difference in this manifest is the arguments passed to the node-local-dns binary. We must explicitly disable its networking setup functions to let Cilium handle the traffic.

The NodeLocal DNSCache deployment also requires the node-local-dns ConfigMap and the kube-dns-upstream Service (plus RBAC/ServiceAccount). For brevity, the snippet below shows only the DaemonSet arguments that differ in the Cilium/LRP approach. The node-cache reads the template Corefile (/etc/coredns/Corefile.base) and generates the active Corefile (/etc/Corefile). The -conf flag points CoreDNS at the active Corefile it should load.

The node-cache binary accepts -localip as an IP list; 0.0.0.0 is a valid value and makes it listen on all interfaces, appropriate for the LRP-based redirection model.

apiVersion: apps/v1 kind: DaemonSet metadata: name: node-local-dns namespace: kube-system labels: k8s-app: node-local-dns spec: selector: matchLabels: k8s-app: node-local-dns template: metadata: labels: k8s-app: node-local-dns annotations: # Optional: policy.cilium.io/no-track-port can be used to bypass conntrack for DNS. # Validate the impact on your Cilium version and your observability/troubleshooting needs. policy.cilium.io/no-track-port: "53" spec: # IMPORTANT for the "LRP + listen broadly" approach: # keep hostNetwork off so you don't hijack node-wide :53 hostNetwork: false # Don't use cluster DNS dnsPolicy: Default containers: - name: node-cache image: registry.k8s.io/dns/k8s-dns-node-cache:1.15.16 args: - "-localip" # Use a bind-all approach. Ensure server blocks bind broadly in your Corefile. - "0.0.0.0" - "-conf" - "/etc/Corefile" - "-upstreamsvc" - "kube-dns-upstream" # CRITICAL: Disable internal setup - "-skipteardown=true" - "-setupinterface=false" - "-setupiptables=false" ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP # Ensure your Corefile includes health :8080 so the liveness probe works livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 volumeMounts: - name: config-volume mountPath: /etc/coredns - name: kube-dns-config mountPath: /etc/kube-dns volumes: - name: kube-dns-config configMap: name: kube-dns optional: true - name: config-volume configMap: name: node-local-dns items: - key: Corefile path: Corefile.base

2. The Cilium Local Redirect Policy (LRP)

Instead of iptables, we define a CRD that tells Cilium: "When you see traffic for `kube-dns`, send it to the `node-local-dns` pod on this same node."

apiVersion: "cilium.io/v2" kind: CiliumLocalRedirectPolicy metadata: name: "nodelocaldns" namespace: kube-system spec: redirectFrontend: # ServiceMatcher mode is for ClusterIP services serviceMatcher: serviceName: kube-dns namespace: kube-system redirectBackend: # The backend pods selected by localEndpointSelector must be in the same namespace as the LRP localEndpointSelector: matchLabels: k8s-app: node-local-dns toPorts: - port: "53" name: dns protocol: UDP - port: "53" name: dns-tcp protocol: TCP

This is an LRP-based NodeLocal DNSCache deployment: we disable node-cache’s iptables/interface setup and let Cilium LRP handle local redirection. This differs from the upstream NodeLocal DNSCache manifest, which uses hostNetwork + dummy interface + iptables.

LRP must be enabled in Cilium (e.g., localRedirectPolicies.enabled=true) before applying the CRD. Official Cilium LRP doc

DNS-Based FQDN Policy Enforcement Flow

The diagram below illustrates how Cilium enforces FQDN-based egress policies using DNS observation and datapath programming. During the DNS resolution phase, queries are redirected to NodeLocal DNS (or CoreDNS), where responses are observed and used to populate Cilium’s FQDN-to-IP cache. Cilium then programs these mappings into eBPF maps in the datapath. In the connection phase, when the client initiates an HTTPS connection to the resolved IP, the datapath checks the IP against the learned FQDN map and applies the policy decision before allowing or denying the connection.

End-to-end flow of DNS resolution, FQDN learning, and eBPF-based policy enforcement in Cilium.

The Network Policy "Gotcha"

If you use CiliumNetworkPolicy to restrict egress traffic, specifically for FQDN filtering, you typically allow access to CoreDNS like this:

- toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: "53" protocol: ANY

This will break with local redirection.

Why? Because LRP redirects the DNS request to the node-local-dns backend endpoint; strict egress policies must therefore allow both kube-dns (upstream) and node-local-dns (the redirected destination).

The Repro Setup

To demonstrate this failure, the cluster is configured with:

NodeLocal DNSCache: Deployed as a DaemonSet (node-local-dns) to cache DNS requests locally on every node.
Local Redirect Policy (LRP): An active LRP intercepts traffic destined for the kube-dns Service IP and redirects it to the local node-local-dns pod.
Incomplete Network Policy: A strict CiliumNetworkPolicy (CNP) is enforced on the client pod. While it explicitly allows egress to kube-dns, it misses the corresponding rule for node-local-dns.

Reveal the issue using Hubble:

In this scenario, the client pod dns-client is attempting to resolve the external domain github.com.

When inspecting the traffic flows, you will see EGRESS DENIED verdicts. Crucially, notice the destination pod in the logs below: kube-system/node-local-dns, not kube-dns.

Although the application originally sent the packet to the Cluster IP of CoreDNS, Cilium's Local Redirect Policy modified the destination to the local node cache. Since strictly defined Network Policies assume traffic is going to the kube-dns identity, this redirected traffic falls outside the allowed rules and is dropped by the default deny stance.

The Fix: You must allow egress to both labels.

- toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns # Add this selector for the local cache - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: node-local-dns toPorts: - ports: - port: "53" protocol: ANY

Without this addition, pods protected by strict egress policies will timeout resolving DNS, even though the cache is running.

Use Hubble to observe the network flows:

After adding matchLabels: k8s:k8s-app: node-local-dns, the traffic is now allowed. Hubble confirms a policy verdict of EGRESS ALLOWED for UDP traffic on port 53. Because DNS resolution now succeeds, the response populates the Cilium FQDN cache, subsequently allowing the TCP traffic to github.com on port 443 as intended.

Real-World Example: Restricting Egress with FQDN Policies

Here is a complete CiliumNetworkPolicy that locks down a workload to only access api.example.com. Note how the DNS rule explicitly allows traffic to both kube-dns (for upstream) and node-local-dns (for the local cache).

apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: secure-workload-policy spec: endpointSelector: matchLabels: app: critical-workload egress: # 1. Allow DNS Resolution (REQUIRED for FQDN policies) - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns # Allow traffic to the local cache redirection target - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: node-local-dns toPorts: - ports: - port: "53" protocol: ANY rules: dns: - matchPattern: "*" # 2. Allow specific FQDN traffic (populated via DNS lookups) - toFQDNs: - matchName: "api.example.com" toPorts: - ports: - port: "443" protocol: TCP

Configuration & Upstream Loops

When configuring the ConfigMap for node-local-dns, use the standard placeholders provided by the image. The binary replaces them at runtime:

__PILLAR__CLUSTER__DNS__: The Upstream Service IP (kube-dns-upstream).
__PILLAR__UPSTREAM__SERVERS__: The system resolvers (usually /etc/resolv.conf).

Ensure kube-dns-upstream exists as a Service selecting the CoreDNS pods so cache misses are forwarded to the actual CoreDNS backends.

Alternative: AKS LocalDNS

LocalDNS is an Azure Kubernetes Services (AKS)-managed node-local DNS proxy/cache.

Pros:

Managed lifecycle at the node pool level.
Support for custom configuration via localdnsconfig.json (e.g., custom server blocks, cache tuning).
No manual DaemonSet management required.

Cons & Limitations:

Incompatibility with FQDN Policies: As noted in the official documentation, LocalDNS isn’t compatible with applied FQDN filter policies in ACNS/Cilium; if you rely on FQDN enforcement, prefer a DNS path that preserves FQDN learning/enforcement.
Updating configuration requires reimaging the node pool.

For environments heavily relying on strict Cilium Network Policies and FQDN filtering, the manual deployment method described above (using LRP) can be more reliable and transparent.

AKS recommends not enabling both upstream NodeLocal DNSCache and LocalDNS in the same node pool, as DNS traffic is routed through LocalDNS and results may be unexpected.

References

Event-Driven to Change-Driven: Low-cost dependency inversion

CollinBrian — Wed, 17 Dec 2025 22:22:37 GMT

Event-driven architectures tout scalability, loose coupling, and eventual consistency. The architectural patterns are sound, the theory is compelling, and the blog posts make it look straightforward.

Then you implement it.

Suddenly you're maintaining separate event stores, implementing transactional outboxes, debugging projection rebuilds, versioning events across a dozen micro-services, and writing mountains of boilerplate to handle what should be simple queries.

Your domain events that were supposed to capture rich business meaning have devolved into glorified database change notifications. Downstream services diff field values to extract intent from "OrderUpdated" events because developers just don't get what constitutes a proper domain event.

The complexity tax is real, don't get me wrong, it's very elegant but for many systems it's unjustified.

Drasi offers an alternative: change-driven architecture that delivers reactive, real-time capabilities across multiple data sources without requiring you to rewrite your application or over complicate your architecture.

What do we mean by “Event-driven” architecture

As Martin Fowler notes, event-driven architecture isn't a single pattern, it's at least four distinct patterns that are often confused, each with its own benefits and traps.

Event Notification is the simplest form. Here, events act as signals that something has happened, but carry minimal data, often just an identifier. The recipient must query the source system for more details if needed. For example, a service emits an OrderPlaced event with just the order ID. Downstream consumers must query the order service to retrieve full order details.

Event Carried State Transfer broadcasts full state changes through events. When an order ships, you publish an OrderShipped event containing all the order details. Downstream services maintain their own materialized views or projections by consuming these events.

Event Sourcing goes further, events become your source of truth. Instead of storing current state, you store the sequence of events that led to that state. Your order isn't a row in a database; it's the sum of OrderPlaced, ItemAdded, PaymentProcessed, and OrderShipped events.

CQRS (Command Query Responsibility Segregation) separates write operations (commands) from read operations (queries). While not inherently event-driven, CQRS is often paired with event sourcing or event-carried state transfer to optimize for scalability and maintainability. Originally derived from Bertrand Meyer's Command-Query Separation principle and popularized by Greg Young, CQRS addresses a specific architectural challenge: the tension between optimizing for writes versus optimizing for reads.

The pattern promises several benefits:

Optimized data models: Your write model can focus on transactional consistency while read models optimize for query performance

Scalability: Read and write sides can scale independently

Temporal queries: With event sourcing, you get time travel for free—reconstruct state at any point in history

Audit trail: Every change is captured as an immutable event

While CQRS isn't inherently tied to Domain-Driven Design (DDD), the pattern complements DDD well. In DDD contexts, CQRS enables different bounded contexts to maintain their own read models tailored to their specific ubiquitous language, while the write model protects domain invariants. This is why you'll often see them discussed together, though each can be applied independently.

The core motivation for these patterns is often to invert the dependency between systems, so that your downstream services do not need to know about your upstream services.

The Developer's Struggle: When Domain Events Become Database Events

Chris Kiehl puts it bluntly in his article "Don't Let the Internet Dupe You, Event Sourcing is Hard": "The sheer volume of plumbing code involved is staggering—instead of a friendly N-tier setup, you now have classes for commands, command handlers, command validators, events, aggregates, and then projections, model classes, access classes, custom materialization code, and so on."

But the real tragedy isn't the boilerplate, it's what happens to those carefully crafted domain events. As developers are disconnected from the real-world business, they struggle to understand the nuances of domain events, a dangerous pattern emerges. Instead of modeling meaningful business processes, teams default to what they know: CRUD.

Your event stream starts looking like this:

OrderCreated

OrderUpdated

OrderUpdated (again)

OrderUpdated (wait, what changed?)

OrderDeleted

As one developer noted on LinkedIn, these "CRUD events" are really just "leaky events that lack clarity and should not be used to replicate databases as this leaks implementation details and couples services to a shared data model."

Dennis Doomen, reflecting on real-world production issues, observes: "It's only once you have a living, breathing machine, users which depend on you, consumers which you can't break, and all the other real-world complexities that plague software projects that the hard problems in event sourcing will rear their heads."

The result? Your elegant event-driven architecture devolves into an expensive, brittle form of self-maintained Change Data Capture (CDC). You're not modeling business processes; you're just broadcasting database mutations with extra steps.

The Anti-Corruption Layer: Your Defense Against the Outside World

In DDD, an Anti-Corruption Layer (ACL) protects your bounded context from external models that would corrupt your domain. Think of it as a translator that speaks both languages, the messy external model and your clean internal model.

The ACL ensures that changes to the external system don't ripple through your domain. If the legacy system changes its schema, you update the translator, not your entire domain model.

When Event Taxonomies Become Your ACL (And Why They Fail)

In most event-driven architectures, your event taxonomy is supposed to serve as the shared contract between services. Each service publishes events using its own ubiquitous language, and consumers translate these into their own models, this translation is the ACL.

The theory looks beautiful:

But reality? Most teams end up with this:

Instead of OrderPaid events that carry business meaning, we get OrderUpdated events that force every consumer to reconstruct intent by diffing fields. When you change your database schema, say splitting the orders table or switching from SQL to NoSQL, every downstream service breaks because they're all coupled to your internal data model.

You haven't built an anti-corruption layer. You've built a corruption pipeline that efficiently distributes your internal implementation details across the entire system, forcing you to deploy all services in lock step and eroding the decoupling benefits you were supposed to get.

Enter Drasi: Continuous Queries

This is where Drasi changes the game. Instead of publishing events and hoping downstream services can make sense of them, Drasi tails the changelog of the data source itself and derives meaning through continuous queries.

A continuous query in Drasi isn't just a query that runs repeatedly, it's a living, breathing projection that reacts to changes in real-time. Here's the key insight: instead of imperative code that processes events ("when this happens, do that"), you write declarative queries that describe the state you care about ("I want to know about orders that are ready and have drivers waiting").

Let's break down what makes this powerful:

Declarative vs. Imperative

Traditional event processing:

Drasi continuous query:

Semantic Mapping from Low-Level Changes

Drasi excels at transforming database-level changes into business-meaningful events. You're not reacting to "row updated in orders table", you're reacting to "order ready for curbside pickup."

This enables the same core benefits of dependency inversion we get from event-driven architectures but at a fraction of the effort.

Advanced Temporal Features

Remember those developers struggling with "OrderUpdated" events, trying to figure out if something just happened or has been true for a while? Drasi handles this elegantly:

This query only fires when a driver has been waiting for more than 10 minutes, no timestamp tracking, no state machines, no complex event correlation logic, imagine trying to manually implement this in a downstream event consumer. 😱

Cross-Source Aggregation Without Code

With Drasi, you can have live projections across PostgreSQL, MySQL, SQL Server, and Cosmos DB as if they were a single graph:

No custom aggregation service. No event stitching logic. No custom downstream datastore to track the sum or keep a materialized projection. Just a query.

Continuous Queries as Your Shared Contract

Drasi's continuous queries, combined with pre-processing middleware, can form the shared contract that your anti-corruption layer can depend on.

The continuous query becomes your contract. Downstream systems don't know or care whether orders come from PostgreSQL, MongoDB, or a CSV file. They don't know if you normalized your database, denormalized it, or moved to event sourcing. They just consume the query results. Clean, semantic, and stable.

Reactions as your Declarative Consumers

Drasi does not simply output a stream of raw change diffs, instead it has a library of interchangeable Reactions, that can act on the output of continuous queries. These are declared using YAML and can do anything from host a web-socket endpoint that provides a live projection to your UI, to calling an Http endpoint or publishing a message on a queue.

Example: The Curbside Pickup System

Let's see how this works in Drasi's curbside pickup tutorial. This example has two independent databases and serves as a great illustration of a real-time projection built from multiple upstream services.

The Business Problem

A retail system needs to:

Match ready orders with drivers who've arrived at pickup zones
Alert staff when drivers wait more than 10 minutes without their order being ready
Coordinate data from two different systems (retail ops in PostgreSQL, physical ops in MySQL)

Traditional Event-Driven Approach

In this architecture, you'd need something like:

That's just the happy path. We haven't handled:

Event ordering issues
Partial failures
Cache invalidation
Service restarts and replay
Duplicate events
Transactional outboxing

The Drasi Approach

With Drasi, the entire aggregation service above becomes two queries:

Delivery Dashboard Query:

Wait Detection Query:

That's it. No event handlers. No caching. No timers. No state management. Drasi handles:

Change detection across both databases
Correlation between orders and vehicles
Temporal logic for wait detection
Pushing updates to dashboards via SignalR

The queries define your business logic declaratively. When data changes in either database, Drasi automatically re-evaluates the queries and triggers reactions for any changes in the result set.

Drasi: The Non-Invasive Alternative to Legacy System Rewrites

Here's perhaps the most compelling argument for Drasi: it doesn't require you to rewrite anything.

Traditional event sourcing means:

Redesigning your application around events
Rewriting your persistence layer
Implementing transactional outboxes
Managing snapshots and replays
Training your team on new patterns, steep learning curve
Migrating existing data to event streams
Building projection infrastructure
Updating all consumers to handle events

As one developer noted about their event sourcing journey: "Event Sourcing is a beautiful solution for high-performance or complex business systems, but you need to be aware that this also introduces challenges most people don't tell you about."

Drasi's approach:

Keep your existing databases
Keep your existing services
Keep your existing deployment model
Add continuous queries where you need reactive behavior
Get the benefits of dependency inversion
Gradually migrate complexity from code to queries

You can start with a single query on a single table and expand from there. No big bang. No feature freeze. No three-month architecture sprint or large multi-year investments, full of risk.

Migration Example: From Polling to Reactive

Let's say you have a legacy order system where a scheduled job polls for ready orders every 30 seconds:

With Drasi, you:

Point Drasi at your existing database
Write the continuous query
Update your dashboard to receive pushes instead of polls
Turn off the polling job

Your database hasn't changed. Your order service hasn't changed. You've just added a reactive layer on top that eliminates polling overhead and reduces notification latency from 30 seconds to milliseconds.

The intellectually satisfying complexity of event sourcing often obscures a simple truth: most systems don't need it. They need to know when interesting things change in their data and react accordingly. They need to combine data from multiple sources without writing bespoke aggregation services. They need to transform low-level changes into business-meaningful events.

Drasi delivers these capabilities without the ceremony.

Where Do We Go from Here?

If you're building a new system and your team has deep event sourcing experience embrace the pattern. Event sourcing shines for certain domains.

But if you're like many teams, trying to add reactive capabilities to existing systems, struggling with data synchronization across services, or finding that your "events" are just CRUD operations in disguise, consider the change-driven approach.

Start small:

Identify one painful polling loop or batch job
Set up Drasi to monitor those same data sources
Write a continuous query that captures the business condition
Replace the polling with push-based reactions
Measure the reduction in latency, overhead, and code complexity

The best architecture isn't the most sophisticated one, it's the one your team can understand, maintain, and evolve. Sometimes that means acknowledging that we've been mid-curving it with overly complex event-driven architectures.

Drasi and change-driven architecture offer the power of reactive systems without the complexity tax. Your data changes. Your queries notice. Your systems react.

It makes it a non-event.

Want to explore Drasi further? Check out the official documentation and try the curbside pickup tutorial to see change-driven architecture in action.

Building Bridges: Microsoft’s Participation in the Fedora Linux Community

bexelbie — Wed, 17 Dec 2025 13:47:31 GMT

At Microsoft, we believe that meaningful open source participation is driven by people, not corporations. But companies can - and should - create the conditions that empower individuals to contribute. Over the past year, our Community Linux Engineering team has been doing just that, focusing on Fedora Linux and working closely with the community to improve infrastructure, tooling, and collaboration. This post shares some of the highlights of that work and outlines where we’re headed next.

Modernizing Fedora Cloud Image Delivery

One of our most impactful contributions this year has been expanding the availability of Fedora Cloud images across major cloud platforms. We introduced support for publishing images to both the Azure Community Gallery and Google Cloud Platform—capabilities that didn’t exist before. At the same time, we modernized the existing AWS image publishing process by migrating it to a new, OpenShift-hosted automation framework. This new system, developed by our team and led by engineer Jeremy Cline, streamlines image delivery across all three platforms and positions the project to scale and adapt more easily in the future.

We partnered with Adam Williamson in Fedora QE to extend this tooling to support container image uploads, replacing fragile shell scripts with a robust, maintainable system. Nightly Fedora builds are now uploaded to Azure, with one periodically promoted to “latest” after manual validation and basic functionality testing. This ensures cloud users get up-to-date, ready-to-run images - critical for workloads that demand fast boot times and minimal setup. As you’ll see , we have ideas for improving this testing.

Enabling Secure Boot on ARM with Sigul

Secure Boot is essential for trusted cloud workloads across architectures. Our current focus includes enabling it on ARM-based systems. Fedora currently signs most artifacts with Sigul, but UEFI applications are handled separately via a dedicated x86_64 builder with a smart card. We’re working to enable Sigul-based signing for UEFI applications across architectures, but Sigul is a complex project with unmaintained dependencies. We’ve stepped in to help modernize Sigul, starting with a Rust-based client and a roadmap to re-architect the code and structure for easier maintenance and improved performance.

This work is about more than just Microsoft’s needs - it’s about enabling Secure Boot support out of the box, like what users expect on x86_64 systems.

Bringing Inspektor Gadget to Fedora

Inspektor Gadget is an eBPF-based toolkit for kernel instrumentation, enabling powerful observability use cases like performance profiling and syscall tracing. The Community Linux Engineering team consulted with the Inspektor Gadget maintainers at Microsoft about putting the project in Fedora. This led to the maintainers natively packaging it for Fedora and assuming ongoing maintenance of the package.

We are encouraging teams to become active Fedora participants, to maintain their own packages, and to engage directly with the community. We believe in bi-directional feedback: upstream contributions should benefit both the project and the contributors.

Azure VM Utils: Simplifying Cloud Enablement

To streamline Fedora’s compatibility with Azure, we’ve introduced a package called azure-vm-utils. It consolidates Udev rules and low-level utilities that make Fedora work better on Azure infrastructure, particularly with NVMe devices. This package is a step toward greater transparency and maintainability and could serve as a model for other cloud providers.

Fedora WSL: A Layer 9 Success

Fedora is now officially available in the Windows Subsystem for Linux (WSL) catalog - a milestone that required both technical and organizational effort. While the engineering work was substantial, the real challenge was navigating the legal and governance landscape. This success reflects deep collaboration between Fedora leadership, Red Hat, and Microsoft.

Looking Ahead: Strategic Participation and Testing

We’re not stopping here. Our roadmap includes:

Replacing Sigul with a modern, maintainable signing infrastructure.
Expanding participation in Fedora SIGs (Cloud, Go, Rust) where Microsoft has relevant expertise.
Improving automated testing using Microsoft’s open source LISA framework to validate Fedora images at cloud scale.
Enhancing the Fedora-on-Azure experience, including exploring mirrors within Azure and expanding agent/extension support.

We’re also working closely with the Azure Linux team, which is aligning its development model with Fedora - much like RHEL does. while Azure Linux has used some Fedora sources in the past, their upcoming 4.0 release is intended to align much more closely with Fedora as an upstream

A Call for Collaboration

While contributing patches is a good start, we intend to do much more. We aim to be a deeply involved member of the Fedora community - participating in SIGs, maintaining packages, and listening to feedback. If you have ideas for where Microsoft can make strategic investments that benefit Fedora, we want to hear them. You’ll find us alongside you in Fedora meetings, forums, and at conferences like Flock.

Open source thrives when contributors bring their whole selves to the table. At Microsoft, we’re working to ensure our engineers can do just that - by aligning company goals with community value.

(This post is based on a talk delivered at Flock to Fedora 2025.)

Beyond the Chat Window: How Change-Driven Architecture Enables Ambient AI Agents

CollinBrian — Wed, 03 Dec 2025 23:09:40 GMT

AI agents are everywhere now. Powering chat interfaces, answering questions, helping with code. We've gotten remarkably good at this conversational paradigm. But while the world has been focused on chat experiences, something new is quietly emerging: ambient agents. These aren't replacements for chat, they're an entirely new category of AI system that operates in the background, sensing, processing, and responding to the world in real time. And here's the thing, this is a new frontier. The infrastructure we need to build these systems barely exists yet.

Or at least, it didn't until now.

Two Worlds: Conversational and Ambient

Let me paint you a picture of the conversational AI paradigm we know well. You open a chat window. You type a question. You wait. The AI responds. Rinse and repeat. It's the digital equivalent of having a brilliant assistant sitting at a desk, ready to help when you tap them on the shoulder.

Now imagine a completely different kind of assistant. One that watches for important changes, anticipates needs, and springs into action without being asked. That's the promise of ambient agents. AI systems that, as LangChain puts it: "listen to an event stream and act on it accordingly, potentially acting on multiple events at a time."

This isn't an evolution of chat; it's a fundamentally different interaction paradigm. Both have their place. Chat is great for collaboration and back-and-forth reasoning. Ambient agents excel at continuous monitoring and autonomous response. Instead of human-initiated conversations, ambient agents operate through detecting changes in upstream systems and maintaining context across time without constant prompting.

The use cases are compelling and distinct from chat. Imagine a project management assistant that operates in two modes: you can chat with it to ask, "summarize project status", but it also runs in the background, constantly monitoring new tickets that are created, or deployment pipelines that fail, automatically reassigning tasks. Or consider a DevOps agent that you can query conversationally ("what's our current CPU usage?") but also monitors your infrastructure continuously, detecting anomalies and starting remediation before you even know there's a problem.

The Challenge: Real-Time Change Detection

Here's where building ambient agents gets tricky. While chat-based agents work perfectly within the request-response paradigm, ambient agents need something entirely different: continuous monitoring and real-time change detection. How do you efficiently detect changes across multiple data sources? How do you avoid the performance nightmare of constant polling? How do you ensure your agent reacts instantly when something critical happens?

Developers trying to build ambient agents hit the same wall: creating a reliable, scalable change detection system is hard. You either end up with:

Polling hell: Constantly querying databases, burning through resources, and still missing changes between polls

Legacy system rewrites: Massive expensive multi-year projects to re-write legacy systems so that they produce domain events

Webhook spaghetti: Managing dozens of event sources, each with different formats and reliability guarantees

This is where the story takes an interesting turn.

Enter Drasi: The Change Detection Engine You Didn't Know You Needed

Drasi is not another AI framework. Instead, it solves the problem that ambient agents need solved: intelligent change detection. Think of it as the sensory system for your AI agents, the infrastructure that lets them perceive changes in the world.

Drasi is built around three simple components:

Sources: Connectivity to the systems that Drasi can observe as sources of change (PostgreSQL, MySQL, Cosmos DB, Kubernetes, EventHub)

Continuous Queries: Graph-based queries (using Cypher/GQL) that monitor for specific change patterns

Reactions: What happens when a continuous query detects changes, or lack thereof

But here's the killer feature: Drasi doesn't just detect that something changed. It understands what changed and why it matters, and even if something should have changed but did not. Using continuous queries, you can define complex conditions that your agents care about, and Drasi handles all the plumbing to deliver those insights in real time.

The Bridge: langchain-drasi Integration

Now, detecting changes is only part of the challenge. You need to connect those changes to your AI agents in a way that makes sense. That's where langchain-drasi comes in, a purpose-built integration that bridges Drasi's change detection with LangChain's agent frameworks. It achieves this by leveraging the Drasi MCP Reaction, which exposes Drasi continuous queries as MCP resources.

The integration provides a simple Tool that agents can use to:

Discover available queries automatically

Read current query results on demand

Subscribe to real-time updates that flow directly into agent memory and workflow

Here's what this looks like in practice:

from langchain_drasi import create_drasi_tool, MCPConnectionConfig

# Configure connection to Drasi MCP server
mcp_config = MCPConnectionConfig(server_url="http://localhost:8083")

# Create the tool with notification handlers 
drasi_tool = create_drasi_tool( 
    mcp_config=mcp_config, 
    notification_handlers=[buffer_handler, console_handler] 
)

# Now your agent can discover and subscribe to data changes 
# No more polling, no more webhooks, just reactive intelligence

The beauty is in the notification handlers: pre-built components that determine how changes flow into your agent's consciousness:

BufferHandler: Queues changes for sequential processing

LangGraphMemoryHandler: Automatically integrates changes into agent checkpoints

LoggingHandler: Integrates with standard logging infrastructure

This isn't just plumbing; it's the foundation for what we might call "change-driven architecture" for AI systems.

Example: The Seeker Agent Has Entered the Chat

Let's make this concrete with my favorite example from the langchain-drasi repository: a hide and seek inspired non-player character (NPC) AI agent that seeks human players in a multi-player game environment.

The Scenario

Imagine a game where players move around a 2D map, updating their positions in a PostgreSQL database. But here's the twist: the NPC agent doesn't have omniscient vision. It can only detect players under specific conditions:

Stationary targets: When a player doesn't move for more than 3 seconds (they're exposed)

Frantic movement: When a player moves more than once in less than a second (panicking reveals your position)

This creates interesting strategic gameplay, players must balance staying still (safe from detection but vulnerable if found) with moving carefully (one move per second is the sweet spot). The NPC agent seeks based on these glimpses of player activity. These detection rules are defined as Drasi continuous queries that monitor the player positions table.

For reference, these are the two continuous queries we will use:

When a player doesn't move for more than 3 seconds, this is a great example of detecting the absence of change use the trueLater function:

MATCH 
    (p:player { type: 'human' }) 
WHERE drasi.trueLater( 
        drasi.changeDateTime(p) <= (datetime.realtime() - duration( { seconds: 3 } )),  
        drasi.changeDateTime(p) + duration( { seconds: 3 } ) 
      ) 
RETURN 
    p.id, 
    p.x, 
    p.y

When a player moves more than once in less than a second is an example of using the previousValue function to compare that current state with a prior state:

MATCH 
    (p:player { type: 'human' }) 
WHERE drasi.changeDateTime(p).epochMillis - drasi.previousValue(drasi.changeDateTime(p).epochMillis) < 1000 
RETURN 
    p.id, 
    p.x, 
    p.y

Here's the neat part: you can dynamically adjust the game's difficulty by adding or removing queries with different conditions; no code changes required, just deploy new Drasi queries.

The traditional approach would have your agent constantly polling the data source checking these conditions: "Any player moves? How about now? Now? Now?"

The Workflow in Action

The agent operates through a LangGraph based state machine with two distinct phases:

1. Setup Phase (First Run Only)

Setup queries prompt - Prompts the AI model to discover available Drasi queries
Setup queries call model - AI model calls the Drasi tool with discover operation
Setup queries tools - Executes the Drasi tool calls to subscribe to relevant queries
This phase loops until the AI model has discovered and subscribed to all relevant queries

2. Main Seeking Loop (Continuous)

Check sensors - Consumes any new Drasi notifications from the buffer into the workflow state
Evaluate targets - Uses AI model to parse sensor data and extract target positions
Select and plan - Selects closest target and plans path
Execute move - Executes the next move via game API
Loop continues indefinitely, reacting to new notifications

No polling. No delays. No wasted resources checking positions that don't meet the detection criteria. Just pure, reactive intelligence flowing from meaningful data changes to agent actions. The continuous queries act as intelligent filters, only alerting the agent when relevant changes occur.

Click here for the full implementation

The Bigger Picture: Change-Driven Architecture

What we're seeing with Drasi and ambient agents isn't just a new tool, it's a new architectural pattern for AI systems. The core idea is profound: AI agents can react to the world changing, not just wait to be asked about it. This pattern enables entirely new categories of applications that complement traditional chat interfaces.

The example might seem playful, but it demonstrates that AI agents can perceive and react to their environment in real time. Today it's seeking players in a game. Tomorrow it could be:

Managing city traffic flows based on real-time sensor data
Coordinating disaster response as situations evolve
Optimizing supply chains as demand patterns shift
Protecting networks as threats emerge

The change detection infrastructure is here. The patterns are emerging. The only question is: what will you build?

Where to Go from Here

Ready to dive deeper? Here are your next steps:

Explore Drasi: Head to drasi.io and discover the power of the change detection platform

Try langchain-drasi: Clone the GitHub repository and run the Hide-and-Seek example yourself

Join the conversation: The space is new and needs diverse perspectives. Join the community on Discord. Let us know if you have built ambient agents and what challenges you faced with real-time change detection.

Project Pavilion Presence at KubeCon NA 2025

lexinadolski — Thu, 04 Dec 2025 11:52:33 GMT

KubeCon + CloudNativeCon NA took place in Atlanta, Georgia, from 10-13 November, and continued to highlight the ongoing growth of the open source, cloud-native community. Microsoft participated throughout the event and supported several open source projects in the Project Pavilion. Microsoft’s involvement reflected our commitment to upstream collaboration, open governance, and enabling developers to build secure, scalable and portable applications across the ecosystem.

The Project Pavilion serves as a dedicated, vendor-neutral space on the KubeCon show floor reserved for CNCF projects. Unlike the corporate booths, it focuses entirely on open source collaboration. It brings maintainers and contributors together with end users for hands-on demos, technical discussions, and roadmap insights. This space helps attendees discover emerging technologies and understand how different projects fit into the cloud-native ecosystem. It plays a critical role for idea exchanges, resolving challenges and strengthening collaboration across CNCF approved technologies.

Why Our Presence Matters

KubeCon NA remains one of the most influential gatherings for developers and organizations shaping the future of cloud-native computing. For Microsoft, participating in the Project Pavilion helps advance our goals of:

Open governance and community-driven innovation
Scaling vital cloud-native technologies
Secure and sustainable operations
Learning from practitioners and adopters
Enabling developers across clouds and platforms

Many of Microsoft’s products and cloud services are built on or aligned with CNCF and open-source technologies. Being active within these communities ensures that we are contributing back to the ecosystem we depend on and designing by collaborating with the community, not just for it.

Microsoft-Supported Pavilion Projects

containerd

Representative: Wei Fu

The containerd team engaged with project maintainers and ecosystem partners to explore solutions for improving AI model workflows. A key focus was the challenge of handling large OCI artifacts (often 500+ GiB) used in AI training workloads. Current image-pulling flows require containerd to fetch and fully unpack blobs, which significantly delays pod startup for large models.

Collaborators from Docker, NTT, and ModelPack discussed a non-unpacking workflow that would allow training workloads to consume model data directly. The team plans to prototype this behavior as an experimental feature in containerd. Additional discussions included updates related to nerdbox and next steps for the erofs snapshotter.

Copacetic

Representative: Joshua Duffney

The Copa booth attracted roughly 75 attendees, with strong representation from federal agencies and financial institutions, a sign of growing adoption in regulated industries. A lightning talk delivered at the conference significantly boosted traffic and engagement. Key feedback and insights included:

High interest in customizable package update sources
Demand for application-level patching beyond OS-level updates
Need for clearer CI/CD integration patterns
Expectations around in-cluster image patching
Questions about runtime support, including Podman

The conversations revealed several documentation gaps and feature opportunities that will inform Copa’s roadmap and future enablement efforts.

Drasi

Representative: Nandita Valsan

KubeCon NA 2025 marked Drasi’s first in-person presence since its launch in October 2024 and its entry into the CNCF Sandbox in early 2025. With multiple kiosk slots, the team interacted with ~70 visitors across shifts. Engagement highlights included:

New community members joining the Drasi Discord and starring GitHub repositories
Meaningful discussions with observability and incident management vendors interested in change-driven architectures
Positive reception to Aman Singh’s conference talk, which led attendees back to the booth for deeper technical conversations

Post-event follow-ups are underway with several sponsors and partners to explore collaboration opportunities.

Flatcar Container Linux

Representatives: Sudhanva Huruli and Vamsi Kavuru

The Flatcar project had some fantastic conversations at the pavilion. Attendees were eager to learn about bare metal provisioning, GPU support for AI workloads, and how Flatcar’s fully automated build and test process keeps things simple and developer friendly. Questions around Talos vs. Flatcar and CoreOS sparked lively discussions, with the team emphasizing Flatcar’s usability and independence from an OS-level API. Interest came from government agencies and financial institutions, and the preview of Flatcar on AKS opened the door to deeper conversations about real-world adoption. The Project Pavilion proved to be the perfect venue for authentic, technical exchanges.

Flux

Representatives: Dipti Pai

The Flux booth was active throughout all three days of the Project Pavilion, where Microsoft joined other maintainers to highlight new capabilities in Flux 2.7, including improved multi-tenancy, enhanced observability, and streamlined cloud-native integrations. Visitors shared real-world GitOps experiences, both successes and challenges, which provided valuable insights for the project’s ongoing development. Microsoft’s involvement reinforced strong collaboration within the Flux community and continued commitment to advancing GitOps practices.

Headlamp

Representatives: Joaquim Rocha, Will Case, and Oleksandr Dubenko

Headlamp had a booth for all three days of the conference, engaging with both longstanding users and first-time attendees. The increased visibility from becoming a Kubernetes sub-project was evident, with many attendees sharing their usage patterns across large tech organizations and smaller industrial teams.

The booth enabled maintainers to:

Gather insights into how teams use Headlamp in different environments
Introduce Headlamp to new users discovering it via talks or hallway conversations
Build stronger connections with the community and understand evolving needs

Inspektor Gadget

Representatives: Jose Blanquicet and Mauricio Vásquez Bernal

Hosting a half-day kiosk session, Inspektor Gadget welcomed approximately 25 visitors. Attendees included newcomers interested in learning the basics and existing users looking for updates. The team showcased new capabilities, including the tcpdump gadget and Prometheus metrics export, and invited visitors to the upcoming contribfest to encourage participation.

Istio

Representatives: Keith Mattix, Jackie Maertens, Steven Jin Xuan, Niranjan Shankar, and Mike Morris

The Istio booth continued to attract a mix of experienced adopters and newcomers seeking guidance. Technical discussions focused on:

Enhancements to multicluster support in ambient mode
Migration paths from sidecars to ambient
Improvements in Gateway API availability and usage
Performance and operational benefits for large-scale deployments

Users, including several Azure customers, expressed appreciation for Microsoft’s sustained investment in Istio as part of their service mesh infrastructure.

Notary Project

Representative: Feynman Zhou and Toddy Mladenov

The Notary Project booth saw significant interest from practitioners concerned with software supply chain security. Attendees discussed signing, verification workflows, and integrations with Azure services and Kubernetes clusters. The conversations will influence upcoming improvements across Notary Project and Ratify, reinforcing Microsoft’s commitment to secure artifacts and verifiable software distribution.

Open Policy Agent (OPA) - Gatekeeper

Representative: Jaydip Gabani

The OPA/Gatekeeper booth enabled maintainers to connect with both new and existing users to explore use cases around policy enforcement, Rego/CEL authoring, and managing large policy sets. Many conversations surfaced opportunities around simplifying best practices and reducing management complexity. The team also promoted participation in an ongoing Gatekeeper/OPA survey to guide future improvements.

ORAS

Representative: Feynman Zhou and Toddy Mladenov

ORAS engaged developers interested in OCI artifacts beyond container images which includes AI/ML models, metadata, backups, and multi-cloud artifact workflows. Attendees appreciated ORAS’s ecosystem integrations and found the booth examples useful for understanding how artifacts are tagged, packaged, and distributed. Many users shared how they leverage ORAS with Azure Container Registry and other OCI-compatible registries.

Radius

Representative: Zach Casper

The Radius booth attracted the attention of platform engineers looking for ways to simplify their developer's experience while being able to enforce enterprise-grade infrastructure and security best practices. Attendees saw demos on deploying a database to Kubernetes and using managed databases from AWS and Azure without modifying the application deployment logic. They also saw a preview of Radius integration with GitHub Copilot enabling AI coding agents to autonomously deploy and test applications in the cloud.

Conclusion

KubeCon + CloudNativeCon North America 2025 reinforced the essential role of open source communities in driving innovation across cloud native technologies. Through the Project Pavilion, Microsoft teams were able to exchange knowledge with other maintainers, gather user feedback, and support projects that form foundational components of modern cloud infrastructure. Microsoft remains committed to building alongside the community and strengthening the ecosystem that powers so much of today’s cloud-native development.

For anyone interested in exploring or contributing to these open source efforts, please reach out directly to each project’s community to get involved, or contact Lexi Nadolski at lexinadolski@microsoft.com for more information.

Azure Linux: Driving Security in the Era of AI Innovation

Sudhanva — Tue, 18 Nov 2025 16:49:02 GMT

Microsoft is advancing cloud and AI innovation with a clear focus on security, quality, and responsible practices. At Ignite 2025, Azure Linux reflects that commitment. As Microsoft’s ubiquitous Linux OS, it powers critical services and serves as the hub for security innovation. This year’s announcements, Azure Linux with OS Guard public preview and GA of pod sandboxing, reinforce security as one of our core priorities, helping customers build and run workloads with confidence in an increasingly complex threat landscape.

Announcing OS Guard Public Preview

We’re excited to announce the public preview of Azure Linux with OS Guard at Ignite 2025! OS Guard delivers a hardened, immutable container host built on the FedRAMP-certified Azure Linux base image. It introduces a significantly streamlined footprint with approximately 100 fewer packages than the standard Azure Linux image, reducing the attack surface and improving performance. FIPS mode is enforced by default, ensuring compliance for regulated workloads right out of the box. Additional security features include dm-verity for filesystem immutability, Trusted Launch backed by vTPM-secured keys, and seamless integration with AKS for container workloads. Built with upstream transparency and active Microsoft contributions, OS Guard provides a secure foundation for containerized applications while maintaining operational simplicity.

During the preview period, code integrity and mandatory access Control (SELinux) are enabled in audit mode, allowing customers to validate policies and prepare for enforcement without impacting workloads.

General Availability: Pod Sandboxing for stronger isolation on AKS

We’re also announcing the GA of pod sandboxing on AKS, delivering stronger workload isolation for multi-tenant and regulated environments. Based on the open source Kata project, Pod Sandboxing introduces VM-level isolation for containerized workloads by running each pod inside its own lightweight virtual machine using Kata Containers, providing a stronger security boundary compared to traditional containers.

Connect with us at Ignite

Meet the Azure Linux team and see these innovations in action:

Ignite: Join us at our breakout session (https://ignite.microsoft.com/en-US/sessions/BRK144) and visit the Linux on Azure Booth for live demos and deep dives.

Session Type	Session Code	Session Name	Date/Time (PST)
Breakout	BRK 143	Optimizing performance, deployments, and security for Linux on Azure	Thu, Nov 20/ 1:00 PM – 1:45 PM
Breakout	BRK 144	Build, modernize, and secure AKS workloads with Azure Linux	Wed, Nov 19/ 1:30 PM – 2:15 PM
Breakout	BRK 104	From VMs and containers to AI apps with Azure Red Hat OpenShift	Thu, Nov 20/ 8:30 AM – 9:15 AM
Theatre	TRH 712	Hybrid workload compliance from policy to practice on Azure	Tue, Nov 18/ 3:15 PM – 3:45 PM
Theatre	THR 701	From Container to Node: Building Minimal-CVE Solutions with Azure Linux	Wed, Nov 19/ 3:30 PM – 4:00 PM
Lab	Lab 505	Fast track your Linux and PostgreSQL migration with Azure Migrate	Tue, Nov 18/ 4:30 PM – 5:45 PM PST Wed, Nov 19/ 3:45 PM – 5:00 PM PST Thu, Nov 20/ 9:00 AM – 10:15 AM PST

Whether you’re migrating workloads, exploring security features, or looking to engage with our engineering team, we’re eager to connect and help you succeed with Azure Linux.

Resources to get started

Azure Linux OS Guard Overview & QuickStart: https://aka.ms/osguard
Pod Sandboxing Overview & QuickStart: https://aka.ms/podsandboxing
Azure Linux Documentation: https://learn.microsoft.com/en-us/azure/azure-linux/

From Policy to Practice: Built-In CIS Benchmarks on Azure - Flexible, Hybrid-Ready

pallakatos — Tue, 18 Nov 2025 08:00:00 GMT

Security is more important than ever. The industry-standard for secure machine configuration is the Center for Internet Security (CIS) Benchmarks. These benchmarks provide consensus-based prescriptive guidance to help organizations harden diverse systems, reduce risk, and streamline compliance with major regulatory frameworks and industry standards like NIST, HIPAA, and PCI DSS. In our previous post, we outlined our plans to improve the Linux server compliance and hardening experience on Azure and shared a vision for integrating CIS Benchmarks. Today, that vision has turned into reality.

We're now announcing the next phase of this work: Center for Internet Security (CIS) Benchmarks are now available on Azure for all Azure endorsed distros, at no additional cost to Azure and Azure Arc customers.

With today's announcement, you get access to the CIS Benchmarks on Azure with full parity to what’s published by the Center for Internet Security (CIS). You can adjust parameters or define exceptions, tailoring security to your needs and applying consistent controls across cloud, hybrid, and on-premises environments - without having to implement every control manually. Thanks to this flexible architecture, you can truly manage compliance as code.

How we achieve parity

To ensure accuracy and trust, we rely on and ingest CIS machine-readable Benchmark content (OVAL/XCCDF files) as the source of truth. This guarantees that the controls and rules you apply in Azure match the official CIS specifications, reducing drift and ensuring compliance confidence.

What’s new under the hood

At the core of this update is azure-osconfig’s new compliance engine - a lightweight, open-source module developed by the Azure Core Linux team. It evaluates Linux systems directly against industry-standard benchmarks like CIS, supporting both audit and, in the future, auto-remediation. This enables accurate, scalable compliance checks across large Linux fleets. Here you can read more about azure-osconfig.

Dynamic rule evaluation

The new compliance engine supports simple fact-checking operations, evaluation of logic operations on them (e.g., anyOf, allOf) and Lua based scripting, which allows to express complex checks required by the CIS Critical Security Controls - all evaluated natively without external scripts.

Scalable architecture for large fleets

When the assignment is created, the Azure control plane instructs the machine to pull the latest Policy package via the Machine Configuration agent. Azure-osconfig’s compliance engine is integrated as a light-weight library to the package and called by Machine Configuration agent for evaluation – which happens every 15-30minutes. This ensures near real-time compliance state without overwhelming resources and enables consistent evaluation across thousands of VMs and Azure Arc-enabled servers.

Future-ready for remediation and enforcement

While the Public Preview starts with audit-only mode, the roadmap includes per-rule remediation and enforcement using technologies like eBPF for kernel-level controls. This will allow proactive prevention of configuration drift and runtime hardening at scale. Please reach out if you interested in auto-remediation or enforcement.

Extensibility beyond CIS Benchmarks

The architecture was designed to support other security and compliance standards as well and isn’t limited to CIS Benchmarks. The compliance engine is modular, and we plan to extend the platform with STIG and other relevant industry benchmarks. This positions Azure as a platform for a place where you can manage your compliance from a single control-plane without duplicating efforts elsewhere.

Collaboration with the CIS

This milestone reflects a close collaboration between Microsoft and the CIS to bring industry-standard security guidance into Azure as a built-in capability. Our shared goal is to make cloud-native compliance practical and consistent, while giving customers the flexibility to meet their unique requirements. We are committed to continuously supporting new Benchmark releases, expanding coverage with new distributions and easing adoption through built-in workflows, such as moving from your current Benchmark version to a new version while preserving your custom configurations.

Certification and trust

We can proudly announce that azure-osconfig has met all the requirements and is officially certified by the CIS for Benchmark assessment, so you can trust compliance results as authoritative. Minor benchmark updates will be applied automatically, while major version will be released separately. We will include workflows to help migrate customizations seamlessly across versions.

Key Highlights

Built-in CIS Benchmarks for Azure Endorsed Linux distributions
Full parity with official CIS Benchmarks content and certified by the CIS for Benchmark Assessment
Flexible configuration: adjust parameters, define exceptions, tune severity
Hybrid support: enforce the same baseline across Azure, on-prem, and multi-cloud with Azure Arc
Reporting format in CIS tooling style

Supported use cases

Certified CIS Benchmarks for all Azure Endorsed Distros - Audit only (L1/L2 server profiles)
Hybrid / On-premises and other cloud machines with Azure Arc for the supported distros
Compliance as Code (example via Github -> Azure OIDC auth and API integration)
Compatible with GuestConfig workbook

What’s next?

Our next mission is to bring the previously announced auto-remediation capability into this experience, expand the distribution coverage and elevate our workflows even further. We’re focused on empowering you to resolve issues while honoring the unique operational complexity of your environments. Stay tuned!

Get Started

Documentation link for this capability
Enable CIS Benchmarks in Machine Configuration and select the “Official Center for Internet Security (CIS) Benchmarks for Linux Workloads” then select the distributions for your assignment, and customize as needed.
In case if you want any additional distribution supported or have any feedback for azure-osconfig – please open an Azure support case or a Github issue here
Relevant Ignite 2025 session:
- Hybrid workload compliance from policy to practice on Azure

Connect with us at Ignite

Meet the Linux team and stop by the Linux on Azure booth to see these innovations in action:

Session Type	Session Code	Session Name	Date/Time (PST)
Theatre	THR 712	Hybrid workload compliance from policy to practice on Azure	Tue, Nov 18/ 3:15 PM – 3:45 PM
Breakout	BRK 143	Optimizing performance, deployments, and security for Linux on Azure	Thu, Nov 20/ 1:00 PM – 1:45 PM
Breakout	BRK 144	Build, modernize, and secure AKS workloads with Azure Linux	Wed, Nov 19/ 1:30 PM – 2:15 PM
Breakout	BRK 104	From VMs and containers to AI apps with Azure Red Hat OpenShift	Thu, Nov 20/ 8:30 AM – 9:15 AM
Theatre	THR 701	From Container to Node: Building Minimal-CVE Solutions with Azure Linux	Wed, Nov 19/ 3:30 PM – 4:00 PM
Lab	Lab 505	Fast track your Linux and PostgreSQL migration with Azure Migrate	Tue, Nov 18/ 4:30 PM – 5:45 PM PST Wed, Nov 19/ 3:45 PM – 5:00 PM PST Thu, Nov 20/ 9:00 AM – 10:15 AM PST

Innovations and Strengthening Platforms Reliability Through Open Source

KashanK — Tue, 18 Nov 2025 06:42:34 GMT

The Linux Systems Group (LSG) at Microsoft is the team building OS innovations in Azure enabling secure and high-performance platforms that power millions of workloads worldwide. From providing the OS for Boost, optimizing Linux kernels for hyperscale environments or contributing to open-source projects like Rust-VMM and Cloud Hypervisor, LSG ensures customers get the best of Linux on Azure. Our work spans performance tuning, security hardening, and feature enablement for new silicon enablement and cutting-edge technologies, such as Confidential Computing, ARM64 and Nvidia Grace Blackwell all while strengthening the global open-source ecosystem. Our philosophy is simple: we develop in the open and upstream first, integrating improvements into our products after they’ve been accepted by the community.

At Ignite we like to highlight a few open-source key contributions in 2025 that are the foundations for many product offerings and innovations you will see during the whole week. We helped bring seamless kernel update features (Kexec HandOver) to the Linux kernel, improved networking paths for AI platforms, strengthened container orchestration and security efforts, and shared engineering insights with global communities and conferences. This work reflects Microsoft’s long-standing commitment to open source, grounded in active upstream participation and close collaboration with partners across the ecosystem. Our engineers work side-by-side with maintainers, Linux distro partners, and silicon providers to ensure contributions land where they help the most, from kernel updates to improvements that support new silicon platforms.

Linux Kernel Contributions

Enabling Seamless Kernel Updates: Persistent uptime for critical services is a top priority. This year, Microsoft engineer Mike Rapoport successfully merged Kexec HandOver (KHO) into Linux 6.16 ¹. KHO is a kernel mechanism that preserves memory state across a reboot (kexec), allowing systems to carry over important data when loading a new kernel. In practice, this means Microsoft can apply security patches or kernel updates to Azure platform and customers VMs without rebooting or with significantly reduced downtime. It’s a technical achievement with real impact: cloud providers and enterprises can update Linux on the fly, enhancing security and reliability for services that demand continuous availability.

Optimizing Network Drivers for AI Scale: Massive AI models require massive bandwidth. Working closely with our partners deploying large AI workloads on Azure, LSG engineers delivered a breakthrough in Linux networking performance. LSG team rearchitected the receive path of the MANA network driver (used by our smart NICs) to eliminate wasted memory and enable recycling of buffers.

2x higher effective network throughput on 64 KB page systems
35% better memory efficiency for RX buffers
15% higher throughput and roughly half the memory use even on standard x86_64 VMs

References

MANA RX optimization patch: net: mana: Use page pool fragments for RX buffers LKML

Linux Plumbers 2025 talk: Optimizing traffic receive (RX) path in Linux kernel MANA Driver for larger PAGE_SIZE systems

Improving Reliability for Cloud Networking: In addition to raw performance, reliability got a boost. One critical fix addressed a race condition in the Hyper-V hv_netvsc driver that sometimes caused packet loss when a VM’s network channel initialized. By patching this upstream, we improved network stability for all Linux guests running on Hyper-V keeping customer VMs running smoothly during dynamic operations like scale-out or live migrations. Our engineers also upstreamed numerous improvements to Hyper-V device drivers (covering storage, memory, and general virtualization).We fixed interrupt handling bugs, eliminated outdated patches, and resolved issues affecting ARM64 architectures. Each of these fixes was contributed to the mainline kernel, ensuring that any Linux distribution running on Hyper-V or Azure benefits from the enhanced stability and performance.

References

Upstream fix: hv_netvsc race on early receive events: kernel.org commit referenced by Ubuntu bug Launchpad
Ubuntu Azure backport write-up: Bug 2127705 – hv_netvsc: fix loss of early receive events from host during channel open Launchpad
Older background on hv_netvsc packet-loss issues: kernel.org bug 81061

Strengthening Core Linux Infrastructure: Several of our contributions targeted fundamental kernel subsystems that all Linux users rely on. For example, we led significant enhancements to the Virtual File System (VFS) layer reworking how Linux handles process core dumps and expanding file management capabilities. These changes improve how Linux handles files and memory under the hood, benefiting scenarios from large-scale cloud storage to local development. We also continued upstream efforts to support advanced virtualization features.Our team is actively upstreaming the mshv_vtl driver (for managing secure partitions on Hyper-V) and improving Linux’s compatibility with nested virtualization on Azure’s Microsoft Hypervisor (MSHV). All this low-level work adds up to a more robust and feature-rich kernel for everyone.

References

Example VFS coredump work: split file coredumping into coredump_file()

mshv_vtl driver patchset: Drivers: hv: Introduce new driver – mshv_vtl (v10) and v12 patch series on patchew

Bolstering Linux Security in the Cloud: Security has been a major thread across our upstream contributions. One focus area is making container workloads easier to verify and control. Microsoft engineers proposed an approach for code integrity in containers built on containerd’s EROFS snapshotter, shared as an open RFC in the containerd project -GitHub. The idea is to use read-only images plus integrity metadata so that container file systems can be measured and checked against policy before they run.

We also engaged deeply with industry partners on kernel vulnerability handling. Through the Cloud-LTS Linux CVE workgroup, cloud providers and vendors collaborate in the open on a shared analysis of Linux CVEs. The group maintains a public repository that records how each CVE affects various kernels and configurations, which helps reduce duplicated triage work and speeds up security responses.

On the platform side, our engineers contributed fixes to the OP-TEE secure OS used in trusted execution and secure-boot scenarios, making sure that the cryptographic primitives required by Azure’s Linux boot flows behave correctly across supported devices. These changes help ensure that Linux verified boot chains remain reliable on Azure hardware.

References

containerd RFC: Code Integrity for OCI/containerd Containers using erofs-snapshotter GitHub

Cloud-LTS public CVE analysis repo: cloud-lts/linux-cve-analysis

Linux CVE workgroup session at Linux Plumbers 2025: Linux CVE workgroup

OP-TEE project docs: OP-TEE documentation

Developer Tools & Experience

Smoother OS Management with Systemd: Ensuring Linux works seamlessly on Azure scale. The core init system systemd saw important improvements from our team this year. LSG contributed and merged upstream support for disk quota controls in systemd services. With new directives (like StateDirectoryQuota and CacheDirectoryQuota), administrators can easily enforce storage limits for service data, which is especially useful in scenarios like IoT devices with eMMC storage on Azure’s custom SoCs. In addition, Sea-Team added an auto-reload feature to systemd-journald, allowing log configuration changes to apply at runtime without restarting the logging service . These improvements, now part of upstream systemd, help Azure and other Linux environments perform updates or maintenance with minimal disruption to running services. These improvements help Azure and other environments roll out configuration updates with less impact on running workloads.

References

systemd quota directives: systemd.exec(5) – StateDirectoryQuota and related options
systemd journald reload behavior: systemd-journald.service(8)

Empowering Linux Quality at Scale: Running Linux on Azure at global scale requires extensive, repeatable testing. Microsoft continues to invest in LISA (Linux Integration Services Automation), an open-source framework that validates Linux kernels and distributions on Azure and other Hyper-V–based environments.

Over the past year we expanded LISA with:

New stress tests for rapid reboot sequences to catch elusive timing bugs
Better failure diagnostics to make complex issues easier to root-cause
Extended coverage for ARM64 scenarios and technologies like InfiniBand networking
Integration of Azure VM SKU metadata and policy checks so that image validation can automatically confirm conformance to Azure requirements

These changes help us qualify new kernels, distributions, and VM SKUs before they are shipped to customers. Because LISA is open source, partners and Linux vendors can run the same tests and share results, which raises quality across the ecosystem.

References

```
LISA GitHub repo: microsoft/lisa
```

LISA documentation: Welcome to Linux Integration Services Automation LISA Documentation

Community Engagement and Leadership

Sharing Knowledge Globally: Open-source contribution is not just about code - it’s about people and knowledge exchange. Our team members took active roles in community events worldwide, reflecting Microsoft’s growing leadership in the Linux community. We were proud to be a Platinum Sponsor of the inaugural Open Source Summit India 2025 in Hyderabad, where LSG engineers served on the program committee and hosted technical sessions. At Linux Security Summit Europe 2025, Microsoft’s security experts shaped the agenda as program committee members, delivered talks (such as “The State of SELinux”), and even led panel discussions alongside colleagues from Intel, Arm, and others. And in Paris at Kernel Recipes 2025, our own SMEs shared kernel insights with fellow developers. By engaging in these events, Microsoft not only contributes code but also helps guide the conversation on the future of Linux. These relationships and public interactions build mutual trust and ensure that we remain closely aligned with community priorities.

References

Event: Open Source Summit India 2025 – Linux Foundation

Paul Moore’s talk archive:  LSS-EU 2025

Conference: Kernel Recipes 2025 and Kernel Recipes 2025 schedule

Closing Thoughts

Microsoft’s long-term commitment to open source remains strong, and the Linux Systems Group will continue contributing upstream, collaborating across the industry, and supporting the upstream communities that shape the technologies we rely on. Our work begins in upstream projects such as the Linux kernel, Kubernetes, and systemd, where improvements are shared openly before they reach Azure. The progress highlighted in this blog was made possible by the wider Linux community whose feedback, reviews, and shared ideas help refine every contribution. As we move ahead, we welcome maintainers, developers, and enterprise teams to engage with our projects, offer input, and collaborate with us. We will continue contributing code, sharing knowledge, and supporting the open-source technologies that power modern computing, working with the community to strengthen the foundation and shape a future that benefits everyone.

References & Resources:

Linux on Azure at Microsoft Ignite 2025: What’s New, What to Attend, and Where to Find Us

shreyabaheti — Mon, 17 Nov 2025 18:55:08 GMT

Microsoft Ignite 2025 is almost here, and we’re heading back to San Francisco from November 17-21 with a full digital experience for those joining online. Every year, Ignite brings together IT pros, developers, security teams, and technology leaders from around the world to explore the future of cloud, AI, and infrastructure. This year, Linux takes center stage in a big way.

From new security innovations in Azure Linux to deeper AKS modernization capabilities and hands-on learning opportunities, Ignite 2025 is packed with content for anyone building, running, or securing Linux-based workloads in Azure. Below is your quick guide to the biggest Linux announcements and the must-see sessions.

Major Linux Announcements at Ignite 2025

Public Preview: Built-in CIS Benchmarks for Azure Endorsed Linux Distributions

CIS Benchmarks are now integrated directly into Azure Machine Configuration, giving you automated and customizable compliance monitoring across Azure, hybrid, and on-prem environments. This makes it easier to continuously govern your Linux estate at scale with no external tooling required.

Public Preview: Azure Linux OS Guard

Azure Linux OS Guard introduces a hardened, immutable Linux container host for AKS with FIPS mode enforced by default, a reduced attack surface, and tight AKS integration. It is ideal for highly regulated or sensitive workloads and brings stronger default security with less operational complexity.

General Availability: Pod Sandboxing for AKS (Kata Containers)

Pod Sandboxing with fully managed Kata Containers is now GA, delivering VM-level isolation for AKS workloads. This provides stronger separation of CPU, memory, and networking and is well-suited for multi-tenant applications or organizations with strict compliance boundaries.

Linux Sessions at Ignite

Whether you are optimizing performance, modernizing with containers, or exploring new security scenarios, there is something for every Linux practitioner.

Breakout Sessions

Session Code	Session Name	Date and Time (PST)
BRK143	Optimizing performance, deployments, and security for Linux on Azure	Thu Nov 20, 1:00 PM to 1:45 PM
BRK144	Build, modernize, and secure AKS workloads with Azure Linux	Wed Nov 19, 1:30 PM to 2:15 PM
BRK104	From VMs and containers to AI apps with Azure Red Hat OpenShift	Thu Nov 20, 8:30 AM to 9:15 AM
BRK137	Nasdaq Boardvantage : AI-driven governance on PostgreSQL and AI Foundry	Wed Nov 19, 11:30 AM to 12:15 PM

Theatre Sessions

Session Code	Session Name	Date and Time (PST)
THR712	Hybrid workload compliance from policy to practice on Azure	Tue Nov 18, 3:15 PM to 3:45 PM
THR701	From Container to Node: Building Minimal-CVE Solutions with Azure Linux	Wed Nov 19, 3:30 PM to 4:00 PM

Hands-on Lab

Lab 505: Fast track your Linux and PostgreSQL migration with Azure Migrate

Tue Nov 18, 4:30 PM to 5:45 PM
Wed Nov 19, 3:45 PM to 5:00 PM
Thu Nov 20, 9:00 AM to 10:15 AM

This interactive lab helps you assess, plan, and execute Linux and PostgreSQL migrations at scale using Azure Migrate’s end-to-end tooling.

Meet the Linux on Azure Team at Ignite

If you are attending in person, come say hello. Visit the Linux on Azure Expert Meetup stations inside the Microsoft Hub. You can ask questions directly to Microsoft’s Linux engineering and product experts, explore demos across Azure Linux, compliance, and migration, and get recommendations tailored to your workloads. We always love meeting customers and partners.

Dalec: Declarative Package and Container Builds

SertacOzercan — Wed, 29 Oct 2025 21:34:49 GMT

Dalec, a Cloud Native Computing Foundation (CNCF) Sandbox project, is a Docker BuildKit frontend that enables users to build system packages and container images from declarative YAML specifications. As a BuildKit frontend, Dalec integrates directly into the Docker build process, requiring no additional tools beyond Docker itself.

Dalec’s primary focus is building native Linux packages (RPM and DEB formats) from source, with optional container image creation from those packages. It supports RPM-based distributions such as Azure Linux, AlmaLinux, and Rocky Linux, as well as DEB-based distributions like Debian and Ubuntu. It also supports pluggable backends for additional operating systems.

By replacing complex spec files and multi-stage Dockerfiles with a single declarative configuration, Dalec simplifies package building while providing built-in support for SBOMs, provenance attestations, and package signing.

Why Dalec?

Building packages traditionally requires:

Manual creation of distribution-specific spec files (.spec for RPM, debian/ directory for DEB)
Multi-stage Dockerfiles with complex build logic
Architecture-specific build configurations
Separate build processes for each target distribution
Manual dependency management and bootstrapping

Dalec replaces this complexity with a single declarative YAML specification that defines:

Sources - Git repositories, HTTP archives, or inline content
Dependencies - Build-time and runtime package dependencies
Build steps - Commands to compile and prepare artifacts
Artifacts - Binaries, configuration files, and systemd units to package
Targets - Distribution-specific customizations
Tests - Validation of the built package
Image config - Optional container image creation from the package

The result is a portable, and auditable build process.

Key Benefits

🐳 Zero Installation - Works with standard Docker. No specialized tooling, package managers, or daemon processes required. If you can run docker build, you can use Dalec.

🚀 Fast and Cacheable - BuildKit’s intelligent caching means rebuilds are lightning fast. Change one source file, rebuild only what’s affected.

🤝 Integration – Dalec integrates at both the package manager level and with language toolchains such as Go and Rust. This allows Dalec to automatically manage caches for items such as Go modules, incremental compiler caches, and packages across builds.

🔒 Secure by Default - Automatic SBOM generation, provenance attestations, and package signing built into the workflow. Supply chain security without extra steps.

📦 Package and Container - Build once, output both. Get installable RPM/DEB packages for traditional deployments AND minimal container images for Kubernetes from the same specification.

🌍 Multi-Distribution - One spec, multiple targets. Build for Ubuntu, Debian, Azure Linux, Rocky Linux, and AlmaLinux without maintaining separate build configurations.

✍️Better Composition - Declarative configurations compose cleanly. You can override specific parts for different distributions or targets without rewriting the entire build logic, and focusing on the package level makes it more natural to break components down into more composable pieces.

🔍 Fully Auditable - Declarative configuration means no hidden build steps. Every artifact’s provenance is traceable, meeting compliance and security requirements.

Who Should Use Dalec?

Dalec makes it easy for different audiences to achieve their goals:

For Application Developers

Convert your source code into distributable packages without learning RPM spec files or Debian packaging conventions. Focus on your application, not build infrastructure.

For Platform Operators

Maintain a consistent build process across your organization. Centralize packaging expertise in reusable specifications instead of scattered Dockerfiles and build scripts. Enforce security and compliance requirements at build time.

For Package Maintainers

Build packages for multiple distributions from a single source of truth. Reduce maintenance burden and ensure consistency across your supported platforms.

How Dalec Works

Dalec is implemented as a BuildKit frontend. When you specify # syntax=ghcr.io/project-dalec/dalec/frontend:latest at the top of your spec file, Docker BuildKit automatically pulls and executes the Dalec frontend, which translates your YAML specification into low-level build instructions (LLB graphs).

The Build Flow:

Source acquisition - Clone git repositories, download archives, generate dependency manifests (like Go modules, npm packages, or Python wheels)
Package build - Execute build steps in isolated environments with dependencies installed
Package creation - Create RPM or DEB packages with artifacts and metadata
Package testing - Install and validate the package in a clean environment
Container creation (optional) - Install packages into a minimal base image

From a single YAML file, you can generate multi-architecture packages that work across distributions, then optionally compose them into minimal container images—all without writing a single line of Dockerfile or distribution-specific spec.

Real-World Use Cases

Open Source Projects - Distribute your software across multiple Linux distributions without maintaining separate packaging workflows. A single Dalec spec replaces RPM spec files, debian/ directories, and custom build scripts.

Enterprise Deployments - Build compliant, auditable packages for both traditional VM-based deployments and modern Kubernetes clusters. The same build produces installable packages for your data center and container images for your cloud infrastructure.

CI/CD Pipelines - Integrate seamlessly into GitHub Actions, GitLab CI, or any CI system that supports Docker. No special runners or build agents required, just standard Docker.

Security-Critical Applications - Leverage BuildKit's built-in SBOM and provenance generation to meet supply chain security requirements. Every build produces attestations that prove what went into your packages and containers.

Multi-Architecture Builds - Build for x86_64, ARM64, and other architectures with a single command. BuildKit handles the complexity of cross-compilation automatically.

A Simple Example

Here’s what a Dalec spec looks like. This example builds go-md2man, a tool that converts Markdown to man pages:

# syntax=ghcr.io/project-dalec/dalec/frontend:latest name: go-md2man version: 2.0.3 revision: "1" packager: Dalec Example vendor: Dalec Example license: MIT description: A tool to convert markdown into man pages (roff). website: https://github.com/cpuguy83/go-md2man sources: src: generate: - gomod: {} # Pre-downloads Go modules (network disabled during build) git: url: https://github.com/cpuguy83/go-md2man.git commit: v2.0.3 dependencies: build: golang: build: env: CGO_ENABLED: "0" steps: - command: | cd src go build -o go-md2man . artifacts: binaries: src/go-md2man: image: entrypoint: go-md2man cmd: --help tests: - name: Check bin files: /usr/bin/go-md2man: permissions: 0755

From this single YAML file, you can build RPM packages, DEB packages, and container images for multiple distributions and architectures using standard docker build commands.

Dalec can also create minimal container images with just runtime dependencies, no source code building required. This is perfect for creating lightweight containers with only the packages you need.

Here’s a minimal example that creates a container with curl and bash:

# syntax=ghcr.io/project-dalec/dalec/frontend:latest name: my-minimal-image version: 0.1.0 revision: "1" license: MIT description: A minimal image with only curl and shell access dependencies: runtime: curl: bash: image: entrypoint: /bin/bash

Build it with:

docker build -f my-minimal-image.yml --target=azlinux3 -t my-minimal-image:0.1.0 .

This produces a minimal image built from scratch containing only curl, bash, and their dependencies.

Pro tip: You can skip creating a spec file entirely by passing dependencies on the command line:

docker build -t my-minimal-image:0.1.0 --build-arg BUILDKIT_SYNTAX=ghcr.io/project-dalec/dalec/frontend:latest --target=azlinux3/container/depsonly - <<<"$(jq -c '.dependencies.runtime = {"curl": {}, "bash": {}} | .image.entrypoint = "/bin/bash"' <<<"{}")"

Learn more in the Container-only builds documentation.

Getting Started

Ready to build your first package and container image? The Dalec Quickstart walks you through building go-md2man, a real-world example that demonstrates:

Creating a declarative YAML specification
Building RPM and DEB packages
Creating minimal container images
Multi-architecture builds

The quickstart shows how a single spec can produce packages for multiple distributions (Azure Linux, Ubuntu, Debian) and architectures (x86_64, ARM64) using standard docker build commands.

CI/CD Integration

Dalec integrates seamlessly into any CI/CD system that supports Docker, including GitHub Actions, GitLab CI, Jenkins, and cloud-native build systems. Since Dalec builds use standard docker build commands, no special runners or build agents are required—just a standard Docker environment.

Thanks to BuildKit, Dalec can also build directly on Kubernetes clusters using the Kubernetes driver. This enables scalable, cloud-native builds without requiring dedicated build VMs, making it ideal for large-scale CI/CD pipelines.

This makes it easy to:

Build and publish packages on every release
Generate multi-architecture artifacts in parallel
Integrate SBOM generation and signing into your pipeline
Push containers to any OCI-compliant registry
Scale builds elastically on Kubernetes infrastructure

Supply Chain Security

SBOM and Provenance Attestations

BuildKit provides integrated support for Software Bill of Materials (SBOM) generation and provenance attestations. SBOMs automatically catalog every package and dependency in your build, providing complete transparency into what’s inside your artifacts. Provenance attestations are cryptographically-signed records that prove how your image was built, including the source repository, build parameters, and execution environment. This enables verification throughout your deployment pipeline. Learn more about configuring these features in the BuildKit attestations documentation.

Package Signing

Dalec supports GPG signing of packages for additional trust and verification:

package_config:
  signer:
    frontend:
      image: <signer-image>

Signed packages ensure recipients can verify authenticity and detect tampering.

Conclusion

Dalec provides a modern, declarative approach to building system packages and containers. By leveraging Docker BuildKit as a frontend, it eliminates the need for complex build toolchains while providing secure builds across multiple Linux distributions.

For open-source projects that need to distribute both packages and containers, Dalec offers a unified build process that simplifies CI/CD while strengthening supply chain security.

Whether you’re migrating from traditional RPM spec files, consolidating Dockerfiles, or building a new project from scratch, Dalec provides the simplicity of modern container tools with the flexibility of native package formats.

Get Started

Ready to try Dalec? You’re just one docker build command away:

Explore the documentation: project-dalec.github.io/dalec
Try the quickstart tutorial: Build your first package and container
Browse examples: See real-world specs in the examples directory
Join the community: Connect with us in the #dalec channel on CNCF Slack
Contribute: Dalec is an open-source project under the CNCF. Contributions are welcome! Check out the contributing guide to get involved

Red Hat Enterprise Linux Software Reservations Now Available

abbottkarl — Mon, 27 Oct 2025 16:00:00 GMT

What's New

After careful collaboration with Red Hat, we've updated our RHEL pay-as-you-go billing meters and reservation pricing to align with Red Hat's latest pricing model. These updates address previous billing meter issues and ensure an accurate, transparent experience for our customers.

Starting today, customers can purchase RHEL software reservations on Azure with updated pricing, allowing organizations to reduce their Linux workload costs with the flexibility and reliability of software reservations.

Key Benefits of RHEL Software Reservations

Cost savings: Save up to 24% compared to pay-as-you-go pricing by committing to a one-year term

Predictable costs: Lock in pricing for your RHEL workloads and optimize your cloud budget

Pricing clarity: Updated meters aligned with Red Hat's current pricing model

Seamless Azure integration: Manage your RHEL software reservations alongside other Azure resources.

RHEL software reservations allow you to pre-purchase RHEL software capacity at a discounted rate, delivering significant savings over standard pay-as-you-go pricing.

Get Started Today

RHEL software reservations are available now on the Azure portal. To learn more about pricing, terms, and how to purchase them, visit the following pages:

Pricing - Linux Virtual Machines | Microsoft Azure

Prepay for software plans - Azure Reservations - Azure Virtual Machines | Microsoft Learn

Red Hat reservation plan discounts - Azure - Microsoft Cost Management | Microsoft Learn

What are Azure Reservations? - Microsoft Cost Management | Microsoft Learn

We're committed to providing transparent, reliable billing for all Azure services and appreciate your continued partnership as we deliver the best cloud platform for your open-source workloads.

For questions or support, please contact Azure Support or your Microsoft account team.

Drasi is Fluent in GQL: Integrating the New Graph Query Standard

CollinBrian — Thu, 09 Oct 2025 18:21:36 GMT

Drasi , the open-source Rust data change processing platform, simplifies the creation of change-driven systems through continuous queries, reactions, and clearly defined change semantics. Continuous queries enable developers to specify precisely what data changes matter, track these changes in real-time, and react immediately as changes occur. Unlike traditional database queries, which provide static snapshots of data, continuous queries constantly maintain an up-to-date view of query results, automatically notifying reactions of precise additions, updates, and deletions to the result set as they happen.

To date, Drasi has supported only openCypher for writing continuous queries; openCypher is a powerful declarative graph query language. Recently, Drasi has added support for Graph Query Language (GQL), the new international ISO standard for querying property graphs. In this article, we describe what GQL means for writing continuous queries and describe how we implemented GQL Support.

A Standardized Future for Graph Queries

GQL is the first officially standardized database language since SQL in 1987. Published by ISO/IEC in April 2024, it defines a global specification for querying property graphs. Unlike the relational model that structures data into tables, the property graph model structures data inside of the database as a graph. With GQL support, Drasi enables users to benefit from a query language that we expect to be widely adopted across the database industry, ensuring compatibility with future standards in graph querying.

Drasi continues to support openCypher, allowing users to select the query language that best fits their requirements and existing knowledge. With the introduction of GQL, Drasi users can now write continuous queries using the new international standard.

Example GQL Continuous Query: Counting Unique Messages

Event-driven architectures traditionally involve overhead for parsing event payloads, filtering irrelevant data, and managing contextual state to identify precise data transitions. Drasi eliminates much of this complexity through continuous queries, which maintain accurate real-time views of data and generate change notifications.

Imagine a simple database with a message table containing the text of each message. Suppose you want to know, in real-time, how many times the same message has been sent. Traditionally, addressing these types of scenarios involves polling databases at set intervals, using middleware to detect state changes, and developing custom logic to handle reactions. It could also mean setting up change data capture (CDC) to feed a message broker and process events through a stream processing system. These methods can quickly become complex and difficult, especially when handling numerous or more sophisticated scenarios.

Drasi simplifies this process by employing a change-driven architecture. Rather than relying on polling or other methods, Drasi uses continuous queries that actively monitor data for specific conditions. The moment a specified condition is met or changes, Drasi proactively sends notifications, ensuring real-time responsiveness.

The following example shows the continuous query in GQL that counts the frequency of each unique message:

MATCH 
   (m:Message)
LET Message = m.Message
 RETURN
   Message,
   count(Message) AS Frequency

You can explore this example in the Drasi Getting Started tutorial.

Key Features of the GQL Language

OpenCypher had a significant influence on GQL and there are many things in common between the two languages; however, there are also some important differences.

A new statement introduced in GQL is NEXT, which enables linear composition of multiple statements. It forms a pipeline where each subsequent statement receives the working table resulting from the previous statement.

One application for NEXT is the ability to filter results after an aggregation. For example, to find colors associated with more than five vehicles, the following query can be used:

MATCH (v:Vehicle)
RETURN v.color AS color, count(v) AS vehicle_count
NEXT FILTER vehicle_count > 5
RETURN color, vehicle_count

Equivalent openCypher:

MATCH (v:Vehicle)
WITH v.color AS color, count(v) AS vehicle_count
WHERE vehicle_count > 5
RETURN color, vehicle_count

GQL introduces additional clauses and statements: LET, YIELD, and FILTER.

The LET statement allows users to define new variables or computed fields for every row in the current working table. Each LET expression can reference existing columns in scope, and the resulting variables are added as new columns.

Example:

MATCH (v:Vehicle)
LET makeAndModel = v.make + '  ' + v.model
RETURN makeAndModel, v.year

Equivalent openCypher:

MATCH (v:Vehicle)
WITH v, v.make + '  ' + v.model AS makeAndModel
RETURN makeAndModel, v.year

The YIELD clause projects and optionally renames columns from the working table, limiting the set of columns available in scope. Only specified columns remain in scope after YIELD.

Example:

MATCH (v:Vehicle)-[e:LOCATED_IN]->(z:Zone)
YIELD v.color AS vehicleColor, z.type AS location
RETURN vehicleColor, location

FILTER is a standalone statement that removes rows from the current working table based on a specified condition. While GQL still supports a WHERE clause for filtering during the MATCH phase, the FILTER statement provides additional flexibility by allowing results to be filtered after previous steps. It does not create a new table; instead, it updates the working table. Unlike openCypher’s WHERE clause, which is tied to a MATCH or WITH, GQL's FILTER can be applied independently at various points in the query pipeline.

Example:

MATCH (n:Person)
FILTER n.age > 30
RETURN n.name, n.age

GQL also provides control in how aggregations are grouped. The GROUP BY clause can be used to explicitly define the grouping keys, ensuring results are aggregated exactly as intended.

MATCH (v:Vehicle)-[:LOCATED_IN]->(z:Zone)
RETURN z.type AS zone_type, v.color AS vehicle_color, count(v) AS vehicle_count
GROUP BY zone_type, vehicle_color

If the GROUP BY clause is omitted, GQL defaults to an implicit grouping behavior, having all non-aggregated columns in the RETURN clause automatically used as the grouping keys.

While many of the core concepts, like pattern matching, projections, and filtering, will feel familiar to openCypher users, GQL’s statements are distinct in their usage. Supporting these differences in Drasi required design changes, described in the following section, that led to multiple query languages within the platform.

Refactoring Drasi for Multi-Language Query Support

Instead of migrating Drasi from openCypher to GQL, we saw this as an opportunity to address multi-language support in the system. Drasi's initial architecture was designed exclusively for openCypher. In this model, the query parser generated an Abstract Syntax Tree (AST) for openCypher. The execution engine was designed to process this AST format, executing the query it represented to produce the resulting dataset. Built‑in functions (such as toUpper() for string case conversion) followed openCypher naming and were implemented within the same module as the engine. This created an architectural challenge for supporting additional query languages, such as GQL.

To enable multi-language support, the system was refactored to separate the parsing, execution, and function management. A key insight was that the existing AST structure, originally created for openCypher, was flexible enough to be used for GQL. Although GQL and openCypher are different languages, their core operations, matching patterns, filtering data, and projecting results, could be represented by this AST.

The diagram shows the dependencies within this new architecture, highlighting the separation and interaction between the components. The language-specific function modules for openCypher and GQL provide the functions to the execution engine. The language-specific parsers for openCypher and GQL produce an AST, and the execution engine operates on this AST. The engine only needs to understand this AST format, making it language-agnostic.

The AST structure is based on a sequence of QueryPart objects. Each QueryPart represents a distinct stage of the query, containing clauses for matching, filtering, and returning data. The execution engine processes these QueryParts sequentially.

pub struct QueryPart {
    pub match_clauses: Vec<MatchClause>,
    pub where_clauses: Vec<Expression>,
    pub return_clause: ProjectionClause,
}

The process begins when a query is submitted in either GQL or openCypher. The query is first directed to its corresponding language-specific parser, which handles the lexical analysis and transforms the raw query string into the standardized AST. When data changes occur in the graph, the execution engine uses the MATCH clauses from the first QueryPart to find affected graph patterns and captures the matched data. This matched data then flows through each QueryPart in sequence. The WHERE portion of the AST filters out data that does not meet the specified conditions. The RETURN portion transforms the data by selecting specific fields, computing new values, or performing aggregations. Each QueryPart's output becomes the next one's input, creating a pipeline that incrementally produces query results as the underlying graph changes.

To support functions from multiple languages in this AST, we introduced a function registry to abstract a function's name from its implementation. Function names can differ (e.g., toUpper() in openCypher versus Upper() in GQL). For any given query, language-specific modules populate this registry, mapping each function name to its corresponding behavior. Functions with shared logic can be implemented once in the engine and registered under multiple names in specific function crates, preventing code duplication. Meanwhile, language-exclusive functions can be registered and implemented separately within their respective modules. When processing an AST, the engine uses the registry attached to that query to resolve and execute the correct function. The separate function modules allow developers to introduce their own function registry, supporting custom implementations or names.

Conclusion

By adding support for GQL, Drasi now offers developers a choice between openCypher and the new GQL standard. This capability ensures that teams can use the syntax that best fits their skills and project requirements. In addition, the architectural changes set the foundation for additional query languages.

You can check out the code on our GitHub organization, dig into the technical details on our documentation site, and join our developer community on Discord.

The Open-Source Paradox: How Microsoft is giving back

LachlanEvenson — Thu, 02 Oct 2025 17:45:28 GMT

The open-source community faces a paradox that threatens its very foundation. While open-source software now powers every major technology platform—from smartphones in our pockets to the cloud infrastructure running the world's largest applications—the developers and maintainers who create and sustain these critical projects are struggling under an unsustainable burden.

The statistics paint a sobering picture of this crisis:

🧑‍💻 60% of open-source maintainers are unpaid for their contributions, volunteering countless hours to maintain code that powers the global economy.

🧠 58% have quit or considered quitting due to burnout, threatening the continuity of projects millions depend on daily.

🏢 90% of enterprises rely heavily on open source, yet only about 60% encourage their employees to contribute back to the projects they depend on.

💸 Only 14% of corporate open-source investment goes directly to funding maintainers and projects, with the majority flowing to consulting and support services.

This creates an unsustainable equation: critical infrastructure maintained by exhausted volunteers while billion-dollar companies extract value without proportional investment back into the ecosystem. At Microsoft, we have witnessed this challenge firsthand—and we believe we have a responsibility to be part of the solution.

Why All Things Open Matters to Microsoft

This is precisely why I am excited to share that Microsoft will be joining the incredible community at All Things Open 2025 in Raleigh, North Carolina, October 12–14. As a first-time presenting sponsor of this premier open-source conference, we are not just displaying our technology, we are demonstrating our commitment to addressing the sustainability challenges facing the open-source ecosystem.

All Things Open represents everything we have come to understand about building healthy open-source communities: collaboration over competition, sustainability over short-term gains, and the shared belief that technology should empower everyone. With attendees from sixty-nine countries and a focus on real developer challenges, this conference provides the perfect platform to discuss not just what we are building, but how we are building it responsibly.

From Consumer to Contributor: Microsoft's Evolution

Our journey with open source began with a fundamental shift in perspective. We moved from being primarily consumers of open-source software to becoming one of the world's largest contributors to open-source projects. This was not just a strategic decision, but it was a recognition that sustainable technology requires sustainable communities.

Today, Microsoft is among the top contributors to open-source projects on GitHub, employs full-time open-source maintainers, and partners with the community to strengthen open source for everyone. This commitment includes investments through GitHub, Azure credits, and direct funding of projects like Alpha Omega—a Linux Foundation initiative working to secure the critical open-source software we all rely on. But numbers only tell part of the story.

What We're Bringing to Raleigh

Our Open-Source Journey: From Kernel to Copilot

Join me for my keynote on Monday morning (9:45–10:00 a.m. ET), "From Kernel to Copilot: Microsoft's Open-Source Journey to AI at Scale," where I'll explore how Microsoft's deep commitment to open source has evolved from contributing to the Linux kernel to building AI services that run at global scale. We will examine how open-source technologies power mission-critical workloads across Azure and GitHub, and how Microsoft became the largest cloud provider contributor to Cloud Native Computing Foundation projects. I will also spotlight new initiatives like Radius, Dalec, and Copacetic that reflect our vision for a more collaborative, cloud-native future.

Practical Solutions in Action

In my track session, "Using AI Agents to Empower Application Modernization for Kubernetes," I will demonstrate how open-source tooling combined with AI agents can reduce the friction of modernizing applications for cloud-native platforms. This is not about making development faster, it is about making complex modernization accessible to more developers, reducing the expertise barrier that often prevents teams from adopting cloud-native technologies.

Real Solutions at Our Booth

Visit us at booths 20 and 21, where our team will be demonstrating how we are tackling sustainability challenges:

Linux on Azure: How enterprise adoption drives investment back to Linux ecosystem development
AI-Powered Developer Tools: Reducing the manual overhead of open-source maintenance.
GitHub Programs: Supporting open-source projects through various community initiatives.
Security Initiatives: Automated vulnerability detection and resources to reduce security burden on maintainers.

Deep-Dive Theater Sessions

Throughout the conference, we will be hosting focused discussions on:

The Economics of Open Source: Creating sustainable funding models for critical projects.
Maintainer Mental Health: Building support systems and prevent burnout in open-source communities.
Enterprise Open-Source Strategies: How companies can contribute meaningfully to the projects they depend on

Also, do not miss the following demos and theater sessions in our booth:

From "Open" to "Operable": ARO Virtualization + Arc-Enabled Hybrid, with OSS-First Workflows
This session shows how OpenShift on Azure (ARO) can run open-source-native workflows—GitOps, KubeVirt-based virtualization—while staying portable across clouds and data centers via Azure Arc. We will deploy and onboard an OpenShift cluster to Arc, bootstrap a GitOps app from a public repo, and spin up a Fedora VM with OpenShift Virtualization in under 10 minutes.
By Joel Sisko, Solution Engineer, Microsoft

AKS Automatic: Streamlining AI-Enabled Application Deployment
Learn how AKS Automatic simplifies deploying AI-enabled applications to Kubernetes using GitHub or Azure DevOps. This session covers automated cluster setup with best practices, built-in monitoring, and alerts, and streamlined Day Zero to Day Two operations—including scaling, GitHub Actions, and observability. Focus on innovation while AKS Automatic handles the infrastructure.
By Joel Schluchter, GBB, Microsoft

Enabling AI Agents with Real-Time Linux and Kubernetes Observability via Model Context Protocol
Discover how Microsoft leverages open-source tools and the Model Context Protocol (MCP) to give AI agents real-time observability in Linux and Kubernetes environments. Learn how Inspektor Gadget and the AKS MCP Server enable seamless diagnostics, monitoring, and security from familiar tools like VS Code and GitHub Copilot—accelerating innovation and democratizing observability for cloud-native apps.
By Ron Abellera, Microsoft GBB

The Open-Source AI Editor
VS Code recently open-sourced its AI capabilities! You might be asking: Why? How can I contribute? What about the forks?! We will answer these questions and more as we walk through VS Code's journey to becoming the open-source AI editor.
By Olivia Guzzardo McVicker, Senior Cloud Advocate, Microsoft

Demonstrate the new GHCP App Modernization tool which allows customers to easily upgrade existing Java & .NET applications to newer versions.

Keeping enterprise applications modern is often a months-long effort—auditing codebases, updating dependencies, applying patches, and preparing services for the cloud. With GitHub Copilot app modernization, that process gets radically simplified. In this session, we will show how you can take a legacy Java application and upgrade it in minutes. Copilot now provides end-to-end support for Java and .NET projects, delivering automated assessment reports, code transformations, build patching, dependency updates, and even containerization for cloud deployment. Whether you are migrating to the cloud or simply keeping pace with evolving frameworks, Copilot streamlines the process and reduces the risk, letting your team focus on delivering value instead of wrestling with upgrades. See how you can modernize faster, safer, and smarter.

By Ayan Gupta, Advocacy Program Manager, Microsoft

And much more...

Beyond Technology: Building Community

What excites me most about All Things Open is not just the opportunity to highlight solutions, it is the chance to learn from maintainers, contributors, and users about what works. No single company or initiative can resolve open source's sustainability crisis. It requires the collective wisdom and effort of the entire community.

The Path Forward

The open-source sustainability challenge is not just a technical problem—it is a community problem that requires community solutions. As we gather in Raleigh, we have an opportunity to move beyond talking about these challenges to collaboratively building solutions.

Whether you are maintaining a critical library used by millions, contributing to emerging AI frameworks, or building the next generation of developer tools, your perspective matters. The solutions we build together at events like All Things Open will determine whether open source continues to thrive for the next generation of developers.

Join the Conversation

If you are attending All Things Open 2025, I invite you to be part of this crucial conversation. Share your experiences, challenge our assumptions, and help us build a more sustainable future for open source.

Find me at our booth, attend my keynote, and track sessions, or connect directly. Because the future of open source is not something that happens to us, it is something we build together.

Lachie Evenson leads Microsoft's open-source strategy and community programs. Connect with him on Bluesky, X, LinkedIn, GitHub, or at All Things Open 2025.

Learn more about Microsoft's open-source initiatives at opensource.microsoft.com. Register for All Things Open 2025 at https://2025.allthingsopen.org/

Sources:

Resources:

Introducing Image Customizer for Azure Linux

Kavya_Nagalakunta — Thu, 18 Sep 2025 15:00:00 GMT

We are excited to release Image Customizer, an open-source tool, built and maintained by the Azure Linux team. Image Customizer lets you customize well-tested existing Azure Linux images for any scenario in just minutes. Already trusted by first party teams like LinkedIn, Azure Frontdoor, and Azure Nexus in production, this tool is designed to make image customization simple, reliable, and fast. With full dm-verity support for enhanced security, it also supports customization of Azure Linux with OS Guard images.

Unlike VM-based image customization, Image Customizer directly modifies the image without booting a VM using a chroot-based approach, making customization faster, more reliable, and easier to integrate into existing workflows.

✨ Get Image Customizer here

✨ Explore our documentation here.

Why Choose Image Customizer?

Direct, Reliable Customization

- Build on top of bootable, tested, and supported base images.
- Lower overhead and fewer side effects by avoiding VM boot.
- No need to rely on the Azure Linux Toolkit. Previously, building from scratch meant your image may fail to boot sometimes. Image Customizer reduces that risk.

Clean and Lightweight

- Minimal dependencies for a streamlined setup (for example, no SSH required).
- You only need to invoke one command to run Image Customizer. It is available as a container with all its dependencies bundled for easy integration into CI/CD pipelines.

Versatile and Powerful

- Supported input formats: vhd, vhdx, qcow2, PXE bootable artifacts, raw and iso created by Image Customizer.
- Supported output formats: vhd, vhd-fixed, vhdx, qcow2, raw, iso, and cosi.
- Perform a wide range of operations: add/remove/update packages, add files and directories, create/update users, enable/disable services, customize partitions, image history, dm-verity and more. Full list of supported operations can be found here.

Cross-Platform Compatibility

- Tested and verified to work on Ubuntu 22.04, Azure Linux 3.0 and WSL2 (Windows Subsystem for Linux). While officially tested on these platforms, Image Customizer will likely work on other Linux distributions as well.

Consistent and Predictable Builds

- Use --package-snapshot-time or snapshotTime to filter packages by publication timestamp, ensuring only packages available at that point in time are considered. This prevents unexpected changes from newer package versions when reusing configuration files across time.

Getting Started with Image Customizer

To use Image Customizer, you’ll need a configuration file that describes the changes you want to make, using the Declarative API provided by Image Customizer. Next, select a base Azure Linux image as your foundation. With these two pieces in hand, you’re ready to run Image Customizer. The easiest way is to use the Image Customizer container, which comes pre-packaged with all necessary dependencies and is recommended for most users. Alternatively, you can use the standalone executable binary if that better fits your workflow. In just a few minutes, Image Customizer will generate a modified Azure Linux image tailored to your needs. This process is designed to be repeatable and user-friendly, making it easy to add packages, files, users, make partition changes, and much more.

To help you get started, we have a Quick Start guide that walks you through your first customization step by step. For those who want to explore further, comprehensive API documentation is available, covering both Command-line usage and Configuration options.

Upcoming Community Call

Join our upcoming community call to learn more about using Image Customizer and see a live demo. We’ll cover best practices, advanced scenarios, and answer any questions you may have.

Date & Time: September 25^th, 2025 at 8:00AM PST

Teams Link: Azure Linux - External Community Call | Meeting-Join | Microsoft Teams

Community Call Schedule: https://learn.microsoft.com/en-us/azure/azure-linux/support-help#stay-connected-with-azure-linux

Help and Feedback

If you’d like to report bugs, request features, or contribute to the tool, you can do so directly through our azure-linux-image-tools GitHub repo. We welcome feedback and contributions from the community!

Acknowledgements

A huge thank you (in no order) to our Image Customizer team ─ Adit Jha, Brian Telfer, Chris Gunn, Deepu Thomas, Elaine Zhao, George Mileka, Himaja Kesari, Jim Perrin, Jiri Appl, Lanze Liu, Roaa Sakr, Kavya Nagalakunta and Vince Perri.