The shadow-traffic pattern we used to replace a runtime under live, multi-tenant load, and prove it was safe before moving anyone.
Summary
Azure Logic Apps Consumption Integration Accounts were served by roughly 60,000 per-customer Azure Functions apps on the end-of-life v1/v2 runtime. Over about two years we migrated the entire fleet to Functions v4 (isolated worker), across all Azure regions, with no migration work required from customers. Here is the whole approach before the deep dive:
- Shadow traffic. Run real production traffic through the new runtime in parallel with the old one and compare every result, without ever letting the unproven path serve the customer (Sections 5 and 6).
- A parity bar we could defend. Compare 100% of eligible traffic, separate genuine bugs from naturally nondeterministic workloads, and fix every real divergence before moving anyone (Section 6).
- A progressive, reversible rollout. Shift traffic by a hash-based rate, region by region, with a configuration-only rollback that takes effect in minutes (Section 7).
- Careful retirement. Stop the old apps first, observe, and only then delete, with a long reversible gap (Section 9).
Why this was possible without customer involvement: we own and operate the compute, the durable customer data (agreements, maps, schemas) never moved, and the actions are pure transforms, so we could run both runtimes on the same traffic and compare them behind our own control plane. The result was an aggregate cutover with no material latency or error regression, with one honest caveat: a small subset of customers hit a brief edge case under high load before we rolled back (Section 10).
1. The hard part wasn't the new runtime, it was proving the old one could be replaced
Azure Logic Apps Consumption Integration Accounts power critical behind-the-scenes work in enterprise integration: schema and map storage, XML and flat-file transforms, partner and agreement resolution, and the connectors that lean on them. That work is live, multi-tenant, and always on. Customers don't schedule downtime for it, and neither can we.
Under the hood, those capabilities were served by Azure Functions apps running on runtime v1/v2. Those apps executed the Integration Account actions (transforms, lookups) while the durable data itself (agreements, maps, schemas) lived in storage they didn't own. Those runtime versions have reached the end of their supported life, and staying on them was its own growing risk. Moving to a supported runtime was part of a broader modernization effort: it restored a healthy security- and dependency-update path immediately, and it opened the door to newer .NET down the road. The destination was clear: Azure Functions v4.
Here's the catch that made this more than a version bump. We couldn't prove compatibility from a test environment. The behavior we had to preserve was the behavior real customer workloads depended on, in production, across many regions. A staging suite gives you confidence; it doesn't give you proof.
So we built a migration path around a simple idea. Let the old runtime keep serving the live path while the new runtime processes the same real traffic in parallel, and don't move anyone until the data says it's safe. This post is how we did that: shadow traffic, a parity bar we could defend, a progressive rollout with a fast way back, and a guardrailed path to finally retire the old apps.
Figure 1. Before and after. A new v4 app is provisioned per classic app (one-to-one) and reads the same durable data, which never moves. Customers stay on classic until parity is proven.
2. Why Functions v4, and why we couldn't just flip a switch
The move to v4 is also a move to the isolated worker model. On v1/v2, our code ran in-process with the Functions host. On v4 it runs in an isolated worker process, decoupled from the host. That decoupling is exactly what a modern, supportable runtime looks like, but it's also a breaking change in how the code is hosted, which is the crux of why this couldn't be a quick switch.
We considered the usual lighter-weight options and ruled each out for this workload:
- In-place upgrade. Going from v1/v2 to the v4 isolated worker isn't an in-place runtime bump; it's a host-model change. There's no safe "upgrade the live app and hope" here.
- Deployment slots or blue-green alone. Slots solve routing, not proof. Even with staged or split routing, moving customer traffic across a fleet of roughly 60,000 per-customer apps before we'd proven the new app matched the old one on real requests would still be a blind behavioral bet.
- Maintenance windows. Integration Accounts are live customer infrastructure. There's no acceptable downtime budget for this surface.
Which points at the thesis of the whole project: the approach we chose is more complex for us, but simpler for the customer. We absorbed the complexity so customers wouldn't have to. This was only possible because of where the work sat: we own and operate these Functions apps, the durable customer data never moves, and the actions are pure transforms, so the whole migration could happen behind our control plane without customers changing, redeploying, or even knowing which runtime served a given call.
3. The shape of the problem
A few properties made this harder than a typical app move:
- Per-account compute at fleet scale. Integration Account capabilities were backed by Functions apps at a per-account granularity, so "migrate the app" really meant "migrate a fleet of roughly 60,000 apps" across all Azure regions.
- A small number of shared platform workloads. Alongside the per-account apps, a few shared platform workloads served cross-cutting functionality. There's only one set of them per region, so they had a different blast radius and needed their own, more cautious targeting (Section 7).
- Stateless compute. The durable Integration Account data (agreements, maps, schemas) lives in storage and did not move; those stateful assets stayed exactly where they were. We were migrating compute, not customer data, which kept the parity surface smaller than it first appears.
- A runtime decaying under time pressure. One risk was easy to underrate: the longer we stayed on the deprecated v1/v2 platform, with no security-patch path and no dependency updates, the more its reliability could erode underneath us. "Do nothing" wasn't a safe holding pattern.
The bar we set ourselves was to keep customer-visible disruption as close to invisible as we could make it, and to prove we were hitting that bar with data rather than assert it.
4. The confidence ladder
Instead of a cutover, we built a progression. Each rung had to hold before we climbed to the next:
- Provision alongside. Stand up a new v4 app next to the classic app; change nothing customers touch.
- Validate parity. Make the v4 app's behavior equivalent to classic (Section 5).
- Shadow real traffic. Send real traffic to v4 in parallel while classic still serves the workflow; compare results (Section 6).
- Analyze mismatches. Triage every divergence, fix forward, and raise the bar until any residual divergence is understood and explainable.
- Rate-gated cutover. Move a controlled, hash-based fraction of traffic to v4, region by region (Section 7).
- Stay rollback-ready. Keep a fast, fine-grained way back at every step (Section 7).
- Stop, then delete, carefully. Retire classic only after guardrails prove it's safe (Section 9).
Figure 2. The confidence ladder. Each rung had to hold before we climbed. Rollback stayed available throughout, and retirement (stop first, delete much later) came last.
This ladder is the spine of everything below.
5. Parity foundation: what had to be equivalent
Because the durable Integration Account data stayed in storage, parity wasn't about copying state. It was about one question: does the v4 code package produce the same observable output as the classic one, for the same input? Every action an app could perform had to behave equivalently between the in-process host (v1/v2) and the isolated worker (v4).
The subtle part is that the host-model change can shift behavior you never explicitly wrote. The clearest example is JSON serialization. The in-process and isolated worlds start from different serializer defaults, so the same object can render differently: timestamp formats, enum representations, and the like. For customer output to stay reliably equivalent, byte-for-byte where customers observed exact serialized values, those defaults couldn't be left to chance. (The concrete bug this caused, and how shadow caught it, is in Section 6.)
So "validate parity" meant pinning the things that change observable behavior (runtime and dependency versions, host and app configuration, and serialization settings) and then letting real traffic be the judge of whether we'd actually achieved it.
6. Shadow mode: the showpiece
This is the heart of the migration. For an app in shadow mode, every eligible call still went to the classic runtime, and the workflow received the classic result. In parallel, we sent the same call to the v4 app and compared the two responses, emitting a match/mismatch signal. The workflow was never given the v4 result while we were still proving it. A v4 shadow error or slow response stayed off the customer path entirely and could only ever influence our telemetry and gating, never the result the workflow received.
For each shadowed request we compared the status code, the action or outcome code, and the output payload, the signals that define a correct result. We also captured request timings, but as a health and regression signal, not as part of the match/mismatch verdict.
Why was this safe to do at all? The obvious objection to running every request twice is side effects: wouldn't the parallel call double real-world actions? For this workload, no. Integration Account actions are essentially pure transforms, meaning transforms and metadata lookups that read the durable Integration Account data and produce an output, without writing to any external customer system. The shadow call computes a result and we compare it; nothing is being written anywhere, so there was nothing to double. That property is what made full-traffic shadowing safe by construction.
Because it was safe, we didn't sample. When an app was in shadow mode we compared 100% of its live traffic, every eligible request. "Eligible" simply excluded apps opted out at the resource level (for example, unsupported special-case apps on an exclusion list); everything else was compared.
We started strict, requiring a full JSON match, then relaxed to a normalized comparison so that formatting noise wasn't miscounted as a real difference. But some customer workloads are legitimately nondeterministic: they embed random values or timestamps, so classic itself won't return an identical response twice. To separate "v4 is wrong" from "this operation is inherently nondeterministic," we added a deterministic-match mode. For a shadowed request, we also called classic a second time. The logic is simple:
- If v4 disagrees with classic's first response and classic's second response also disagrees with its first, the operation is nondeterministic, and the v4 "mismatch" is discounted.
- If v4 disagrees but the two classic calls agree with each other, that's a real v4 divergence to fix.
Crucially, that extra classic call ran off the customer path, and only on the handful of apps already showing mismatches, so it added no customer-visible latency and was never an always-on tax. Shadowing itself (classic plus one v4 call) covered 100% of eligible traffic; the extra classic call existed only for those already-mismatching apps.
When the two disagreed, the migration team triaged the divergence. Cases sorted, informally, into a few recognizable buckets: serialization and format differences, customer nondeterminism, transient infrastructure blips, and genuine v4 logic bugs. We didn't formalize that into a tracked taxonomy. We also watched the asymmetric cases, where classic succeeds but the v4 shadow call errors, or the reverse. We read these as ordinary infrastructure noise rather than v4 defects because of their shape: they stayed at very low rates, correlated in time with known regional blips, and showed no app-specific concentration, so they didn't materially gate an app's readiness.
There was no single magic number to advance, but the judgment wasn't loose either. Before an app's traffic could move we wanted a sustained, near-100% normalized match with no unexplained systemic divergence, its asymmetric failures reviewed and understood, and stable v4 error and latency signals through a bake period. Within those guardrails the final call was operational judgment, and we also weighed known quirks like cold starts occasionally causing a failure or a retry alongside the overall match rate.
The best evidence the method earned its cost is a real bug it surfaced before any customer was affected: the serializer-defaults divergence from Section 5. In-process v1 and isolated v4 serialized certain values differently out of the box, so v4's output didn't match classic's for things like timestamps and enums. Shadow flagged the mismatch; the fix was to explicitly pin v4's JSON serialization options to match the classic runtime, making customer output identical again. Without shadow, that's exactly the kind of quiet difference that ships and generates confused support tickets weeks later.
Running both worlds isn't free. Shadowing roughly doubles compute and the read-only dependency calls that go with it. We absorbed that by standing up the v4 apps on their own dedicated App Service Plans, isolating their capacity so the shared downstream services stayed healthy. Relative to each region's overall load the added traffic wasn't a large delta, and because it ran on isolated, monitored capacity behind the same switches, we could cut it instantly if it ever strained anything downstream. Per app, shadow typically ran for weeks before cutover, and longer in the early stages while we built confidence that the pattern held across the fleet.
Figure 3. Runtime routing. Solid lines are the customer path; dashed lines run off it. In shadow, the workflow always gets the classic result while v4 runs in parallel and only emits a match/mismatch signal.
7. Controlling blast radius, rolling out, and watching it land
Parity earns the right to move traffic; control is what lets you move it safely.
Traffic moved by a hash-based exposure rate keyed on a stable resource or tenant identifier. The hash is deterministic: for a stable identifier, the same account maps to the same bucket, so an app stays put on classic or v4 and never flip-flops between them as the rate ramps or as we roll region to region. That stability matters, because it prevents apps from oscillating across deployments.
Layered control came from four switches: a master switch, a mode selector (classic, shadow, or v4-only), the percentage-based rate, and resource-level exclusion controls. The exclusion controls were the emergency brake: name an affected resource and it snaps back to classic, with no deployment required, effective within a few minutes.
We advanced through safe-deployment rings, and we deliberately started where a mistake would hurt least: early canary and test regions, then internal-to-Microsoft accounts, and only then the broader regions. Advancing a ring was a monitor-and-judge decision against rich telemetry. We checked the prior ring's rates, confirmed the target region was healthy, let earlier regions bake, and watched how they behaved under customer-traffic spikes before moving on.
| Stage / ring | Advance gate | Signals watched | Halt / rollback action |
|---|---|---|---|
| Canary + test regions | Sustained near-100% normalized match; no open systemic mismatch; stable v4 error/latency through bake | Match rate, v4 error rate, latency, customer-reported incidents | Exclude resource(s) to classic in minutes |
| Internal-to-Microsoft accounts | Prior ring's gates still green at higher scenario variety; asymmetric failures reviewed and understood | same | Hold ring or roll back via config |
| Broad regions (safe-deployment rings) | Prior-ring gates held, target region healthy, bake time elapsed | same plus behavior under traffic spikes | Exclusion controls or lower the rate |
When a signal turned, whether a dip in match rate, a rise in v4-only errors, a latency regression, or a customer-reported incident, we could hold the ring or roll back. The rollback is just a configuration change that takes effect in a few minutes, and it wasn't theoretical: we exercised it for real several times (most notably for a high-load edge case, and on some internal apps), which means it was battle-tested under real load, not just on paper.
The migration team owned the rollout day to day, driving ring advancement, watching the dashboards, and making the advance, halt, and rollback calls, with standard on-call and incident processes layered on top, so a customer-reported incident paged the on-call engineer through the normal channels.
None of this is safe if you can't see it. We invested heavily in dashboards, metrics, and counters, and fed mismatch and health signals (non-reversible telemetry, not payload logging) into health and mismatch dashboards. The loop was simple and strict: signal, triage, gate decision. A batch advanced only when the telemetry said it had earned it.
Figure 4. The observability loop. Signal, triage, then gate decision: a batch advanced only when telemetry (parity and v4 health) said it had earned it; otherwise it held, rolled back, or was fixed forward.
8. Keeping shadow traffic safe
Shadowing real customer traffic is a privacy responsibility, not just an engineering technique, so we designed the comparison to need no customer data at rest. The comparison happened in memory, in the moment: we never logged PII or customer payloads. What we emitted were signals, not data, meaning match/mismatch outcomes and metrics designed not to reveal or reconstruct the underlying content. Because comparisons were transient and no payload was ever logged or retained, the design avoided creating a new customer-data-at-rest surface, so there's no store of sensitive traces to govern. The platform's standard runtime access controls and operational safeguards still applied to the live path itself. And as covered in Section 6, because the actions are pure transforms, the shadow path produced no duplicate customer-visible side effects.
That privacy choice came with a real tradeoff. Because we never captured payloads, a handful of mismatches simply couldn't be reproduced from the signals alone. For those, we reached out to the affected customers directly and, with their help, collected example maps and inputs that let us reproduce the divergence in a controlled setting, root-cause it, and fix it. Trading a little investigative convenience for keeping customer data out of our telemetry was the right call, and customers were glad to help us close real bugs.
9. Retiring the old world: stop, then delete, with guardrails
A migration isn't done when traffic moves; it's done when the old system is safely gone. We deliberately separated stopping from deleting, with a long, reversible gap between them.
- Confirm, then disable. After an app cut over, we first confirmed via telemetry that no traffic was still reaching the classic app, then disabled it through a per-region configuration. Disabled but not deleted means quickly restorable if anything surfaced.
- Observe. We waited again after disabling, watching for anything to break, before considering deletion at all.
- Delete last, and only in bulk confidence. We only began deleting classic apps after most apps across most regions had been disabled for a sustained period. Deletion is the final, irreversible stage, gated on a long, successful disabled window rather than done eagerly per app.
Retirement safety, in one line: traffic fully moved, no remaining callers, disabled (reversible), observation window elapsed, and only then delete.
10. Results, lessons, and what's next
By the numbers (rounded):
- Scale. Roughly 60,000 Functions apps migrated, across all Azure regions.
- Shadow volume. At peak, the fleet compared on the order of hundreds of millions of requests per week, with 100% of eligible traffic compared, not a sample.
- Parity arc. The match rate climbed in step-changes, each driven by a big, systemic fix: serialization pinning chief among them, alongside alignments in host and DI configuration and dependency versions. After those landed, it was mostly smooth, with only minor or nondeterministic residue.
- Cutover impact (fleet level). Cutover was intentionally uneventful in aggregate: aggregate latency and error-rate telemetry showed no material regression at the moment of switch, because shadow had already proven parity. That's a fleet-level signal, not a promise of zero individual impact; see the honest caveat below.
- Timeline. A little over two years: roughly year one achieving and confirming parity via shadow, and year two moving real traffic and then retiring the old apps.
An honest caveat, because "zero impact" would be too strong. Safe deployment and shadowing caught the overwhelming majority of issues before customers saw them, and most customers had to do nothing. But it wasn't perfectly invisible: a small subset of customers hit an edge case under high load and experienced brief, real impact before we rolled back and coordinated a minor adjustment. Rollback did its job; we used it several times early on, then rarely once the big fixes were in. The incident trend followed the same shape: a few incidents up front while the systemic fixes landed, then quiet.
Lessons worth keeping:
- A parity bar is only credible if it accounts for nondeterminism. The "call classic twice" trick was what let us tell a real v4 bug apart from a workload that simply never repeats itself, and enabling it selectively kept it from costing customers latency.
- The host-model change hides in serialization. The most important fix wasn't in business logic; it was pinning serializer settings so v4's output matched the classic runtime exactly.
- No side effects, no duplication risk. Because the actions only read data and compute, with nothing written to any external system, we could shadow all traffic safely. That single property is the biggest reason this approach was even available to us.
- Signals beat payloads. Comparing in memory and emitting only non-reversible signals gave us the confidence of full-traffic comparison without the liability of storing customer data.
- Separate "stop" from "delete." The long, reversible disabled window turned the scariest, irreversible step into a calm one.
What's next: customer traffic is on v4, and we have already deleted the large majority of the classic v1 apps. What remains is the final cleanup, removing the last disabled classic apps in the trailing regions and finishing the more cautious retirement of the shared platform workloads. The migration itself is done; the remaining deletion is deliberately unhurried.
The broader takeaway is that shadow traffic plus a progressive, reversible rollout is a repeatable pattern for replacing a runtime under live, multi-tenant load. Customers still carry real risk when the infrastructure under them changes; what this pattern shifts is control. The team that owns the migration absorbs the complexity and holds the levers to prove parity, gate the rollout, and reverse it, so customers don't have to manage the change or its mitigation.