How Azure Storage Mover turned a manually operated AzCopy fleet into a managed, petabyte-scale cloud migration.
Every wedding, every newborn session, every graduation portrait a professional photographer delivers has to live somewhere safe, fast, and affordable—often for years. For one leading professional-photography SaaS platform—which powers galleries, client proofing, and print sales for tens of thousands of professional photographers worldwide—“somewhere” had grown into roughly 550 million high-resolution image objects—about 1.1 PB—spread across three AWS regions.
The company had already modernized its application platform on Azure, running on Azure Kubernetes Service. The last and largest piece was the data itself, still in Amazon S3. This is the story of how the team moved that estate to Azure Blob Storage across three continents—and why, after standing up a hand-built migration fleet of 50 virtual machines running AzCopy, it finished the job faster, more safely, and with a fraction of the operational effort by switching to Azure Storage Mover.
By the numbers ~550M objects · ~1.1 PB scope · 3 AWS regions → 3 Azure regions · up to ~7.6 GB/s · zero data-integrity failures · Cold tier on landing
The challenge: petabytes, three regions, and every storage class
The image estate wasn’t uniform. It spanned the full range of S3 storage classes—from S3 Standard and Standard-IA through Glacier Instant Retrieval, Glacier Flexible Retrieval, and Glacier Deep Archive—with object sizes from tiny thumbnails to originals approaching 100 MB. The data lived in three AWS regions, each mapped to an Azure destination:
|
AWS source region |
Azure destination |
|
US East 2 (Ohio) |
East US 2 |
|
Europe (Ireland) |
North Europe |
|
Asia Pacific (Sydney) |
Australia Southeast |
Europe alone illustrates the scale and complexity. A single source bucket inventoried 52,650,507 objects / 242.2 TiB:
|
S3 storage class |
Objects |
Size |
|
STANDARD_IA |
40,587,546 |
182.7 TiB |
|
GLACIER (Flexible Retrieval) |
8,459,463 |
40.2 TiB |
|
GLACIER_IR (Instant Retrieval) |
3,165,828 |
15.5 TiB |
|
STANDARD |
368,186 |
3.4 TiB |
|
DEEP_ARCHIVE |
69,484 |
379.5 GiB |
|
Total |
52,650,507 |
242.2 TiB |
The customer set a high bar:
- Total fidelity. Every object had to arrive intact, validated with MD5 checksums against the AWS source.
- No disruption. Photographers and their clients upload and download around the clock; the migration couldn’t degrade their experience.
- Cost discipline. The company meticulously tracks API transaction counts and egress on both clouds. The approach had to minimize redundant operations and data-out charges.
- Cold on landing. Data had to arrive directly in the Azure Blob Cold tier, matching the cost profile of long-lived photo archives.
- Archival-aware. Frozen Glacier and Deep Archive objects can’t simply be copied; they had to be rehydrated at the source and handled separately.
Phase 1: the do-it-yourself AzCopy fleet—and what it cost in people and infrastructure
The team’s first approach was a high-performance, build-it-yourself pipeline based on AzCopy in server-to-server copy mode. It worked—but at this scale the operational weight was enormous.
Moving just the accessible (non-archived) data meant designing, building, and operating an entire fleet:
- 50 Azure virtual machines (Standard_D8ls_v6, NVMe, Gen2) deployed as a Virtual Machine Scale Set.
- A custom Ubuntu 24.04 image built with HashiCorp Packer, published through an Azure Compute Gallery with the right NVMe driver support.
- Terraform to stand up the VNet, subnet, public IPs, Key Vault (for AWS credentials), system-assigned Managed Identities, and RBAC role assignments on every target account.
- The dataset sharded into ~176 batches of ≤300,000 objects each, generated from AWS S3 Inventory CSVs by custom Python tooling.
Then the human part. Five architects each connected to 10 VMs, pulled a batch file, opened a tmux session, exported AWS and Azure credentials, validated connectivity, launched an AzCopy job (--check-md5=FailIfDifferent, --overwrite=false), and manually tracked every job in a shared Excel sheet—marking each batch In Progress, then Done, watching for failures and restarting by hand.
The fleet did move real data. The wave-tracking spreadsheet records the completed AzCopy effort for Europe’s accessible tiers:
|
AzCopy migration (Europe, accessible tiers) |
Value |
|
Finished AzCopy jobs |
135 |
|
Objects transferred |
35,367,514 |
|
Data transferred |
~163 TB |
|
Operators |
5 (each managing ~10 VMs) |
But the model didn’t scale gracefully. Every additional region meant more VMs to build, patch, and secure; more credentials to broker; more batches to shard and assign; and more human hours spent watching azcopy jobs show and updating a spreadsheet. Integrity checking, archival handling, restart logic, and progress tracking were all the team’s responsibility. For ~550 million objects across three regions, the infrastructure footprint and the human investment were simply overwhelming.
Phase 2: Azure Storage Mover—the same job, managed
In parallel, the team worked with the Azure Storage Mover product group, whose cloud-to-cloud capability moves data directly from AWS S3 to Azure Blob Storage through a multi-cloud S3 connector—with no migration VMs at all. Over the engagement, the product group rolled out the specific capabilities the customer’s environment demanded, turning a fleet-heavy operation into a managed one.
The contrast was night and day:
|
|
DIY AzCopy fleet |
Azure Storage Mover |
|
Compute to build & operate |
50 VMs (VMSS), custom Packer image, NVMe tuning |
None — fully managed, agentless data plane |
|
Credentials & security |
Per-fleet Key Vault, Managed Identity, RBAC wiring |
Managed by the service |
|
Work distribution |
~176 batches manually sharded & assigned |
Folder-aligned jobs, defined once |
|
Integrity |
Manual --check-md5 per job |
Content-MD5 computed & stored server-side, validated against AWS |
|
Archival data |
Manual filtering/exclusion |
Restore-state detection: non-restored Glacier/Deep Archive automatically skipped |
|
Tiering |
Set per command |
Direct Cold tier placement on ingest |
|
Restart / resume |
Manual, risk of duplicate PUTs & egress |
Delta-aware SYNC — only missing objects copied |
|
Progress tracking |
Excel sheet, 5 operators |
Job run status & counters in the service |
|
People required |
5 architects babysitting 50 VMs |
One engineer orchestrating many jobs |
A Storage Mover instance was deployed in East US 2 to minimize latency to the source buckets.
Why single-part upload to ~100 MB is the key to end-to-end MD5 validation MD5 only works as an integrity proof if both clouds describe the whole object with the same hash. In Amazon S3, an object’s ETag equals the MD5 of its contents only when the object was uploaded as a single part—a multipart upload instead yields a composite “MD5-of-MD5s” ETag (with a -N suffix) that can’t be compared to a plain MD5. Azure Blob behaves the same way: a blob carries a true whole-object Content-MD5 only when it’s written in one operation; if a large object is split into blocks, no single hash covers the entire blob. So to prove a migrated photo is byte-for-byte identical to its S3 source, each object must be written as a single part on both sides. Because the customer’s originals run up to ~100 MB—well above the typical 8 MB chunking threshold—Storage Mover’s ability to upload objects that large in a single part is exactly what enables a direct, 1:1 MD5 comparison against AWS, without re-downloading a petabyte of data just to re-checksum it.
Prove it first: an Australia proof of concept
Before committing the bulk of the estate, the team ran a controlled proof of concept against the Australia region, deliberately seeding it with the hard cases: hybrid S3 storage classes and objects larger than 8 MB, up to ~100 MB.
|
Capability validated |
Outcome |
|
Single-part upload for objects >8 MB to ~100 MB, with correct Content-MD5 |
Validated |
|
Automatic skip of non-restored Glacier / Deep Archive objects |
Validated (247 archival files correctly skipped in one job) |
|
Restore-state detection for archival objects |
Validated via job logs |
|
Target Cold tier enforcement |
Validated — 166/166 blobs confirmed on Cold tier |
Scale up: tens of jobs, massive parallelism, zero failures
With evidence in hand, the team scaled to the US region—the largest scope—segmented into 49 folder-aligned jobs, up to 10 running in parallel. A single mid-flight snapshot of four parallel jobs shows the throughput:
|
Job |
Files transferred |
Bytes transferred |
Throughput |
|
copy-pictures-44 |
2,788,632 |
13.26 TB |
2,131 MB/s |
|
copy-pictures-45 |
2,733,084 |
13.17 TB |
1,794 MB/s |
|
copy-pictures-46 |
2,751,199 |
13.29 TB |
1,841 MB/s |
|
copy-pictures-47 |
2,771,981 |
13.66 TB |
1,865 MB/s |
Those four jobs alone had moved ~53 TB, scanned ~249 TB, and were sustaining roughly 7.6 GB/s combined—with zero data-integrity failures. Across the full Storage Mover effort, the service handled on the order of 180 million items / ~977 TB of transfer activity (including delta re-runs) across the US, Australia, and rehydrated European archival data.
When end users briefly saw HTTP 5xx responses during one high-parallelism window, the team simply paused the offending jobs and resumed them later. Because re-runs are delta-aware, resuming cost almost nothing—already-copied files were detected and skipped, with no duplicate PUTs and no duplicate egress. No spreadsheet, no 50 VMs, no manual restart logic.
Making the product better: feedback that shipped
Working at genuine production scale, the team surfaced real issues that the Azure Storage Mover product group turned into improvements:
- Control-plane status accuracy. During the US migration, the Azure Resource Manager (ARM) view reported several jobs as Queued—with empty agent names and null counters—while the data-plane logs showed them actively running and transferring with healthy throughput. The team captured a clean reproduction with full diagnostics; the product group root-caused a divergence tied to a specific delete-then-rerun scenario and deployed a fix across all regions.
- New capabilities validated at scale. Cold-tier targeting, large-object single-part upload with MD5, and archival skip/restore-state detection were all exercised in a live, multi-region, multi-hundred-terabyte migration—exactly the validation that hardens a service.
As the product group put it during the engagement, the customer’s feedback was “very valuable in helping us improve the quality of the service.”
Results at a glance
- ~550 million objects / ~1.1 PB of photo data targeted for migration from 3 AWS regions to Azure Blob Storage.
- Phase 1 (DIY AzCopy fleet) required 50 VMs and 5 architects to move ~35 million objects / ~163 TB of Europe’s accessible data—an unsustainable operational load at full scale.
- Phase 2 (Azure Storage Mover) handled the bulk—~180 million items / ~977 TB of transfer activity across US, Australia, and rehydrated European archives—with no migration fleet at all.
- Sustained up to ~7.6 GB/s aggregate throughput, with zero data-integrity failures (MD5-validated against AWS).
- Cold tier on landing, and budget protected through delta-aware SYNC re-runs that avoid duplicate PUTs and egress.
- Multiple product improvements delivered back into Azure Storage Mover.
“[Customer quote to be confirmed — suggested themes: moving hundreds of millions of our photographers’ images to Azure with zero data loss and no disruption, and how a managed service let a small team finish what a 50-VM fleet started.]” — [Title], the customer
Lessons learned for your own S3-to-Blob migration
- Weigh the operational cost, not just transfer speed. A hand-built AzCopy fleet can move data fast, but at petabyte scale the VMs, credentials, sharding, tracking, and human babysitting dominate. A managed service can finish the same work with a fraction of the people and infrastructure.
- Start with a representative proof of concept. Pick your smallest region but your hardest data—mixed storage classes and your largest objects—and validate checksums and tier placement before you scale.
- Rehydrate archival tiers first. Glacier and Deep Archive objects must be restored at the source before they can move; plan that timeline and let Storage Mover skip what isn’t ready yet.
- Segment large scopes into many jobs. Folder-aligned jobs give you parallelism, granular restart, and clean progress tracking—without a spreadsheet.
- Lean on delta-aware re-runs. Treat stop/resume as a first-class operation—SYNC means resuming is cheap and safe.
- Land data in the right tier the first time. Targeting Cold on ingest avoids a second, costly re-tiering pass.
Get started
- Learn about Azure Storage Mover
- Read the cloud-to-cloud (AWS S3 to Azure Blob) migration guidance
- Explore Azure Blob Storage access tiers
- Modernizing the apps too? Ask your Microsoft team about the Azure Storage app-modernization Agent Skill for moving S3 (Boto3) applications to Azure.
Thanks to the customer team, the Azure Storage Mover product group for the feature rollout, and the CSU CSA Apps & Data team for their partnership on this migration.