As organizations increasingly adopt multi cloud and hybrid cloud strategies, one recurring challenge is how to make data available across public clouds without duplicating datasets or incurring high data egress costs. Traditional approaches often rely on exporting, copying, or syncing data between platforms—introducing latency, governance overhead, and unnecessary cost. Databricks Delta Sharing addresses this challenge by enabling secure, real time data sharing across cloud boundaries without requiring data replication.
What Is Delta Sharing?
Delta sharing is an open protocol for secure data sharing that allows organizations to share live data stored in Delta Lake with external consumers—across cloud providers such as Azure, AWS, and Google Cloud—while keeping the data in its original location. Consumers access the same up‑to‑date data without the provider having to copy or move it to another cloud.
Why Delta Sharing Matters in Multi‑Cloud Architectures
In multi‑cloud environments, data sharing typically leads to increased storage duplication and cross‑cloud data transfer charges. Delta Sharing changes this model by exposing data in place, enabling access rather than replication.
Key architectural benefits include:
- Cross‑cloud data availability without replication
Data producers share tables directly from their existing data lake, eliminating the need for extract‑and‑load pipelines. - Reduced egress costs compared to copy‑based approaches
Since data is not duplicated by default, Delta Sharing avoids large-scale data movement that typically drives cloud egress charges. Egress occurs only when data is queried across clouds. - Secure and governed access
Access is managed using fine‑grained permissions, auditability, and centralized governance through Unity Catalog, ensuring shared data remains secure and compliant. - Vendor‑neutral and open
Consumers do not need to run Databricks. Delta Sharing supports multiple clients such as Spark, Pandas, and BI tools, making it suitable for heterogeneous analytics ecosystems.
Delta Sharing vs. Data Replication
A common misconception is that cross‑cloud analytics always require data replication. With Delta Sharing:
- Live access is used for shared analytics and reporting use cases
- Replication (deep clone) becomes an explicit design choice only when local copy or offline processing is required—rather than a default requirement
This gives architects the flexibility to optimize for cost, performance, or isolation depending on the workload, instead of paying replication costs upfront.
Figure: Databricks Delta Sharing Architecture:
Delta Sharing enables secure, real‑time data access across public clouds without replicating data. Consumers query data in place through a secure control plane, ensuring governance, reduced operational overhead, and optimized egress costs.
This diagram visually represents Databricks Delta Sharing architecture with a clear left‑to‑right flow that works well for blogs and technical readers:
- Left – Data Producer (Azure Databricks + Unity Catalog + Delta Lake Storage)
- Center – Delta Sharing Service (secure control plane, metadata & access control)
- Right – Data Consumers (AWS / GCP / Spark / BI tools / Pandas)
- Bottom flow – Read‑only, on‑demand access with no data replication and egress only on read
Architecture Diagram Explanation: Databricks Delta Sharing
Overview
The diagram illustrates how Databricks Delta Sharing enables secure, real‑time data access across public clouds (Azure, AWS, GCP) without replicating data, helping reduce operational overhead and data‑egress costs.
At a high level, data remains in the producer’s cloud storage, while consumers in other clouds access it securely on demand using the Delta Sharing protocol
1️⃣ Data Producer (Azure Databricks)
- The Data Provider hosts curated datasets in a Delta Lake (for example, on Azure Data Lake Storage Gen2).
- Data is governed using Unity Catalog, which controls:
- Which tables are shared
- Who can access them
- What operations are allowed (read‑only access)
No data is copied or exported during sharing—datasets remain in the producer’s storage account.
2️⃣ Delta Sharing Service (Control Plane)
- Delta Sharing acts as the secure control layer, not a data storage layer.
- It manages:
- Authentication and authorization of consumers
- Metadata exchange (schemas, table versions)
- Auditing and access tracking
The control plane ensures that only authorized consumers can discover and query shared datasets.
3️⃣ Secure Network Access
- Consumers connect to the producer’s storage using HTTPS-based secure endpoints.
- Depending on architecture:
- Access may be public endpoint–based
- Or secured further using private networking patterns (VPN, private endpoints, or controlled IP ranges)
All access occurs under the governance policies defined by the data producer.
4️⃣ Data Consumers (AWS / GCP / Azure)
- Consumers may run:
- Databricks on another cloud
- Spark clusters
- Python (Pandas)
- BI and analytics tools that support Delta Sharing
Consumers query the data in place, directly from the producer’s Delta tables
Importantly, consumers do not need to copy or store the data locally unless they explicitly choose to.
5️⃣ Data Flow (Read‑Only, On‑Demand)
- When a consumer runs a query:
- Metadata is resolved via Delta Sharing
- Data blocks are read directly from the producer’s cloud storage
- Results are returned to the consumer
This “access‑instead‑of‑replicate” model avoids continuous data synchronization pipelines and minimizes unnecessary data movement.
6️⃣ Cost Optimization and Egress Considerations
- No default data replication means:
- No extra storage costs
- No background sync jobs
- Data egress charges occur only when data is actually read across clouds, not upfront or continuously.
Compared to traditional copy‑based sharing, this significantly reduces overall egress exposure for many analytics workloads
7️⃣ Optional: Local Replication (Explicit Choice)
- If required, consumers can perform a deep clone to bring a local copy into their own cloud.
- This is an explicit architectural decision, used only when:
- Low‑latency local access is required
- Data isolation is mandatory
Replication is optional — not a prerequisite for sharing.
Common Use Cases
Delta Sharing is particularly effective for:
- Sharing curated datasets across cloud platforms
- Enabling partner or third‑party analytics without data duplication
- Supporting centralized data platforms while allowing decentralized consumption
- Reducing operational overhead in cross‑cloud data ecosystems
Key Takeaway
Databricks Delta Sharing enables organizations to provide secure, real‑time cross‑cloud data access without data duplication—helping reduce operational complexity and egress costs in multi‑cloud architectures.
Microsoft