The “Why”

Do you manage data in a complex multi-cloud environment? Are you struggling with data silos, evolving regulations, and the pressure to maintain control and compliance across on-prem and multiple clouds? Do you ever wish an intelligent assistant could help shoulder the load of data governance? If so, I can relate. Let me tell you a story that might sound familiar.

Meet Mark (pictured above). He is a data governance officer at Contoso (a fictional but very representative enterprise).  Mark’s day job is ensuring data governance and compliance across his company’s vast hybrid cloud estate – think around ~2 million data assets sprawled across 12+ datacenters on-premises and in different public clouds. Regulatory requirements are constantly shifting. Customer data is increasingly sensitive. Each department and region has its own way of doing things. Mark is fighting an uphill battle with data silos and disconnected cloud operations. He bounces between a patchwork of tools – spreadsheets, cloud consoles, governance portals – trying to answer basic questions:

Where is our data?

Who’s using it?

Are we in compliance?

Armed with an old desk calculator and a pile of paper-based reports (a perfect 1990s backdrop), he is dealing with the data around him that has exploded in volume and complexity.

What if Mark had a single pane of glass. The glass that reflects and acts. It reflects your governance state and enforces compliance – a self-hydrating pane of glass accompanied by a conversational AI.

And he’s not alone. We’re all living in a data overload era. Every day, organizations generate and ingest more information than ever before. Transistors and mainframes gave way to the internet boom of the ’90s, then an explosion of mobile devices in the 2000s, social media in the 2010s, and now widespread cloud computing – all funneling data into our systems at an exponential rate. On top of that, a new wave of AI and conversational interfaces has arrived here in the mid-2020s, making data more accessible but also increasing expectations for real-time insight. It’s no wonder modern IT leaders feel overwhelmed.

But these challenges are also opportunities. The way I see it, the incredible growth of data and cloud capabilities means we have a chance to reimagine data governance. The fact that I’m writing about this right now is no coincidence. My customers are looking to resolve problems in this space. In my conversations with them, I hear the same needs: We want better governance, more visibility, streamlined oversight… and cherry on top, we want it in an “agentic” fashion. In other words, they want to delegate the grunt work to the platform toolset augmented by AI, so they can focus on higher-value tasks.

The “What”

That vision – agentic data governance with hybrid cloud flexibility – became the driver for this work. This is a modular solution, and you have these building block style components (cloud services, governance tools, AI agents), which you can snap them together into an intended solution. Think of it as a jumpstart kit for continuous data governance across multiple clouds, with autonomous (“agentic”) assistance baked in that you can leverage and build upon. It’s not the final, productized solution – more a vision of what’s possible.

Contoso’s Requirements

These are the high-level requirements from Contoso:

• Data governance across clouds under one roof
• A single pane of glass dashboard consolidating reporting on the 5 governance domains:

o   Visibility on data residency and lineage

o   PII (Personally Identifiable Information) must run on a CC (Confidential Compute)

o   Security software (Defender) compliance

o   Resource tagging compliance (foundational for a good governance posture)

o   OS updates compliance

• Ability to enforce compliance in an agentic manner with a human in the loop
• Agentic enforcement of compliance pertaining to residency and confidential compute

Solution – The breakdown

The solution is comprised of 8 modules addressing these requirements. These solution modules are:

1. Foundational (Landing zones, Data Sources, Operational setup, Policies, etc.)
2. Dashboard Hydration + Agentic Reporting – Residency Compliance
3. Dashboard Hydration + Agentic Reporting – Confidential Compute for PII Compliance
4. Dashboard Hydration + Agentic Reporting – MS Defender Compliance
5. Dashboard Hydration + Agentic Reporting – Resource Tag Compliance
6. Dashboard Hydration + Agentic Reporting – OS Updates/Patch Compliance
7. Enforce Compliance via Copilot Agent - Residency Compliance
8. Enforce Compliance via Copilot Agent – CC PII Compliance

Solution – The architecture view

These are the main technical components that make up the solution architecture:

• Data sources of all shapes and sizes on the left, governed by the native Azure or the Arc plane.
• Additional Azure services across the bottom layer for the foundational governance posture
• Microsoft Purview, in the top middle, as the unified data governance platform
• Microsoft Fabric, in the bottom middle, as the end-to-end ingestion and analytics platform
• Microsoft Power Platform, on the right, as the low code/no code business flow and the copilot agent experience

Solution – The end user view

So how does Mark see this solution as a data governance officer? He doesn’t see all the intricacies of the solution integration and the logic execution. He sees two things:

1. A Power BI dashboard running on Microsoft Fabric with

• • A compliance dashboard with an overall score in each of the five compliance domains alongside scores for each of the data products across these domains
  • Additional reporting views for more granular reporting
  • Fabric-based pipeline that hydrates the underlying semantic models from various sources to keep the reports fresh and current

2. A Copilot agent (in Teams) for both:

• • Reporting on all compliance domains
  • Enforcing in-scope compliance across selected domains

The agent takes care of it - queries Fabric’s semantic model, calls Azure Function endpoints, updates Purview glossary terms, applies Azure tags, and sends Teams notifications.

The “How” – Residency Compliance

Let’s pick a few modules to walk through how these solution modules work together to give a cohesive agentic governance experience to Mark.

It’s Monday morning, and Mark logs into the Contoso governance portal with a cup of coffee in hand. Instead of a dozen browser tabs, he has two main tools opened: the Data Governance Dashboard and the Contoso Governance Copilot agent.

To address some inquiries that came as an assigned action to him, he interacted with the agent. During this interaction, not only did he validate if there were any residency missing in the unified data governance platform (Purview), but he was also able to address a mismatch between Purview and Azure resource, based on the designed principles. Here is the snippet of the chat:

 

Now, under the hood, several components have worked on behalf of the agent in performing this governance checking and applying the necessary course of action:

Even before Mark's conversation with the agent, an ongoing hydration process keeps the Fabric Power BI dashboard up to date.

Dashboard Hydration + Agentic Reporting – Residency Compliance

 

 

1. A Fabric notebook runs the residency scorecard code block through a pipeline.
2. It reads two Lakehouse tables containing latest residency information from Purview and the approved region list
3. Then, the notebook gets a Microsoft Entra bearer token
4. Once acquired, the notebook then calls an Azure Function endpoint
5. This endpoint, then searches for the Azure resources associated with the data products in Purview using an Azure resource tag.
6. The notebook then compares the declared Purview residency with the approved region list and the associated resource’s region
7. The notebook then calculates the final 0 / 25 / 50 / 75 / 100 residency compliance score and a reason. For example: A data product without an associated Azure resource gets a 0, while a data product whose residency in Purview is an approved region by Contoso, and also matches with the associated Azure resource, gets a 100.
8. It then writes the results to the relevant residency compliance Lakehouse tables
9. The dedicated compliance table then feeds to the semantic model for reporting
10. The compliance Power BI dashboard is hydrated

Enforce Compliance via Copilot Agent - Residency Compliance

 With the dashboard data regularly updated, the agent follows this logic, the updated reporting data, and the actions at its disposal, during the earlier conversation with Mark :

1. Mark initiates the conversation with the agent
2. The agent calls a Power Automate flow
3. This flow retrieves Purview’s residency information stored in the Fabric semantic model
4. 5, 6, 7 and 8. When Mark asks to investigate further on a data product, the agent carries the conversation using a topic, which then leverages a flow, which uses a Power Automate custom connector to access an Azure Function endpoint. This endpoint then retrieves latest glossary (residency) information about the data product in question, from Purview, and provides a preview back to the user
5. 10, 11, 12, and 13. If the update criteria are met, and if there is no conflict, and with Mark’s blessings, the topic then calls another flow to access the Functions Purview Update endpoint, and make the glossary (residency) update in Purview for that data product

The “How” – Confidential Compute for PII Compliance

Dashboard Hydration + Agentic Reporting – Confidential Compute for PII Compliance

The following snippet shows how Mark addresses the compliance risk with a critical data product (application), S/4 HANA, and performed the necessary compliance actions, such as tagging the associated resources and notifying the data product owners via Teams channel.

The following diagram shows the under-the-hood hydration flow for confidential compute compliance:

Enforce Compliance via Copilot Agent – CC PII Compliance

Finally, the diagram below shows how Mark’s conversation flows through the main solution components:

Outcome

Stepping back, what did we accomplish for Mark and Contoso? We turned an onslaught of governance challenges into an opportunity to modernize how data is managed. This gave Mark:

• Centralized Visibility into data assets across the landscape through Purview and a unified dashboard
• Proactive compliance enabled with automated checks - controlled with Purview exports and Fabric pipeline schedules
• And compliance enforcement using an agent
• Hybrid Cloud Consistency. By using Azure Arc and a foundational data plane management setup
• Reduced Operational overhead with agentic reporting and compliance

Though the solution is comprised of wide variety of components/services, it is built from standard building blocks and is relatively simple to implement. In total, the solution combined around a dozen Azure services and over 40 distinct components (from Purview catalogs to data pipelines, to custom functions and flows). You can choose to implement some or all the compliance domains. Or, better yet, build upon and create new domains and pave new paths.

Wrap-up

I believe many enterprises could take a similar journey. If you’re facing these issues, consider this an invitation to think differently about data governance. Start with the pieces you already have – your own building blocks of cloud services and data – and imagine what you could build. Chances are that a lot of the heavy lifting can be orchestrated with today’s technology. And with the rise of AI copilots, the dream of agentic data governance – where your policies are continuously enforced by smart agents – is no longer science fiction. It’s here, right now, waiting for you to take it for a spin.

 Next steps

• Watch the video narrative on SAP on Azure YouTube channel: 
• Build it with the GitHub Repository: https://github.com/moazmirza/data-sov-and-hyb-cloud
• Comments/questions: Here, or @ LinkedIn /moazmirza

Solution Selfies