This is a 360-degree view into how to bring big data from Microsoft 365 and related services into Azure Synapse Analytics with Microsoft Graph Data Connect. This works internally and externally with your network of customers, suppliers and partners. Harness information that goes above and beyond what’s possible compared to other reporting options in Microsoft 365, and get available solution templates to quickly get started.
Principal Program Manager Lead from the Microsoft Graph team, Jeremy Thake, joins Jeremy Chapman to show how to keep people and your organization connected with Microsoft Graph Data Connect and Azure Synapse.
Use your M365 data for clear internal and external collaboration.
See how to connect M365 data into Azure Synapse from Microsoft Graph Data Connect.
Get new hires up-to-speed
Build a customer 360 solution with Azure Synapse linked by Microsoft Graph Data Connect.
Load, transform, and analyze M365 data at scale.
Your steps to link Microsoft Graph Data Connect with Azure Synapse Analytics.
Watch our video here.
QUICK LINKS:
00:00 — Introduction
00:53 — Internal and external collaboration
02:24 — Behind Microsoft Graph and Azure Synapse
03:55 — Build a customer 360 solution
06:29 — Data environment behind Power BI dashboard
07:40 — How to bring Graph data into Synapse
09:14 — How to build out a B2B example
10:33 — Wrap up
Link References:
Set it up with a step-by-step guide at https://aka.ms/MGDCsetup
Unfamiliar with Microsoft Mechanics?
As Microsoft’s official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft.
- Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries
- Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog
- Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/website
- To get the newest tech for IT in your inbox, subscribe to our newsletter: https://www.getrevue.co/profile/msftmechanics
Keep getting this insider knowledge, join us on social:
- Follow us on Twitter: https://twitter.com/MSFTMechanics
- Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/
- Enjoy us on Instagram: https://www.instagram.com/msftmechanics/
- Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics
Video Transcript:
- Coming up, we’ll take a look at updates to Microsoft Graph Data Connect that lets you bring in big data from Microsoft 365 and related services to Azure Synapse Analytics for a 360-degree view of how your organization works both internally and externally with your network of customers, suppliers, and partners. Now we’re going to show you how harnessing this information goes above and beyond what’s possible compared to other reporting options in Microsoft 365, as well as available solution templates that you can use today to quickly get started. So joining us today is Jeremy Thake, from the Microsoft Graph Team, to show us what’s possible. Welcome to the show.
- Thanks. Happy to be here.
- And it’s really great to have you on, especially now where there’s so much focus on figuring out this new way of working that we’re all experiencing today, where we work more hybrid and digitally than ever. So solving for how people in the organization, you know, are connected both internally and externally is a huge challenge. And that’s where the Microsoft Graph Data Connect with Azure Synapse comes in, right? So can you explain what we’re solving for and kind of give us an example of it?
- Right, and there is so much intelligence tied to everyday activities just in the day-to-day running of an organization. We’re talking about an exponential amount of big data. This can be really useful to understand how communication works inside and outside of your organization. For example, you might have a strategic partnership with an external organization and want to try and understand the strength of that relationship. Your CRM tool will capture a subset of that data. But if you have multiple teams working together internally across the departments, there are often so many connection points across your organization that you can’t see and analyze. And so with existing systems that are reliant on people inputting information, it’s hard to build an accurate and true picture. Yet the data on activities with specific domains including interactions like emails, meetings, Teams chats, file shares, and other collaboration activities, key data that would help you form an accurate view all exists within the Microsoft Graph. It’s just that getting to it has always been a challenge. You could try to connect to it via the Microsoft Graph API, but, again, this is big data, so the throughput needed to harness this data at a meaningful scale will be limited. All this changes with Microsoft Graph Data Connect using Azure Synapse Analytics, which now makes it seamless to access the data and perform large scale analytics and machine learning, all with the privacy and security built in.
- All right, so now you can kind of unlock all of that valuable insight that’s held into that data and tap into that. So for those that are new to the Microsoft Graph and Azure Synapse, let me explain a bit more further in terms of what those are. So the Microsoft Graph is the gateway to data and intelligence that’s in Microsoft 365 and really provides this unified programmability model and API to access your Graph data across Microsoft 365, Windows, Azure Active Directory, and Intune, all while enabling user privacy by default. Now it can securely tap into the data that you own and manage from calendaring, file sharing, email, directory services, collaboration, and more. Then Microsoft Graph Data Connect actually lets you bring all of that big data into your Azure Storage and directly connect to it via Azure Synapse, Microsoft’s limitless analytics platform. Now this all provides a single managed environment that your data engineers, your data analysts, and also your data scientists can use to analyze and integrate insights into their apps or reporting solutions. In fact, it’s how Microsoft apps themselves, like the Microsoft Viva suite and also Microsoft 365 apps and much of the reporting that you’re going to find in the Microsoft 365 admin center work as well to bring you insights unique to your organization.
- Right, and we give you access the same secure access to the breadth of data in Microsoft Graph versus the out-of-the-box experience. We give you access to all the organizational data for your analysis. And because this is can be a massive amount of data to load, transform, and analyze, Azure Synapse is a big data analytics service that’s able to bring everything together for you at scale.
- So can you walk us through then what’s possible when you bring the Microsoft Graph together with Azure Synapse?
- Absolutely, and to be honest, there are limitless possibilities. One of the most powerful use cases we’ve seen organizations start with is building a customer 360 solution. Here you’re looking at a Power BI dashboard we’ve built as a 360-degree view of our sales accounts. The nice thing about this is that the data isn’t reliant on the seller logging information into a CRM system. And if there are multiple connection points, you have the visibility into activities a seller wasn’t directly involved in. You can see the number of high priority accounts, account of the ones that need some attention, and how many are good in standing. Below that, you’ll see we’ve developed a connectivity score that rates how well-connected we are across our teams to these accounts. It’s looking at things like number of activities, touchpoints with influential stakeholders from leadership’s teams, transactions and more. We can see what’s going well and what’s not going so well from the score standpoint. Below that, you can see the gap since our last activities and immediately you’ll see some correlation. For example, Fabrikam residences here has the longest gap, and they also have the lowest connectivity score. And this is an interactive report. So I can filter on the transaction size using this slider. And these now represent our biggest accounts. If I dig into the details here for this account, Munson’s pickles, you’ll also see we’ve mapped our sellers, account contacts, and executive contacts. The event tracker shows activities by month. And on the right we see the top relationships, interactions, and whether we’ve met with our leadership in the last six months, and that’s looking red and it shows the next best action to re-engage with this account. So as you can see, we need to do some work to improve our connection with our leadership teams.
- And this is really powerful information. But besides kind of reminding the seller, they need to get back into contact with that account. What else can it do?
- Yeah, absolutely. There are so many different ways to use this, whether you’re a sales manager or a seller. This is really helpful as a seller on our team who has left their role and now we can give this dashboard to the new account manager. Now instead of all the ramp up time it takes to meet a bunch of people and maybe go down some dead ends, they’ll see exactly who’s active. They can see the best people to get in touch with at Munson’s as well as executive contacts and other sellers. This will really help the new seller get up to speed much faster, especially if the person who left wasn’t diligent at logging all these interactions with Munson’s team in the CRM system.
- Right, and this is something that I can really relate to having managed CRM systems in the past. They only show a small snapshot of all the activities that are going on between a couple of accounts, but can you show us the data that’s kind of behind all of this?
- Yeah, absolutely. All of this data behind this is running in Azure Synapse Analytics and we are pulling raw information from the Microsoft Graph. I’m in the Azure Synapse Workspace here. And first let me show you what’s in the data tab. I’ll go straight to the primary data from the Microsoft Graph. Most of what you’re seeing here has been flattened into parquet files so that we can reason over it. Next, I’ll show you what’s in the Integrate tab with our pipelines for events, which are looking across counter activities and messages. And for this scenario, we are only bringing the sender, receiver, and the date properties of the messages. We don’t need to have the message content. You can configure what properties you bring in to meet your data privacy needs, something you cannot do with the Microsoft Graph APIs. Then in the develop tab, you’ll see all of our notebooks to process the data from the Graph. These are all Python-based running in a Spark pool and they are all part of that template. And while we do provide the Power BI reporting as part of the template, bringing this data together or unlocks an unlimited number of additional possibilities like you might get for merging things like a ticketing system or an auditing system.
- And these are all really great examples, especially if you’re a developer or an analyst. But how did you bring all of that Graph data into Synapse?
- Yeah, that’s a really good question because this isn’t the type of data that is structured or semi-structured data that can easily be ingested into Azure Synapse with its built-in data pipelines. This is where Microsoft Graph Data Connect comes in. Included with Microsoft Graph Data Connect are native controls to precisely filter the data you need, with granular consent as mentioned, down to the property level within each data entity. Then it uses Azure Synapse to copy Microsoft 365 data to your Azure Storage at configurable intervals. Because when you’re accessing such large amounts of data from multiple users at once, to avoid throttling we break up the data requests into several concurrent API calls and continuously copy the data in batches. This all happens at a rate designed to avoid throttling. And once the data starts to roll in, you can use Azure controls to manage permissions for users and app access as well as generate audit reports for all activities and operations against the data. You can find a detailed step-by-step guide at aka.ms/MGDCsetup. The steps set everything up are pretty straightforward. First, you need to enable the Microsoft Graph Data Connect service in the Microsoft 365 Admin Center. In the Entra portal, you’ll create a Microsoft Azure Active Directory app registration that will be used as a security principle to run the data extraction process. And then you’ll need a place to store your data using an Azure Storage account. Then you’ll set up your Azure Synapse Workspace that you’ll use for ETL and analytics against the Graph data in your Azure Storage. And once everything is running, data will flow in and updates are automatically triggered on the schedule that you can configure.
- Okay, so now that you’ve got the foundation in place, how would you build out a B2B solution like we saw before?
- Right. It’s pretty straightforward. We’ve made it really easy for you and have built an Azure Synapse template for you. You can find it in a workspace gallery to quickly get started. You’ll walk through a few basic steps to get this set up and point to it your own Azure Storage account where the graph data is coming in. And once everything is running in addition to the Power BI reporting and visualizations I showed earlier, you can combine it with other data sources and pipe this same data into your own custom applications or create directly within the Azure Synapse workspace. In fact, let me show you a few more examples with some raw data queries. Here, I have a SQL script that will perform a basic count of message data, or emails, in our storage account. So I’ll go ahead and run it, and you’ll see this is a large dataset with 2.1 million messages.
- Yeah, ’cause that sounds like my inbox.
- Sounds like mine too. Now I’ll do one more query to show the most recent hundred transactions with our Munson account from earlier. And it’s instantly enumerated everything here. Again, you would use queries like this in an app connected to an Azure Synapse backend. So the Microsoft Graph Data Connect together with Azure Synapse unlocks a lot of valuable data that you can use to improve collaboration and help drive that new business insight.
- And this really unlocks a whole new dimension of collaboration insights as you build out analytics and solutions. So for anyone who is watching right now looking to harness Microsoft Graph with Azure Synapse, what do you recommend?
- Sure. So everything we’ve shown today is available now. We have a complete guided tutorial in Microsoft Learn to get everything set up. Check out aka.ms/MGDCsetup and there are Synapse templates you can use to get quickly started. I showed the customer 360 template today as part of a customer relationship scenario. In addition, we also have people productivity, business process, and a security and compliance analytics scenarios to meet your business needs.
- Thanks so much, Jeremy, for joining us today and showing us some of the possibilities when you connect Microsoft Graph together with Azure Synapse. And be sure to keep checking back to Microsoft Mechanics for all the latest tech updates. And subscribe to our channel if you haven’t already, and as always, thank you for watching.
