The amount of information we create has grown exponentially, and more often is distributed across multiple sources, making finding the right information, at the right time, increasingly difficult. Large enterprises typically use 15-20 disparate systems to store their information- including popular industry standard applications as well as custom line of business applications.
Enterprises want to search across these data sources with minimal context switching. Microsoft Graph connectors help enterprises merge all these data sources with Microsoft 365 content, to surface the information across multiple endpoints using Microsoft Search. Microsoft Graph connectors help you seamlessly bring in data from all these applications through our catalog of connectors by following a few simple steps for configuration.
Why custom connectors?
Occasionally constraints like lack of ready-made solutions to connect this data may limit the ability to merge or store this data in the Microsoft cloud. For example, Microsoft provides built-in connectors for popular data sources like file share, Salesforce, ServiceNow. However custom data sources or line of business applications (homegrown or otherwise) may not have available connectors built by Microsoft.
Figure 2: Microsoft built connectors available for data source types shown in Grey; Custom/line-of-business data sources shown in Blue
Custom connectors can be built today using:
- Microsoft Graph Search & Indexing APIs: The APIs are generally available, and you can read more about it here: Build your first custom Microsoft Graph connector | Microsoft Docs
- Microsoft Graph connectors SDK: The SDK is now generally available and you can read more about it here: Build your first custom Microsoft Graph connector - Microsoft Graph | Microsoft Docs
Why Graph connectors SDK?
The Microsoft Graph connectors SDK enables you to build custom connectors for your line of business applications and allow your information to participate in Microsoft 365 experiences like Microsoft Search. The Microsoft Graph connectors SDK is designed to simplify building high-quality, efficient, and resilient connectors that access Microsoft Graph. The SDK enables you to create custom Graph connectors directly within the Visual Studio integrated development environment (IDE) or through your own project setup, guides you through the process and provides everything you need to build, debug, and deploy your custom connector.
The Graph connectors SDK provides 4 key benefits:
- Fast & efficient development- Write minimal connector code in 11 supported languages by providing the below solutions to get them started quickly. (https://grpc.io/docs/languages/)
|
Solution |
Developer experience |
|
C# template available in Visual Studio Marketplace |
|
|
gRPC contracts for languages other than C# available in GitHub |
|
- Flexibility of data sources- Cloud or on-prem data sources can be indexed by using Connector Platform hosted on any VM to index.
- Access to connector platform- Robust Connector Platform capabilities include full/incremental crawl support, crawl scheduling, crawl recovery, graph ingestion. Leverage the robust Microsoft Graph connectors platform which runs Microsoft’s built-in connectors
- Access to M365 Admin Center- Leverage Microsoft Admin Center to manage custom and Out of the Box connectors in one place. Use the “Search & intelligence” section of M365 Admin Center to monitor and manage your custom connectors along with Microsoft’s built-in connectors. All the dashboards for crawl statistics and error monitoring are available for your custom connectors as well.
Figure 3: Custom connector option on the M365 Admin Center
What is Graph connectors SDK?
The Microsoft Graph connectors SDK works with the following components:
Figure 4: Microsoft Graph connectors SDK components
- Your LoB Connector: Your custom connector code contains the logic to return the data source schema, authenticate with the data source, read the data from the source along with identities by implementing the connector interfaces which act as a bridge between your custom connector and Microsoft platform. Your custom connector hosts a gRPC server to which a gRPC client from the Microsoft platform communicates with via the defined connector interfaces.
Figure 5: Your LoB connector implementing methods with which the Microsoft platform communicates via the bridge.
You can get started with developing your custom connector using:
- C# Custom connector template: If you wish to develop your custom connector in C#, you can download a sample template from Vio (Extensions -> Manage Extensions -> Search for “GraphConnectorTemplate”) and get started quickly.
Figure 6: C# custom connector project template in Visual Studio
- gRPC contracts: You may also choose to develop your custom connector in the following languages (https://grpc.io/docs/languages/:(
- C++
- Dart
- Go
- Java
- Kotlin
- Node
- Objective-C
- PHP
- Python
- Ruby
To get started, you can refer to the connector interfaces shared in the form of contracts. These are gRPC ( https://grpc.io/ ) protocol buffer files that contain the contracts for interaction between the Graph connector platform and your custom connector code.
Given below is a sample of how a gRPC contract for managing authentication of data source, validating configurations, and getting the data source schema looks like. These methods need to be implemented by the connector so that the platform can communicate through these interfaces as required.
Figure 7: A sample gRPC protocol buffer file containing the contracts
- Microsoft Platform: The Microsoft Graph connector agent (Microsoft Graph connector agent | Microsoft Docs) performs all the platform tasks for managing your custom connector and integrating with Microsoft Graph. This a lightweight software that takes care of orchestration capabilities and coordinating between Microsoft 365 Admin Center and your custom connector code. The Microsoft Graph connector agent is built on the same robust Graph connector platform used for running Microsoft’s built-in connectors.
Figure 8: Microsoft Platform capabilities
The Graph connector agent comes with the following capabilities:
- Get data from connector: The agent brings the capability of getting data from your custom connector using the logic implemented by your connector to connect to the data source.
- M365 Admin Center integration: The agent brings the capability of managing, monitoring errors & statistics, and configuring your custom connectors through the M365 Admin Center.
- Crawl Scheduling & Management: The agent brings the capability of crawling the data source through full crawls and incremental crawls. Full crawls traverse (list based or graph based) through your entire data periodically at an interval defined by you. Incremental crawls are more frequent and crawls the data source from the previous timestamp/checkpoint at an interval defined by you.
Note: Graph-based data source traversal (eg: traversing through folder structures) capability is not available.
- Delete, difference and cycle detection: The agent is capable of detecting items deleted from your data source based on the items sent by the custom connector during periodic full crawl and the agent takes care of deleting the item from the Microsoft index as well. The agent has the functionality to detect only items that have changed since the last crawl by computing a hash of the item and comparing with the hash of the item seen previously. To make crawls faster, the agent sends the data to be indexed only when the item has changed. The agent also detects duplicate items that arise from data sources that have items linked by graph structure (like websites) and skips crawling these items.
- Identity mapping: The agent brings the capability of stamping the Access Control Lists (ACLs) on your data for enabling security trimming. You can enable security trimming based on AAD or ACLs from your data source.
Note: Search permissions based on access control from your data source is not available.
- Graph ingestion: The agent takes care of ingesting the data into graph after receiving it from your custom connector.
The Graph connector Agent also comes with an in-built SDK test utility. This includes pre-built test scenarios that you can use to test your custom connector code and communication with the Graph connector agent.
Figure 9: Test scenarios in Microsoft Graph connectors SDK Test Utility
How to get started with Graph connectors SDK?
You can get started with the Microsoft Graph connectors SDK and try out this new experience by following the documentation that provides a good overview of the SDK: Microsoft Graph connectors SDK overview - Microsoft Graph | Microsoft Learn.
You can also get started quickly with our sample connector documentation here: Build your first custom Microsoft Graph connector - Microsoft Graph | Microsoft Docs. We provide a sample connector which you can download from our GitHub repository (microsoftgraph/msgraph-connectors-sdk (github.com))and customize for your specific scenarios using Visual Studio IDE.
Figure 10: Sample graph connector project in Visual Studio
Once customized and deployed you'll be able to index your content from the data source into Microsoft Graph. When enabled, employees in your organization will be to search results from your custom data source alongside your Microsoft 365 information in Microsoft Search. This will also allow you to enable other content experiences from Microsoft Graph in the future.
If you need help with troubleshooting, you can raise an issue or start a discussion on relevant topics through our GitHub repository (microsoftgraph/msgraph-connectors-sdk (github.com)).