How one Microsoft team streamlines Viva Insights data upload and analysis

Microsoft’s HR Business Intelligence (HRBI) team has long used Viva Insights to enhance decision-making, productivity, and operational efficiency across the company. By adopting recently released tools for automating data upload in Viva Insights, Microsoft HRBI has reduced the time and manual effort required to use Viva Insights to uncover timely workforce trends, including the impact of Microsoft 365 Copilot adoption. 

This blog post describes their automation journey.  

Automated (API-based) data upload in Viva Insights 

In November 2024, Viva Insights launched in public preview API-based org data upload capabilities. This method is ideal for consolidating HR data that lives in multiple locations. It’s also a great option for scenarios when data transformation is required prior to upload. Learn more.

The case study that follows is based on that functionality. 

A summary of existing and planned Viva Insights data upload options can be found at the end of this post.

Microsoft’s HR Data Automation Journey 

Before the advent of automated connectors, the Microsoft HRBI team undertook a time-consuming, manual process for managing HR data. Each week and month, dedicated team members meticulously extracted, transformed, and loaded essential HR data, providing accuracy and compliance. This manual operation not only placed significant demands on time but also introduced potential delays and errors inherent in any human-dependent process. 

With the new API-based automation, Microsoft HRBI has substantially simplified the data upload process, reducing the manual time and coordination required while maintaining stringent privacy and compliance standards. This API-based import option allows customers to decide what data is sent to Viva Insights and how frequently it’s refreshed, offering Microsoft the flexibility and security essential for optimal performance. 

This automated method helps customers’ HR data stay current and accurate for reporting. With a strong logging strategy and proactive data issue resolution, Microsoft has fresh HR data without extra manual effort. 

Here’s a more detailed look at the team’s setup process: 

1. Set up Communication between Data Source and Viva Insights 

Microsoft set up a secure connection between the storage location and Viva Insights using Managed Identities. Other options include: 

• API Keys 

• Certificates 

2. HR Data Gathering and ETL Setup

Sources: 

• Microsoft’s HR data originated from multiple sources, including: Four SQL Server databases.  

• A mix of Azure-hosted and on-premises systems.  

Process:  

• Collaborated with HR business analysts to identify required data sources.  

• Used Azure Data Factory (ADF) to consolidate and manage diverse datasets.  

• Handled on-premises data through ADF’s integration runtimes and native connectors. 

3. Data Preparation

Format Requirements: Data must be in a compressed ZIP file containing:  

• data.csv: Tabular representation of HR data with one row per employee.  

• metadata.json: Defines dataset type, bootstrap status, and column mapping.  

Implementation: Used Azure Data Factory to:  

• Aggregate data into a single data.csv file stored in Azure Blob Storage.  

• Store the metadata.json file in a separate blob container.  

• Leveraged an Azure Function App to package the files into a ZIP and ensure API compatibility. 

4. Data Export via Viva Insights Push API 

Azure Function App Development:  

• Accessed file streams using BlobTrigger and BlobInput bindings.  

• Compressed files into a ZIP format with .NET’s System.IO.Compression.  

• Authenticated API requests via Azure’s DefaultAzureCredential.  

• Submitted the ZIP file as multipart form data to the Viva Insights Push API. 

Learn how to set up this automated data import method, which requires some coding to call the Viva Insights API.

Other automated data import methods in development 

In addition to the API-based method described above, we’re also working to roll out two other automated data ingestion methods, currently in private preview. One of these methods lets Viva Insights retrieve data directly from the customer’s storage location, while the other retrieves data from the customer’s human capital management system like Workday. Use this form to sign up for the preview. 

And, for a refresher on our existing data import methods apart from the newly automated options, refer to the table below.