Ingest Custom Data into Azure Log Analytics via API Using PowerShell
Data can be sent to Log Analytics in various ways, depending on the type of data you want to ingest. For example, you can send custom logs or system information from your computers to Log Analytics, or any data relevant to your business needs. Ways of Ingesting Data into Log Analytics Azure Log Analytics supports several data ingestion methods, including: 1. Agent-Based Ingestion: Azure Monitor Agent (AMA) or Log Analytics Agent for system logs and metrics. 2. Diagnostic Settings: Directly stream Azure resource diagnostic logs to Log Analytics. 3. Data Collection Rules (DCR): Configure rules to filter, transform, and send data to Log Analytics. 4. Custom Logs: Upload custom log files for ingestion into specific Log Analytics tables. 5. Azure Monitor Data Sources: Integration with Azure services like App Insights or Azure Security Center. 6. Logs Ingestion API: Send custom data programmatically into Log Analytics via APIs. In the example below, I demonstrate how to ingest Entra Application details into Azure Log Analytics, including key information such as Application Name, Application ID, and Creation Date, by leveraging the Log Ingestion API with PowerShell for seamless and efficient data integration. Example: Ingesting Entra Application Data into Log Analytics Custom Table Using APIs When not using the Data Collection Endpoint, you must leverage DCR endpoints to connect via APIs. However, note that DCR endpoints will only appear if you avoid creating tables through the Azure Portal. Instead, use ARM templates for this setup. Below is a step-by-step guide for ingesting Entra Application data into a Log Analytics table. Step1: Create an Entra Application having (Application.Read.All) API Permissions Step 2: Define and Create a Custom Table in Log Analytics First, create a table in Log Analytics and define the required columns. For this example, we aim to ingest the following details: Application Name Application ID Application Created Date Publisher Domain Identifier URIs PowerShell Script to Create the Table: <# Disclaimer Test this thoroughly before using it in production. The sample code is for illustration only and provided "as is," without warranties or support. #> $tableParams = @' { "properties": { "schema": { "name": "EntraApplications_CL", "columns": [ { "name": "TimeGenerated", "type": "datetime", "description": "The time at which the data was generated" }, { "name": "AppName", "type": "string", "description": "Entra Application Name" }, { "name": "AppId", "type": "string", "description": "Entra Application ID" }, { "name": "AppCreatedDate", "type": "datetime", "description": "Application Creation Date" }, { "name": "PublisherDomain", "type": "string", "description": "Publisher Domain Details" }, { "name": "IdentifierUris", "type": "string", "description": "Entra Application Identifier URIs" } ] } } } '@ Invoke-AzRestMethod -Path "/subscriptions/<subscription-id>/resourcegroups/<resource-group>/providers/microsoft.operationalinsights/workspaces/<workspace-name>/tables/EntraApplications_CL?api-version=2022-10-01" -Method PUT -payload $tableParams This script creates the EntraApplications_CL table with the specified schema. Step 3: Create a Data Collection Rule (DCR) ARM Template for Data Collection Rule: Change the defaultValue in below template: Like Location, DCR Rule Name, ResourceID { "$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "dataCollectionRuleName": { "type": "string", "defaultValue": "AADDCR" }, "location": { "type": "string", "defaultValue": "Central India" }, "workspaceResourceId": { "type": "string", "defaultValue": "/subscriptions/<subscription-id>/resourcegroups/<resource-group>/providers/microsoft.operationalinsights/workspaces/<workspace-name>" } }, "resources": [ { "type": "Microsoft.Insights/dataCollectionRules", "name": "[parameters('dataCollectionRuleName')]", "location": "[parameters('location')]", "apiVersion": "2023-03-11", "properties": { "streamDeclarations": { "Custom-EntraApplicationsRAW": { "columns": [ { "name": "TimeGenerated", "type": "datetime" }, { "name": "AppName", "type": "string" }, { "name": "AppId", "type": "string" }, { "name": "AppCreatedDate", "type": "datetime" }, { "name": "PublisherDomain", "type": "string" }, { "name": "IdentifierUris", "type": "string" } ] } }, "destinations": { "logAnalytics": [ { "workspaceResourceId": "[parameters('workspaceResourceId')]", "name": "signinlogs-la" } ] }, "dataFlows": [ { "streams": ["Custom-EntraApplicationsRAW"], "destinations": ["signinlogs-la"], "transformKql": "source | project TimeGenerated, AppName, AppId, AppCreatedDate, PublisherDomain, IdentifierUris", "outputStream": "Custom-EntraApplications_CL" } ] } } ] } Deploy the template using the Azure Portal's Custom Template Deployment. Step 4: Review DCR Click on the Json View you've the log ingestion endpoint available. We will be using it in the script to ingest the data. Step 5: Assign Permissions to Your Application Ensure the Entra application has the necessary Monitoring Metrics Publisher permissions. As of now there is no data in the table Step 6: Use the Script to Ingest Data To ingest data, you’ll need two tokens: Graph API Token: For retrieving application data. Logs Ingestion API Token: For sending data to Log Analytics. PowerShell Script for Data Ingestion: <# Disclaimer Test this thoroughly before using it in production. The sample code is for illustration only and provided "as is," without warranties or support. #> $DcrURI = "https://<dcr-endpoint>.ingest.monitor.azure.com" # DCR Endpoint, highlighted in DCR review section $tenantId = "<tenant-id>" $appId = "<app-id>" $appSecret = "<app-secret>" $Table = "Custom-EntraApplicationsRAW" $DcrImmutableId = "<dcr-immutable-id>" # As higligted in DCR review section # Function to Retrieve Access Token (for Microsoft Graph) function Get-AccessToken { Write-Host "Obtaining Azure AD bearer token for Microsoft Graph..." -ForegroundColor Cyan $tokenUri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token" $body = @{ client_id = $appId client_secret = $appSecret scope = "https://graph.microsoft.com/.default" grant_type = "client_credentials" } $headers = @{ "Content-Type" = "application/x-www-form-urlencoded" } try { $response = Invoke-RestMethod -Uri $tokenUri -Method Post -Body $body -Headers $headers return $response.access_token } catch { Write-Host "Failed to retrieve bearer token for Microsoft Graph: $_" -ForegroundColor Red exit } } function Get-AllApplications { $applications = @() $url = "https://graph.microsoft.com/v1.0/applications" $headers = @{ "Authorization" = "Bearer $graphAccessToken" "Content-Type" = "application/json" } do { try { # Fetch the current batch of applications $response = Invoke-RestMethod -Uri $url -Method Get -Headers $headers if ($response.value) { # Add applications to the collection $applications += $response.value } # Update the URL to the next page if it exists $url = $response.'@odata.nextLink' } catch { Write-Host "Failed to fetch applications: $_" -ForegroundColor Red exit } } while ($url) # Return the complete list of applications return $applications } Write-Host "Total applications retrieved: $($applications.Count)" -ForegroundColor Green # Transform Applications Data for Logs Ingestion API Write-Host "Transforming application data for ingestion..." -ForegroundColor Cyan $payload = @() foreach ($app in $applications) { $payload += @{ TimeGenerated = ([datetime]::UtcNow).ToString("o") AppName = $app.displayName AppId = $app.appId AppCreatedDate = $app.createdDateTime PublisherDomain = $app.publisherDomain IdentifierUris = ($app.identifierUris -join ",") } } # Exit if no data to ingest if ($payload.Count -eq 0) { Write-Host "No applications found. Exiting..." -ForegroundColor Yellow exit } $logIngestionScope = [System.Web.HttpUtility]::UrlEncode("https://monitor.azure.com//.default") $logIngestionBody = "client_id=$appId&scope=$logIngestionScope&client_secret=$appSecret&grant_type=client_credentials" $logIngestionHeaders = @{ "Content-Type" = "application/x-www-form-urlencoded" } $tokenUri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token" try { $logIngestionToken = (Invoke-RestMethod -Uri $tokenUri -Method Post -Body $logIngestionBody -Headers $logIngestionHeaders).access_token Write-Host "Bearer token for Logs Ingestion API retrieved successfully." -ForegroundColor Green } catch { Write-Host "Failed to retrieve bearer token for Logs Ingestion API: $_" -ForegroundColor Red exit } # Send Data to Log Analytics (using the correct token and body) Write-Host "Sending data to Log Analytics via Logs Ingestion API..." -ForegroundColor Cyan $ingestionBody = $payload | ConvertTo-Json # Make sure this is the correct data $ingestionHeaders = @{ "Authorization" = "Bearer $logIngestionToken" "Content-Type" = "application/json" } $ingestionUri = "$DcrURI/dataCollectionRules/$DcrImmutableId/streams/$Table"+"?api-version=2023-01-01" try { $uploadResponse = Invoke-RestMethod -Uri $ingestionUri -Method Post -Body $ingestionBody -Headers $ingestionHeaders Write-Host "Data successfully pushed to Log Analytics custom table: $Table" -ForegroundColor Green Write-Output $uploadResponse } catch { Write-Host "Failed to push data to Log Analytics: $_" -ForegroundColor Red } You can now view the data in your Log Analytics workspace. Conclusion This process helps you easily send custom data to Azure Log Analytics. By using Data Collection Rules and the Logs Ingestion API, you can manage and monitor your data efficiently.
