Migrating Splunk Logs to Azure Application Insights on VMs

Phase 0 – Understanding the Current Environment

Before touching any code or agents, map out the existing architecture.

Current setup:

• Applications write logs via log4net → local log files on VMs
• Splunk Universal Forwarders read logs → send to Splunk Indexers
• Splunk dashboards, alerts, and integrations with PagerDuty/ServiceNow
• Logic Monitor monitors VMs for CPU, memory, disk, uptime (not logs)

Developer tasks:

1. Document all log file locations:

grep -r "log4net" /path/to/apps

   2. List all Splunk forwarder configurations:

cat /opt/splunkforwarder/etc/system/local/inputs.conf

1. Record all dashboards, alerts, and their dependencies.
2. Note any compliance requirements (PHI-sensitive logs, retention policy).

Note: Capture Splunk dashboard screenshot with example alerts and log search query to reference during migration.

Phase 1 – Prepare Azure Environment

Step 1.1 – Create Application Insights and Log Analytics Workspace

Developer Actions:

1. Go to Azure Portal → Create Resource → Application Insights
2. Fill in:
   • Name: AppName-Insights
   • Resource Group: Observability-RG
   • Region: closest to your VMs
   • Application Type: .NET, Java, or Other
3. Click Review + Create → Create
4. Navigate to Log Analytics Workspace
5. Create workspace for centralizing logs and metrics
   • Note Workspace ID and Primary Key for agents

Step 1.2 – Retrieve Instrumentation Key / Connection String

1. Open Application Insights resource → Properties
2. Copy Instrumentation Key or Connection String

Note: You will use this for SDK integration later and optional log appender configuration.

Phase 2 – File-Based Ingestion Migration 

Since your apps already write logs to files, we can replace Splunk forwarders with Azure Monitor Agent (AMA).

Step 2.1 – Install Azure Monitor Agent (AMA) on VMs

Windows VM:

# PowerShell script to install AMA Install-Agent.ps1 -WorkspaceId "<WorkspaceID>" -WorkspaceKey "<WorkspaceKey>"

Linux VM:

sudo ./install-ama.sh --workspace-id <WorkspaceID> --workspace-key <WorkspaceKey>

Verify installation:

# Windows Get-Service -Name "AzureMonitorAgent" # Linux sudo systemctl status azuremonitoragent

Note: Capture AMA service status to confirm it is running.

Step 2.2 – Configure AMA to Read Log Files

1. Navigate to Azure Portal → Log Analytics Workspace → Agents Configuration
2. Add Data Collection Rule:
   • Select Custom Logs
   • Add path for log4net log files on the VM
   • Map fields like timestamp, log level, message
3. Assign the rule to your VM(s)

Step 2.3 – Validate Log Ingestion

1. Navigate to Application Insights → Logs
2. Run a simple query:

traces | order by timestamp desc | limit 50

  3. Compare with Splunk logs:

Metric	Splunk Count	Azure Count	Status
Errors last 1h	105	103

Phase 3 – Alert Migration

Step 3.1 – Map Splunk Alerts to Azure Monitor Alerts

Example Mapping:

Splunk Alert	Azure Equivalent	Frequency
Error threshold > 10	Log query alert (`traces	where severityLevel == "Error"

Developer Steps:

1. Navigate to Azure Monitor → Alerts → + New alert rule
2. Select Resource → Application Insights → Condition → Custom log search
3. Configure Action Group:
   • PagerDuty, Email, ServiceNow
4. Set Alert Frequency and Severity

Screenshot Tip: Capture alert creation screen with query and action group.

Phase 4 – Optional SDK-Based Migration (Full Observability)

For deep insights, tracing, and metrics.

Step 4.1 – .NET Applications

1. Install SDK:

dotnet add package Microsoft.ApplicationInsights.AspNetCore dotnet add package Microsoft.ApplicationInsights.Log4NetAppender

2. Configure in Program.cs:

builder.Services.AddApplicationInsightsTelemetry();

3. Integrate log4net with Application Insights:

<appender name="aiAppender" type="Microsoft.ApplicationInsights.Log4NetAppender.ApplicationInsightsAppender"/> <root> <level value="INFO"/> <appender-ref ref="aiAppender"/> </root>

4. Add Instrumentation Key in appSettings.json:

{ "ApplicationInsights": { "InstrumentationKey": "YOUR_KEY_HERE" } }

Step 4.2 – Java Applications

1. Add Maven dependency:

<dependency> <groupId>com.microsoft.azure</groupId> <artifactId>applicationinsights-web</artifactId> <version>3.4.15</version> </dependency>

2. Configure ApplicationInsights.xml with Instrumentation Key
3. Enable automatic telemetry: requests, exceptions, dependencies

Step 4.3 – Python Applications

from opencensus.ext.azure.log_exporter import AzureLogHandler import logging logger = logging.getLogger(__name__) logger.addHandler(AzureLogHandler(connection_string='InstrumentationKey=YOUR_KEY')) logger.setLevel(logging.INFO)

Phase 5 – Dual Logging Validation

• Keep Splunk forwarders running temporarily
• Compare logs and alerts:

services.Configure<TelemetryConfiguration>(config => { config.DefaultTelemetrySink.TelemetryProcessorChainBuilder .UseSampling(20.0); // 20% sample });