How to leverage Azure Monitor to meet functional/non-functional requirements-No.3 Compute/Inside OS

[daisami]

This article is a part of series articles for Azure Monitor. Please refer to How to leverage Azure Monitor to meet functional and non-functional requirements - No.1 overview first before reading this post. This post dives deeply for Compute/Inside category among monitoring categories as highlighted blue. 

Article No	monitoring category	monitoring target	Note
2	Compute	Reboot	monitor reboot frequency
		CPU	monitor CPU usage
		Memory	monitor memory usage
3	Compute/Inside OS	Log file	monitor event log and syslog
		Process	monitor available process
4	Storage/Disk	Disk	monitor disk usage
		folder/file	monitor folder usage and file size
5	Endpoint/IPv4 address	response/service	monitor specific address and port
	Web site	Scenario	monitor web scenario
6	Network	Connectivity	monitor vNiC and VNET peering
		Firewall	monitor Azure Firewall rule usage
7	Backup	Backup	monitor backup status
	Azure Resources	Resource health	monitor resource availability

3.1.1 Log file monitoring: EventLog and Syslog

Configure eventlog of Windows Server or Syslog of Linux on Legacy agent management menu on your Log Analytics workspace. A screenshot below shows Linux Syslog configuration and added facilities.

Please note the Log Analytics agents won’t be supported as of August 31, 2024. Plan to migrate to Azure Monitor Agent prior to this date.

Syslog data will be stored to LogManagement on Log Analytics workspace after a short wait. Log data on /var/log/rsyslog will be stored to SyslogMessage column as each line if you configure syslog facility.

Here is Kusto to retrieve logs which SyslogMessage value contains "journal" string.

Syslog
| where Facility == "syslog"
| where SyslogMessage contains "journal"
| order by TimeGenerated

In this way, you can extract specific logs from Syslog table which stores /var/log/rsyslog data. Create a script which executes logger -p local0.critical "Critical message hoge" and configure crontab for the script to execute periodically, then we can find specific messages among stored logs and call an alert rule. We can also disable the alert rule and pause it in this case.

3.1.2 Log file monitoring: custom log

There are various types of logs which are generated by variety of applications. Azure Monitor allows you to collect such logs with Custom Logs feature. Refer to Collect text logs with the Log Analytics agent in Azure Monitor for the detail.

In this post, we will setup Custom Logs for a Java application using logback. Here is part of logback-spring.xml for this use case. This Java application will generate a log file on /var/log/demo2/applog.log. 

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <springProperty name="LOG_DIR_PATH" source="log.dir.path" defaultValue="/var/log/demo2/"></springProperty>
    <appender name="ROLLING" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOG_DIR_PATH}/applog.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>${LOG_DIR}/demo2-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <charset>UTF-8</charset>
            <pattern>%date %-4relative [%thread] %-5level %logger{35} - %msg %n</pattern>
        </encoder>
    </appender>

Setup Custom Logs configuration from Custom log menu on your Log Analytics workspace as follows.  

Run your Java application and let logback generates its log, then you can retrieve your logs with Kusto query.

RawData column stores full text of collected entry and you can parse the text data by leveraging Kusto query if you need. Refer to Parse text data in Azure Monitor logs for the detail.

Note: The log must have specific format for each timestamp to use Azure Monitor. Refer to Collect text logs with the Log Analytics agent in Azure Monitor for the detail.

Finally, here is check result of Log file monitoring. 

Type	category	Goal and outcome	Result
1	monitoring	Azure Monitor can satisfy functional requirements	OK
2		Azure Monitor can setup short granularity for detections	1 min
3		Azure Monitor can setup thresholds detections	OK
4		Azure Monitor can setup retry detections	OK
5		Azure Monitor can suspend and resume for checking threshold	OK
6		Azure Monitor can send a mail for detection results	OK
7	statistics	Azure Monitor can retrieve workspace logs with specific duration	OK
8		Azure Monitor can visualize statistic data	OK
9	automation	Azure Monitor can have primary action based on alert rules	OK
10		Azure Monitor can send validation results	OK

3.2.1 Process monitoring: running processes inside OS

Open Legacy agent management menu on your Log Analytics workspace. A screenshot below shows Linux Syslog configuration and added local0 facility. This is an example for local0, thus add other facilities if you need.

Create and setup simple script which output logs when the process number is less than expected number. Enable crontab to periodically run the script. As an example, this script outputs "apache2 process down! please check" message to /var/log/syslog when httpd process number is less than 3.

#!/bin/sh
HTTPCOUNT=`ps -ef | grep -i httpd | grep -vc grep`
if [ ${HTTPCOUNT} -lt 3 ]
then
	logger -p local0.crit "apache2 process down! please check"
fi

We can see that the syslog data is stored on Log Analytics workspace. This message says that httpd processes is less than 3, thus we will configure an alert rule to send an email notification when this line is output. We can also disable the alert rule and pause it in this case.

Syslog
| where Computer == "gcp-instance-01"
| where Facility == "local0"
| where SyslogMessage contains "apache2"

Optional - 3.2.2 Process monitoring: Leverage VM Insights

We can identify with VM Insights tables that a process is running even though we can't check the number of the process. This example checks whether httpd process runs or not with VMProcess and InsightsMetrics tables. Log data is sent to Log Analytics workspace per minute.

let ApacheProcessId = toscalar (
    VMProcess
    | where Computer startswith "gcp-instance-01"
    | where ExecutablePath contains "httpd"
    | top 1 by TimeGenerated
    | project Process
);
InsightsMetrics
| where Name == "Heartbeat" and Origin == "vm.azm.ms/map"
| where Computer startswith "gcp-instance-01"
| where isnotempty(ApacheProcessId) and Tags contains ApacheProcessId
| where TimeGenerated > ago(5m)

Optional - 3.2.3 Process monitoring: Port monitoring with VM Insights

VMBoundPort table allows us to check status of a port. 

We can check whether the port is open or not, but we can't identify its hang up status. We can leverage Connection Monitor of Network Watcher in the case. Refer to Create a monitor in Connection Monitor by using the Azure portal for the detail.

The VM Insights Map allows you to check status of each process and its ports, and outbound network communication. This feature can adopt to outside Microsoft Azure for example this screenshot is a VM on GCP.

However, note that main purpose of this feature is not for sending notification when a specific process is down, but visualize outbound network communication from the VM. Overview of VM insights 

Finally, here is check result of Process monitoring. 

Type	category	Goal and outcome	Result
1	monitoring	Azure Monitor can satisfy functional requirements	OK*1
2		Azure Monitor can setup short granularity for detections	1 min
3		Azure Monitor can setup thresholds detections	OK
4		Azure Monitor can setup retry detections	OK
5		Azure Monitor can suspend and resume for checking threshold	OK
6		Azure Monitor can send a mail for detection results	OK
7	statistics	Azure Monitor can retrieve workspace logs with specific duration	OK
8		Azure Monitor can visualize statistic data	OK
9	automation	Azure Monitor can have primary action based on alert rules	OK
10		Azure Monitor can send validation results	OK

*1 Process monitoring is sometimes required to check a process per seconds. You might be needed to consider other solution including 3rd party tools.