How to leverage Azure Monitor to meet functional and non-functional requirements - No.4 Storage/Disk

This article is a part of series articles for Azure Monitor. Please refer to How to leverage Azure Monitor to meet functional and non-functional requirements - No.1 overview first before reading this post. This post dives deeply for Disk category among monitoring categories as highlighted blue. 

To monitor disk capacity is mandatory topics for monitoring requirements because disk space depletion can cause serious errors. It's also essential to monitor size of specific file or folder, where is frequently used for example file upload or  logging. We can check Disk and Folder/File several ways with Azure Monitor, thus this post describes how to leverage Azure Monitor for the purpose.

4.1 Disk

There are some several ways to monitor Disk as follows. We will deeply dive there methods.

• Disk free space usage with VM Insights
• Disk free space usage with Perf 

4.1.1 Disk free space usage with VM Insights

Here is an example query to check disk usage with InsightsMetrics. This feature allows us to check the disk usage per drive and mountpoint.

InsightsMetrics
| where Computer == "gcp-instance-01"
| where Namespace == "LogicalDisk"
| where Name == "FreeSpacePercentage"
| extend d=parse_json(Tags)
| project TimeGenerated, Computer, Namespace, Name, Val, d["vm.azm.ms/mountId"]

Right screenshot visualizes free space percentage of the mount id within 24 hours. vm.azm.ms/mountId indicates mountpoints on operation systems, thus we can monitor all devices across OS disk, Data disk, and temporary disk. Note that the mountpoint name is different from Disk resource name on Azure Portal. We can use Val for alert threshold, thus create a query to trigger notification when value of Val is less than 10% or others.

We can create a simple graph with project function setting up x-axis as time series and y-axis as expected value for example free space percentage here. We can also disable the alert rule and pause it in this case. 

4.1.2 Disk free space usage with Perf

Open Legacy agent management menu on your Log Analytics workspace. A screenshot below shows Linux performance counters configuration and added "% Free Space" and "Free Megabytes". It initially takes a couple of hours to reflect this configuration on Log Analytics workspace.

As next, run a Kusto query as follows to check free disk space. Update each parameter for your use. 

Perf
| where Computer == "gcp-instance-01"
| where ObjectName == "Logical Disk"
| where CounterName  == "% Free Space"
| where InstanceName == "/" or InstanceName == "/mountdisk"
| order by TimeGenerated
| project TimeGenerated, Computer, ObjectName, CounterName, InstanceName, CounterValue, CounterPath

We can check all mountpoints across OS disk, Data disk, and temporary disk. You can create a simple graph and disable an alert as same with VM Insights.

Finally, here is check result of Disk monitoring. 

Type	category	Goal and outcome	Result
1	monitoring	Azure Monitor can satisfy functional requirements	OK
2		Azure Monitor can setup short granularity for detections	1 min
3		Azure Monitor can setup thresholds detections	OK
4		Azure Monitor can setup retry detections	OK
5		Azure Monitor can suspend and resume for checking threshold	OK
6		Azure Monitor can send a mail for detection results	OK
7	statistics	Azure Monitor can retrieve workspace logs with specific duration	OK
8		Azure Monitor can visualize statistic data	OK
9	automation	Azure Monitor can have primary action based on alert rules	OK
10		Azure Monitor can send validation results	OK

 

4.2 Folder/File

There are some several ways to monitor Folder/File as follows. We will deeply dive there methods.

• Leverage EventLog and Syslog
• File Integrity Monitoring of Microsoft Defender for Cloud 

4.2.1 Leverage EventLog and Syslog

Please note the Log Analytics agents won’t be supported as of August 31, 2024. Plan to migrate to Azure Monitor Agent prior to this date.

This method leverages Syslog and a script file. Configure Linux Syslog on Legacy agent management menu on your Log Analytics workspace. A screenshot below shows Linux Syslog configuration and added local0 facility. You can use an other facility depending on your script file. 

Then, create a script to check folder or file usage and configure crontab to periodically run the script. This example script put its logs to /var/log/message and an agent sends the data to Log Analytics workspace, thus we can retrieve the log with query and enable an alert.

#!/bin/sh

# folder size check
ETC_HTTPD_DIR=`du -sk /etc/httpd | awk '{ print $1 }'`
if [ ${ETC_HTTPD_DIR} -gt 600 ]
then
        logger -sp local0.err "/etc/httpd dir size over (<700k)"
fi

# file size check
HTTPD_CONF_FILE=`ls -l /etc/httpd/conf/httpd.conf | awk '{ print $5 }'`
if [ ${HTTPD_CONF_FILE} -gt 6224 ]
then
        logger -sp local0.err "/etc/httpd/conf/httpd.conf size over (<7224byte)"
fi

This script does not use any Azure specific commands, thus you can tailor the script for your use cases. This is not built-in feature but more flexible than following method.

 

4.2.2 File Integrity Monitoring of Microsoft Defender for Cloud

File Integrity Monitoring(FIM) can monitor changes of file or directory. FIM can track OS configuration, registry, setting file and setting folder across all VMs associated to Log Analytics workspace. The setting takes a time to be reflected on Azure Portal at first time, thus I recommend to wait a day. Open FIM from Microsoft Defender for Cloud and setup configuration.

Here is an example screenshot for FIM. This screenshot shows that FIM can monitor OS configuration, registry, setting file and setting folder across environments for example Azure, AWS, and GCP - you can definitely extend this to on-premise.

We can retrieve this information with Kusto query. Here is an example.

ConfigurationChange
| where Computer == "gcp-instance-01"
| where ConfigChangeType in("Files", "Registry")
| order by TimeGenerated
| render table

This feature is perfect to track changes, but you should consider to use 4.2.1 Leverage EventLog and Syslog if you have to check size of a file or a folder, which does not have mountpoint itself.

 

Finally, here is check result of File/Folder monitoring. 

Type	category	Goal and outcome	Result
1	monitoring	Azure Monitor can satisfy functional requirements	OK
2		Azure Monitor can setup short granularity for detections	1 min
3		Azure Monitor can setup thresholds detections	OK
4		Azure Monitor can setup retry detections	OK
5		Azure Monitor can suspend and resume for checking threshold	OK
6		Azure Monitor can send a mail for detection results	OK
7	statistics	Azure Monitor can retrieve workspace logs with specific duration	OK
8		Azure Monitor can visualize statistic data	OK
9	automation	Azure Monitor can have primary action based on alert rules	OK
10		Azure Monitor can send validation results	OK