SAP Applications and Microsoft Defender for Linux

Cameron_MSFT_SAP_PM · ‎Jan 11 2023

What is “Defender”?

This blog is about “Microsoft Defender for Endpoint” for Linux, hereafter referred to as MDE. The term “Defender” is used across multiple products and technologies.

An overview of Microsoft 365 Defender is illustrated here What is Microsoft 365 Defender? | Microsoft Learn

The typical audience for this blog is SAP Basis administrators and consultants. Enterprise Security is a specialist role and the activities described in this blog should be planned in conjunction with the Security Administrators. The objective of this blog is to provide a basic understanding of MDE on Linux and how to operate, check and troubleshoot problems on SAP VMs running MDE. It is generally recommended that the Enterprise Security Team coordinate with the SAP team and jointly design the MDE configuration, exclusions and scheduling.

Before continuing it is strongly recommended to watch the video in the link below. Microsoft Defender for Endpoint (MDE) is one component in the set of Defender solutions and in turn has multiple subcomponents Microsoft Defender for Endpoint | Microsoft Docs

This blog is focusing on two subcomponents: Next-generation protection (AntiVirus) and Endpoint detection and response (EDR). Next-generation protection is an AntiVirus (AV) product similar to AV solutions for Windows environments. Endpoint detection and response (EDR) detects and can block suspicious activity and system calls.

Microsoft Defender for Endpoint Subcomponents

Core Defender Vulnerability Management

Attack surface reduction

Next-generation protection

Endpoint detection and response

Automated investigation and remediation

Microsoft Threat Experts

Centralized configuration and administration, APIs

Microsoft 365 Defender

How is Microsoft Defender for Endpoint on Linux Deployed?

MDE for Linux may be deployed to VMs running SAP applications automatically in a subscription if Microsoft Defender for Cloud is activated. The SAP administrators and/or infrastructure team may not be aware MDE for Linux will be automatically deployed as a VM Extension. SAP administrators may observe that MDE is not installed when a new VM is first created, but after some time the following extension can be seen in the “Extensions + applications” blade in the Azure Portal.

Microsoft Defender for Cloud may be activated for the subscription containing SAP resources and MDE for Windows and Linux may be deployed by default. Further information can be found here: Using Microsoft Defender for Endpoint in Microsoft Defender for Cloud to protect native, on-premises...

MDE for Linux can also be deployed manually via tools such as yum and zypper, or via ansible, chef and puppet

Microsoft Defender for Endpoint on Linux | Microsoft Learn

SAP administrators and consultants should check with the Enterprise Security team for details about which deployment mode is used for the Azure subscription running SAP VMs.

Prerequisites & Default Deployment Configuration

Prerequisites for deploying MDE for Linux on SAP VMs:

MDE version 101.88.48 or higher must be deployed. Do not use lower releases
MDE for Linux supports all the Linux releases used by SAP applications
MDE for Linux requires connectivity to Internet from VMs to update AV Definitions
MDE for Linux requires some crontab (or other task scheduler) entries to schedule scans, log rotation and MDE updates. Enterprise Security team will normally manage these entries How to schedule an update of the Microsoft Defender for Endpoint (Linux) | Microsoft Learn

Internet connectivity can be confirmed with the command below

To view MDE status run mdatp health

"real_time_protection_enabled : false" means that the AV component of MDE will not intercept IO calls. There is no IO interception and no scheduled AV scanning therefore MDE for Linux will not cause IO performance degradation on SAP DBMS or Application servers.

Note: if MDE for Linux is deployed by methods other Azure Extension the AntiVirus functionality may be enabled by default.

The Linux crontab is typically used to schedule MDE AV scan and log rotation tasks

How to schedule scans with Microsoft Defender for Endpoint (Linux) | Microsoft Learn

EDR functionality will be Active whenever MDE for Linux is installed. There is no way to disable EDR functionality through command line or configuration.

See the section “Checklist for Troubleshooting Problems on SAP VMs Running MDE on Linux” for more information on troubleshooting EDR

How to Check Defender Deployment and Configuration

It is recommended to check the installation and configuration of MDE with the command mdatp health

SAP Application and DBMS servers MDE configuration should be similar the screenshot below. The key parameters are:

healthy = true
release_ring = Production. Pre-release and insider rings should not be used with SAP Applications
real_time_protection_enabled = false. This prevents realtime IO interception
automatic_definition_update_enabled = true
definition_status = “up_to_date”. Run a manual update if another value is seen
edr_early_preview_enabled = “disabled”. Do not enable on SAP systems. This may lead to system instability
conflicting_applications = [ ]. Other AV or security software installed on a VM
MDE engine_version = must be 101.88.48 or higher otherwise issues with NFS or sudo may occur in some cases

This article has some useful hints on troubleshooting installation issues for MDE:

Troubleshoot installation issues for Microsoft Defender for Endpoint on Linux | Microsoft Docs

How to Setup MDE AntiVirus Exclusions

It is generally recommended to enable real_time_protection_enabled = true after identifying the relevant DBMS and SAP exclusions. This provides the optimal protection while at the same time avoiding performance problems.

This article details how to configure AV exclusions for processes, files and folders per individual VM Configure and validate exclusions for Microsoft Defender for Endpoint on Linux | Microsoft Learn

SAP administrators should contact the Enterprise Security Team to discuss how to configure AV exclusions for all SAP VMs in an Azure Resource Group or Subscription.

Warning: If real time scanning is enabled on MDE releases lower than 101.88.48 sudo may be blocked. It is strongly recommended to update to the latest version of MDE and verify the MDE release before enabling real time scanning.

It is recommended to exclude:

DBMS data files, log files and temp files, including disks containing backup files
The entire contents of the SAPMNT directory
The entire contents of the SAPLOC directory
The entire contents of the TRANS directory
The entire contents of directories for standalone engines such as TREX

Note: It is recommended to have database files on the separate mountpoint with read and write permissions only (exec permission to mount point).

Hana systems should exclude /hana/data, /hana/log and /hana/shared – see Note 1730930.

Oracle ASM systems do not need exclusions as MDE cannot read ASM disks.

Common mistakes to avoid when defining exclusions | Microsoft Learn

Recommended SAP OSS Notes

2248916 - Which files and directories should be excluded from an antivirus scan for SAP BusinessObje...

1984459 - Which files and directories should be excluded from an antivirus scan for SAP Data Service...

2808515 - Installing security software on SAP servers running on Linux - SAP ONE Support Launchpad

1730930 - Using antivirus software in an SAP HANA appliance - SAP ONE Support Launchpad

1730997 - Unrecommended versions of antivirus software - SAP ONE Support Launchpad

Note: MDE for Linux folder exclusions are now recursive Configure and validate exclusions for Microsoft Defender for Endpoint on Linux | Microsoft Learn

After configuring exclusions it is possible to test with the EICAR test file. The EICAR test file can be placed in a temporary location to confirm MDE AV is functioning correctly

Configure and validate exclusions based on extension, name, or location | Microsoft Learn

In the example below the standard EICAR test file is downloaded with wget and a scan is run manually.

Threats can be listed with the command mdatp threat list and then the file(s) removed with the command below

Checklist for Troubleshooting Problems on SAP VMs Running MDE on Linux

If there are performance, stability or installation problems on an SAP VM running MDE for Linux it is recommended to follow the checklist below:

Run mdatp health and confirm all settings are set
1. healthy = true
2. release_ring = Production
3. real_time_protection_enabled = false (or true with appropriate exclusions configured)
4. automatic_definition_update_enabled = true
5. definition_status = “up_to_date”
6. edr_early_preview_enabled = “disabled”
7. conflicting_applications = [ ]
8. confirm definitions and MDE versions are up to date
Run zypper, yum or dnf to update mdatp. Deploy updates for Microsoft Defender for Endpoint on Linux | Microsoft Learn
Run mdatp definitions update to update AV definitions
Run mdatp connectivity test. If there are any connectivity issues follow the procedure Troubleshoot cloud connectivity issues for Microsoft Defender for Endpoint on Linux | Microsoft Lear...
Confirm that process, file and folder exclusions are appropriately configured with mdatp exclusion list
Behavior Monitoring is disabled. It can be enabled or disabled via managed config. Use the command: mdatp config behavior-monitoring --value disabled
Microsoft support may request to create a managed config file in the path /etc/opt/microsoft/mdatp/managed/mdatp_managed.json

This file can be configured with additional debugging/support options

Restart mdatp service with the command sudo service mdatp restart

Review the MDE on Linux logs for unusual events or warnings. Log files are located under "/var/log/microsoft/mdatp"

After completing the above checklist try to reproduce the problem. If the problem reproduces and MDE is a possible cause then follow the procedure below to open a support case.

How to Collect Logs & Open Support Cases

In rare cases MDE for Linux may impact performance or stability of an SAP VM. If this is suspected follow this checklist:

Download the Client Analyzer tool https://aka.ms/XMDEClientAnalyzer

Run the client analyzer on macOS or Linux | Microsoft Docs

Increase logging level if required Microsoft Defender for Endpoint on Linux resources | Microsoft Learn
Collect any other logs
Create a support request via the Defender 365 Portal

MDE for Linux support cases should be opened by the Enterprise Security Team via the Defender 365 Portal and not via the typical Azure Portal page. The support message should mention “Defender on Linux performance issues”

Review this link before sending the support case Troubleshoot performance issues for Microsoft Defender for Endpoint on Linux | Microsoft Learn

How to Uninstall Defender

In rare cases it may be necessary to uninstall MDE to isolate a problem. SAP support may also occasionally request that a problem is reproduced without any AntiVirus or security software installed.

MDE for Linux can be uninstalled using yum, zypper or dnf. Microsoft Defender for Endpoint on Linux resources | Microsoft Learn

Another option is to use the installer script Deploy Microsoft Defender for Endpoint on Linux manually | Microsoft Learn

Troubleshooting Steps

The complete list of log and config files for MDE is:

"/var/log/microsoft/mdatp"

"/var/opt/microsoft/mdatp"

"/etc/opt/microsoft/mdatp"

"/etc/opt/microsoft/mdatp/managed"

"/var/opt/microsoft/mdatp/crash"

AV or EDR events such as finding a Virus are logged into the Defender 365 Portal.

The deployment and health status of a subscription is also visible within the Defender 365 Portal, an example is illustrated below

It is generally recommended to install nmon and activate sysstat (SAR) on SAP servers. These tools are useful for determining if MDE for Linux or other processes are causing high CPU or disk utilization.

Unfortunately NMON is not available in some repositories such as zypper, dnf, yum and must be downloaded http://nmon.sourceforge.net/pmwiki.php
NMON also has the ability to record to a log file that can be analyzed in Excel. Execute the command nmon -f -s1 -c600 (this will record every 1 second for 600 count, or 10 minutes). The log file can then be analyzed in nmon analyzer Excel Macro nmon for Linux | Site / Nmon-Analyser (sourceforge.net)
sysstat or SAR may or may not be installed and activated by default. Suse gallery images may have SAR running by default. Check the directory /var/log/sa. If the directory does not exist or does not contain recent sarXX files then follow the steps below
KSAR is a graphical tool that presents historical system performance information in a simple and easy to interpret way. This tool requires a runtime JVM https://github.com/vlsi/ksar/releases (download the latest pre-release version)

If sysstat needs to be installed follow the steps below:

# sudo yum install sysstat

# sudo service sysstat restart

Redirecting to /bin/systemctl restart sysstat.service

The /var/log/sa/sarXX files can be copied onto a Windows PC with sftp

sftp -i <keyfilename>.pem azureuser@<xx.xx.xx.xx>

get /var/log/sa/sar<XX>

On a Windows PC run this command and open the SAR file "Java -jar C:\sap_media\ksar.jar"

KSAR shows long term trends and NMON is a realtime tool.

When reviewing KSAR graphs problems with AV software may be indicated by high “Waiting I/O” times.

Example of NMON logging with analysis in Excel via Nmon-Analyzer macros. CPU shown in blue line and IOPS in pink.