New network-based detections and improved device discovery using Zeek

Organizations are finding network-based attacks becoming an increasingly popular way of infiltrating systems because they often leave minimal traces on source and target devices. At Microsoft Ignite 2022, we announced partnering with Zeek, an open-source network security monitoring platform, and its corporate sponsor, Corelight, to help security teams combat these attacks more effectively.  As a result, Zeek is now integrated as a component within Microsoft Defender for Endpoint.    

 

The integration of Zeek into Microsoft Defender for Endpoint provides new levels of network analysis capabilities based on deep inspection of network traffic powered by Zeek, a powerful open-source network analysis engine that allows researchers to tackle sophisticated network-based attacks in ways that weren't possible before. Administrators onboarding endpoints to Defender for Endpoint can now monitor inbound and outbound traffic with a novel engine that is capable of:

 

Session Awareness - Being able to aggregate network protocol data across an entire TCP/UDP session, such as NTLM and Kerberos authentications, SSH sessions, FTP connections, and RPC. These aggregated protocol insights provide much richer metadata and extracted payloads that can be used to enhance the detection capabilities of network-based attacks, as well as the passive classification of discovered devices.

 

Dynamic Protocol Detection - Being able to detect attacks even on non-default ports, a common pattern attackers use to hide their network traffic.

 

Dynamic Scripting Content - Being able to add new detections on the fly using Zeek scripts, backed by a wide community of security advocates. This unlocks the ability to react to emerging network-based threats such as Log4Shell and PrintNightmare at unprecedented speed. In a reality where new vulnerabilities are discovered on a weekly basis, this is a true game changer.

 

While Zeek has been around for over 20 years, the software has traditionally run on Unix-like operating systems such as Linux, FreeBSD, and macOS. As part of the new partnership between Microsoft and Corelight, we extended Zeek to support Windows-based systems. This is a non-trivial engineering effort - which we are excited to contribute back to the open-source community.

 

Supercharging Defender for Endpoint with Zeek

 

The integration of Zeek into Microsoft Defender for Endpoint provides a powerful ability to detect malicious activity in a way that enhances our existing endpoint security capabilities, as well as enables a more accurate and complete discovery of endpoints & IoT devices. 
Using Zeek, Defender for Endpoint will collect network events used for detections, posture and device discovery and will adhere to the Microsoft privacy practices that Defender for Endpoint upholds today.

 

Network-based Detection of Malicious Activity

 

With many attacks only visible at the network layer, continuously monitoring and analyzing network activity is critical. Attackers that utilize their own network stack can often bypass existing endpoint-based detections, but they cannot hide their network footprint. Providing visibility into the network layer, using both incoming and outgoing traffic from each endpoint, broadens the ability to protect devices operating on the network even if they are not onboarded to Defender for Endpoint, by detecting attacks initiated by these devices, as well as discovering vulnerable services and operating system versions running on them. Moreover, being deployed on endpoints allows admins to compose non-network telemetry such as process and file actions into network-based detections to give richer context into suspicious network activity.

 

Accurate signals of malicious network activity are important for identifying an attack campaign at the early stages of exploration and lateral movement, which can lead to stopping an attack in its infancy.

 

Below are more details on the first two Zeek-based detections now deployed to Defender for Endpoint:

 

PrintNightmare detection - This detection identifies PrintNightmare exploitation attempts. The PrintNightmare Zeek script identifies the usage of the RPC functions used to install a remote printer driver. We further contextualize this action with additional endpoint and network-based telemetry and rely on the behavioral profiles of existing network entities in the organization to cover both inbound and outbound attacks and reduce false positive rates to the lowest possible extent.

 

 

Figure 1 - A PrintNightmare Alert Example

 

Proprietary password spray detection - Using Zeek’s out-of-the-box NTLM analyzer, Microsoft Defender for Endpoint can now identify attackers that are trying to authenticate to a machine with many different users as part of a password spray attack, while using different NTLM-based protocols such as SMB, Telnet, HTTP, RPC, or WINRM. Zeek’s ability to provide the session context comes into play and allows the detection logic to take different handshake parameters into account, thus making it much more accurate.

 

 

Figure 2 - A Password Spray Alert Example

 

The two above-mentioned detections are just a few examples that demonstrate how the new traffic parsing and processing capabilities of this new integration boost the detection coverage, reaching attack types that were so far undetectable.

 

Device Discovery Enhancements

 

In addition to these new detections, the integration also enhances Defender for Endpoint’s passive device discovery capabilities by utilizing many widely used protocols that are supported out of the box, including the below:

 

NTLM - The NTLM authentication protocol involves both client and server devices sending their hostname, domain name, and operating system version. This is highly valuable data when it comes to device discovery. Zeek aggregates and reports this information for both sides on the NTLM transaction.

 

 

SSH - Zeek monitors SSH protocol traffic and parses out the server version string. This string often includes the version of the SSH server software and the host operating system version.

 

 

FTP - FTP servers usually respond with a code 220 response after a successful TCP handshake. This means that the server is ready to serve a new user. As part of the code 220 response, a response message is sent which typically contains identifying information about the FTP server.

 

 

Zeek Performance on Endpoints

 

Although the Zeek engine runs very efficiently, it is designed to run on a dedicated machine where it can consume all its resources to achieve the highest processing throughput of network traffic. To minimize the system resource requirements for the Zeek integration, within the Microsoft Defender for Endpoint agent, Zeek runs in a system-controlled process that automatically regulates its consumed resources. Using an intelligent throttling mechanism that kicks in whenever there’s a spike in network traffic, the agent can automatically reduce the number of packets Zeek processes depending on the system resource utilization. This also means that the agent’s impact on the network volume is being regulated. We were able to test the resource utilization across various large-scale organizations and determine that the impact was minimal as shown in Figure 3.

                    

 

Figure 3 - The effect that intelligent throttling had on memory usage of Zeek integration in MDE

 

In conclusion, the integration of Zeek into Microsoft Defender for Endpoint delivers a new platform that provides a set of powerful network discovery and detection capabilities across Windows endpoints using unprecedented visibility into network traffic that allows new ways to protect your endpoints against advanced network-based attacks. On top of that, it also unlocks the ability to react to new network-based threats faster than ever thanks to its scripting ability and community-backed content, so when “the next Log4Shell” comes, Defender for Endpoint will be even better prepared to protect.

 

Differentiation with current NDR solutions

 

Customers that have a Network Detection and Response (NDR) solution deployed in their network might wonder how the new integration impacts the need for NDR. To that end, we wish to shed additional light on the different goals of NDR and the new integration:      

 

• Zeek in Microsoft Defender for Endpoint does not replace traditional NDR solutions. On the contrary, it’s a complementary data source providing network signals that otherwise might be missed.

 

• Neither data source is sufficient on its own, given the sophistication of advanced attacks and the rising number of unmanaged/unmanageable endpoints on the Internet. Microsoft recommends that security teams combine both data sources - endpoint for depth, and network for breadth - to gain full visibility across all parts of the network.

 

• If you've already deployed Zeek / Corelight for NDR, or are planning to, this integration enhances your investment because analysts can rely on the same underlying tech and data format, regardless of the point of collection.

 

More information

 

• Understand which versions of the Microsoft Defender for Endpoint agent support the new integration here:
  • What's new in Microsoft Defender for Endpoint | Microsoft Learn
• Find out more details about the integration in our ZeekWeek 2022 presentations:
  • ZeekWeek 2022 - Zeek for Windows: The Journey to Run on All Endpoints - Elad Solomon
  • ZeekWeek 2022 - Zeek for Endpoint: Detection and Device Discovery - Boaz Wasserman
• View the open-source contribution in Zeek’s GitHub repository:
  • Microsoft's Port of Zeek for Windows - zeek/zeek (github.com)
• Previous announcements:
  • Ignite: Microsoft Defender for Endpoint Announcements at Microsoft Ignite 2022
  • Corelight: Zeek is Now a Component of Microsoft Windows (corelight.com)
  • Zeek Project: A few questions and answers on Zeek and Windows