Blog Post

Core Infrastructure and Security Blog

11 MIN READ

Forward On-Premises Windows Security Event Logs to Microsoft Sentinel

paulberg

Microsoft

Dec 16, 2021

Hello,

It has been a while since Raven, and I have blogged on security. My little buddy Raven (miniature Schnauzer) has been dealing with genetic back problems that have made it difficult to run or jump, so her days of roaming the yard and scaring off squirrels has been curtailed.

This past summer I was able to spend a lot of time in my backyard with Raven quietly resting alongside me. Raven has never given up on protecting the yard, but she needs help from me to find the intruders. She lays there quietly but when I say “Squirrel” and point, her back problems vanish temporarily as she vanquishes the little critter (don’t worry she never gets close to one).

As I initially sat and work on the technical topic of this blog, it dawned on me how much Raven needing help finding intruders and what Microsoft Sentinel (Formerly Azure Sentinel) can provide to our customers. Microsoft Sentinel is the alerting mechanism that finds the anomalies in your environment and can alert you to go evict them.

Windows Event Forwarding (WEF) isn’t something new, I believe it has been around for more than 20 years, but the ability to query has never been its strong point, plus storage can be an issue. Having the ability to get access to all of the enterprises Windows Event logging data without having to load a client (WEF is built into the o/s) has two major advantages.

No cost
No agent management

Imagine a customer with close to 200,000 endpoints and having to maintain the installed client base, that could be a real headache and client costs are very high (I am working with such a scenario). A WEC server can’t have that large of a number of clients so it has to be split out, and I have been asked “how many clients could connect to a single WEC server?” There is no precise answer to that question. Since there are many factors that enter into that question. Size of WEC server, amount of traffic being sent,… I have seen that the number of a clients that a WEC server can handle, could go as high as 10,000 clients but again the environment factors enter into this.

Once one or more WEC server have been stood up then you will need to add an “Azure Arc” connection to Azure, so Microsoft Sentinel can “Connect” to the WEC server. The Microsoft Sentinel connector “Windows Forwarded Events (Preview)” requires AMA, as it is not supported for MMA, and AMA requires the deployment of Azure Arc.

This will then provide the customer complete access to the logs from the hosts that exist outside of Azure (On-Premises, AWS, GCP for example) that were aggregated with WEF.

Below I have walked through the steps needed to help deploy a WEF to Microsoft Sentinel infrastructure.

Windows Event Forwarding Log Collector to Microsoft Sentinel Rollout

There is no need to load an agent on every device to capture the Windows Security Event Logs from your on-premises Windows workstations & servers. Windows hosts already have this built into the operating system. To capture the events without having to load the Azure Monitoring Agent (AMA) the Windows Event Forwarding process can be used to send logs to a “Windows Event Collector” (WEC). The WEC will then need the AMA loaded to send the events to a Log Analytics Workspace (LAW) that is monitored by Microsoft Sentinel.

Note: Microsoft Sentinel must be enabled/deployed prior to the deployment of the AMA agent.

From a high-altitude view:

Deploy Microsoft Sentinel
Build a Windows Event Collector (WEC) server to host the security event logs from client (source) computers
Create a Group Policy to define where the clients are to request the logs and events (Subscription), they are to send to the WEC
Create a subscription on the WEC to define what logs and events to receive
For on-premises WEC server(s), enroll it/them in the Azure Arc service
Add the Microsoft Sentinel, “Windows Forwarded Events (Preview)” connector
- Define the WEC hosts
- Define the “Forwarding Event Logs” log to collect from
Browse/Query (KQL) the LAW for Security Events

High Level Steps in Graphic Format

Create and Push GPO To Clients So They Can Find the WEF Collector

Clients Request WEF Policy from WEF Server

WEF Collector Pushes Policy to Clients

Clients Continuously Push Security Events to WEF Collector

Collected Security Events are Continuously Forwarded to the Azure Log Analytics Workspace

Building a Windows Event Collector

This is a resource requirement. The size of the host will depend on the number of source clients and logs being forwarded to the WEC.

“You deploy EventLog Forwarding in a large environment. For example, you deploy 40,000 to 100,000 source computers. In this situation, we recommend that you deploy more than one collector that has 2,000 clients to not more than 4,000 clients per collector.

Note: AMA can handle up to 5,000 EPS, but be aware that it is important to have enough WEC servers as if the limit of EPS is reached the agent won’t be able to handle the load.

Additionally, we recommend that you install at least 16 GB of RAM and four (4) processors on the collector to support an average load of 2,000 to 4,000 clients that have one or two subscriptions configured.

Fast disks are recommended, and the ForwardedEvents log can be put onto another disk for better performance.

The memory usage of the Windows Event Collector service depends on the number of connections that are received by the client. The number of connections depends on the following factors:

The frequency of the connections
The number of subscriptions
The number of clients
The operating system of the clients

For example, for the default values of 4,000 clients and five to seven subscriptions, the memory that is used by the Windows Event Collector service may quickly exceed 4 GB and continue to grow. This can make the computer unresponsive.”
Best practice of configuring EventLog forwarding performance - Windows Server | Microsoft Docs

Ensure Events can be forwarded if running on a Windows Server

“Symptoms

You configure a Windows Server 2019 or Windows Server 2016 computer as an event collector. You also configure a source-initiated subscription (and related Group Policy Objects) for event forwarding. However, the events are not forwarded and the event source computers log event messages that resemble the following:

Log Name: Microsoft-Windows-Forwarding/Operational

Event ID: 105

Task Category: None

User: NETWORK SERVICE

Description:

The forwarder is having a problem communicating with subscription manager at address http://W19SRV.contoso.com:5985/wsman/SubscriptionManager/WEC. Error code is 2150859027 and Error Message is The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol.

Cause

This behavior is caused by the permissions that are configured for the following URLs:

http://+:5985/wsman/
http://+:5986/wsman/

On the event collector computer, both the Windows Event Collector service (WecSvc) and the Windows Remote Management service (WinRM) use these URLs. However, the default access control lists (ACLs) for these URLs allow access for only the svchost process that runs WinRM. In the default configuration of Windows Server 2016, a single svchost process runs both WinRM and WecSvc. Because the process has access, both services function correctly. However, if you change the configuration so that the services run on separate host processes, WecSvc no longer has access and event forwarding no longer functions.

Resolution

To view the URL permissions, open an elevated Command Prompt window and run the command netsh http show urlacl.

To fix the URL permissions, use the elevated Command Prompt window and run the following commands:

netsh http delete urlacl url=http://+:5985/wsman/

netsh http add urlacl url=http://+:5985/wsman/ sddl=D:(A;;GX;;;S-1-5-80-569256582-2953403351-2909559716-1301513147-412116970)(A;;GX;;;S-1-5-80-4059739203-877974739-1245631912-527174227-2996563517)

netsh http delete urlacl url=https://+:5986/wsman/

netsh http add urlacl url=https://+:5986/wsman/ sddl=D:(A;;GX;;;S-1-5-80-569256582-2953403351-2909559716-1301513147-412116970)(A;;GX;;;S-1-5-80-4059739203-877974739-1245631912-527174227-2996563517)”

Event collector doesn't forward events - Windows Server | Microsoft Docs

Create a Group Policy Object (GPO) and link it to an Organization Unit (OU)

WEF uses WINRM, which uses ports 5985 for http or 5986 for https. Ensure that you have the winrm service running on clients before you start capturing traffic. Winrm is started by default on Windows Server 2008 and beyond.

If the goal is to capture the Security event logs as one of the logs (In our demo we will need to capture the Security Event Logs), then it will be required to grant the “Network Service” access to the Security event log, by default access is denied. From an Active Directory domain machine, run the following command, from an elevated command line:
wevtutil gl security

This will list out the ACL’s defined on the Security Event Log. Look for “channelAccess” the "O:BAG:SYD:" is where the permissions on the log are stored. Copy from the O through the last parenthesis and paste it into Notepad. If there isn’t a (A;;0x1;;;NS) on the end like the example below, then append that on to the line in Notepad. This last part provides the Network Service (NS), access to the Security Event log.

Start up Group Policy Management Editor

There are 2 settings that will need to be added, to point the clients to the WEC server
Computer>Policies>Admin Templates>Windows Components>Event Forwarding>Configure target subscription manager

This will need to be updated with the address of your WEC server in the format shown below:

Server=http://fqdnofWECserver:5985/wsman/SubscriptionManager/WEC,Refresh=60

Replace the red highlighted area with the fqdn of the WEC server.

Note: “Server=” is needed in the line defined above

The refresh interval on the end indicates how often clients should check in to see if new subscriptions are available. In this example 60 seconds is extremely chatty, but during testing you only have to wait 1 minute for updated configuration. Setting to hourly (Refresh=3600) in production should work just fine.

Once the defined WEC has been completed, the Network Service needs to be granted access to the Security Event Log. This step is not needed if you won’t be reading that log file.

Computer>Policies>Admin Templates>Windows Components>Event Log Service>Security> Configure log access

From a previous step where the Security Event log permissions were built and stored in Notepad, this value will now be updated in the GPO.

Note: This will replace any previous settings on this Event Log, so just be aware of this update.

Once this GPO has been built, it will be up to the admin to decide how to apply the policy to the workstations/servers so they can check in with the WEC server to get the subscription definition.

Create a WEC subscription

Now that client GPO has been defined a subscription needs to be built to tell these clients what logs and Events should be “Forwarded”.

From the WEC server, start up “Event Viewer”.

Right click on “Subscriptions” and select “Create Subscription...”.

“Subscription name:” Enter a unique name for the subscription (try to avoid spaces)
“Description:” is optional
“Destination log:” Select the log file “Forwarded Events”
Select “Source Computer Initiated”
Click on “Select Computer Groups…”

Select the “Add Domain Computers” button and walk through the Active Directory (AD) picker to populate the Computers to be added. In the example below, there are just individual machines but AD groups can also be used. Once all objects have been selected click the “Ok” button”

From the “Subscription Properties” main page, click on the “Select events” button.

The “Query Filter” page allows the admin of the filter the ability to only forward events interested in capturing. This filter will be used by all client subscribers that are forwarding events. These events will all be sent to the WEC server. If the admin would like the WEC server to capture all events but filter this list before sending to Microsoft Sentinel, there is a second filter definition on the Microsoft Sentinel connector.

Note: Events are continuously sent to the WEF collector

Once filtering has been completed, select “Ok” and select “Ok” again on the Subscription properties page.

Waiting approximately 15 minutes (After the GPO has applied to the clients), the “Forwarded Events” log should begin to populate from subscribers to the WEC subscription.

If you look closely at the screen capture below you will see that the “Forwarded Events” log resides on vm2016-01 (DOS prompt), yet the reporting in the event itself belongs to VM2019DC-01.

Enroll WEC server(s) into Azure Arc services

In order to capture events within Microsoft Sentinel, there has to be a connection to the Log Analytics workspace that Microsoft Sentinel monitors. To do this we need to enroll our WEC server into Azure Arc. This is completed by installing the Azure Monitor Agent.

Log onto the Azure portal:
https://portal.azure.com
From the search portal enter, “Servers - Azure Arc” and select this to go to the “Servers - Azure Arc” blade.
Click on “+ Add”

For this initial trial rollout, select the “Generate Script” from the “Add a single server” box.

Select “Next”
Complete the Resource Details and click on “Next”

Note: In the example from my lab below, I am using a “Public” endpoint. I would strongly encourage organizations to not expose ANY log files on the public internet!

Enter any required “Tags” for your organization and select “Next”
Select the “Download” button
Select the “Copy” button
- The “Download” button will probably be blocked by your organization

From the on-premises WEC collector desktop, open a script editor (Notepad for example) and paste the contents of the clipboard and save it as “WEC-Sentinel.ps1”.
- Open an elevated PowerShell command prompt
- Change directories to where you saved “WEC-Sentinel.ps1”
- Execute the script: .\WEC-Sentinel.ps1

You will be prompted to sign into a web browser and enter a code

Follow the on screen prompts to logon and approve the joining of this machine to Azure Arc

Move back to the “Servers – Azure Arc” blade, hit refresh and the newly onboarded host should now be a part of this subscriptions “Azure Arc” as seen with the Data Collector (“vm2016-01”) below.

Add the Microsoft Sentinel, “Windows Forwarded Events (Preview)” connector

Once events are being collected, the events now need to be imported into a “Log Analytics Workspace” (LAW) for Sentinel to be able to monitor and report on them.

Log onto the Azure portal:
https://portal.azure.com
Select Microsoft Sentinel

Select the “Data Connectors” blade
Select the LAW that you would like to aggregate events to from the WEC

Enter “Windows Forwarded Events” in the “Search by name or provider” box
- Click on “Windows Forwarded Event”

Select “Open connector page”

Select “+Add data collection rule”

On the “Basics” tab enter
- “Rule Name”, “Subscription” and “Resource Group”

On the “Resources” select the “+Add Resource(s)”

Browse to the “Collector(s)” that will be capturing on-premises Security event logs
- Click the “Apply” button

On the “Collect” tab select the “+Add Resource(s)”
- Browse to the on-premises Data Collector (VM2016-01)
- Select the “Apply” button

Choose if you want to send ALL security events or just a filtered list
- Click the “Next” button

Click the “Create” button

Browse the LAW for Security Events

Log onto the Azure portal:
https://portal.azure.com
Select Azure Sentinel
Select the LAW
Select the “Logs” blade
- Close the “Favorites” query page

To query the WEF logs imported into Microsoft Sentinel, the administrator can open a KQL query with the “WindowsEvent” table

There are 3 hosts that are currently reporting to my LAW, which is defined within the WEC subscription(s)
- VM2016-01
- Win10-1809-01
- WV2019DC-02

This document won’t dive any deeper into KQL, if that is needed a separate document can be built to assist with filtering.

Hopefully this has provided you with some options to reduce costs and get your log data in a SIEM. I know Raven really appreciates me “Alerting” her to intruders in the backyard. 😊

References

Updated Jan 04, 2022

Version 6.0

PaulBergson

paulberg

Microsoft

Joined October 30, 2017

View Profile

Core Infrastructure and Security Blog

Follow this blog board to get notified when there's new activity

james765
Brass Contributor
Dec 21, 2021
This is crying out for a Microsoft hosted event log collector.

Imagine each Application Insights instance had a WEC URL ready to go? 🙂
Ciyaresh
Brass Contributor
Feb 23, 2023
paulberg What happens in an event when WEF collector server is down for either update reasons or just plain crashed? Do servers that are sending logs keep the current state and then forward the logs once collector becomes available? or is it a fire and forget case where all the logs sent when WEF collector was down are lost?

edit: for those who are wondering about this, we have tested it by taking down the WEF collector for few minutes and then bring it back online. We have timed the power-off and power-on and it looks like the logs are kept locally on clients until WEF collector becomes available and then forwarded. So we don't lose logs if WEF is down, it will just forward when it becomes available without losing the track. Also the timestamp on the logs stays true. So if WEF goes down on 15:00 and come backs online at 15:10, when clients send logs that were generated in between those times, in sentinel you will see logs time as 15:01, 15:03 etc. So keep this in mind when you are building analytic rules.
Ciyaresh
Brass Contributor
Apr 25, 2022
gautamgupta did you find a solution this? We are thinking of using this method to forward our on-prem server logs to sentinel but we were not sure if we are getting everything that we would get if we deployed individual agents on servers rather than this method. Also, another disadvantage is that the microsoft's analytic rule templates are based on AMA or legacy agent agent.
jeremyhAUS
Brass Contributor
Mar 29, 2023
Would it be possible to use a KQL transform on the DCR to divert events from the Security Channel into the Security Events table in the LAW instead of all the events being in the wrong table? Surely this would allow the ASIM normalisation to work properly and the analytics rules to apply to these events too?

I will try to lab this out.

EDIT:
Sadly it did not work:
{"error":{"code":"InvalidPayload","message":"Data collection rule is invalid","details":[{"code":"InvalidDataFlow","message":"Transformation of stream 'Microsoft-WindowsEvent' to 'Microsoft-SecurityEvent' is not supported","target":"properties.dataFlows[0]"}]}}

bujhie

Copper Contributor

Sep 05, 2023

jeremyhAUS markj995 To split Security logs into SecurityEvent and non-Security logs into WindowsEvent tables you'll need two DCRs created manually and associated with the AMA agent installed on your WECs. I went with bicep, but you can use ARM, Terraform or there are blog posts how to update existing DCRs using Azure Resource Manager REST API. Some links to get started: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview

Below are relevant sections of the DCRs - you should be able to figure out the rest

dataSources definition for DCR1:

windowsEventLogs: [
        {
          name: 'nonSecurityEvents'
          streams:[
            'Microsoft-WindowsEvent'
          ]
          xPathQueries: [
            /*Filter out Security logs*/
            '''ForwardedEvents!*[System[(Channel!='Security') and (Channel!='Microsoft-Windows-AppLocker/EXE and DLL') and (Channel!='Microsoft-Windows-AppLocker/MSI and Script') ]]'''
          ]
        }
      ]

dataSources definition for DCR2:

windowsEventLogs: [
        {
          name: 'SecurityEvents'
          streams:[
            'Microsoft-SecurityEvent'
          ]
          xPathQueries: [
            '''ForwardedEvents!*[System[(Channel='Security')]]'''
            '''ForwardedEvents!*[System[(Channel='Microsoft-Windows-AppLocker/EXE and DLL') or (Channel='Microsoft-Windows-AppLocker/MSI and Script')]]'''
          ]
        }
      ]

Hope this helps you and others.

gautamgupta
Copper Contributor
Jan 19, 2022
Hi Paul
We have deployed this WEC Server in one of the client environment. We are getting logs in WindowsEvent table but the EventData field is unparsed. We deployed ASIM also in the connector page, but still same results.
gautamgupta
Copper Contributor
Apr 29, 2022
Ciyaresh No, there is no permanent solution to this yet . The logs from WEF will go to WindowsEvent table. You will get all the info using this but you have to write new detection rules which will parse the information from this table. Microsoft said they are working on writing similar analytics rules for this but there is no ETA
AndrewX
Iron Contributor
Aug 13, 2022
Hi paulberg thank you for effort put into this article, I've read them all and this stands out.

If we are collecting security events on premise with WEF then forwarding them to LAW & Sentinel, and also have the windows agent install on some servers also collecting the security events through the Windows Security Sentinel Connector, will this duplicate events and cost?
paulberg
Microsoft
Aug 15, 2022
Yes. So the key would be to use the AMA agent (With the Azure ARC agent it now uses this) and filter on those events you want and hopefully prevent duplicating data.
TokeSR
Copper Contributor
Aug 18, 2022
Hey folks. I'm not 100% sure why I need the WEF Preview connector to be deployed if AMA is also needed on the machine. Can't I just configure the AMA agent to pick up these forwarded logs? If I can, what benefit does this new WEF Preview connector provide over it?