Hello,
It has been a while since Raven, and I have blogged on security. My little buddy Raven (miniature Schnauzer) has been dealing with genetic back problems that have made it difficult to run or jump, so her days of roaming the yard and scaring off squirrels has been curtailed.
This past summer I was able to spend a lot of time in my backyard with Raven quietly resting alongside me. Raven has never given up on protecting the yard, but she needs help from me to find the intruders. She lays there quietly but when I say “Squirrel” and point, her back problems vanish temporarily as she vanquishes the little critter (don’t worry she never gets close to one).
As I initially sat and work on the technical topic of this blog, it dawned on me how much Raven needing help finding intruders and what Microsoft Sentinel (Formerly Azure Sentinel) can provide to our customers. Microsoft Sentinel is the alerting mechanism that finds the anomalies in your environment and can alert you to go evict them.
Windows Event Forwarding (WEF) isn’t something new, I believe it has been around for more than 20 years, but the ability to query has never been its strong point, plus storage can be an issue. Having the ability to get access to all of the enterprises Windows Event logging data without having to load a client (WEF is built into the o/s) has two major advantages.
No cost
No agent management
Imagine a customer with close to 200,000 endpoints and having to maintain the installed client base, that could be a real headache and client costs are very high (I am working with such a scenario). A WEC server can’t have that large of a number of clients so it has to be split out, and I have been asked “how many clients could connect to a single WEC server?” There is no precise answer to that question. Since there are many factors that enter into that question. Size of WEC server, amount of traffic being sent,… I have seen that the number of a clients that a WEC server can handle, could go as high as 10,000 clients but again the environment factors enter into this.
Once one or more WEC server have been stood up then you will need to add an “Azure Arc” connection to Azure, so Microsoft Sentinel can “Connect” to the WEC server. The Microsoft Sentinel connector “Windows Forwarded Events (Preview)” requires AMA, as it is not supported for MMA, and AMA requires the deployment of Azure Arc.
This will then provide the customer complete access to the logs from the hosts that exist outside of Azure (On-Premises, AWS, GCP for example) that were aggregated with WEF.
Below I have walked through the steps needed to help deploy a WEF to Microsoft Sentinel infrastructure.
There is no need to load an agent on every device to capture the Windows Security Event Logs from your on-premises Windows workstations & servers. Windows hosts already have this built into the operating system. To capture the events without having to load the Azure Monitoring Agent (AMA) the Windows Event Forwarding process can be used to send logs to a “Windows Event Collector” (WEC). The WEC will then need the AMA loaded to send the events to a Log Analytics Workspace (LAW) that is monitored by Microsoft Sentinel.
Note: Microsoft Sentinel must be enabled/deployed prior to the deployment of the AMA agent.
From a high-altitude view:
This is a resource requirement. The size of the host will depend on the number of source clients and logs being forwarded to the WEC.
“You deploy EventLog Forwarding in a large environment. For example, you deploy 40,000 to 100,000 source computers. In this situation, we recommend that you deploy more than one collector that has 2,000 clients to not more than 4,000 clients per collector.
Note: AMA can handle up to 5,000 EPS, but be aware that it is important to have enough WEC servers as if the limit of EPS is reached the agent won’t be able to handle the load.
Additionally, we recommend that you install at least 16 GB of RAM and four (4) processors on the collector to support an average load of 2,000 to 4,000 clients that have one or two subscriptions configured.
Fast disks are recommended, and the ForwardedEvents log can be put onto another disk for better performance.
The memory usage of the Windows Event Collector service depends on the number of connections that are received by the client. The number of connections depends on the following factors:
For example, for the default values of 4,000 clients and five to seven subscriptions, the memory that is used by the Windows Event Collector service may quickly exceed 4 GB and continue to grow. This can make the computer unresponsive.”
Best practice of configuring EventLog forwarding performance - Windows Server | Microsoft Docs
Ensure Events can be forwarded if running on a Windows Server
“Symptoms
You configure a Windows Server 2019 or Windows Server 2016 computer as an event collector. You also configure a source-initiated subscription (and related Group Policy Objects) for event forwarding. However, the events are not forwarded and the event source computers log event messages that resemble the following:
Log Name: Microsoft-Windows-Forwarding/Operational
Event ID: 105
Task Category: None
User: NETWORK SERVICE
Description:
The forwarder is having a problem communicating with subscription manager at address http://W19SRV.contoso.com:5985/wsman/SubscriptionManager/WEC. Error code is 2150859027 and Error Message is The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol.
This behavior is caused by the permissions that are configured for the following URLs:
On the event collector computer, both the Windows Event Collector service (WecSvc) and the Windows Remote Management service (WinRM) use these URLs. However, the default access control lists (ACLs) for these URLs allow access for only the svchost process that runs WinRM. In the default configuration of Windows Server 2016, a single svchost process runs both WinRM and WecSvc. Because the process has access, both services function correctly. However, if you change the configuration so that the services run on separate host processes, WecSvc no longer has access and event forwarding no longer functions.
To view the URL permissions, open an elevated Command Prompt window and run the command netsh http show urlacl.
To fix the URL permissions, use the elevated Command Prompt window and run the following commands:
netsh http delete urlacl url=http://+:5985/wsman/
netsh http add urlacl url=http://+:5985/wsman/ sddl=D:(A;;GX;;;S-1-5-80-569256582-2953403351-2909559716-1301513147-412116970)(A;;GX;;;S-1-5-80-4059739203-877974739-1245631912-527174227-2996563517)
netsh http delete urlacl url=https://+:5986/wsman/
netsh http add urlacl url=https://+:5986/wsman/ sddl=D:(A;;GX;;;S-1-5-80-569256582-2953403351-2909559716-1301513147-412116970)(A;;GX;;;S-1-5-80-4059739203-877974739-1245631912-527174227-2996563517)”
Event collector doesn't forward events - Windows Server | Microsoft Docs
WEF uses WINRM, which uses ports 5985 for http or 5986 for https. Ensure that you have the winrm service running on clients before you start capturing traffic. Winrm is started by default on Windows Server 2008 and beyond.
If the goal is to capture the Security event logs as one of the logs (In our demo we will need to capture the Security Event Logs), then it will be required to grant the “Network Service” access to the Security event log, by default access is denied. From an Active Directory domain machine, run the following command, from an elevated command line:
wevtutil gl security
This will list out the ACL’s defined on the Security Event Log. Look for “channelAccess” the "O:BAG:SYD:" is where the permissions on the log are stored. Copy from the O through the last parenthesis and paste it into Notepad. If there isn’t a (A;;0x1;;;NS) on the end like the example below, then append that on to the line in Notepad. This last part provides the Network Service (NS), access to the Security Event log.
Start up Group Policy Management Editor
There are 2 settings that will need to be added, to point the clients to the WEC server
Computer>Policies>Admin Templates>Windows Components>Event Forwarding>Configure target subscription manager
This will need to be updated with the address of your WEC server in the format shown below:
Server=http://fqdnofWECserver:5985/wsman/SubscriptionManager/WEC,Refresh=60
Replace the red highlighted area with the fqdn of the WEC server.
Note: “Server=” is needed in the line defined above
The refresh interval on the end indicates how often clients should check in to see if new subscriptions are available. In this example 60 seconds is extremely chatty, but during testing you only have to wait 1 minute for updated configuration. Setting to hourly (Refresh=3600) in production should work just fine.
Once the defined WEC has been completed, the Network Service needs to be granted access to the Security Event Log. This step is not needed if you won’t be reading that log file.
Computer>Policies>Admin Templates>Windows Components>Event Log Service>Security> Configure log access
From a previous step where the Security Event log permissions were built and stored in Notepad, this value will now be updated in the GPO.
Note: This will replace any previous settings on this Event Log, so just be aware of this update.
Once this GPO has been built, it will be up to the admin to decide how to apply the policy to the workstations/servers so they can check in with the WEC server to get the subscription definition.
Now that client GPO has been defined a subscription needs to be built to tell these clients what logs and Events should be “Forwarded”.
From the WEC server, start up “Event Viewer”.
Right click on “Subscriptions” and select “Create Subscription...”.
Select the “Add Domain Computers” button and walk through the Active Directory (AD) picker to populate the Computers to be added. In the example below, there are just individual machines but AD groups can also be used. Once all objects have been selected click the “Ok” button”
From the “Subscription Properties” main page, click on the “Select events” button.
The “Query Filter” page allows the admin of the filter the ability to only forward events interested in capturing. This filter will be used by all client subscribers that are forwarding events. These events will all be sent to the WEC server. If the admin would like the WEC server to capture all events but filter this list before sending to Microsoft Sentinel, there is a second filter definition on the Microsoft Sentinel connector.
Note: Events are continuously sent to the WEF collector
Once filtering has been completed, select “Ok” and select “Ok” again on the Subscription properties page.
Waiting approximately 15 minutes (After the GPO has applied to the clients), the “Forwarded Events” log should begin to populate from subscribers to the WEC subscription.
If you look closely at the screen capture below you will see that the “Forwarded Events” log resides on vm2016-01 (DOS prompt), yet the reporting in the event itself belongs to VM2019DC-01.
In order to capture events within Microsoft Sentinel, there has to be a connection to the Log Analytics workspace that Microsoft Sentinel monitors. To do this we need to enroll our WEC server into Azure Arc. This is completed by installing the Azure Monitor Agent.
Note: In the example from my lab below, I am using a “Public” endpoint. I would strongly encourage organizations to not expose ANY log files on the public internet!
Once events are being collected, the events now need to be imported into a “Log Analytics Workspace” (LAW) for Sentinel to be able to monitor and report on them.
This document won’t dive any deeper into KQL, if that is needed a separate document can be built to assist with filtering.
Hopefully this has provided you with some options to reduce costs and get your log data in a SIEM. I know Raven really appreciates me “Alerting” her to intruders in the backyard. :smiling_face_with_smiling_eyes:
Event collector doesn't forward events - Windows Server | Microsoft Docs
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.