User Profile
TS-noodlemctwoodle
Brass Contributor
Joined May 05, 2020
User Widgets
Recent Discussions
Grouping Azure Sentinel - Azure Active Directory Identity Protection alerts
Is there a way to group Azure Active Directory Identity Protection alerts such as "Unfamiliar sign-in properties" in Azure Sentinel? We are seeing hundreds of these alerts being raised on a daily basis and it is causing quite a lot of noise in the incidents panel of Azure Sentinel. What would be really useful is a way to group all these alerts into a single incident, however, I do not see a way to do this. Any guidance would be greatly appreciated.8.2KViews1like10CommentsParsing XML in Azure Sentinel
CliveWatson I wonder if you can give me some pointers for how to parse XML syslog information in Azure Sentinel? Here is an sample of the redacted syslog message formatted into XML 05:19.0Z Some-Server-Name Events - EventFwd [agentInfo@3401 tenantId="0" bpsId="0" tenantGUID="{00000000-0000-0000-0000-000000000000}" tenantNodePath="1\2"] �<?xml version="1.0" encoding="utf-8"?> <UpdateEvents> <MachineInfo> <AgentGUID>{00000000-0000-0000-0000-000000000000}</AgentGUID> <MachineName>Some-Machine</MachineName> <RawMACAddress>112233445566</RawMACAddress> <IPAddress>1.1.2.3</IPAddress> <AgentVersion>1.2.3.123</AgentVersion> <OSName>Windows 41</OSName> <TimeZoneBias>-10</TimeZoneBias> <UserName>myName</UserName> </MachineInfo> <BrandCommonUpdater ProductName="Brand Agent" ProductVersion="1.0.0" ProductFamily="AVP"> <UpdateEvent> <EventID>1234</EventID> <Severity>0</Severity> <GMTTime>2020-00-00T06:41:02</GMTTime> <ProductID>SomeName1999</ProductID> <Locale>0001</Locale> <Error>0</Error> <Type>SomeCore</Type> <Version>1234.0</Version> <InitiatorID>SOMEAGENT3000</InitiatorID> <InitiatorType>OnDemand</InitiatorType> <SiteName>Some-Server-Name</SiteName> <Description>N/A</Description> </UpdateEvent> </BrandCommonUpdater> </UpdateEvents> Many thanksSolved11KViews1like9CommentsRe: Split userPrincipalName in Azure Sentinel Playbook
GaryBushey The template language expression 'split(triggerBody()?['object']?['properties']?['owner']?['userPrincipalName'], '@')['0']' cannot be evaluated because property '0' cannot be selected. Array elements can only be selected using an integer index. This isnt possible, I have tried many variations of the same split(triggerBody()?['object']?['properties']?['owner']?['userPrincipalName'], '@'),['0']' split(triggerBody()?['object']?['properties']?['owner']?['userPrincipalName'], '@')(['0']')2.1KViews0likes1CommentSplit userPrincipalName in Azure Sentinel Playbook
Please could someone give me some pointers on how I can split a userPrincipalName from mailto:user.name@doamin.com to user.name? Essentially I want to extract everything before the '@' symbol. I attempted using this syntax however it is not working, so any pointers would be greatly appreciated. split(triggerBody()?['object']?['properties']?['owner']?['userPrincipalName'], '@') This leaves me with ["user.name", "domain.com"], how can I escape the "domain.com"Solved2.2KViews0likes3CommentspfSense syslog to Azure Sentinel Guide
I've seen various posts across the internet of people trying to get pfSense working with Azure Sentinel and I wanted to share this project I have been working on myself. I would firstly like to say thank you to some of the great open-source projects out there on GitHub which made this entire process possible. I would like to call out https://github.com/pfelk/pfelk for his awesome work with PFELK, I have used his GROK patterns which parse the pfSense data and add additional context the messages, such as GeoIP data, rule types, friendly names and more. Further information about this project and some KQL functions can be found on my https://github.com/noodlemctwoodle/pf-azure-sentinel page Configuration Ubuntu 18.04-20.04 Server onPrem 1. Install Ubuntu Server 20.04 on a Virtual Machine or Computer and update the OS sudo apt update; sudo apt upgrade -y 2. Disabling Swap - Swapping can be disabled for performance and stability. (Optional) sudo swapoff -a 3. Configuration Date/Time Zone The box running this configuration will reports firewall logs based on its clock. The command below will set the timezone to Eastern Standard Time (EST). To view available timezones type "sudo timedatectl list-timezones" sudo timedatectl set-timezone Europe/London 4. Download and install the public GPG signing key wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - 5. Download and install apt-transport-https package sudo apt install apt-transport-https 6. Add Elasticsearch|Logstash Repositories (version 7+) echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list 7. Install Java 14 LTS sudo apt install openjdk-14-jre-headless Install and Configure MaxMind (Optional) 1. Add MaxMind Repository This step is optional, you can skip this step and the configuration will default to the built-in GeoIP lookups for Elastic sudo add-apt-repository ppa:maxmind/ppa 2. Install MaxMind sudo apt install geoipupdate 3. Configure MaxMind Create a https://www.maxmind.com/en/geolite2/signup Login to your MaxMind Account; navigate to "My License Key" under "Services" and Generate a new license key 4. Add the new license keys to the configuration file sudo nano /etc/GeoIP.conf 5. Modify lines 7 & 8 as follows (without < >): AccountID <Input Your Account ID> LicenseKey <Input Your LicenseKey> 6. Modify line 13 as follows: EditionIDs GeoLite2-City GeoLite2-Country GeoLite2-ASN 7. Modify line 18 as follows: DatabaseDirectory /usr/share/GeoIP/ 8. Download Maxmind Databases sudo geoipupdate 9. Add cron (automatically updates Maxmind every week on Sunday at 1700hrs) sudo nano /etc/cron.weekly/geoipupdate 10. Add the following and save/exit 00 17 * * 0 geoipupdate Logstash Configuration 1. Install Logstash sudo apt update && sudo apt install logstash 2. Create Required Directories sudo mkdir /etc/logstash/conf.d/{databases,patterns,templates} 3. Download the following configuration files (Required) sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/01-inputs.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/02-types.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/03-filter.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/05-firewall.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/07-interfaces.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/10-apps.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/30-geoip.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/45-cleanup.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/50-outputs.conf -P /etc/logstash/conf.d/ 4. Download the following configuration files (Optional) sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/pfSense/35-rules-desc.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/36-ports-desc.conf -P /etc/logstash/conf.d/ 5. Download the grok pattern (Required) sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/patterns/pfelk.grok -P /etc/logstash/conf.d/patterns/ 6. Download the Database(s) (Optional) sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/databases/rule-names.csv -P /etc/logstash/conf.d/databases/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/databases/service-names-port-numbers.csv -P /etc/logstash/conf.d/databases/ These databases will be required if you carried out the optional step (4) 7. Configure Firewall Rule Database (Optional) Go to your pfSense GUI and go to Firewall -> Rules. Ensure the rules have a description, this is the text you will see in Azure Sentinel. Block rules normally have logging on, if you want to see good traffic also, enable logging for pass rules. Extract rule descriptions with associated tracking number In pfSense and go to diagnostics -> Command Prompt Enter the following command in the execute shell command box and click the execute button pfctl -vv -sr | grep label | sed -r 's/@([[:digit:]]+).*(label "|label "USER_RULE: )(.*)".*/"\1","\3"/g' | sort -V -u | awk 'NR==1{$0="\"Rule\",\"Label\""RS$0}7' The results will look something like this: "55","NAT Redirect DNS" "56","NAT Redirect DNS" "57","NAT Redirect DNS TLS" "58","NAT Redirect DNS TLS" "60","BypassVPN" Copy the entire results to your clipboard and past within the rule-names.csv as follows: "Rule","Label" "55","NAT Redirect DNS" "56","NAT Redirect DNS" "57","NAT Redirect DNS TLS" "58","NAT Redirect DNS TLS" "60","BypassVPN" 8. Update the Logstash configuration Go back to the server you installed Logstash. sudo nano /etc/logstash/conf.d/databases/rule-names.csv 9. Paste the results from pfSense into the first blank line after "0","null" Example: "0","null" "1","Input Firewall Description Here You must repeat step 1 (Rules) if you add new rules in pfSense and then restart Logstash 10. Update firewall interfaces Amend the 05-firewall.conf file sudo nano /etc/logstash/conf.d/05-firewall.conf Adjust the interface name(s) to correspond with your hardware the interface below is referenced as igb0 with a corresponding “WAN” and friendly name of "ISP Provider". Add/remove sections, depending on the number of interfaces. ### Change interface as desired ### if [interface][name] =~ /^igb0$/ { mutate { add_field => { "[interface][alias]" => "WAN" } add_field => { "[network][name]" => "ISP Provider" } } } Forwarding pfSense Logs to Logstash 1. In pfSense navigate to Status -> System Logs -> Settings 2. General Logging Options Show log entries in reverse order (newest entries on top) 3. General Logging Options > Log firewall default blocks (optional) Log packets matched from the default block rules in the ruleset Log packets matched from the default pass rules put in the ruleset Log packets blocked by 'Block Bogon Networks' rules Log packets blocked by 'Block Private Networks' rules Log errors from the webserver process 4. Remote Logging Options: check "Send log messages to remote syslog server" Select a specific interface to use for forwarding (optional) Select IPv4 for IP Protocol Enter the Logstash server local IP into the field Remote log servers with port 5140 (eg 192.168.1.50:5140) Under "Remote Syslog Contents" check "Everything" Install Log Analytics Plugin 1. Run the command to install the Azure Log Analytics plugin sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-azure_loganalytics 2. Configuration sudo nano /etc/logstash/conf.d/50-outputs.conf Amend the output to match your Sentinel workspace output { azure_loganalytics { customer_id => "<OMS WORKSPACE ID>" shared_key => "<CLIENT AUTH KEY>" log_type => "<LOG TYPE NAME>" } } Example: output { azure_loganalytics { customer_id => "1234567-7654321-345678-12334445" shared_key => "kflsdjkgfslfjsdf0ife0f0efe0-09f0we9f-ef-w00e-0w-f0w-0fwe-f0d0-w==" log_type => "pfsense_logstash" } } 3. Restart Logstash sudo systemctl restart logstash 4. If you run into any problems use the Logstash plain.log to troubleshoot cat /var/log/logstash/logstash-plain.log 5. Wait for data to show in Azure Sentinel Query the data 1. Using KQL we can now query the data // PFSesne GeoIp Traffic pfsense_logstash_CL | where TimeGenerated > ago(1m) | where tags_s contains "GeoIP" | project TimeGenerated, interface_alias_s, network_name_s, interface_name_s, source_ip_s, source_port_s, source_geo_region_name_s, source_geo_country_iso_code_s, source_geo_country_name_s, destination_ip_s, destination_port_s, destination_geo_region_name_s, destination_geo_country_code3_s, network_direction_s, event_action_s, event_reason_s, ruleName, destination_service_s, network_transport_s14KViews1like4CommentsRe: Grouping Azure Sentinel - Azure Active Directory Identity Protection alerts
GaryBushey- I was indeed referring to the Microsoft Security (Preview) rule. Rod_Trentas GaryBushey says you can't edit the Microsoft Security (Preview) rules like you can with scheduled rules, so this wasn't possible, unfortunately. I think I have overcome the problem now, I'm just testing it. 🙂7.8KViews1like0CommentsRe: Grouping Azure Sentinel - Azure Active Directory Identity Protection alerts
I have created a custom Scheduled rule and used this KQL to capture the same information that Identity Protection captures in the event. SecurityAlert | summarize arg_max(TimeGenerated, *) by SystemAlertId | where DisplayName has "Unfamiliar sign-in properties" | where AlertSeverity has "Low" | project SystemAlertId, Entities, ExtendedProperties | extend Entities = iff(isempty(Entities), todynamic('[{"dummy" : ""}]'), todynamic(Entities)) | extend ExtendedProperties = iff(isempty(ExtendedProperties), todynamic('[{"dummy" : ""}]'), todynamic(ExtendedProperties)) | mvexpand Entities, ExtendedProperties | evaluate bag_unpack(Entities) | evaluate bag_unpack(ExtendedProperties) | extend userName = columnifexists("User Account", "") | extend ipAddress = columnifexists("Address", "") I mapped the entities from the KQL in the rule I grouped all alerts into a single incident7.8KViews0likes6CommentsRe: Parsing XML in Azure Sentinel
CliveWatsonThank you very much with your help on this, your a legend. Here is the working solution based upon your suggestion print syslogmsg = '05:19.0Z Some-Server-Name Events - EventFwd [agentInfo@3401 tenantId="0" bpsId="0" tenantGUID="{00000000-0000-0000-0000-000000000000}" tenantNodePath="1\2"] <?xml version="1.0" encoding="utf-8"?><UpdateEvents><MachineInfo><AgentGUID>{00000000-0000-0000-0000-000000000000}</AgentGUID><MachineName>Some-Machine</MachineName><RawMACAddress>112233445566</RawMACAddress><IPAddress>1.1.2.3</IPAddress><AgentVersion>1.2.3.123</AgentVersion><OSName>Windows 41</OSName><TimeZoneBias>-10</TimeZoneBias><UserName>myName</UserName></MachineInfo><BrandCommonUpdater ProductName="Brand Agent" ProductVersion="1.0.0" ProductFamily="AVP"><UpdateEvent><EventID>1234</EventID><Severity>0</Severity><GMTTime>2020-00-00T06:41:02</GMTTime><ProductID>SomeName1999</ProductID><Locale>0001</Locale><Error>0</Error><Type>SomeCore</Type><Version>1234.0</Version><InitiatorID>SOMEAGENT3000</InitiatorID><InitiatorType>OnDemand</InitiatorType><SiteName>Some-Server-Name</SiteName><Description>N/A</Description></UpdateEvent></BrandCommonUpdater></UpdateEvents>' | parse syslogmsg with * " tenantNodePath" * " " xml | extend xml=parse_xml(xml) | extend MachineName = xml.UpdateEvents.MachineInfo.MachineName | extend IPAddress = xml.UpdateEvents.MachineInfo.IPAddress | where isnotempty(MachineName) | project MachineName, IPAddress Edit: Just to clean up the query I have made an adjustment to the solution as suggested by CliveWatson and Ofer10KViews0likes2CommentsRe: Parsing XML in Azure Sentinel
CliveWatsonYes that is whole string syslogmessge like in the Print statement.. Would it be possible for you to show me how to extract the data values after this value 05:19.0Z Some-Server-Name Events - EventFwd [agentInfo@3401 tenantId="0" bpsId="0" tenantGUID="{00000000-0000-0000-0000-000000000000}" tenantNodePath="1\2"] <?xml version="1.0" encoding="utf-8"?> I'm guessing I would need to RegEx out the above header to get to the data values below. Although I am not sure how to proceed with that? <MachineName>Some-Machine</MachineName> <RawMACAddress>112233445566</RawMACAddress> <IPAddress>1.1.2.3</IPAddress> <AgentVersion>1.2.3.123</AgentVersion> <OSName>Windows 41</OSName> <TimeZoneBias>-10</TimeZoneBias> <UserName>myName</UserName> <EventID>1234</EventID> <Severity>0</Severity> <GMTTime>2020-00-00T06:41:02</GMTTime> <ProductID>SomeName1999</ProductID> <Locale>0001</Locale> <Error>0</Error> <Type>SomeCore</Type> <Version>1234.0</Version> <InitiatorID>SOMEAGENT3000</InitiatorID> <InitiatorType>OnDemand</InitiatorType> <SiteName>Some-Server-Name</SiteName> <Description>N/A</Description> Many Thanks for your help so far 🙂9.9KViews0likes3CommentsRe: Parsing XML in Azure Sentinel
CliveWatson Would you be able to assist how I might format your example for SecurityEvent into Syslog using the message example? GaryBushey I looked at this documentation, although I dont fully understand the examples provided 😐 I also looked at this post https://www.systemcenterautomation.com/2020/01/extracting-nested-fields-kusto/ but i haven't been able to replicate the output with the data I have10KViews0likes5CommentsRe: Parsing XML in Azure Sentinel
The raw string looks like this: 05:19.0Z Some-Server-Name Events - EventFwd [agentInfo@3401 tenantId="0" bpsId="0" tenantGUID="{00000000-0000-0000-0000-000000000000}" tenantNodePath="1\2"] <?xml version="1.0" encoding="utf-8"?><UpdateEvents><MachineInfo><AgentGUID>{00000000-0000-0000-0000-000000000000}</AgentGUID><MachineName>Some-Machine</MachineName><RawMACAddress>112233445566</RawMACAddress><IPAddress>1.1.2.3</IPAddress><AgentVersion>1.2.3.123</AgentVersion><OSName>Windows 41</OSName><TimeZoneBias>-10</TimeZoneBias><UserName>myName</UserName></MachineInfo><BrandCommonUpdater ProductName="Brand Agent" ProductVersion="1.0.0" ProductFamily="AVP"><UpdateEvent><EventID>1234</EventID><Severity>0</Severity><GMTTime>2020-00-00T06:41:02</GMTTime><ProductID>SomeName1999</ProductID><Locale>0001</Locale><Error>0</Error><Type>SomeCore</Type><Version>1234.0</Version><InitiatorID>SOMEAGENT3000</InitiatorID><InitiatorType>OnDemand</InitiatorType><SiteName>Some-Server-Name</SiteName><Description>N/A</Description></UpdateEvent></BrandCommonUpdater></UpdateEvents> I have this KQL so far to at leastquery the computer and create a data table of just the Syslog message Syslog | where Computer contains "Some-Server-Name" | project SyslogMessage | extend NewField=parse_xml(SyslogMessage)10KViews0likes8Comments
Recent Blog Articles
No content to show