User Profile
TS-noodlemctwoodle
Brass Contributor
Joined 5 years ago
User Widgets
Recent Discussions
pfSense syslog to Azure Sentinel Guide
I've seen various posts across the internet of people trying to get pfSense working with Azure Sentinel and I wanted to share this project I have been working on myself. I would firstly like to say thank you to some of the great open-source projects out there on GitHub which made this entire process possible. I would like to call out a3ilson for his awesome work with PFELK, I have used his GROK patterns which parse the pfSense data and add additional context the messages, such as GeoIP data, rule types, friendly names and more. Further information about this project and some KQL functions can be found on my GitHub page Configuration Ubuntu 18.04-20.04 Server onPrem 1. Install Ubuntu Server 20.04 on a Virtual Machine or Computer and update the OS sudo apt update; sudo apt upgrade -y 2. Disabling Swap - Swapping can be disabled for performance and stability. (Optional) sudo swapoff -a 3. Configuration Date/Time Zone The box running this configuration will reports firewall logs based on its clock. The command below will set the timezone to Eastern Standard Time (EST). To view available timezones type "sudo timedatectl list-timezones" sudo timedatectl set-timezone Europe/London 4. Download and install the public GPG signing key wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - 5. Download and install apt-transport-https package sudo apt install apt-transport-https 6. Add Elasticsearch|Logstash Repositories (version 7+) echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list 7. Install Java 14 LTS sudo apt install openjdk-14-jre-headless Install and Configure MaxMind (Optional) 1. Add MaxMind Repository This step is optional, you can skip this step and the configuration will default to the built-in GeoIP lookups for Elastic sudo add-apt-repository ppa:maxmind/ppa 2. Install MaxMind sudo apt install geoipupdate 3. Configure MaxMind Create a MaxMind Account Login to your MaxMind Account; navigate to "My License Key" under "Services" and Generate a new license key 4. Add the new license keys to the configuration file sudo nano /etc/GeoIP.conf 5. Modify lines 7 & 8 as follows (without < >): AccountID <Input Your Account ID> LicenseKey <Input Your LicenseKey> 6. Modify line 13 as follows: EditionIDs GeoLite2-City GeoLite2-Country GeoLite2-ASN 7. Modify line 18 as follows: DatabaseDirectory /usr/share/GeoIP/ 8. Download Maxmind Databases sudo geoipupdate 9. Add cron (automatically updates Maxmind every week on Sunday at 1700hrs) sudo nano /etc/cron.weekly/geoipupdate 10. Add the following and save/exit 00 17 * * 0 geoipupdate Logstash Configuration 1. Install Logstash sudo apt update && sudo apt install logstash 2. Create Required Directories sudo mkdir /etc/logstash/conf.d/{databases,patterns,templates} 3. Download the following configuration files (Required) sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/01-inputs.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/02-types.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/03-filter.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/05-firewall.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/07-interfaces.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/10-apps.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/30-geoip.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/45-cleanup.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/50-outputs.conf -P /etc/logstash/conf.d/ 4. Download the following configuration files (Optional) sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/pfSense/35-rules-desc.conf -P /etc/logstash/conf.d/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/36-ports-desc.conf -P /etc/logstash/conf.d/ 5. Download the grok pattern (Required) sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/patterns/pfelk.grok -P /etc/logstash/conf.d/patterns/ 6. Download the Database(s) (Optional) sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/databases/rule-names.csv -P /etc/logstash/conf.d/databases/ sudo wget https://raw.githubusercontent.com/noodlemctwoodle/pfsense-azure-sentinel/main/Logstash-Configuration/etc/logstash/conf.d/databases/service-names-port-numbers.csv -P /etc/logstash/conf.d/databases/ These databases will be required if you carried out the optional step (4) 7. Configure Firewall Rule Database (Optional) Go to your pfSense GUI and go to Firewall -> Rules. Ensure the rules have a description, this is the text you will see in Azure Sentinel. Block rules normally have logging on, if you want to see good traffic also, enable logging for pass rules. Extract rule descriptions with associated tracking number In pfSense and go to diagnostics -> Command Prompt Enter the following command in the execute shell command box and click the execute button pfctl -vv -sr | grep label | sed -r 's/@([[:digit:]]+).*(label "|label "USER_RULE: )(.*)".*/"\1","\3"/g' | sort -V -u | awk 'NR==1{$0="\"Rule\",\"Label\""RS$0}7' The results will look something like this: "55","NAT Redirect DNS" "56","NAT Redirect DNS" "57","NAT Redirect DNS TLS" "58","NAT Redirect DNS TLS" "60","BypassVPN" Copy the entire results to your clipboard and past within the rule-names.csv as follows: "Rule","Label" "55","NAT Redirect DNS" "56","NAT Redirect DNS" "57","NAT Redirect DNS TLS" "58","NAT Redirect DNS TLS" "60","BypassVPN" 8. Update the Logstash configuration Go back to the server you installed Logstash. sudo nano /etc/logstash/conf.d/databases/rule-names.csv 9. Paste the results from pfSense into the first blank line after "0","null" Example: "0","null" "1","Input Firewall Description Here You must repeat step 1 (Rules) if you add new rules in pfSense and then restart Logstash 10. Update firewall interfaces Amend the 05-firewall.conf file sudo nano /etc/logstash/conf.d/05-firewall.conf Adjust the interface name(s) to correspond with your hardware the interface below is referenced as igb0 with a corresponding “WAN” and friendly name of "ISP Provider". Add/remove sections, depending on the number of interfaces. ### Change interface as desired ### if [interface][name] =~ /^igb0$/ { mutate { add_field => { "[interface][alias]" => "WAN" } add_field => { "[network][name]" => "ISP Provider" } } } Forwarding pfSense Logs to Logstash 1. In pfSense navigate to Status -> System Logs -> Settings 2. General Logging Options Show log entries in reverse order (newest entries on top) 3. General Logging Options > Log firewall default blocks (optional) Log packets matched from the default block rules in the ruleset Log packets matched from the default pass rules put in the ruleset Log packets blocked by 'Block Bogon Networks' rules Log packets blocked by 'Block Private Networks' rules Log errors from the webserver process 4. Remote Logging Options: check "Send log messages to remote syslog server" Select a specific interface to use for forwarding (optional) Select IPv4 for IP Protocol Enter the Logstash server local IP into the field Remote log servers with port 5140 (eg 192.168.1.50:5140) Under "Remote Syslog Contents" check "Everything" Install Log Analytics Plugin 1. Run the command to install the Azure Log Analytics plugin sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-azure_loganalytics 2. Configuration sudo nano /etc/logstash/conf.d/50-outputs.conf Amend the output to match your Sentinel workspace output { azure_loganalytics { customer_id => "<OMS WORKSPACE ID>" shared_key => "<CLIENT AUTH KEY>" log_type => "<LOG TYPE NAME>" } } Example: output { azure_loganalytics { customer_id => "1234567-7654321-345678-12334445" shared_key => "kflsdjkgfslfjsdf0ife0f0efe0-09f0we9f-ef-w00e-0w-f0w-0fwe-f0d0-w==" log_type => "pfsense_logstash" } } 3. Restart Logstash sudo systemctl restart logstash 4. If you run into any problems use the Logstash plain.log to troubleshoot cat /var/log/logstash/logstash-plain.log 5. Wait for data to show in Azure Sentinel Query the data 1. Using KQL we can now query the data // PFSesne GeoIp Traffic pfsense_logstash_CL | where TimeGenerated > ago(1m) | where tags_s contains "GeoIP" | project TimeGenerated, interface_alias_s, network_name_s, interface_name_s, source_ip_s, source_port_s, source_geo_region_name_s, source_geo_country_iso_code_s, source_geo_country_name_s, destination_ip_s, destination_port_s, destination_geo_region_name_s, destination_geo_country_code3_s, network_direction_s, event_action_s, event_reason_s, ruleName, destination_service_s, network_transport_sGrouping Azure Sentinel - Azure Active Directory Identity Protection alerts
Is there a way to groupAzure Active Directory Identity Protection alerts such as "Unfamiliar sign-in properties" in Azure Sentinel? We are seeing hundreds of these alerts being raised on a daily basis and it is causing quite a lot of noise in the incidents panel of Azure Sentinel. What would be really useful is a way to group all these alerts into a single incident, however, I do not see a way to do this. Any guidance would be greatly appreciated.
