pfSense syslog to Azure Sentinel Guide

Brass Contributor

I've seen various posts across the internet of people trying to get pfSense working with Azure Sentinel and I wanted to share this project I have been working on myself.


I would firstly like to say thank you to some of the great open-source projects out there on GitHub which made this entire process possible. I would like to call out a3ilson for his awesome work with PFELK, I have used his GROK patterns which parse the pfSense data and add additional context the messages, such as GeoIP data, rule types, friendly names and more.


Further information about this project and some KQL functions can be found on my GitHub page




Ubuntu 18.04-20.04 Server onPrem
1. Install Ubuntu Server 20.04 on a Virtual Machine or Computer and update the OS



sudo apt update; sudo apt upgrade -y



2. Disabling Swap - Swapping can be disabled for performance and stability. (Optional)



sudo swapoff -a



3. Configuration Date/Time Zone

  • The box running this configuration will reports firewall logs based on its clock. The command below will set the timezone to Eastern Standard Time (EST).
    To view available timezones type "sudo timedatectl list-timezones"



sudo timedatectl set-timezone Europe/London



4. Download and install the public GPG signing key



wget -qO - | sudo apt-key add -



5. Download and install apt-transport-https package



sudo apt install apt-transport-https



6. Add Elasticsearch|Logstash Repositories (version 7+)



echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list



7. Install Java 14 LTS



sudo apt install openjdk-14-jre-headless​




Install and Configure MaxMind (Optional)

1. Add MaxMind Repository 

This step is optional, you can skip this step and the configuration will default to the built-in GeoIP lookups for Elastic 



sudo add-apt-repository ppa:maxmind/ppa​



2. Install MaxMind



sudo apt install geoipupdate



3. Configure MaxMind

Create a MaxMind Account

  • Login to your MaxMind Account; navigate to "My License Key" under "Services" and Generate a new license key

4. Add the new license keys to the configuration file



sudo nano /etc/GeoIP.conf



5. Modify lines 7 & 8 as follows (without < >):



AccountID <Input Your Account ID>
LicenseKey <Input Your LicenseKey>



6. Modify line 13 as follows:



EditionIDs GeoLite2-City GeoLite2-Country GeoLite2-ASN



7. Modify line 18 as follows:



DatabaseDirectory /usr/share/GeoIP/



8. Download Maxmind Databases



sudo geoipupdate



9. Add cron (automatically updates Maxmind every week on Sunday at 1700hrs)



sudo nano /etc/cron.weekly/geoipupdate



10. Add the following and save/exit



00 17 * * 0 geoipupdate




Logstash Configuration

1. Install Logstash



sudo apt update && sudo apt install logstash



2. Create Required Directories



sudo mkdir /etc/logstash/conf.d/{databases,patterns,templates}



3. Download the following configuration files (Required)



sudo wget -P /etc/logstash/conf.d/
sudo wget -P /etc/logstash/conf.d/
sudo wget -P /etc/logstash/conf.d/
sudo wget -P /etc/logstash/conf.d/
sudo wget -P /etc/logstash/conf.d/
sudo wget -P /etc/logstash/conf.d/
sudo wget -P /etc/logstash/conf.d/
sudo wget -P /etc/logstash/conf.d/
sudo wget -P /etc/logstash/conf.d/



4. Download the following configuration files (Optional)



 sudo wget -P /etc/logstash/conf.d/

sudo wget -P /etc/logstash/conf.d/



5. Download the grok pattern (Required)



sudo wget -P /etc/logstash/conf.d/patterns/



6. Download the Database(s) (Optional) 



sudo wget -P /etc/logstash/conf.d/databases/
sudo wget -P /etc/logstash/conf.d/databases/



These databases will be required if you carried out the optional step (4)


7. Configure Firewall Rule Database (Optional)

  1. Go to your pfSense GUI and go to Firewall -> Rules.
  2. Ensure the rules have a description, this is the text you will see in Azure Sentinel.
  3. Block rules normally have logging on, if you want to see good traffic also, enable logging for pass rules.
  4. Extract rule descriptions with associated tracking number

In pfSense and go to diagnostics -> Command Prompt

Enter the following command in the execute shell command box and click the execute button



 pfctl -vv -sr | grep label | sed -r 's/@([[:digit:]]+).*(label "|label "USER_RULE: )(.*)".*/"\1","\3"/g' | sort -V -u | awk 'NR==1{$0="\"Rule\",\"Label\""RS$0}7'



The results will look something like this:



"55","NAT Redirect DNS"
"56","NAT Redirect DNS"
"57","NAT Redirect DNS TLS"
"58","NAT Redirect DNS TLS"



Copy the entire results to your clipboard and past within the rule-names.csv as follows:



"55","NAT Redirect DNS"
"56","NAT Redirect DNS"
"57","NAT Redirect DNS TLS"
"58","NAT Redirect DNS TLS"



8. Update the Logstash configuration
Go back to the server you installed Logstash.



sudo nano /etc/logstash/conf.d/databases/rule-names.csv



9. Paste the results from pfSense into the first blank line after "0","null"



"1","Input Firewall Description Here



You must repeat step 1 (Rules) if you add new rules in pfSense and then restart Logstash

10. Update firewall interfaces

Amend the 05-firewall.conf file



sudo nano /etc/logstash/conf.d/05-firewall.conf



Adjust the interface name(s) to correspond with your hardware the interface below is referenced as igb0 with a corresponding “WAN” and friendly name of "ISP Provider". Add/remove sections, depending on the number of interfaces.



### Change interface as desired ###
if [interface][name] =~ /^igb0$/ {
mutate {
        add_field => { "[interface][alias]" => "WAN" }
        add_field => { "[network][name]" => "ISP Provider" }




Forwarding pfSense Logs to Logstash

1. In pfSense navigate to Status -> System Logs -> Settings

2. General Logging Options

  • Show log entries in reverse order (newest entries on top)

3. General Logging Options > Log firewall default blocks (optional)

  • Log packets matched from the default block rules in the ruleset
  • Log packets matched from the default pass rules put in the ruleset
  • Log packets blocked by 'Block Bogon Networks' rules
  • Log packets blocked by 'Block Private Networks' rules
  • Log errors from the webserver process

4. Remote Logging Options:

  • check "Send log messages to remote syslog server"
  • Select a specific interface to use for forwarding (optional)
  • Select IPv4 for IP Protocol
  • Enter the Logstash server local IP into the field Remote log servers with port 5140 (eg
  • Under "Remote Syslog Contents" check "Everything"



Install Log Analytics Plugin

1. Run the command to install the Azure Log Analytics plugin



sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-azure_loganalytics



2. Configuration


sudo nano /etc/logstash/conf.d/50-outputs.conf


Amend the output to match your Sentinel workspace



output {
        azure_loganalytics {
        customer_id => "<OMS WORKSPACE ID>"
        shared_key => "<CLIENT AUTH KEY>"
        log_type => "<LOG TYPE NAME>"






output {
        azure_loganalytics {
        customer_id => "1234567-7654321-345678-12334445"
        shared_key => "kflsdjkgfslfjsdf0ife0f0efe0-09f0we9f-ef-w00e-0w-f0w-0fwe-f0d0-w=="
        log_type => "pfsense_logstash"



3. Restart Logstash



sudo systemctl restart logstash



4. If you run into any problems use the Logstash plain.log to troubleshoot



cat /var/log/logstash/logstash-plain.log



5. Wait for data to show in Azure Sentinel



Query the data

1. Using KQL we can now query the data



// PFSesne GeoIp Traffic
| where TimeGenerated > ago(1m)
| where tags_s contains "GeoIP"
| project TimeGenerated, interface_alias_s, network_name_s, interface_name_s, source_ip_s, source_port_s, source_geo_region_name_s, source_geo_country_iso_code_s,
source_geo_country_name_s, destination_ip_s, destination_port_s, destination_geo_region_name_s, destination_geo_country_code3_s,
network_direction_s, event_action_s, event_reason_s, ruleName, destination_service_s, network_transport_s





4 Replies

@TS-noodlemctwoodle the links in section 3. Download the following configuration files (Required)

are not longer working. 

@George__Wilburn There is an updated guide on my github :) 

Eeeeeep careful sending logs plain TCP no TLS.

I know you are doing to on-prem logstash but even so, it would be nice to have a TLS solution ;)

Nice Guide @TS-noodlemctwoodle !


Does anyone know if there will be an adapter or service for Sentinal that will allow native integration for platforms that cannot natively send anything other than syslog? To have such an awesome product as Sentinal not be able to ingest one of the oldest (if not the oldest) standardized logging capabilities seems a bit unusual. And yes, I know I can stand up a vm to bridge the gap, but that really seems to be a counter cloud pattern. Paying to run an OS or event a container as a log adapter just seems very 90's.