Microsoft Secure Tech Accelerator
Apr 03 2024, 07:00 AM - 11:00 AM (PDT)
Microsoft Tech Community
Azure Sentinel: The connectors grand (CEF, Syslog, Direct, Agent, Custom and more)
Published Aug 13 2019 11:53 PM 199K Views
Microsoft

(Last updated Apr 20th, 2021)

 

Please note that as the built-in list of connectors in Azure Sentinel is growing, this list is not actively maintained anymore. Refer to the Azure Sentinel connector documentation for more information. 

 

Source types

 

Built-in

Built-in connectors are included in the Azure Sentinel documentation and the data connectors pane in the product itself. Those connectors are based on one of the technologies listed below. Therefore a built-in connector will have a type: CEF, Syslog, Direct, and so forth.

 

Syslog and CEF

Most network and security systems support either Syslog or CEF (which stands for Common Event Format) over Syslog as means for sending data to a SIEM. This makes Syslog or CEF the most straightforward ways to stream security and networking events to Azure Sentinel.

 

  • Want to learn more about best practices for CEF collection? see here.
  • Want to scale CEF or Syslog collection?  Use a VM scale set as described here.

 

The advantage of CEF over Syslog is that it ensures the data is normalized, making it more immediately useful for analysis using Sentinel. However, unlike many other SIEM products, Sentinel allows ingesting unparsed Syslog events and performing analytics on them using query time parsing. 

 

The number of systems supporting Syslog or CEF is in the hundreds, making the table below by no means comprehensive. We will update this list continuously. The table provides links to the source device's vendor documentation for configuring the device to send events in Syslog or CEF.

 

Tip: Want to ingest test CEF data? here is how to do that.

 

Direct

Most Microsoft cloud sources and many other clouds and on-prem systems can send to Azure Sentinel natively. For Microsoft Azure sources, this often uses their diagnostics feature, on which you can read more here.

 

Agent

The Log Analytics agent can collect different types of events from servers and endpoints listed here. To learn more about the agent, read Azure Sentinel Agent: Collecting telemetry from on-prem and IaaS server.

 

Threat Intelligence (TI)

You can use one of the threat intelligence connectors:

  • Platform, which uses the Graph Security API
  • TAXII, which uses the TAXII 2.0 protocol

to ingest threat intelligence indicators, which are used by Azure Sentinel's built-in TI analytics rules, and to build your own rules. You can read more about the Threat Intelligence connectors in module #6 of the Azure Sentinel Ninja Training 

 

Custom: Logic Apps, Logstash, Azure Functions, and others

In addition to CEF and Syslog, many solutions are based on Sentinel's data collector API and create custom log tables in the workspace. Those belong to 3 groups:

  • Sources that support Logstash, which in turn has an output plug-in that can send the events to Azure Sentinel.
  • Sources that have native support for the API.
  • Sources for which there is a community or Microsoft field created solution that uses the API, usually using Logic Apps or an Azure function.

You can read more about custom connectors here.

 

Automation and integration

While all the types above focused on getting telemetry into Azure Sentinel, connectors marked as automation/integration enable Azure Sentinel to implement other use cases such as sending information to another system or performing an action on another system. Those might be API-based on integration or Logic App-based integrations. 

 

The Grand List

 

Vendor

Product

Connector
Type

Connecting and using

Agari Phishing Defense and Brand Protection Built-in (Function, Graph Security API) Instructions
AI Vectra Detect Built-in (CEF) Instructions
Akamai   Built-in (CEF) Instructions

Alcide

kAudit

Built-in (API)

Instructions

AlgoSec

ASMS

CEF

Instructions and examples

Anomali

Limo

Built-in (TAXII)

Instructions

Anomali

ThreatStream

Built-in (TI Platform)

Instructions

Anomali

Match

Integration

Overview and instructions

Apache

httpd

Built-in (Agent custom logs)

Instructions

Also, read using rsyslog or logger as a file forwarder for an alternative method.

Apache

Kafka

Logstash

See Logstash plug-in. Use to get events sent using Kafka, not for Kafka's own audit events.

Aruba

ClearPass

CEF

Instructions

AT&T Cyber

AlienVault OTX

TI (Platform)

Using Logic Apps, See instructions

AWS

CloudTrail

Built-in

Sentinel built-in connector

AWS

CloudTrail S3 logs

Custom

Using an Azure Function. See here.

Using an AWS Lambda Function. See here.

AWS

CloudWatch

Logstash

See Logstash Plug-in.

AWS

Kinesis

Logstash

See Logstash Plug-in.

AWS

Object Level S3 Logging

Logstash 

See here.

AWS

Security Hub

Custom

Azure Function. See here.

Barracuda

WAF

Built-in (API)

Instructions

Barracuda

CloudGen Firewall

API

Sentinel built-in connector

BETTER Mobile

Threat Defense

Built-in (API)

Instructions

Beyond Security

beSECURE

Built-in (API)

Instructions

Carbon Black

Cloud Endpoint Standard (Cb Defense)

Built-in (Function)

Syslog

Sentinel built-in connector 

 

Instructions

Carbon Black

(Cb Response)

Syslog

Instructions

Checkpoint   CEF

Sentinel Built-in connector

Cisco ACS Syslog

Instructions

Cisco ASA Cisco (CEF)

Sentinel built-in connector

Notes:

- Cisco ASA support uses Sentinel's CEF pipeline. However, Cisco's logging is not in CEF format.

- Make sure you disable logging timestamp using "no logging timestamp". See here for more details.

Cisco Cloud Security Gateway (CWS) CEF Use the Cisco Advanced Web Security Reporting.
Cisco FTD Cisco (CEF) FTP Platform logs are compatible with ASA logs and can use the same connector (see here).
Cisco IOS Syslog Instructions
Cisco ISE  (NAC) Syslog Instructions
Cisco Web Security Appliance (WSA) CEF Use the Cisco Advanced Web Security Reporting.

Cisco

Meraki

Syslog

Instructions

Event Types and Log Samples

Cisco eStreamer CEF

Using enCore

Cisco Firepower Threat Defense

CEF

Syslog

Using eStreamer enCore

Instructions, Event reference

Cisco FireSight

CEF

Using eStreamer enCore

Cisco IronPort Web Security Appliance Syslog

Instructions

Cisco Nexus Syslog

Instructions

Cisco Umbrella Built-in (Function)

Instructions

Also, see this blog post

for a custom solution

Cisco Unified Computing System (UCS) Built-in (Syslog)

Instructions

Cisco Viptela SD-WAN Syslog

Instructions

Citrix Analytics Built-in (Direct)

Instructions

Citrix NetScaler  Syslog

Instructions

Message format

Citrix NetScaler App FW Built-in (CEF) Instructions

Clearswift

Web Security Gateway

Syslog

Instructions

Cloudflare

 

 

Use Cloudflare Logpush to send to storage and a custom connector to read events from storage (for example, reading AWS S3 buckets).

Cribl

LogStream

Direct

Instructions

CrowdStrike

Falcon

CEF

Instructions. Use a SIEM connector installed on-premises.

CyberArk

Endpoint Privilege Manager (EPM)

Syslog

Logstash

Instructions (for both)

CyberArk

Privileged Access Security (PTA)

CEF

Instructions

Message format

Darktrace

Immune

CEF

See announcement. Contact vendor for instructions.

Digital Guardian

 

CEF

3rd party instructions

DocuSign

Monitor

Custom

See this blog post

Duo Security

 

CEF

Using Duo LogSync

Extrahop

Reveal

Built-in (CEF)

Instructions

F5

ASM (WAF)

Built-in (CEF)

Instructions

F5

BigIP (System, LTM, AFM, ASM, APM, AVR)

Built-in (Direct)

Instructions 

Fastly

WAF Custom

See this blog post (Logic Apps or Azure Function)

Forcepoint

Web Security (WebSense) CEF

Instructions

Detailed reference

Forcepoint

CASB CEF

Sentinel built-in connector

Forcepoint

DLP Direct

Sentinel built-in connector

Forcepoint

NGFW CEF

Sentinel built-in connector

Forescout

CounterAct CEF

Instructions

Fortinet

  CEF

Sentinel built-in connector

Log message reference

CEF mapping and examples

Fortinet

FortiSIEM

CEF

Instructions

Fortinet

FortiSOAR

Integration

Instructions

GitHub

 

Custom

See connector, rules, and hunting queries 

here

GCP

Cloud Storage

Logstash

See Plug-in. Use to get events stored in GCP Cloud Storage, not for Cloud Storage own audit events.

GCP

Pub/Sub

Logstash

See Plug-in. Use to get events sent using Pub/Sub, not for Pub/Sub own audit events.

GCP

Stacdriver

Logstash

 

Custom

Through GCP Cloud Storage or GCP Pub/Sub as described above. 

Using GCP Cloud Function. See here.

Group-IB

 

Custom (TI Platform)

Using Logic Apps. See instructions

GuardiCore

Centra

CEF

Contact vendor for instructions

HP

Printers

Syslog

Instructions

IBM

iSeries

CEF

See here.

IBM

QRadar events

Syslog

Forward raw events or correlation events in raw, parsed, or JSON format. See instructions.

IBM

QRadar offenses

Custom (Function)

Blog post

IBM

X-Force

TI (TAXII)

Instructions

IBM

zSecure

CEF

See What's new for zSecure V2.3.0

Note that it supports alerts only.

Illusive 

Attack Management System

Syslog

Sentinel built-in connector

Imperva

SecureSphere

CEF

Instructions

Infoblox NIOS

Built-in (Syslog)

Instructions

InSights  

TI (TAXII)

TAXII Instructions and related workbook

Jamf Pro

Syslog

Instructions

Juniper ATP

CEF

Instructions

Juniper JunOS based devices

Built-in (Syslog)

Instructions

Kaspersky Security Center  CEF Instructions

ManageEngine

AD Audit Plus

CEF

Instructions (use ArcSight instructions)

ManageEngine

Exchange Reporter Plus

Syslog

Instructions

McAfee

ePO

Syslog

Instructions (Note: TLS only (requires rsyslog TLS configuration)

McAfee

MVISION EDR

Syslog

Instructions

McAfee

Web Gateway

CEF

Instructions

Microfocus

Fortify AppDefender

CEF

Instructions (require authentication; contact vendor for further details).

Microsoft

Active Directory

Agent

Most AD events are logged as part of security events. 

Also, See in this list:

  • LDAP auditing
  • SMBv1 auditing

Microsoft

Advanced Threat Protection (ATA)

CEF

Microsoft

Azure Active Directory (AAD)

Built-in (Diagnostics)

Microsoft

Azure Active Directory Domain Services

Diagnostics

Microsoft

Azure Active Directory Identity Protection

 

Microsoft

Azure

Azure Activity

Azure Subscriptions

Azure Management Groups

Direct

Microsoft

Application Insights

Direct

Microsoft

App Services & Web Application monitoring 

Direct

Instructions and reference architecture 

Microsoft

Azure B2B

Direct

Included as part of AAD events

Microsoft

Azure B2C

Direct

collect B2C logs from your B2C tenant to your primary tenant AAD logs as described here

Microsoft

Azure Cosmos DB

Direct

Instructions

Microsoft

Azure Data Lake Gen 1

Direct

Microsoft

Azure Data Factory

Direct

Instructions

Microsoft

Azure Databricks

Direct

Instructions

Microsoft

Azure DDOS

Built-in (diagnostics)

Microsoft Azure Defender  and Azure Security Center (ASC)

Direct

Microsoft

Azure Defender for IoT

Built-in (Direct)

Microsoft

Azure DevOps

Direct

Instructions

Microsoft

Azure Event Hub (subscription)

Logstash

See Logstash Plug-in. Use to get events sent using an Event Hub, not for Event Hub own audit events.

Microsoft

Azure Files

Direct (Diagnostics)

Instructions

Schema information

Microsoft

Azure Firewall

Built-in (diagnostics)

Microsoft

Azure Front Door

Direct

Instructions
Microsoft Azure Key Vault (AKV)

Built-in (Diagnostics)

Connect:

Use:

Microsoft Azure Information Protection (Classic and Unified Labeling)

Built-in (Direct)

Instructions
Microsoft Azure Kubernetes Service (AKS)

Direct

Microsoft Azure Log Analytics

Direct

Collect query auditing and other metrics: Instructions
Microsoft Azure Logic Apps

Direct

Instructions
Microsoft Azure Network Security Groups (NSG)

Direct

Microsoft Azure SQL

Built-in (diagnostics)

Microsoft Azure SQL Managed Instance

Direct

Instructions
Microsoft Azure Site Recovery

Direct

Instructions
Microsoft Azure Storage

Direct

Instructions

Blog: Blob and File Storage Investigations

Microsoft Azure Storage Content

Custom (Azure Function)

Ingest the content of Azure Storage Blobs. See GitHub.
Microsoft Azure Synapse

Direct

Instructions
Microsoft Azure Web Application Firewall (WAF)

Built-in (Diagnostics)

Microsoft

BitLocker / MBAM

Agent

Using Windows Event collection. Blog post

Microsoft

Cloud App Security (Alerts, Discovery logs)

Built-in (Direct)

Microsoft

Cloud App Security (Activity Log)

CEF

Instructions

Microsoft

Defender for Office

Built-in

Custom

 

 

 

For AIRs alerts: instructions

For other alerts: Use Either a Logic App or an Azure function custom connector. For the Azure Function connector, query for RecordType_d == "28", "41" or "47" .

Microsoft

Defender for Identity (Azure ATP) Alerts

Built-in

Microsoft

Defender for Identity (Azure ATP) Events

CEF

Microsoft

Desktop Analytics

Direct

Connect

Microsoft

DNS

Agent

Sentinel built-in connector

Microsoft

Dynamics 365

Built-in

Sentinel built-in connector

Microsoft

Dynamics (not 365)

Agent

Using IIS logs

Using Dynamics Trace Files

Microsoft

IIS

Agent

Instructions

Microsoft

Intune

Direct

Connect

Use cases

Microsoft

LDAP (Windows Server)

Agent

Configure AD diagnostics logging and set "16 LDAP Interface Events" to 2 or above.

Microsoft

Office 365 (Exchange, SharePoint, OneDrive, DLP Alerts)

Built-in

 

Sentinel built-in connector

For details about DLP alerts, read here

Microsoft 

Office 365 (Microsoft Defender for Office; formerly Office ATP, PowerBI, Yammer, Sway, Forms, eDiscovery, and others)

Custom (Azure Function, Logic Apps)

Use Either a Logic App or an Azure function custom connector

Microsoft

Office 365 e-mail trace logs

Custom (Logic Apps)

See Blog Post.

Microsoft

PowerBI Embedded

Direct (Diagnostics)

Instructions

Microsoft

SMBv1 (Windows Server)

Agent

See Enable Auditing on SMB Servers, and the CmdLet reference 

Microsoft

Teams (Call Logs)

Custom

Using Logic Apps

Microsoft

Teams (Management Activity)

Built-in

Microsoft

Teams Shifts

Custom

Use Either a Logic App or an Azure function custom connector. For the Azure Function connector, query for RecordType_d == "73"

Microsoft

SCCM

Agent

Instructions

Microsoft

SQL Server

Agent

Instructions, parser, rules, and hunting queries

You can also audit at the engine level.

Microsoft

Sysmon

Agent

Using Windows Event collection. Blog post

Microsoft

Windows (Security Events)

Agent

Microsoft

Windows (Other Events, Sysmon)

Agent

Instructions

Microsoft

Windows network connections

Agent

VM Insights

Wire Data

Microsoft

Windows Firewall

Agent

Sentinel built-in connector

Microsoft

Windows Virtual Desktop

Direct

Mimecast

 

Agent

Announcement. For technical instructions, contact the vendor.

Minerva Labs

 

CEF

Please ask the vendor for instructions.

MISP

 

TI (Platform)

Sentinel built-in connector

NetApp

ONTAP

Syslog

Instructions

Note that those are management activity audit logs and not file usage activity logs.

Netflow

 

Logstash

Use the Netflow codec plug-in

Nexthink

 

CEF

Instructions

Nozomi

Guardian

CEF

Contact vendor for details

NXlog

 

Direct

Instructions

Okta

SSO

Built-in (Function)

Instructions

One Identity

Safeguard

Built-in (CEF)

Instructions

Oracle

Cloud (OCI)

Custom (Azure Function)

Available Here

Oracle

DB

Syslog

Instructions

Orca

 

Built-in (API)

Instructions

OSSEC

 

CEF

Instructions

Pager Duty

 

Automation (Playbook)

Blog post

Palo Alto

Cloudgenix

Syslog

Instructions

Palo Alto

Minemeld

TI (Platform)

Sentinel built-in connector

Palo Alto

PanOS

CEF

Sentinel built-in connector

Palo Alto

Panorama

CEF

Instructions

Palo Alto

Prisma

Syslog

Custom

Instructions, Fields

Logic Apps using a Webhook and clarification

Palo Alto

Traps through Cortex

Syslog

Instructions

Notes:

- Require rsyslog configuration to support RFC5424

- TLS only (requires rsyslog TLS configuration)

- The certificate has to be signed by a public CA

Palo Alto

XDR

CEF

Instructions

Palo Alto

XSOAR

Integration

Forward Azure Sentinel incidents to Palo Alto XSOAR 

Perimeter 81

 

Built-in (API)

Instructions

Ping Identity

Federate

CEF

Instructions

Ping Identity

Provisioner

CEF

Instructions

Postgress DB Syslog, Windows Event log

Instructions

Proofpoint On Demand Built-in (API)

Instructions

Proofpoint TAP Built-in (Function)

Instructions

Pulse Connect Built-in (Syslog)

Instructions

Qualys VM Built-in (Function)

Instructions

Radware Cloud WAF Logstash

Instructions

RedHat OpenShift Syslog
API

Instructions for Syslog
Fluentd Log Analytics plugin for API

RedHat Azure OpenShift Syslog
Custom

Instructions for Syslog
Fluentd Log Analytics plugin for API

RiskIQ   Action (Logic Apps)

Azure Logic-Apps built-in connector

Salesforce Service Cloud Built-in (Function)

Instructions

SAP Hana Syslog

Instructions (requires an SAP account)

SentinelOne   CEF

Please consult the vendor for instructions

SNMP   Syslog

Instructions

Snort   Agent

Instructions

SonicWall   CEF

Instructions

Make sure you:
- Select local use 4 as the facility.

- Select ArcSight as the Syslog format.

Sophos Central CEF Instructions. Note that the script provided by Sophos has to be scheduled using a cron job, which is not documented on the reference page.
Sophos XF Firewall Built-in (Syslog) Instructions
Squadra  secRMM Built-in (API) Instructions
Squid Proxy  

Built-in (Agent)

Syslog

Instructions

 

Configure access logs with either the TCP or UDP modules. Sentinel's built-in queries use the default log format.

Symantec

DLP

Syslog

CEF

Instructions. Note that only UDP is supported

Instructions. Uses response automation.

Symantec

ICDX

Built-in (API)

Instructions

Symantec

Proxy SG (Bluecoat)

Built-in (Syslog)

Instructions

Symantec   Endpoint Protection Manager Syslog Instructions  
Symantec Cloud Workload Protection API Instructions
Symantec VIP Built-in (Syslog) Instructions
TheHive  

Integration

Send new incidents to TheHive

Thinkst Canary

Syslog

Instructions

ThreatConnect  

TI (Platform)

Sentinel built-in connector

ThreatQuotient  

TI (Platform)

Sentinel built-in connector

Thycotic Secret Server

CEF

Instructions

TitanHQ WebTitan Cloud

Syslog

Instructions

Trend Micro  

CEF

Using Control Manager

Using LogForwarder

Trend Micro Apax Central (Cloud and On-prem)

CEF

Instructions

Trend Micro Deep Security

CEF

Sentinel built-in connector

Tufin SecureTrack

Syslog

Instructions

Varonis

DatAlert

CEF

Instructions

WatchGuard   CEF Instructions
Zimperium  
Mobile Threat Defense Built-in (API) Instructions 
zScaler Internet Access (ZIA) Built-in (CEF) Instructions
zScaler Private Access (ZPA) Logstash Use LSS. Since LSS sends raw TCP but not Syslog, you will have to use Logstash and not Azure Sentinel's native connector. 
Zoom   Custom Using Azure Function. See blog post.

 

78 Comments
Copper Contributor

Is Azure sentinel planning on Normalising ingested logs? Other players in this space are normalising ingested logs (see Elastic Common Schema) and CEF being a legacy example. Is the Azure Sentinel Team planning on defining a normalised data model for ingested Azure and legacy logs ? This would make querying data sets a lot simpler.

 

At the moment logs are disparately sprayed across different log Analytics workspaces tables (this might be the wrong name):

SignInLogs -- AAD logs

AzureDiagnostics - SQL PaaS logs

SecurityEvent - Windows server logs - Split across windows and

Unix VM logs - Syslog

 

Otherwise if MS team can provide some guidance per Azure service and where the logs are recorded and how you can link or query across these unique Log Analytics tables?

 

Thanks in advance for your assistance. 

 

 

Copper Contributor

The last two Fortinet links are dead.

Microsoft

@arvkris : fixed. I hope they don't change their links again...

Copper Contributor

Can a single Syslog/CEF server be used to stream CEF and syslog data sources?

Microsoft

@Chi_Duong : Yes, but it would require direct edit to the agent and syslog daemon configuration files.

 

Update (Dec 26th 2019): You no longer need to directly edit the configuration files:

  1. Install the CEF connector VM using the instructions in the connector page.
  2. Configure the facilities & priorities that you want to get Syslog messages of using Settings -> Workspace Settings -> Advanced Settings -> Data -> Syslog
  3. Make sure that the facility/priority combination used by your CEF source is not configured for Syslog collection

That’s it. If #3 is not doable, we will have to revert to config file editing on the VM.

Copper Contributor

 

*NOTE* We already have a support case with the vendor (Fortinet) but so far all we've got is "we cannot help you now, we have only tested this out on virtual appliances". *NOTE*

 

Is there any way to change the "default query" of a connector?

 

We have a bunch of physical FortiGate appliances, from whcih logshipping in CEF format to Sentinel works fine (We can see the entries in CommonSecurityLog) but they're not logged as "Fortinet" per se;

 

An example log post:

`Oct 24 14:27:07 DEVICE_HOSTNAME CEF: 0|Fortinet|FortiGate-300E|6.0.5,build0268 (GA)|0000000013|forward traffic close|5|start=Oct 24 2019 14:27:07 logver=60 deviceExternalId=FG....`

 

However, the Fortinet connector says "not connected".

clipboard_image_0.png

 

 

Our guess is because Sentinel is looking for something like this (as one of the example queries):

 

clipboard_image_1.png

... where DeviceProduct == “Fortigate” …
We assume the culprit is that it’s looking for “Fortigate”, not a wildcard “Fortigate*”, and the Fortinet physical appliances report their model as Fortigate-$MODEL.

 

So.. can we somehow change the “default query” for the connector to either search for “Fortigate*” or simply remove the “where DeviceProduct == “Fortigate”” clause completely?

 

Thank you in advance.

 

Microsoft

@arvkris : we are aware of this bug and are working to resolve. As you mentioned, it affects only the connector page.

Copper Contributor

Hi,

 

We have a Fortigate, we can see on TCPDump that logs are received by syslog deamon and forwarded to sentinet agent on port 25226.

On log analytics we see that logs are arriving, with the correct format:

 

0|Fortinet|Fortigate|v6.2.0|00013|traffic:forward deny|3|

 

but the connector of Fortinet isn't showing any received log. 

 

we are facing the same issue as the @arvkris, and we think this is a parsing issue.

 

@Ofer_Shezaf  is this bug that you mention corrected?

Microsoft

@hpinto

 

I think @arvkris's challenge was somewhat different

  • In his case, the second "Fortigate" (bolded in your example) was different and we missed on identifying it as Fortigate.
  • In your case, if I understand correctly, you get the information as CEF rather than parsed in the workspace. 

 

To that end, you see the value "0|Fortinet|Fortigate|v6.2.0|00013|traffic:forward deny|3|" in which field in which table?

 

~ Ofer

Copper Contributor

Hello,

 

Have Infoblox DNS Query/Response logs been tested with Azure Sentinel ?

I am trying to test it, so far i found the following :

 

1.  Infoblox DNS seems to generate only Threat Logs in CEF. The other logging categories, such as DNS Queries/Responses, are logged in some non-CEF format over syslog, like the following:

#<166>Dec 23 12:54:05 infoblox1.localdomain named[12821]: client @0x7fbc3c0cc6e0 192.168.80.1#57296 (server1.fwd1): query: server1.fwd1 IN A + (192.168.80.200)

 

I am not even seeing these logs in the Sentinel Workspace. The logs arrive at the Syslog Agent and get forwarded to omsagent process over port 25226, but beyond that i don't see them anywhere

 

Please advise:

1. Should we create a custom parser for Infoblox query/response logs or Microsoft has already addressed them  ?

2. How to troubleshoot logs processing and ingestion after the logs are delivered from the syslog daemon to the omsagent daemon? Any troublehsoot files or tables to look into ?

3. By having a vendor connector listed in Azure Sentinel connector list, such as ASA, Fortigate, .., does this mean having "parser" in the background ? the thing is all such vendor connectors do query the CommonSecurityLog with filter of "device vendor" , so i don't fully understand the technical meaning of "having an xx vendor connector"

 

Thanks in advance.

 

Microsoft

@majo1 :

 

First to your specific challenge: since the events are Syslog, they require setting up the Syslog connector rather than, or in addition to, the CEF connector. As things are now, the Syslog messages are rejected.

 

To have a single connector VM support both CEF and Syslog:

  1. Install the CEF connector VM using the instructions in the connector page (the new procedure in case yours was setup before October).
  2. Configure the facilities & priorities that you want to get Syslog messages of using Settings -> Workspace Settings -> Advanced Settings -> Data -> Syslog
  3. Make sure that the facility/priority combination used by your CEF source is not configured for Syslog collection

 

That’s it. If #3 is not doable, we will have to revert to config file editing on the VM.

 

As to your question:

  • You will need custom parsers as described in the custom connector blog post.
  • A troubleshooting script is available for CEF. For Syslog I suggest working with support.
  • Having a connector listed in the connector page implies parsing, however most of them are CEF, which means parsed as sent. This does not hold true for the list here.
Copper Contributor

Hi @Ofer_Shezaf 

 

In our case our Fortigate send syslog message in CEF Format, we have installed Azure Onboard Agent and CEF Connector on Linux Machine.

 

On Log Analytics, we can see that the Fortigate logs are arraiving.

 

Syslog Message: 0|Fortinet|Fortigate|v6.2.0|00013|traffic:forward deny|

Facility: local4

Process Name: CEF

Type: syslog

 

When we go to Data Connectors (Fortinet) we din't see anything last receiving log, on CEF Connector either too.

 

THis is the only device that we send syslog with CEF Format.

Microsoft

@hpinto : 

 

I assume you also enabled, or at least modified the Syslog facilities as described in my response to @majo1 above. If the facilities include local4, you will receive the CEF message *also* in the Syslog table. To avoid this you need to make sure that CEF events use a facility which is not configured for Syslog. for Fortinet use:

config log settings
set facility <facility_name>
end

 

This still leaves the question of why you did not get a CEF copy. Did you go through the steps here: https://docs.microsoft.com/en-us/azure/sentinel/connect-common-event-format#step-3-validate-connecti...?

Copper Contributor

Hi @Ofer_Shezaf 

 

Yes we did that seps on CEF connector, this is why we comment post, because we can't put the CEF working, its frustanting, because we OMS Agent says that collects logs on 25256.

 

The events are observed by the CEF Troubleshooter.

 

Security-config-omsagent.conf contains rsyslog.d routing configuration
rsyslog daemon configuration was found valid.
Trying to restart syslog daemon
Restarting rsyslog daemon - 'sudo service rsyslog restart'
Redirecting to /bin/systemctl restart rsyslog.service
rsyslog daemon restarted.
This will take a few seconds.
Omsagent restarted.
This will take a few seconds.
Incoming port grep: 0.0.0.0:514
tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN
udp 0 0 0.0.0.0:514 0.0.0.0:*

Daemon incoming port 514 is open
Incoming port grep: 25226
tcp 0 0 127.0.0.1:25226 0.0.0.0:* LISTEN

Omsagent is listening to incoming port 25226
Validating CEF\ASA into rsyslog daemon - port 514
This will take 60 seconds.
sudo tcpdump -A -ni any port 514 -vv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
15:30:50.745647 IP (tos 0x0, ttl 64, id 55478, offset 0, flags [DF], proto TCP (6), length 1335)
10.35.72.145.13129 > 10.35.72.147.shell: Flags [P.], cksum 0x7dcb (correct), seq 24964634:24965917, ack 15089686, win 229, options [nop,nop,TS val 1370415842 ecr 324117405], length 1283
E..7..@.@...
#H.
#H.3I...|....@.....}......
Received CEF\ASA message in daemon incoming port.[514]
Notice: To tcp dump manually execute the following command - 'tcpdump -A -ni any port 514 -vv'
Fetching CEF messages from daemon files.

 

Then we need to add to the DataConnectos -> syslog -> add syslog facility, or otherwise the log don't appear the message on logAnalytics.

 

On Fortinet we can only specified facility as syslog, alert, auth, kernel and Local0, etc... we have specified the facility Syslog facility.

 

This is a parsing issue, because the message is send is syslog, and sentinel read the CEF, and map as Process Name: CEF.

 

But on data connectors we din't see any green connector to CEF or Fortinet.

 

 

 

Copper Contributor

My mistake i din't attach the tcpdump of OMS Agent:

 

udo tcpdump -A -ni any port 25226 -vv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
15:30:58.743394 IP (tos 0x0, ttl 64, id 61856, offset 0, flags [DF], proto UDP (17), length 904)
127.0.0.1.35443 > 127.0.0.1.25226: [bad udp cksum 0x0188 -> 0x84d8!] UDP, length 876
Received CEF message in agent incoming port.[25226]
Notice: To tcp dump manually execute the following command - 'tcpdump -A -ni any port 25226 -vv'

 

on logs analytics we can only see message when we put data connector facility as syslog, other wise we din't see nothing as Syslog message os CEF Message.

 

Here is a TCP Dump 

 

127.0.0.1.35443 > 127.0.0.1.25226: [bad udp cksum 0x0138 -> 0xbaba!] UDP, length 796
E..8v.@.@..0.........sb..$.8<190>Dec 26 16:04:23 xxxx-xxx CEF: 0|Fortinet|Fortigate|v6.2.0|28704|utm:app-ctrl app-ctrl-all 

 

on logs analytics

 

ProcessName: CEF

SyslogMessage: 0|Fortinet|Fortigate|v6.2.0|0001

Facility: Syslog

 

Witch facility did MS recommend for this to work?

Microsoft

@hpinto : I think that a support ticket might be a better option to resolve this. One thing I did notice in the data you sent is that it seems that rsyslog forwards on UDP 25226 while the default (new) configuration for the OMS agent is to listen to TCP 25226.

Copper Contributor

 

 

Microsoft

@majo1 : your comment came out empty.

Copper Contributor
Hey Ofer, Is there any way to change the OMS agent to listen for syslog traffic on a different port ie;6514 for syslog-TLS I can't seem to find the configuration change for that even after configuring my rsyslog.conf file to listen on that port and recieve packets. Any ideas? Thanks, US
Copper Contributor

@Ofer_Shezaf going back to the dual CEF/Syslog server. How should the configuration files look? (assuming rsyslog)

security-config-omsagent.conf - should they have both entries for syslog / cef?

local4.debug @127.0.0.1:25226         (should this be over 25224 for syslog?)

:rawmsg, regex, "CEF\|ASA" ~
*.* @@127.0.0.1:25226"

 

security_events.conf - should this have both entries for syslog / cef as well?'

syslog:

<source>
type syslog
port 25224
bind 127.0.0.1
protocol_type tcp
tag oms.security
format /^(?<time>(?:\w+ +){2,3}(?:\d+:){2}\d+):? ?(?:(?<host>[^: ]+) ?:?)? (?<ident>[a-zA-Z0-9_%\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?: *(?<message>.*)$/
message_length_limit 4096
</source>

cef:

<source>
type syslog
port 25226
bind 127.0.0.1
protocol_type tcp
tag oms.security
format /(?<time>(?:\w+ +){2,3}(?:\d+:){2}\d+|\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[\w\-\:\+]{3,12}):?\s*(?:(?<host>[^: ]+) ?:?)?\s*(?<ident>.*CEF.+?(?=0\|)|%ASA[0-9\-]{8,10})\s*:?(?<message>0\|.*|.*)/
<parse>
message_format auto
</parse>
</source>

 

Thanks in advance,

Chi

Microsoft

@UnixStricken : I would modify the install script by changing daemon_default_incoming_port to your desired port.

Copper Contributor

@Ofer_Shezaf We can't get the omsagent to accept syslog messages from Meraki, getting "pattern not match" errors in omsagent.log. It appears that the agent is attempting to match <ident> to "CEF" or "ASA". If we exclude the <ident> part of the regex we get a nil error, if we choose anything else in the log message as <ident> we get an error saying "failed to find ident: {string}". Do you have any idea how we can get the omsagent to accept the raw syslog messages? We can't find any clear documentation regarding this unfortunately...

 

Many thanks in advance!

Microsoft

@wadstromdev : I think that the events are treated as CEF events, i.e. sent to port 25226 instead of 25224 by the Syslog daemon. I suggest opening a support ticket to help resolve this configuration issue.

Copper Contributor

How do we normalize these logs? No documentation on a very important topic.

Microsoft

Hi @josephabraham : This would depend on the source. CEF sources are parsed and normalized at the source. For Syslog sources, see the section on parsing in the custom connectors blog post.

Brass Contributor

The McAfee ePO "Instructions" link 404s, this link should fix it "https://docs.mcafee.com/bundle/epolicy-orchestrator-5.9.x-product-guide/page/GUID-5C5332B3-837A-4DDA..."

 

The Okta Logstash input has been deprecated and replaced by a newer version using the System Log API (written by the same author) - https://rubygems.org/gems/logstash-input-okta_system_log

 

Can also add Algosec:

Vendor

Product

Connector

Information

Algosec

ASMS

CEF

AFA instructions

FireFlow instructions

 

 

Microsoft

Thanks @pemontto for the updates! post updated.

Copper Contributor

Is there a Workday integration in progress ?

Copper Contributor

New to Sentinel, but see Juniper firewalls are a notable omission, is everyone just using CEF?

Copper Contributor

I am trying to integrate Trend micro Inter message scan which does not have default data connector for sentinel, while configuring syslog for trend micro still data not sent to Azure log analytics work space for Sentinel.

 

Log forwarder deployed on-prem configured as per MS guidance for syslog and cef on same machine, please guide what could be next step bcz using same server Cisco and Paloalto log forwardin working Fine.

 

Guidance will be really appreciated.  

Microsoft

@Shoaib365 : since this issue is bound to require deeper look into your system I think that a support ticket is the best route.

Copper Contributor

I see that SentinelOne says "Please consult vendor for instructions," so I've reached out to support.  Is there anyone who is using ingesting from SentinelOne's EDR into Azure Sentinel that would be able to discuss what you are seeing and any workflows you may have behind it?  Thanks!

Brass Contributor
On Sentinel console we can see number of connectors available increased to 54.
can you also update this list here as well. Do we have SAP log supported and connectors for Vulnerability solution like Nessus, Qualys 
Microsoft

@Dev_Choudhary : list updated. Qualyis is already there in the 54.... We are looking into SAP And Nessus but do not have an ETA.

Iron Contributor

great content @Ofer_Shezaf  have shared with my LinkedIn Network

Brass Contributor

Thanks @Ofer_Shezaf 

Copper Contributor

Hi again @Ofer_Shezaf 
I may be missing something obvious here but how does the IronPort Instructions link which points to the Splunk implementation of the Cisco WSA, help with integrating IronPort to Sentinel?   

CiscoIronPort Web Security ApplianceSyslog

Instructions

Regards - Col.

Microsoft

@Col_Sanders : Thanks for pointing this out. I am not sure whether the Splunk page has changed or that I was hallucinating in the first place. Anyways, I updated it to a Cisco documentation page.

Brass Contributor

Do we have Sentinel connector for Juniper and Box in pipeline.

Microsoft

@Dev_Choudhary : Juniper supports Syslog. I have now added it to the list above. Box would require a custom connector. A a workaround see: 

Ingest Box.com activity events via Microsoft Cloud App Security into Azure Sentinel

Brass Contributor

Thanks @Ofer_Shezaf  for your response.

For Box, I can use Logstash to retrieve and send event to Sentinel. 

For Juniper can you please confirm is it directly supported or do we need to first configure rsyslog to get and write events to a file and than configure OMS agent to read this file.

Also if it directly supported can we expect it will also do parsing ?

Copper Contributor

@Ofer_Shezaf Any plans to make syslog integration easier using an agent alone and removing the need for a syslog server.

 

"If you want to send data from a TCP or UDP source such as syslog, use the Splunk Universal Forwarder to listen to the source and forward the data to your Splunk Cloud deployment."

https://community.splunk.com/t5/Splunk-Enterprise-Security/Can-a-Splunk-Heavy-Forwarder-send-data-vi...

 

https://docs.splunk.com/Documentation/SplunkCloud/8.0.2006/Data/UsingforwardingagentsCloud#Use_a_Uni...

 

Would make life a bit easier :)

Copper Contributor

We seem to be having problems with FTD integrations in Sentinel. We are receiving Syslog message in Azure Sentinel but we do not seem to be able to parse it. Also the using Estreamer Encore link does not seem to work.

Copper Contributor

Does MS has any plans or in the roadmap support for Sybase database ? We have a requirement to forward application logs residing in Sybase to Sentinel, any way we can achieve this ? 

In past this used to work by using ArcSight flex connector, is there a similar framework that can be used. 

Copper Contributor

Hi @Ofer_Shezaf,

Is there a way to send Ubuntu Auth.log data to Azure Sentinel? 

Regards,

Muhammad

 

 

Brass Contributor

@m-waqar have you configure your OMS agent to read auth.log file (from Azure Sentinel, Workspace Setting --> Advanced setting --> Data --> Syslog)

 

Dev_Choudhary_0-1602148960176.png

 

 

 

Copper Contributor

Surely I'm missing the obvious, but where is Microsoft ATP? How does Sentinel collect the events from the endpoints armed with WD ATP, and how the Azure ATP or Office 365 ATP communicate with Sentinel? 

And last but not least - is Microsoft ATA supported log source, or is it too old or too non-cloud? ;)

Copper Contributor

Hi @Ofer_Shezaf

I am trying to ingest data from Cisco Meraki, but it seems that Azure Sentinel does not fully support the format of the message sent by Meraki. The message is truncated and you lose all the information to the left of the first colon. There is a discussion about it in Sentinel & Cisco Meraki? - Microsoft Tech Community and it seems to be because Meraki logs don't follow the RFC standard for syslog messages.

Should Cisco Meraki be in this table if the logs are not ingested correctly? Or there's something I am missing to make it work properly?

Copper Contributor

Is there a way to check if netscaler syslogs are being uploaded.  I have installed the agent to forward logs and in the console shows as connected but when I go to syslog logs in Sentinel only show logs from the local machine and not the Netscaler logs.  Also does not pick up any custom logs? 

Copper Contributor

@Jwcoxy1973, I had the same problem. In my case, with rsyslog, the problem was in the /etc/rsyslog.conf file. That is the general configuration file of rsyslog and by default it has the TCP and UDP syslog reception commented in the Modules section of the file. If you uncomment the one you need or both, and restart the rsyslog service, then you should receive the logs in Sentinel after a couple of minutes. You can also comment there the reception of syslog from the local machine, that way you will only have in sentinel the logs from netscaler in your case.

About the custom logs, I can't help you as I am having some issues of my own about that. 

Co-Authors
Version history
Last update:
‎Sep 29 2021 11:29 PM
Updated by: