%3CLINGO-SUB%20id%3D%22lingo-sub-1064158%22%20slang%3D%22en-US%22%3EIngest%20Sample%20CEF%20data%20into%20Azure%20Sentinel%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1064158%22%20slang%3D%22en-US%22%3E%3CP%3EAn%20Azure%20Sentinel%20Proof%20of%20Concept%20(PoC)%20is%20a%20great%20opportunity%20to%20effectively%20evaluate%20technical%20and%20business%20benefits.%20The%20onboarding%20of%20Microsoft%20cloud%20services%20is%20mostly%20a%20one-click%20experience%3B%20and%20thus%2C%20the%20ingestion%20of%20Syslog%2FCEF%20events%20presents%20the%20most%20notable%20challenge.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20this%20blog%2C%20I%20will%20go%20through%20the%20required%20steps%20showing%20how%20to%20ingest%20CEF%20events%20into%20Azure%20Sentinel%20for%20evaluation%20purposes.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EPreparation%20%26amp%3B%20Use%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EThe%20following%20tasks%20describe%20the%20necessary%20preparation%20steps.%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EInstall%20an%20Ubuntu%20server%20-%20it%20does%20not%20matter%20whether%20in%20Azure%20or%20On-Prem.%20This%20server%20will%20be%20used%20as%20a%20connector%20server.%3C%2FLI%3E%0A%3CLI%3EDeploy%20and%20configure%20the%20CEF%20agent%20on%20the%20connector%20server.%3C%2FLI%3E%0A%3CLI%3EValidate%20the%20connectivity%20between%20connector%20server%20and%20Azure%20Sentinel.%3C%2FLI%3E%0A%3CLI%3EIngest%20data%20into%20Azure%20Sentinel%20via%20connector%20server.%3C%2FLI%3E%0A%3CLI%3EHow%20to%20use%20the%20ingested%20data%20in%20Azure%20Sentinel.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EInstallation%20of%20connector%20server%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EThe%20installation%20of%20the%20connector%20server%20is%20quite%20straight%20forward.%20After%20successful%20installation%2C%20make%20sure%20that%20the%20connector%20server%20is%20deployed%20with%20latest%20updates%20and%20includes%20Python.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20connector%20server%20does%20not%20require%20an%20incoming%20connection%2C%20it%20should%20only%20have%20an%20outside%20connection%20to%20the%20Azure%20Sentinel%20instance.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EDeploy%20and%20validate%20the%20CEF%20agent%20on%20the%20connector%20server%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EThe%20connector%20CEF%20connector%20page%20describes%20the%20required%20steps%20for%20installation%20and%20validation%20%E2%80%93%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Fconnect-common-event-format%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ELink%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20604px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F161596i9BC2E54AC98495A1%2Fimage-dimensions%2F604x329%3Fv%3D1.0%22%20width%3D%22604%22%20height%3D%22329%22%20alt%3D%22clipboard_image_1.png%22%20title%3D%22clipboard_image_1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ESample%20events%20for%20evaluation%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EFor%20this%20blog%2C%20I%20decided%20to%20use%20the%20sample%20CEF%20events%20generated%20by%20Advanced%20Threat%20Analytics%20-%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fadvanced-threat-analytics%2Fcef-format-sa%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ELink%3C%2FA%3E.%20For%20evaluation%20purposes%2C%20any%20vendor's%20CEF%20events%20can%20be%20used.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EImport%20CEF%20events%20into%20Azure%20Sentinel%20via%20connector%20server%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EI%20created%20the%20following%20command%20lines%20to%20ingest%20four%20sample%20Advanced%20Threat%20Analytics%20CEF%20events%20into%20Azure%20Sentinel.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CEM%3Elogger%20-p%20local4.warn%20-t%20CEF%3C%2FEM%3E%3C%2FSTRONG%3E%3CEM%3E%20%22CEF%3A0%7CMicrosoft%7CATA%7C1.9.0.0%7CAbnormalSensitiveGroupMembershipChangeSuspiciousActivity%7CAbnormal%20modification%20of%20sensitive%20groups%7C5%7Cstart%3D2018-12-12T18%3A52%3A58.0000000Z%20app%3DGroupMembershipChangeEvent%20suser%3Dkrbtgt%20msg%3Dkrbtgt%20has%20uncharacteristically%20modified%20sensitive%20group%20memberships.%20externalId%3D2024%20cs1Label%3Durl%20cs1%3D%3CA%20href%3D%22https%3A%2F%2F192.168.0.220%2FsuspiciousActivity%2F5c113d028ca1ec1250ca0491%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2F192.168.0.220%2FsuspiciousActivity%2F5c113d028ca1ec1250ca0491%3C%2FA%3E%22%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CEM%3Elogger%20-p%20local4.warn%20-t%20CEF%20%3C%2FEM%3E%3C%2FSTRONG%3E%3CEM%3E%22CEF%3A0%7CMicrosoft%7CATA%7C1.9.0.0%7CLdapBruteForceSuspiciousActivity%7CBrute%20force%20attack%20using%20LDAP%20simple%20bind%7C5%7Cstart%3D2018-12-12T17%3A52%3A10.2350665Z%20app%3DLdap%20msg%3D10000%20password%20guess%20attempts%20were%20made%20on%20100%20accounts%20from%20W2012R2-000000-Server.%20One%20account%20password%20was%20successfully%20guessed.%20externalId%3D2004%20cs1Label%3Durl%20cs1%3D%3CA%20href%3D%22https%3A%2F%2F192.168.0.220%2FsuspiciousActivity%2F5c114acb8ca1ec1250cacdcb%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2F192.168.0.220%2FsuspiciousActivity%2F5c114acb8ca1ec1250cacdcb%3C%2FA%3E%22%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CEM%3Elogger%20-p%20local4.warn%20-t%20CEF%3C%2FEM%3E%3C%2FSTRONG%3E%3CEM%3E%20%22CEF%3A0%7CMicrosoft%7CATA%7C1.9.0.0%7CEncryptionDowngradeSuspiciousActivity%7CEncryption%20downgrade%20activity%7C5%7Cstart%3D2018-12-12T18%3A10%3A35.0334169Z%20app%3DKerberos%20msg%3DThe%20encryption%20method%20of%20the%20TGT%20field%20of%20TGS_REQ%20message%20from%20W2012R2-000000-Server%20has%20been%20downgraded%20based%20on%20previously%20learned%20behavior.%20This%20may%20be%20a%20result%20of%20a%20Golden%20Ticket%20in-use%20on%20W2012R2-000000-Server.%20externalId%3D2009%20cs1Label%3Durl%20cs1%3D%3CA%20href%3D%22https%3A%2F%2F192.168.0.220%2FsuspiciousActivity%2F5c114f938ca1ec1250cafcfa%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2F192.168.0.220%2FsuspiciousActivity%2F5c114f938ca1ec1250cafcfa%3C%2FA%3E%22%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CEM%3Elogger%20-p%20local4.warn%20-t%20CEF%3C%2FEM%3E%3C%2FSTRONG%3E%3CEM%3E%20%22CEF%3A0%7CMicrosoft%7CATA%7C1.9.0.0%7CEncryptionDowngradeSuspiciousActivity%7CEncryption%20downgrade%20activity%7C5%7Cstart%3D2018-12-12T17%3A00%3A31.2975188Z%20app%3DKerberos%20msg%3DThe%20encryption%20method%20of%20the%20Encrypted_Timestamp%20field%20of%20AS_REQ%20message%20from%20W2012R2-000000-Server%20has%20been%20downgraded%20based%20on%20previously%20learned%20behavior.%20This%20may%20be%20a%20result%20of%20a%20credential%20theft%20using%20Overpass-the-Hash%20from%20W2012R2-000000-Server.%20externalId%3D2010%20cs1Label%3Durl%20cs1%3D%3CA%20href%3D%22https%3A%2F%2F192.168.0.220%2FsuspiciousActivity%2F5c113eaf8ca1ec1250ca0883%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2F192.168.0.220%2FsuspiciousActivity%2F5c113eaf8ca1ec1250ca0883%3C%2FA%3E%22%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20600px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F161598iF18EED81956C8C32%2Fimage-dimensions%2F600x210%3Fv%3D1.0%22%20width%3D%22600%22%20height%3D%22210%22%20alt%3D%22clipboard_image_2.png%22%20title%3D%22clipboard_image_2.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETo%20validate%20whether%20the%20CEF%20events%26nbsp%3Bare%20received%20by%20Syslog%20server%2C%20use%20the%20following%20command%20line.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CEM%3Esudo%20tac%20%2Fvar%2Flog%2Fsyslog%20%7C%20grep%20CEF%20-m%2010%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20605px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F161597iA0BC2F5AFBDB2052%2Fimage-dimensions%2F605x215%3Fv%3D1.0%22%20width%3D%22605%22%20height%3D%22215%22%20alt%3D%22clipboard_image_3.png%22%20title%3D%22clipboard_image_3.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EHow%20to%20use%20the%20ingested%20data%20in%20Azure%20Sentinel%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EOnce%20the%20ingestion%20is%20processed%2C%20you%20can%20query%20the%20data.%20The%20CEF%20logs%20will%20reside%20within%20the%20data%20in%20the%20%3CSTRONG%3ECommonSecurityLog%20t%3C%2FSTRONG%3Eable.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20601px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F161599iC3B57B262A2E687C%2Fimage-dimensions%2F601x356%3Fv%3D1.0%22%20width%3D%22601%22%20height%3D%22356%22%20alt%3D%22clipboard_image_4.png%22%20title%3D%22clipboard_image_4.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1064158%22%20slang%3D%22en-US%22%3E%3CP%3EIn%20this%20blog%2C%20I%20will%20go%20through%20the%20required%20steps%20showing%20how%20to%20ingest%20CEF%20events%20into%20Azure%20Sentinel%20for%20evaluation%20purposes.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F161595iFAFB3F4AB7EA2203%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20alt%3D%22clipboard_image_0.png%22%20title%3D%22clipboard_image_0.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1064158%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Sentinel%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1087347%22%20slang%3D%22en-US%22%3ERe%3A%20Ingest%20Sample%20CEF%20data%20into%20Azure%20Sentinel%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1087347%22%20slang%3D%22en-US%22%3EHey%2C%20Question%3A%20Has%20anyone%20got%20the%20CEF%20collector%20to%20work%20on%20CentOS%20(7)%3F%20I%20tried%20on%20three%20fresh%20Azure%20CentOS%20VM%20instances%20and%20they%20never%20worked.%20I%20was%20sniffing%20the%20TCP%2F25226%20to%20see%20whether%20CEF%20logs%20were%20forwarded%20from%20the%20rsyslog%20deamon%20but%20no%20luck..%20Yesterday%2C%20I%20decided%20to%20'change%20religion%20%3A)%3C%2Fimg%3E%20'%20and%20went%20with%20a%20Ubuntu%20image%20and%20it%20worked%20in%205'..%20thanks!%20Maarten.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1337901%22%20slang%3D%22en-US%22%3ERe%3A%20Ingest%20Sample%20CEF%20data%20into%20Azure%20Sentinel%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1337901%22%20slang%3D%22en-US%22%3E%3CP%3ESame%20issue%20with%20CentOS.%20it%20sends%20the%20cef%20logs%20to%20syslog%20table%20instead%20of%20CommonSecurityLog%20in%20Azure%20Sentinel.%20Works%20for%20Ubuntu%20Server%20though.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

An Azure Sentinel Proof of Concept (PoC) is a great opportunity to effectively evaluate technical and business benefits. The onboarding of Microsoft cloud services is mostly a one-click experience; and thus, the ingestion of Syslog/CEF events presents the most notable challenge.

 

In this blog, I will go through the required steps showing how to ingest CEF events into Azure Sentinel for evaluation purposes.

 

Preparation & Use

The following tasks describe the necessary preparation steps.

  • Install an Ubuntu server - it does not matter whether in Azure or On-Prem. This server will be used as a connector server.
  • Deploy and configure the CEF agent on the connector server.
  • Validate the connectivity between connector server and Azure Sentinel.
  • Ingest data into Azure Sentinel via connector server.
  • How to use the ingested data in Azure Sentinel.

 

Installation of connector server

The installation of the connector server is quite straight forward. After successful installation, make sure that the connector server is deployed with latest updates and includes Python.

 

The connector server does not require an incoming connection, it should only have an outside connection to the Azure Sentinel instance.

 

Deploy and validate the CEF agent on the connector server

The connector CEF connector page describes the required steps for installation and validation – Link.

 

clipboard_image_1.png

 

Sample events for evaluation

For this blog, I decided to use the sample CEF events generated by Advanced Threat Analytics - Link. For evaluation purposes, any vendor's CEF events can be used.

 

Import CEF events into Azure Sentinel via connector server

I created the following command lines to ingest four sample Advanced Threat Analytics CEF events into Azure Sentinel.

 

logger -p local4.warn -t CEF "CEF:0|Microsoft|ATA|1.9.0.0|AbnormalSensitiveGroupMembershipChangeSuspiciousActivity|Abnormal modification of sensitive groups|5|start=2018-12-12T18:52:58.0000000Z app=GroupMembershipChangeEvent suser=krbtgt msg=krbtgt has uncharacteristically modified sensitive group memberships. externalId=2024 cs1Label=url cs1=https://192.168.0.220/suspiciousActivity/5c113d028ca1ec1250ca0491"

logger -p local4.warn -t CEF "CEF:0|Microsoft|ATA|1.9.0.0|LdapBruteForceSuspiciousActivity|Brute force attack using LDAP simple bind|5|start=2018-12-12T17:52:10.2350665Z app=Ldap msg=10000 password guess attempts were made on 100 accounts from W2012R2-000000-Server. One account password was successfully guessed. externalId=2004 cs1Label=url cs1=https://192.168.0.220/suspiciousActivity/5c114acb8ca1ec1250cacdcb"

logger -p local4.warn -t CEF "CEF:0|Microsoft|ATA|1.9.0.0|EncryptionDowngradeSuspiciousActivity|Encryption downgrade activity|5|start=2018-12-12T18:10:35.0334169Z app=Kerberos msg=The encryption method of the TGT field of TGS_REQ message from W2012R2-000000-Server has been downgraded based on previously learned behavior. This may be a result of a Golden Ticket in-use on W2012R2-000000-Server. externalId=2009 cs1Label=url cs1=https://192.168.0.220/suspiciousActivity/5c114f938ca1ec1250cafcfa"

logger -p local4.warn -t CEF "CEF:0|Microsoft|ATA|1.9.0.0|EncryptionDowngradeSuspiciousActivity|Encryption downgrade activity|5|start=2018-12-12T17:00:31.2975188Z app=Kerberos msg=The encryption method of the Encrypted_Timestamp field of AS_REQ message from W2012R2-000000-Server has been downgraded based on previously learned behavior. This may be a result of a credential theft using Overpass-the-Hash from W2012R2-000000-Server. externalId=2010 cs1Label=url cs1=https://192.168.0.220/suspiciousActivity/5c113eaf8ca1ec1250ca0883"

 

clipboard_image_2.png

 

To validate whether the CEF events are received by Syslog server, use the following command line.

 

sudo tac /var/log/syslog | grep CEF -m 10

 

clipboard_image_3.png

 

How to use the ingested data in Azure Sentinel

Once the ingestion is processed, you can query the data. The CEF logs will reside within the data in the CommonSecurityLog table.

 

clipboard_image_4.png

 

2 Comments
Occasional Contributor
Hey, Question: Has anyone got the CEF collector to work on CentOS (7)? I tried on three fresh Azure CentOS VM instances and they never worked. I was sniffing the TCP/25226 to see whether CEF logs were forwarded from the rsyslog deamon but no luck.. Yesterday, I decided to 'change religion :) ' and went with a Ubuntu image and it worked in 5'.. thanks! Maarten.
Occasional Visitor

Same issue with CentOS. it sends the cef logs to syslog table instead of CommonSecurityLog in Azure Sentinel. Works for Ubuntu Server though.