Home

Parsing syslog

%3CLINGO-SUB%20id%3D%22lingo-sub-917451%22%20slang%3D%22en-US%22%3EParsing%20syslog%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-917451%22%20slang%3D%22en-US%22%3E%3CP%3E1.%20i%20am%20ingesting%20firewall%20logs%20as%20syslog%20and%20trying%20to%20parse%20out%20the%20fields%20accordingly%20using%20the%20split%20command%2C%20i%20have%20a%20problem%20that%20the%20beginig%20of%20the%20logs%20is%20not%20piped%20and%20i%20have%20made%20the%20split%20in%202%20occasions.%3C%2FP%3E%3CP%3Eas%20you%20can%20see%20in%20the%20attached%20pic%20the%26nbsp%3B%3CSPAN%3EFWD%7CUDP%7Cp4%7C%20fields%20are%20nit%20parsed%20out.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3Ethis%20is%20the%20_raw%20syslog%20message%3A%3C%2FP%3E%3CP%3E%3CSPAN%3ESecurity%20F180%20Block%3A%20FWD%7CUDP%7Cp4%7C192.168.x%2Cx%7C67%7C00%3A15%3A5d%3A0f%3Ac4%3A01%7C255.255.255.255%7C68%7Cbootpc%7C%7CLAN-2-INTERNET%7C4017%7C0.0.0.0%7C0.0.0.0%7C0%7C1%7C%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E2.%20can%20you%20show%20me%20the%20same%20using%20normal%20regex%20i%20cant%20see%20in%20MSFT%20doc%20how%20to%20do%20it%20the%20old%20way%20%3A)%3C%2Fimg%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E3.%20should%20i%20do%20the%20parsing%20on%20search%20time%20of%20the%20query%3F%20doesnt%20it%20increase%20the%20search%20time%3F%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-917451%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3Eparser%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESentiel%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3Esyslog%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-917534%22%20slang%3D%22en-US%22%3ERe%3A%20Parsing%20syslog%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-917534%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F423940%22%20target%3D%22_blank%22%3E%40omrip%3C%2FA%3E%26nbsp%3B%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOptions%20like%3A%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Eprint%20SyslogMessage%20%3D%20%22Security%20F180%20Block%3A%20FWD%7CUDP%7Cp4%7C192.168.x%2Cx%7C67%7C00%3A15%3A5d%3A0f%3Ac4%3A01%7C255.255.255.255%7C68%7Cbootpc%7C%7CLAN-2-INTERNET%7C4017%7C0.0.0.0%7C0.0.0.0%7C0%7C1%7C%22%0A%7C%20project%20SyslogMessage%20%0A%7C%20extend%20device%20%20%20%20%20%20%20%3D%20extract(%22Security%20(.*%3F)Block%3A%22%2C%201%2C%20SyslogMessage)%20%0A%7C%20extend%20deviceaction%20%3D%20extract(%22USER%3D(.*%3F)COMMAND%22%2C%201%2C%20SyslogMessage)%20%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3Eor%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Eprint%20SyslogMessage%20%3D%20%22Security%20F180%20Block%3A%20FWD%7CUDP%7Cp4%7C192.168.x%2Cx%7C67%7C00%3A15%3A5d%3A0f%3Ac4%3A01%7C255.255.255.255%7C68%7Cbootpc%7C%7CLAN-2-INTERNET%7C4017%7C0.0.0.0%7C0.0.0.0%7C0%7C1%7C%22%0A%7C%20extend%20p%20%3D%20split(SyslogMessage%2C%20%22%7C%22)%20%0A%2F%2F%7C%20extend%20cleanp%20%20%3D%20trim(%40%22%5B%5E%5Cw%5D%2B%22%2Ctostring(p))%0A%7C%20extend%20pos1%20%3D%20split(p.%5B0%5D%2C%20%22%20%22)%0A%7C%20extend%20FWactivity%20%20%20%3D%20trim(%40%22%5B%5E%5Cw%5D%2B%22%2Ctostring(pos1.%5B0%5D))%0A%7C%20extend%20Device%20%20%20%20%20%20%20%3D%20trim(%40%22%5B%5E%5Cw%5D%2B%22%2Ctostring(pos1.%5B1%5D))%0A%7C%20extend%20DeviceAction%20%3D%20trim(%40%22%5B%5E%5Cw%5D%2B%22%2Ctostring(pos1.%5B2%5D))%0A%7C%20extend%20srcMAC%20%20%20%20%20%20%20%3D%20trim(%40%22%5B%5E%5Cw%5D%2B%22%2Ctostring(p.%5B5%5D))%0A%7C%20extend%20DestPort%20%20%20%20%20%3D%20trim(%40%22%5B%5E%5Cw%5D%2B%22%2Ctostring(p.%5B4%5D))%0A%2F%2F%20etc...%0A%7C%20project-away%20SyslogMessage%20%2C%20p%2C%20pos1%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EKQL%20is%20good%20at%20doing%20parsing%20like%20this%20at%20execution%20time.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-920293%22%20slang%3D%22en-US%22%3ERe%3A%20Parsing%20syslog%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-920293%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%26nbsp%3B%20that%20is%20very%20helpful%2C%20tnx%3C%2FP%3E%3CP%3Ewhen%20ingesting%20the%20logs%20to%20the%20syslog%20instead%20of%20CEF%26nbsp%3B%20connector%20i%20am%20very%20limited%20due%20to%20the%20small%20amount%20of%20fileds%20that%20exists%20on%20the%20syslog%20table%20in%20comparison%20with%20the%20CEF%3C%2FP%3E%3CP%3Ehow%20can%20i%20overcome%20it%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-920672%22%20slang%3D%22en-US%22%3ERe%3A%20Parsing%20syslog%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-920672%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F423940%22%20target%3D%22_blank%22%3E%40omrip%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3ECommonSecurityLog%0A%7C%20getschema%20%0A%7C%20summarize%20count(ColumnName)%20%0A%0ASyslog%0A%7C%20getschema%20%0A%7C%20summarize%20count(ColumnName)%20%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20make%20the%20difference%20152%20vs%2015%20columns%20of%20data%2C%26nbsp%3B%20are%20there%20certain%20columns%20you%20are%20missing%20in%20Syslog%3F%26nbsp%3B%20Is%20the%20data%20you%20require%20in%20the%20Syslog%20but%20needs%20extracting%20%2F%20parsing%20which%20is%20I%20believe%20one%20of%20the%20things%20CEF%20does%20for%20you%3F%26nbsp%3B%20%26nbsp%3BBTW%2C%20I'm%20no%20expert%20on%20CEF%20or%20Syslog%2C%20but%20keen%20to%20understand%20your%20use%20case.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThanks%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1006981%22%20slang%3D%22en-US%22%3ERe%3A%20Parsing%20syslog%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1006981%22%20slang%3D%22en-US%22%3E%3CP%20style%3D%22line-height%3A%2012.0pt%3B%20background%3A%20%23FFFFFE%3B%22%3EParse%20Examples%20and%20you%20can%20extend%20and%20create%20new%20columns%3C%2FP%3E%0A%3CP%20style%3D%22line-height%3A%2012.0pt%3B%20background%3A%20%23FFFFFE%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22line-height%3A%2012.0pt%3B%20background%3A%20%23FFFFFE%3B%22%3E%3CSPAN%20style%3D%22font-size%3A%2010.5pt%3B%20font-family%3A%20Consolas%3B%20color%3A%20black%3B%22%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fkusto%2Fquery%2Fparseoperator%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fkusto%2Fquery%2Fparseoperator%3C%2FA%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20style%3D%22line-height%3A%2012.0pt%3B%20background%3A%20%23FFFFFE%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22line-height%3A%2012.0pt%3B%20background%3A%20%23FFFFFE%3B%22%3E%3CSPAN%20style%3D%22font-size%3A%2010.5pt%3B%20font-family%3A%20Consolas%3B%20color%3A%20black%3B%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20style%3D%22line-height%3A%2012.0pt%3B%20background%3A%20%23FFFFFE%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F423940%22%20target%3D%22_blank%22%3E%40omrip%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
omrip
Occasional Contributor

1. i am ingesting firewall logs as syslog and trying to parse out the fields accordingly using the split command, i have a problem that the beginig of the logs is not piped and i have made the split in 2 occasions.

as you can see in the attached pic the FWD|UDP|p4| fields are nit parsed out.

this is the _raw syslog message:

Security F180 Block: FWD|UDP|p4|192.168.x,x|67|00:15:5d:0f:c4:01|255.255.255.255|68|bootpc||LAN-2-INTERNET|4017|0.0.0.0|0.0.0.0|0|1|

2. can you show me the same using normal regex i cant see in MSFT doc how to do it the old way :)

3. should i do the parsing on search time of the query? doesnt it increase the search time?

 

4 Replies

@omrip  

Options like:

print SyslogMessage = "Security F180 Block: FWD|UDP|p4|192.168.x,x|67|00:15:5d:0f:c4:01|255.255.255.255|68|bootpc||LAN-2-INTERNET|4017|0.0.0.0|0.0.0.0|0|1|"
| project SyslogMessage 
| extend device       = extract("Security (.*?)Block:", 1, SyslogMessage) 
| extend deviceaction = extract("USER=(.*?)COMMAND", 1, SyslogMessage) 

 

or

print SyslogMessage = "Security F180 Block: FWD|UDP|p4|192.168.x,x|67|00:15:5d:0f:c4:01|255.255.255.255|68|bootpc||LAN-2-INTERNET|4017|0.0.0.0|0.0.0.0|0|1|"
| extend p = split(SyslogMessage, "|") 
//| extend cleanp  = trim(@"[^\w]+",tostring(p))
| extend pos1 = split(p.[0], " ")
| extend FWactivity   = trim(@"[^\w]+",tostring(pos1.[0]))
| extend Device       = trim(@"[^\w]+",tostring(pos1.[1]))
| extend DeviceAction = trim(@"[^\w]+",tostring(pos1.[2]))
| extend srcMAC       = trim(@"[^\w]+",tostring(p.[5]))
| extend DestPort     = trim(@"[^\w]+",tostring(p.[4]))
// etc...
| project-away SyslogMessage , p, pos1

 

KQL is good at doing parsing like this at execution time.

@Clive Watson  that is very helpful, tnx

when ingesting the logs to the syslog instead of CEF  connector i am very limited due to the small amount of fileds that exists on the syslog table in comparison with the CEF

how can i overcome it?

 

@omrip 

 

CommonSecurityLog
| getschema 
| summarize count(ColumnName) 

Syslog
| getschema 
| summarize count(ColumnName) 

 

I make the difference 152 vs 15 columns of data,  are there certain columns you are missing in Syslog?  Is the data you require in the Syslog but needs extracting / parsing which is I believe one of the things CEF does for you?   BTW, I'm no expert on CEF or Syslog, but keen to understand your use case.

 

Thanks 

 

 

Parse Examples and you can extend and create new columns

 

https://docs.microsoft.com/en-us/azure/kusto/query/parseoperator

 

 

 

@omrip 

Related Conversations
Tabs and Dark Mode
cjc2112 in Discussions on
46 Replies
Extentions Synchronization
Deleted in Discussions on
3 Replies
Stable version of Edge insider browser
HotCakeX in Discussions on
35 Replies
How to Prevent Teams from Auto-Launch
chenrylee in Microsoft Teams on
30 Replies
flashing a white screen while open new tab
Deleted in Discussions on
14 Replies
Security Community Webinars
Valon_Kolica in Security, Privacy & Compliance on
13 Replies