SOLVED

MS Threat Intel matching with custom logs

%3CLINGO-SUB%20id%3D%22lingo-sub-1294551%22%20slang%3D%22en-US%22%3EMS%20Threat%20Intel%20matching%20with%20custom%20logs%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1294551%22%20slang%3D%22en-US%22%3E%3CP%3EMS%20Threat%20Intel%20matches%20only%20with%20default%20Sentinel%20tables%20(like%20CommonSecurityLog%20)%3C%2FP%3E%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Dev_Choudhary_0-1586407347679.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F183283i3D7D14D999758496%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22Dev_Choudhary_0-1586407347679.png%22%20alt%3D%22Dev_Choudhary_0-1586407347679.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3EHow%20same%20can%20be%20utilized%20with%20custom%20table%20events.%3C%2FP%3E%3CP%3EIf%20you%20run%20below%20command%2C%20you%20will%20get%20the%20result%3C%2FP%3E%3CDIV%3E%3CDIV%3E%3CFONT%20size%3D%222%22%20color%3D%22%23993366%22%3E%3CSPAN%3ECommonSecurityLog%3C%2FSPAN%3E%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%3E%3CFONT%20size%3D%222%22%20color%3D%22%23993366%22%3E%3CSPAN%3E%7C%20%3C%2FSPAN%3E%3CSPAN%3Esummarize%3C%2FSPAN%3E%20%3CSPAN%3Ecount%3C%2FSPAN%3E%3CSPAN%3E()%20%3C%2FSPAN%3E%3CSPAN%3Eby%3C%2FSPAN%3E%3CSPAN%3E%20CommunicationDirection%2C%20MaliciousIPCountry%3C%2FSPAN%3E%3C%2FFONT%3E%3C%2FDIV%3E%3C%2FDIV%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Ewhile%20if%20you%20run%20same%20for%20custom%20table%20like%20below%2C%20you%20will%20get%20the%20error%26nbsp%3B%3C%2FP%3E%3CDIV%3E%3CDIV%3E%3CFONT%20size%3D%222%22%20color%3D%22%23FF0000%22%3E%3CSPAN%3ECustomLog_CL%3C%2FSPAN%3E%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%3E%3CFONT%20size%3D%222%22%20color%3D%22%23FF0000%22%3E%7C%20summarize%20count()%20by%20MaliciousIPCountry%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-left%22%20image-alt%3D%22err.PNG%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F183287i452146BED0218290%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22err.PNG%22%20alt%3D%22err.PNG%22%20%2F%3E%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3CDIV%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1335389%22%20slang%3D%22en-US%22%3ERe%3A%20MS%20Threat%20Intel%20matching%20with%20custom%20logs%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1335389%22%20slang%3D%22en-US%22%3E%3CP%3EHello%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F536739%22%20target%3D%22_blank%22%3E%40Dev_Choudhary%3C%2FA%3E%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAs%20mentionned%20on%20the%20CEF%20Connector%26nbsp%3B%3CEM%3E%22%3CSPAN%3EBy%20connecting%20your%20CEF%20logs%20to%20Azure%20Sentinel%2C%20you%20can%20take%20advantage%20of%20search%20%26amp%3B%20correlation%2C%20alerting%2C%20and%20threat%20intelligence%20enrichment%20for%20each%3C%2FSPAN%3E%3C%2FEM%3E%20%3CEM%3Elog%22.%26nbsp%3B%3C%2FEM%3ESo%20the%20MS%20Threat%20Intelligence%20is%20applied%20only%20when%20using%20the%20associated%20connector.%20However%2C%20imported%20logs%20%3CSTRONG%3Edo%20not%3C%2FSTRONG%3E%20get%20into%20the%20same%20%3CEM%3Econnection%3C%2FEM%3E%20process.%20This%20explains%20why%20the%26nbsp%3B%3CSPAN%3E%3CEM%3EMaliciousIPCountry%3C%2FEM%3E%26nbsp%3Bcolumn%20is%20not%20added%20in%20the%20imported%20logs.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThus%2C%20your%20custom%20log%20need%20to%20be%20analyzed%20with%20Threat%20Intelligence%20(not%20necessarily%20MS)%20before%20being%20imported%20into%20the%20Log%20Analytics%20workspace.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1335401%22%20slang%3D%22en-US%22%3ERe%3A%20MS%20Threat%20Intel%20matching%20with%20custom%20logs%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1335401%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F536739%22%20target%3D%22_blank%22%3E%40Dev_Choudhary%3C%2FA%3E%26nbsp%3BIt%20appears%20that%20your%20custom%20log%20is%20missing%20that%20field.%26nbsp%3B%20Where%20did%20you%20get%20the%20log%20from%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1336071%22%20slang%3D%22en-US%22%3ERe%3A%20MS%20Threat%20Intel%20matching%20with%20custom%20logs%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1336071%22%20slang%3D%22en-US%22%3E%3CP%3Eit%20can%20be%20any%20source%20like%20Cisco%20Meraki%2C%20Okta%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1338133%22%20slang%3D%22en-US%22%3ERe%3A%20MS%20Threat%20Intel%20matching%20with%20custom%20logs%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1338133%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F536739%22%20target%3D%22_blank%22%3E%40Dev_Choudhary%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThere%20are%20ways%20to%20do%20it%20but%20it%20requires%20a%20bit%20of%20creative%20log%20ingestion.%20I%20am%20not%20sure%20how%20you%20get%20the%20data%20from%20that%20custom%20log%20into%20Sentinel%20but%20in%20our%20case%2C%20when%20we%20want%20to%20have%20a%20log%20enriched%20with%20the%20Microsoft%20threat%20intel%2C%20we%20%22force%22%20it%20through%20the%20CEF%20connector.%20The%20challenge%20of%20course%2C%20is%20to%20process%20the%20data%20that%20is%20not%20available%20in%20CEF.%20For%20that%20we%20use%20Logstash%20to%20get%20the%20raw%20logs%2C%20parse%20and%20adjust%20the%20format%20to%20match%20CEF%20standard%20and%20send%20it%20as%20syslog%20to%20the%20Sentinel%20collector%20(using%20the%20syslog%20facility%20configured%20for%20the%20CEF%20collector).%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHere%20is%20an%20example%20of%20our%20custom%20threat%20intel%20feed%20(that%20we%20compile%20and%20saved%20as%20csv%20file%20on%20a%20server%20on%20hourly%20basis)%20converted%20as%20CEF%20through%20Logstash%20and%20sent%20to%20Sentinel%20through%20the%20CEF%20data%20collector%20(and%20you%20can%20see%20that%20some%20entries%20match%20Microsoft's%20threat%20intel%20as%20well)%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22firegen_cef.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F186678i3AA5C859E0823CD9%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22firegen_cef.png%22%20alt%3D%22firegen_cef.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20Logstash%20part%20is%20not%20as%20complicated%20as%20one%20may%20think%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22firegen_cef_logstash.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F186679iFA24F894D3CBFBCC%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22firegen_cef_logstash.png%22%20alt%3D%22firegen_cef_logstash.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAdrian%20Grigorof%3C%2FP%3E%3CP%3E%3CA%20href%3D%22http%3A%2F%2Fwww.managedsentinel.com%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ewww.managedsentinel.com%3C%2FA%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1385287%22%20slang%3D%22en-US%22%3ERe%3A%20MS%20Threat%20Intel%20matching%20with%20custom%20logs%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1385287%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F353788%22%20target%3D%22_blank%22%3E%40AdiGrio%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20for%20your%20response.%20your%20suggestion%20is%20helpful%20and%20even%20I%20was%20looking%20to%20onboard%20custom%20threat%20Intel%20with%20Sentinel.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1388826%22%20slang%3D%22en-US%22%3ERe%3A%20MS%20Threat%20Intel%20matching%20with%20custom%20logs%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1388826%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F353788%22%20target%3D%22_blank%22%3E%40AdiGrio%3C%2FA%3E%26nbsp%3B%20have%20you%20tried%20using%20the%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-codecs-cef.html%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3ECEF%20codec%3C%2FA%3E%20instead%20of%20manually%20constructing%20the%20message%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E
Occasional Contributor

MS Threat Intel matches only with default Sentinel tables (like CommonSecurityLog )

 

Dev_Choudhary_0-1586407347679.png

How same can be utilized with custom table events.

If you run below command, you will get the result

CommonSecurityLog
| summarize count() by CommunicationDirection, MaliciousIPCountry

 

while if you run same for custom table like below, you will get the error 

CustomLog_CL
| summarize count() by MaliciousIPCountry

 

err.PNG 

 
6 Replies

Hello @Dev_Choudhary,

 

As mentionned on the CEF Connector "By connecting your CEF logs to Azure Sentinel, you can take advantage of search & correlation, alerting, and threat intelligence enrichment for each log". So the MS Threat Intelligence is applied only when using the associated connector. However, imported logs do not get into the same connection process. This explains why the MaliciousIPCountry column is not added in the imported logs.

 

Thus, your custom log need to be analyzed with Threat Intelligence (not necessarily MS) before being imported into the Log Analytics workspace.

 

 

 

@Dev_Choudhary It appears that your custom log is missing that field.  Where did you get the log from?

it can be any source like Cisco Meraki, Okta 

Best Response confirmed by Dev_Choudhary (Occasional Contributor)
Solution

@Dev_Choudhary 

 

There are ways to do it but it requires a bit of creative log ingestion. I am not sure how you get the data from that custom log into Sentinel but in our case, when we want to have a log enriched with the Microsoft threat intel, we "force" it through the CEF connector. The challenge of course, is to process the data that is not available in CEF. For that we use Logstash to get the raw logs, parse and adjust the format to match CEF standard and send it as syslog to the Sentinel collector (using the syslog facility configured for the CEF collector). 

 

Here is an example of our custom threat intel feed (that we compile and saved as csv file on a server on hourly basis) converted as CEF through Logstash and sent to Sentinel through the CEF data collector (and you can see that some entries match Microsoft's threat intel as well):

 

firegen_cef.png

 

The Logstash part is not as complicated as one may think:

 

firegen_cef_logstash.png

 

Adrian Grigorof

www.managedsentinel.com

Hi @AdiGrio 

Thanks for your response. your suggestion is helpful and even I was looking to onboard custom threat Intel with Sentinel.

@AdiGrio  have you tried using the CEF codec instead of manually constructing the message?