Logstash collector vs UEBA and Exploration queries

%3CLINGO-SUB%20id%3D%22lingo-sub-1706816%22%20slang%3D%22en-US%22%3ELogstash%20collector%20vs%20UEBA%20and%20Exploration%20queries%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1706816%22%20slang%3D%22en-US%22%3E%3CP%3EHello%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWhen%20using%20the%20official%20and%20supported%20Logstash%20output%20to%20ingest%20events%20from%20a%20WEC%20server%2C%20the%20table%20is%20not%20named%20%60SecurityEvent%60%20(gets%20%60_CL%60%20appended)%20and%20the%20fields%20are%20all%20appended%20with%20their%20types%20(due%20to%20the%20LogAnalytics%20API%2C%20documented%20behaviour).%20This%20breaks%20features%20such%20as%20the%20Exploration%20Queries%20(to%20pivot%20from%20investigation%20blade)%20which%20all%20expect%20the%20table%20to%20be%20named%20%60SecurityEvent%60%20with%20specific%20fields.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDo%20you%20plan%20to%20allow%20to%20create%20SecurityEvent%20table%20with%20proper%20fields%20through%20your%20official%20Logstash%20output%20or%20do%20you%20plan%20to%20allow%20to%20define%20mapping%20so%20that%20we%20could%20define%20that%20SecurityEvent%20table%20is%20corresponding%20to%20(example)%20Windows_CL%20and%20that%20field%20%60EventID%60%20is%20mapped%20to%20%60EventID_d%60%20(mapping%20to%20be%20defined%20by%20contributor%20for%20all%20fields%20required%20by%20UEBA%2FExploration%20Queries)%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EBest%20regards%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1706816%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3Eexploration%20queries%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ELogstash%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EUEBA%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1752886%22%20slang%3D%22en-US%22%3ERe%3A%20Logstash%20collector%20vs%20UEBA%20and%20Exploration%20queries%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1752886%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F779640%22%20target%3D%22_blank%22%3E%40milkmix_%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHave%20you%20considered%20using%20a%20Parser%20-%20this%20is%20an%20example%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Fblob%2Fmaster%2FParsers%2FTeams_parser.txt%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Fblob%2Fmaster%2FParsers%2FTeams_parser.txt%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20used%20this%20syntax%20recently%20to%20map%20a%20_CL%20with%20another%20Table%20-%20saved%20as%20a%20function%20called%20InfoP.%26nbsp%3B%20You%20will%20probably%20want%20to%20map%20multiple%20columns%2C%20see%20the%20example%20for%20ideas.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-cpp%22%3E%3CCODE%3Eunion%20withsource%3Dtt%20InformationProtectionLogs_CL%2C%20InformationProtectionEvents%0A%7C%20extend%20User%20%3D%20iif(isempty(User)%2CUserId_s%2CUser)%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EInfoP%3CBR%20%2F%3E%7C%20project%20User%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
Occasional Contributor

Hello,

 

When using the official and supported Logstash output to ingest events from a WEC server, the table is not named `SecurityEvent` (gets `_CL` appended) and the fields are all appended with their types (due to the LogAnalytics API, documented behaviour). This breaks features such as the Exploration Queries (to pivot from investigation blade) which all expect the table to be named `SecurityEvent` with specific fields.

 

Do you plan to allow to create SecurityEvent table with proper fields through your official Logstash output or do you plan to allow to define mapping so that we could define that SecurityEvent table is corresponding to (example) Windows_CL and that field `EventID` is mapped to `EventID_d` (mapping to be defined by contributor for all fields required by UEBA/Exploration Queries)?

 

Best regards

4 Replies
Highlighted

@milkmix_ 

 

Have you considered using a Parser - this is an example: https://github.com/Azure/Azure-Sentinel/blob/master/Parsers/Teams_parser.txt

 

I used this syntax recently to map a _CL with another Table - saved as a function called InfoP.  You will probably want to map multiple columns, see the example for ideas.

 

union withsource=tt InformationProtectionLogs_CL, InformationProtectionEvents
| extend User = iif(isempty(User),UserId_s,User)

 

InfoP
| project User 

Highlighted
Hi Clive,

Thanks for the tips! I wasn't aware of this feature.
Could this be used to map it against a table like SecurityEvents or CommonSecurityEvents used internally by Sentinel in Entity Behavior?
Highlighted
It's a feature that helps you with Queries or features that support queries, like Workbooks & Rules. If you create a function called: "mySecurityEvents" none of the built-in blades would be aware of that new name, so a solution like Entity Behaviour wont use this new 'virtual' table. However you could adapt the Entity Behaviour workbook to include this new Table (parser), or include it in custom rules you write.
Highlighted
indeed, just tested and got this in return:
'''Detected a function and a table with the same name: 'SecurityEvent'. Rename the function to allow it to be used in a query.'''

Will work with LA agent then :)