How to access Null fields in Parsers

%3CLINGO-SUB%20id%3D%22lingo-sub-2668285%22%20slang%3D%22en-US%22%3EHow%20to%20access%20Null%20fields%20in%20Parsers%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2668285%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20are%20working%20on%20creating%20a%20custom%20connector%20to%20ingest%20the%20data%20in%20Azure%20Sentinel.%20We%20are%20now%20working%20on%20parsers%2C%20and%20we%20are%20trying%20to%20replicate%20the%20data%20fields%20as%20in%20Splunk.%20In%20Splunk%2C%20even%20though%20if%20all%20the%20values%20of%20a%20particular%20field%2C%20say%20name%2C%20are%20null%20in%20all%20the%20events%2C%20then%20that%20field%20will%20be%20shown%20null%20in%20all%20the%20events.%3CBR%20%2F%3EBut%20in%20Azure%20if%20all%20the%20events%2C%20of%20a%20particular%20field%20is%20null%2C%20then%20Azure%20removes%20that%20particular%20field.%3CBR%20%2F%3ESo%20as%20we%20want%20to%20replicate%20fields%20to%20as%20that%20of%20parsers%2C%20we%20can't%20access%20that%20particular%20field%2C%20say%20name%2C%20as%20azure%20removed%20that%20field.%20So%20In%20parser's%20if%20we%20create%20a%20new%20field%20statically%2C%20name%3D%22%22%2C%20then%20that%20field%20will%20be%20present.%26nbsp%3B%3CBR%20%2F%3EBut%20if%20the%20value%20comes%20in%20any%201%20event%20out%20of%201000's%2C%20then%20this%20static%20representation%20would%20cause%20Data%20Loss.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESo%2C%20Can%20someone%20help%20me%20determine%20how%20can%20we%20import%20those%20null%20fields%20into%20our%20logs%2C%20or%20method%20can%20we%20use%20to%20get%20those%20null%20fields%2C%20which%20Azure%20removed.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2669170%22%20slang%3D%22en-US%22%3ERe%3A%20How%20to%20access%20Null%20fields%20in%20Parsers%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2669170%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F1106811%22%20target%3D%22_blank%22%3E%40Ronak_Shah%3C%2FA%3E%26nbsp%3BTake%20a%20look%20at%20the%20KQL%20command%26nbsp%3B%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-explorer%2Fkusto%2Fquery%2Fcolumnifexists%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ecolumn_ifexists()%20-%20Azure%20Data%20Explorer%20%7C%20Microsoft%20Docs%20%3C%2FA%3E%3C%2FP%3E%3CP%3EIt%20will%20check%20to%20see%20if%20a%20column%20exists%20and%2C%20if%20it%20does%2C%20will%20return%20its%20value%2C%20otherwise%20return%20a%20default%20value%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2677840%22%20slang%3D%22en-US%22%3ERe%3A%20How%20to%20access%20Null%20fields%20in%20Parsers%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2677840%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F46875%22%20target%3D%22_blank%22%3E%40Gary%20Bushey%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%20data-unlink%3D%22true%22%3EHi%2C%20Thanks%20for%20your%20response%3CBR%20%2F%3EActually%2C%20the%20command%26nbsp%3B%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Ecolumn_ifexists()%2C%20will%20be%20implemented%20in%20the%20parsers%2C%20and%20it%20will%20be%20useful%20when%20we%20actually%20know%20the%20column%20name.%20In%20column_ifexists('column_name'%2C'')%2C%20it%20will%20check%20if%20the%20'column_name'%20column%20is%20present%20or%20not%2C%20if%20not%20then%20null%20value%20will%20be%20added%2C%20But%20this%20is%20all%20possible%20when%20we%20actually%20know%20the%20'column_name'.%26nbsp%3B%3CBR%20%2F%3EAnd%20as%20Azure%20has%20removed%20the%20column%20with%20null%20field%2C%20so%20it%20has%20actually%20removed%20our%20access%20to%20the%20column_name%2C%20So%20how%20do%20we%20know%20then%20what%20to%20actually%20type%20in%20the%20'column_name'%20name.%3C%2FP%3E%3CP%20data-unlink%3D%22true%22%3E%3CBR%20%2F%3ESo%20is%20their%20any%20method%20in%20which%2C%20we%20don't%20allow%20azure%20to%20remove%20that%20column's%20with%20null%20fields.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2678673%22%20slang%3D%22en-US%22%3ERe%3A%20How%20to%20access%20Null%20fields%20in%20Parsers%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2678673%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F1106811%22%20target%3D%22_blank%22%3E%40Ronak_Shah%3C%2FA%3E%26nbsp%3BWouldn't%20the%20system%20that%20is%20sending%20the%20data%20have%20a%20list%20of%20the%20fields%20it%20is%20sending%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E
Occasional Contributor

Hi,

 

We are working on creating a custom connector to ingest the data in Azure Sentinel. We are now working on parsers, and we are trying to replicate the data fields as in Splunk. In Splunk, even though if all the values of a particular field, say name, are null in all the events, then that field will be shown null in all the events.
But in Azure if all the events, of a particular field is null, then Azure removes that particular field.
So as we want to replicate fields to as that of parsers, we can't access that particular field, say name, as azure removed that field. So In parser's if we create a new field statically, name="", then that field will be present. 
But if the value comes in any 1 event out of 1000's, then this static representation would cause Data Loss.

 

So, Can someone help me determine how can we import those null fields into our logs, or method can we use to get those null fields, which Azure removed.

5 Replies

@Ronak_Shah Take a look at the KQL command  column_ifexists() - Azure Data Explorer | Microsoft Docs

It will check to see if a column exists and, if it does, will return its value, otherwise return a default value

@Gary Bushey 

 

Hi, Thanks for your response
Actually, the command  column_ifexists(), will be implemented in the parsers, and it will be useful when we actually know the column name. In column_ifexists('column_name',''), it will check if the 'column_name' column is present or not, if not then null value will be added, But this is all possible when we actually know the 'column_name'. 
And as Azure has removed the column with null field, so it has actually removed our access to the column_name, So how do we know then what to actually type in the 'column_name' name.


So is their any method in which, we don't allow azure to remove that column's with null fields.

@Ronak_Shah Wouldn't the system that is sending the data have a list of the fields it is sending?

@Gary Bushey 

Hi, Thanks for your response


Actually yes, the system that is sending the data have a list of the fields it is sending, But when we pass the data through API to the Azure sentinel, Azure while parsing the data, changes the field names.

Like for eg. In our system we sent fields "name" --> then azure checked for the values of those fields

              --> If the values are null for all the events, then that field is removed

              --> If the field is not null, then that field is present in azure, with a new name say "name_s"

So our concern is, if the fields is removed by azure, how do we know whether that field will come with the name of "name_s" or "something_else"?,

so that we can then write column_ifexists('name_s','') or column_ifexists('something_else','')

@Ronak_Shah Because you are writing this yourself and it is going to a custom table, the fields have the data type appended to them (no idea why, personally I find that quite annoying).  So you will know that a field being sent as "name" will come through as "name_s" if it is a string so you will be able to use column_ifexists('name_s','')