Creating new field in logs based on existing one

[majo1]

Hello,

So as the logs are ingested in Azure Sentinel, i want to add a new key/value to the logs table based on a key that already exists in the logs.

For example, the new key is "Country", and if the Tenant-ID value existing in the logs is XYZ then the country should be added "United Stated".

How can i add such new key and value to Azure Sentinel Schemas ?

In other SIEM Solution, this is achieved by using Feeds.

Thanks.

[#2 · Accepted solution]

@majo1 

Log Analytics (and therefore Sentinel as it uses the same data store) processes raw data.
"...the Log Analytics service processes the raw data and ingests it into the database."

Source: section 3:
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-security

So you have to add or amend the data before you send it, custom logs are one feature that may help. see https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs

---------------------------------------------------------------------

However after the data is in the data store you can enrich it (use the Extend option for example to add columns)  https://docs.microsoft.com/en-us/azure/kusto/query/extendoperator

There are other commands as well.

Example - here when we find country and I add Column for its country code:

WireData
| where isnotempty(RemoteIPCountry)
| extend CountryCode = case (
                            RemoteIPCountry == "United States", "US",
                            RemoteIPCountry == "United Kingdom", "UK",
                            strcat("No Country Code for"," : ", RemoteIPCountry)
                            )
| project RemoteIPCountry , CountryCode

Go to Log Analytics and Run Query

RemoteIPCountry	CountryCode
United States	US
Latvia	No Country Code for : Latvia
Latvia	No Country Code for : Latvia
United States	US
Denmark	No Country Code for : Denmark

You could bring that data in from another Table or even a remote file as well, see this more real example for adding country codes from a file online: https://cloudblogs.microsoft.com/industry-blog/en-gb/cross-industry/2019/08/13/azure-log-analytics-h...

[#3]

@CliveWatson 

Thank you for the answer.

Customizing logs or modifying them before sending to Azure doesn't seem to be a possible option.

The Extend feature is close to my requirement.

Question:

After the Extend function is done, like the example you provided, is there a way to insert the Extended value back in logs in the database so that the Extended value permanently becomes part of the log?

To clarify by example, after i populate the value of CountryCode = 'US', i want the other Analysts to find that value already in the logs whenever they perform queries to the affected table.

The requirement can be probably described as Extend at the ingestion time, rather than query time.

Thanks in advance

[#4]

Sorry but its the Extend at Query time option. Log Analytics storage is WORM (Write Once Read Many).

So you will need to let the Analysts know that they need to use an Extend each time (save the work in a repeatable query and share that with them).

[#5]

@majo1 : 

You would need to think differently with Sentinel. The need is not to create physical field, but rather the enable an analyst to access the field. Sentinel's query time parsing, which @CliveWatson described enables this by using functions. A function encompasses the field extraction in a view that analysts can use without reinventing the field. 

You can read more about how to use functions for this purpose here: https://techcommunity.microsoft.com/t5/Azure-Sentinel/Using-KQL-functions-to-speed-up-analysis-in-Az...

~ Ofer