Data Collector API and TimeGenerated property.

[sirkillnotalot]

Hi guys,

I'm trrying to ingest some logs in to a Sentinel workspace but I'm having issues with populating the correct TimeGenerated property.

According to the ol' docs:

time-generated-field is the name of a field in the data that contains the timestamp of the data item. If you specify a field, its contents are used for TimeGenerated. If you don't specify this field, the default for TimeGenerated is the time that the message is ingested.

We also know from that page that that contents of the message field should follow the ISO 8601 format YYYY-MM-DDThh:mm:ssZ.

So, doing a cheeky copy paste, I've setup a task in a LogicApp to do just that:

I can see the property was formatted and sent:

and confirmed in the rawOutput:

{
"method": "post",
"headers": {
"Log-Type": "Example",
"time-generated-field": "2022-02-17T11:25:40Z"
}

However when querying the table the TimeGenerated is reflecting the Ingestion time only. To confirm this I added a couple of properties to the JSON payload to include utcNow() as RunTime_t and the original event time as TimeStamp_s:

I can work around it in the short-term by using a function to do something like the below. But would much rather preserve the original logs' TimeStamp.

Example
| extend TimeGenerated = TimeStamp_s
| where TimeGenerated <= ago(1h)

Have I missed something _really_ obvious here or this simply not working as intended?

References:

Azure Monitor HTTP Data Collector API - Azure Monitor | Microsoft Docs

Azure Log Analytics Data Collector - Connectors | Microsoft Docs

Edits: Added screenshot, formatting, references

[#2]

@sirkillnotalot Looking at this documentation: Azure Log Analytics Data Collector - Connectors | Microsoft Docs  it states that the Time-generated-field "Represents the original date and time of the record." but it doesn't actually state that this is the TimeGenerated field (although it does lead one to think it would be).

Without adding any of the other date/time related fields, was there any other field that had a date/time in it?   Maybe something like TimeGenerated_t or TimeStamp_t?

[#3]

@Gary Bushey 

The only fields that are date/time was the TimeGenerated passed in the original payload. As a bit of a test I stripped that field out using removeProperty(items('For_each'),'TimeGenerated') and used a static entry in the time-generated-field field instead:

but all I get is:

I initially thought the same as you and that the documentation may be misleading or incorrect but I've had a response from an MS support rep who've confirmed that the field is intended to preserve the record time stamp, not the ingested timestamp.

Think I might go create a new basic LogicApp and see if I can reproduce using a payload I control via PostMan and see if it's something in the way it processes records in a for_each loop.

[#4]

OK so I've found the issue and posting resolution here in case anyone comes across this in the future.

The input we are receiving from our source system:

  {
    "type": "HTTP Outbound",
    "creator": "admin",
    "TimeGenerated": "2022-02-18 11:56:27",
    "table": "web_connections",
    "sys_id": "d2443d3c1b31451095a74195d34bcb65",
    "payload": {
      "hostname": "exampleurl.com",
      "sequence": "5310",
      "method": "GET",
      "url": "https://exampleurl.com",
      "response_status": "200",
      "source_table": "web_connections",
      "source_record": "2454c2271ba22c10fda632e8cd4bcb99",
      "transaction_name": "HTTP Outbound",
      "user_name": "root"
    }
  }

Issue 1

The connector doesn't like it if the payload with the date/time you're using is called TimeGenerated so I needed to:

• add a new property with the TimeGenerated values:
  addProperty(triggerBody(),'TimeStamp',formatDateTime(triggerBody()?['TimeGenerated']))
• to remove the timegenerated property:
  removeProperty(outputs('Compose'),'TimeGenerated')

Issue 2

Usual formating mismatch so ran the formatdatetime() on the TimeGenerated field to convert to ISO8601 format

Issue 3

Probably a bit of a misunderstanding on my part - the 'time-generated-field' is looking for the property name, not the actual value.

When it all works it'll populate the table using the TimeStamp field as both TimeGenerated, and TimeGenerated_s. The original TimeStamp property is actually removed.

Interesting, and also frustratingly, despite the input for the 'time-generated-field' being wrong the API still returns "statusCode: 200" making you think everything is good to go.