Forum Discussion

sirkillnotalot's avatar
sirkillnotalot
Copper Contributor
Feb 17, 2022

Data Collector API and TimeGenerated property.

Hi guys, I'm trrying to ingest some logs in to a Sentinel workspace but I'm having issues with populating the correct TimeGenerated property. According to the ol' docs: time-generated-field is...
GaryBushey's avatar
GaryBushey
Bronze Contributor
Feb 18, 2022

sirkillnotalot Looking at this documentation: Azure Log Analytics Data Collector - Connectors | Microsoft Docs  it states that the Time-generated-field "Represents the original date and time of the record." but it doesn't actually state that this is the TimeGenerated field (although it does lead one to think it would be).

 

Without adding any of the other date/time related fields, was there any other field that had a date/time in it?   Maybe something like TimeGenerated_t or TimeStamp_t?

sirkillnotalot's avatar
sirkillnotalot
Copper Contributor
Feb 18, 2022

GaryBushey 

The only fields that are date/time was the TimeGenerated passed in the original payload. As a bit of a test I stripped that field out using removeProperty(items('For_each'),'TimeGenerated') and used a static entry in the time-generated-field field instead:

but all I get is:

 

I initially thought the same as you and that the documentation may be misleading or incorrect but I've had a response from an MS support rep who've confirmed that the field is intended to preserve the record time stamp, not the ingested timestamp.

 

Think I might go create a new basic LogicApp and see if I can reproduce using a payload I control via PostMan and see if it's something in the way it processes records in a for_each loop.

 

  • sirkillnotalot's avatar
    sirkillnotalot
    Copper Contributor
    Feb 18, 2022

    OK so I've found the issue and posting resolution here in case anyone comes across this in the future.

     

    The input we are receiving from our source system:

      {
    "type": "HTTP Outbound",
    "creator": "admin",
    "TimeGenerated": "2022-02-18 11:56:27",
    "table": "web_connections",
    "sys_id": "d2443d3c1b31451095a74195d34bcb65",
    "payload": {
      "hostname": "exampleurl.com",
      "sequence": "5310",
      "method": "GET",
      "url": "https://exampleurl.com",
      "response_status": "200",
      "source_table": "web_connections",
      "source_record": "2454c2271ba22c10fda632e8cd4bcb99",
      "transaction_name": "HTTP Outbound",
      "user_name": "root"
    }
  }

     

    Issue 1

    The connector doesn't like it if the payload with the date/time you're using is called TimeGenerated so I needed to:

    • add a new property with the TimeGenerated values:
      addProperty(triggerBody(),'TimeStamp',formatDateTime(triggerBody()?['TimeGenerated']))
    • to remove the timegenerated property:
      removeProperty(outputs('Compose'),'TimeGenerated')

     

    Issue 2

    Usual formating mismatch so ran the formatdatetime() on the TimeGenerated field to convert to ISO8601 format

     

    Issue 3

    Probably a bit of a misunderstanding on my part - the 'time-generated-field' is looking for the property name, not the actual value.

     

     

    When it all works it'll populate the table using the TimeStamp field as both TimeGenerated, and TimeGenerated_s. The original TimeStamp property is actually removed.

     

    Interesting, and also frustratingly, despite the input for the 'time-generated-field' being wrong the API still returns "statusCode: 200" making you think everything is good to go.

     

Resources