Hi,
We have a setup where we send IoT data to Azure IoT Hub, which then stores this data as AVRO blobs to ADSL Gen2, batching the IoT messages over a few minutes into a blob. We have set up ADX to ingest this data, where each blob is read on creation via Event Grids and Event Hubs. All of this has worked nicely until recently, each IoT message getting its own record in ADX, and the Body field in each IoT message automatically stored as a dictionary, which is convenient since it is originally JSON data from the IoT device.
Then since around 2022-07-25T06:46:00Z the Body fields are no longer parsed as dictionaries, but stored as a raw byte array instead, see examples below. The array can however be transformed to a dictionary with
parse_json(make_string(Body))
in a query, giving the expected result.
We have not changed the setup, redeployed or done anything that should trigger any changes around this time. No commands or operations were done. We have also checked various logs and insights, and not seen anything out of the ordinary. The only thing we noticed around the time that data changed, was AzureActivity logs at 2022-07-25T06:52:11Z, i.e. 6 minutes later, where opreations with OperationNameValue Microsoft.Resourcehealth/healthevent/Activated/action were triggered for all resources, perhaps indicating a restart.
Does anyone know what can have caused this? Is there something we can do to fix this? Or do we simply have to set up a mapping transformation manually now for some reason, which was not necessary before?
Here is a redacted example (production data) of a record in ADX before the change:
"EnqueuedTimeUtc": 2022-07-25T06:46:18.0700000Z,
"Properties": {"isKnown":"true"},
"SystemProperties": {
"connectionDeviceId": <REDACTED>,
"connectionAuthMethod": "{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"connectionDeviceGenerationId": <REDACTED>,
"contentType": "application/json",
"contentEncoding": "utf-8",
"enqueuedTime": "2022-07-25T06:46:18.0700000Z"
},
"Body": {
"objectType": "FlowMonitorEvent",
"objectVersion": 2,
"timestamp": "2022-07-25T06:46:18.1560000Z",
<MORE DATA, REDACTED>
},
Here is a redacted example (production data) of a record in ADX after the change:
"EnqueuedTimeUtc": 2022-07-25T06:46:26.2640000Z,
"Properties": {"isKnown":"true"},
"SystemProperties": {
"connectionDeviceId": <REDACTED>,
"connectionAuthMethod": "{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"connectionDeviceGenerationId": <REDACTED>,
"contentType": "application/json",
"contentEncoding": "utf-8",
"enqueuedTime": "2022-07-25T06:46:26.2640000Z"
},
"Body": [
123,
13,
10,
32,
32,
34,
111,
98,
106,
101,
<MORE DATA, REDACTED>
13,
10,
125
],
Thanks in advance for any hints or solutions,
Øyvind T. Hovdsveen
