Search-UnifiedAuditLog returns records with duplicate ID fields

[Stephen_Tyrrell]

Microsoft's defines the unified audit log ID field as "The ID of the report entry. The ID uniquely identifies the report entry" (ref: Detailed properties in the audit log - https://learn.microsoft.com/en-us/microsoft-365/compliance/audit-log-detailed-properties?view=o365-w... (updated 2022-07-12). 

However, I am downloading records where some have Identical Id values. I have a basic script for exporting all records from the unified audit log within an hour (shown at end of post). It is never executed unless 24 hours has passed since the record was generated. So far my testing has confirmed that each record's Identity and AuditData.Id fields match, as expected.

I also check every record for an error value (i.e. negative) in the ResultIndex field and trigger a timeout and retry if found (ref: https://techcommunity.microsoft.com/t5/office-365/searching-audit-log-strange-behavior/m-p/2079399#M...). I have yet to actually encounter an instance of this particular scenario.

At first I was thinking of just filtering out the duplicates from the logs downloaded, but decided to verify that the AuditData (JSON) payload between each duplicate record is really identical. In most cases they do match, but much every now and then I find ones that do not!

A unified diff of two (pretty-printed) record's AuditData JSON reveals the only difference (in this case) is in the value associated with the RequestType key:

--- C:\temp\non-identical_json_with_same_id2.json
+++ C:\temp\non-identical_json_with_same_id1.json
@@ -25,7 +25,7 @@
        },
        {
            "Name": "RequestType",
-           "Value": "SAS:EndAuth"
+           "Value": "SAS:BeginAuth"
        }
    ],
"ModifiedProperties":

The full (anonymized) AuditData content is shown below, in minified form:

C:\temp\non-identical_json_with_same_id1.json:

 {"CreationTime":"2022-11-14T08:06:42","Id":"18fXXXXX-XXXX-XXXX-XXXXXXXXXX","Operation":"UserLoginFailed","OrganizationId":"05bXXXXX-XXXX-XXXX-XXXXXXXXXX","RecordType":"AzureActiveDirectoryStsLogon","ResultStatus":"Success","UserKey":"49cXXXXX-XXXX-XXXX-XXXXXXXXXX","UserType":"System","Version":1,"Workload":"AzureActiveDirectory","ClientIP":"XXX.35.XXX.10X","ObjectId":"Unknown","UserId":"Not Available","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Success"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.35"},{"Name":"RequestType","Value":"SAS:BeginAuth"}],"ModifiedProperties":[],"Actor":[{"ID":"49cXXXXX-XXXX-XXXX-XXXXXXXXXX","Type":0}],"ActorContextId":"05bXXXXX-XXXX-XXXX-XXXXXXXXXX","ActorIpAddress":"XXX.35.XXX.10X","InterSystemsId":"69fXXXXX-XXXX-XXXX-XXXXXXXXXX","IntraSystemId":"18fXXXXX-XXXX-XXXX-XXXXXXXXXX","SupportTicketId":"","Target":[{"ID":"Unknown","Type":0}],"TargetContextId":"05bXXXXX-XXXX-XXXX-XXXXXXXXXX","DeviceProperties":[{"Name":"OS","Value":"Windows 10"},{"Name":"BrowserType","Value":"Edge"},{"Name":"IsCompliantAndManaged","Value":"False"}],"ErrorNumber":"50074","LogonError":"UserStrongAuthClientAuthNRequiredInterrupt"}

C:\temp\non-identical_json_with_same_id2.json:

{"CreationTime":"2022-11-14T08:06:42","Id":"18fXXXXX-XXXX-XXXX-XXXXXXXXXX","Operation":"UserLoginFailed","OrganizationId":"05bXXXXX-XXXX-XXXX-XXXXXXXXXX","RecordType":"AzureActiveDirectoryStsLogon","ResultStatus":"Success","UserKey":"49cXXXXX-XXXX-XXXX-XXXXXXXXXX","UserType":"System","Version":1,"Workload":"AzureActiveDirectory","ClientIP":"XXX.35.XXX.10X","ObjectId":"Unknown","UserId":"Not Available","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Success"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.35"},{"Name":"RequestType","Value":"SAS:EndAuth"}],"ModifiedProperties":[],"Actor":[{"ID":"49cXXXXX-XXXX-XXXX-XXXXXXXXXX","Type":0}],"ActorContextId":"05bXXXXX-XXXX-XXXX-XXXXXXXXXX","ActorIpAddress":"XXX.35.XXX.10X","InterSystemsId":"69fXXXXX-XXXX-XXXX-XXXXXXXXXX","IntraSystemId":"18fXXXXX-XXXX-XXXX-XXXXXXXXXX","SupportTicketId":"","Target":[{"ID":"Unknown","Type":0}],"TargetContextId":"05bXXXXX-XXXX-XXXX-XXXXXXXXXX","DeviceProperties":[{"Name":"OS","Value":"Windows 10"},{"Name":"BrowserType","Value":"Edge"},{"Name":"IsCompliantAndManaged","Value":"False"}],"ErrorNumber":"50074","LogonError":"UserStrongAuthClientAuthNRequiredInterrupt"}

The following code fragment is representative of how I am obtaining the data:

# retrieve hour's records... PowerShell 7.3.1

$MAX_TOTAL_LOG_RECORDS = 50000
$RESULT_PAGE_SIZE = 5000

$StartDate = (Get-Date).AddDays(-2)
$EndDate = $StartDate.AddHours(1)
$SessionId = (New-Guid).Guid

Do
{
    [array]$HoursLogRecords += Search-unifiedAuditLog -StartDate $StartDate -EndDate $EndDate `
        -SessionId $SessionId -SessionCommand ReturnLargeSet -ResultSize $RESULT_PAGE_SIZE `
        -Formatted -Verbose -ErrorAction "Stop" -WarningAction Stop

    If (-not $?)
    {
        Throw "Error executing Search-UnifiedActionLog"   
    }
    ElseIf ($null -ne $HoursLogRecords -and $HoursLogRecords.Count -gt $MAX_TOTAL_LOG_RECORDS)
    {
        Throw "Maximum log record quantity ($MAX_TOTAL_LOG_RECORDS) reached: $($HoursLogRecords.Count)"
    }
    ElseIf ($HoursLogRecords.Count -eq 0)
    {
        Write-Warning "Retrieved $($HoursLogRecords.Count) records for the current hour; an unlikely but possible scenario"
        break
    }
}
while ($HoursLogRecords.Count % $RESULT_PAGE_SIZE -eq 0 -and $HoursLogRecords.Count -ne 0)

# check ResultIndex fields for error values

Foreach ($Record in $HoursLogRecords)
{
    If ($Record.ResultIndex -lt 0)
    {
        Throw "Error value detected in'ResultIndex' property"
    }
}

Can anyone please shed some light on this? Specifically:

Are duplicate records redundant or do they represent discrete/independent events in M365?
Why are records with non-identical content being given the same Id?