SOLVED

AIP Log Analytics duplicated records

%3CLINGO-SUB%20id%3D%22lingo-sub-1421743%22%20slang%3D%22en-US%22%3EAIP%20Log%20Analytics%20duplicated%20records%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1421743%22%20slang%3D%22en-US%22%3E%3CP%3E%3CFONT%3EHi%20all%2C%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CFONT%3EDo%20you%20know%20why%20I%20have%20duplicates%20records%20on%20some%20of%20my%20scanned%20files%20in%20my%20Log%20Analytics%3F%20%2C%20the%20only%20difference%20I%20see%20is%20the%20LogId_g%20which%20is%20different.%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CFONT%3EThank%20you.%3C%2FFONT%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1421743%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1423889%22%20slang%3D%22en-US%22%3ERe%3A%20AIP%20Log%20Analytics%20duplicated%20records%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1423889%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F681405%22%20target%3D%22_blank%22%3E%40VoTran%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20have%20answered%20your%20own%20question%2C%20if%20there%20is%20a%20difference%20then%20it%20will%20be%20sent%20(even%20if%20the%20time%2Fdate%20is%20the%20same)%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1424628%22%20slang%3D%22en-US%22%3ERe%3A%20AIP%20Log%20Analytics%20duplicated%20records%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1424628%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ET%3CFONT%3Ehank%20you%20Clive%20for%20the%20answer.%20But%20do%20you%20have%20an%20explanation%20why%20the%20scanned%20files%20would%20have%20another%20different%20logid_g%20if%20the%20AIP%20scanner%20has%20only%20scanned%201%20time%20in%20this%20repository%3F%3C%2FFONT%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1424803%22%20slang%3D%22en-US%22%3ERe%3A%20AIP%20Log%20Analytics%20duplicated%20records%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1424803%22%20slang%3D%22en-US%22%3ESorry%20I%20don't%20know%20this%20particular%20table%2C%20or%20what%20logid_g%20contains.%20Its%20possible%20that%20its%20an%20artefact%20of%20a%20retry%2C%20some%20logs%20do%20get%20multiple%20rows%20when%20there%20is%20a%20retry.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1761773%22%20slang%3D%22en-US%22%3ERe%3A%20AIP%20Log%20Analytics%20duplicated%20records%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1761773%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%2C%3C%2FP%3E%3CP%3ELike%26nbsp%3B%3CSPAN%3E%3Ca%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F681405%22%3E%40VoTran%3C%2Fa%3E%26nbsp%3B%3C%2FSPAN%3EI%20am%20also%20seeing%20duplicate%20rows%20in%20my%20Log%20Analytics%20workspace%20for%20AIP.%26nbsp%3B%20This%20is%20from%20an%20AIP%20Scanner%20job.%20Every%20file%20discovered%20by%20the%20scanner%20has%20a%20duplicate%20row.%3C%2FP%3E%3CP%3ELogid_g%20looks%20like%20some%20sort%20of%20GUID%2C%20but%20I%20cannot%20find%20a%20complete%20schema%20reference%20for%20InformationProtectionLogs_CL%20table%20(some%20of%20the%20columns%20are%20described%20here%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Finformation-protection%2Freports-aip%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Finformation-protection%2Freports-aip%3C%2FA%3E)%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWith%20the%20Distinct%20operator%20in%20KQL%20I've%20managed%20to%20de-dupe%20the%20rows%20for%20reporting%2C%20however%20it%20would%20be%20fantastic%20if%20we%20could%20get%20an%20explantation%20for%20the%20duplicates%20and%20complete%20schema%20reference%20if%20possible%20please%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E
New Contributor

Hi all,

 

Do you know why I have duplicates records on some of my scanned files in my Log Analytics? , the only difference I see is the LogId_g which is different.

 

Thank you.

4 Replies

@VoTran 

 

You have answered your own question, if there is a difference then it will be sent (even if the time/date is the same)

@Clive Watson 

 

Thank you Clive for the answer. But do you have an explanation why the scanned files would have another different logid_g if the AIP scanner has only scanned 1 time in this repository?

Best Response confirmed by VoTran (New Contributor)
Solution
Sorry I don't know this particular table, or what logid_g contains. Its possible that its an artefact of a retry, some logs do get multiple rows when there is a retry.

Hi @Clive Watson,

Like @VoTran I am also seeing duplicate rows in my Log Analytics workspace for AIP.  This is from an AIP Scanner job. Every file discovered by the scanner has a duplicate row.

Logid_g looks like some sort of GUID, but I cannot find a complete schema reference for InformationProtectionLogs_CL table (some of the columns are described here: https://docs.microsoft.com/en-us/azure/information-protection/reports-aip)

 

With the Distinct operator in KQL I've managed to de-dupe the rows for reporting, however it would be fantastic if we could get an explantation for the duplicates and complete schema reference if possible please?