Downloading mass of Unified Audit Logs (UAL) data

New Contributor

Hi,

 

I work in data forensics. We often need to download months of UAL data from customers' Office 365 environment to analyze incidents. For example, I recently had to download 3 months of data, which summed up to 30 GB of data (CSV w/ embedded JSON). We do not want to filter on RecordType, UserIds, Operations, Workload, etc. We need everything.

We are currently downloading data, by slices of 15 minutes, using the Search-UnifiedAuditLog PowerShell command. If we use bigger intervals, we experience errors (e. g., missing/empty data, timeouts, crashes, etc). Even then, we still experience errors every now and then. Also, we can never be sure that our data is 100% complete.

All in all, it can take us up to 4 work days to be able to download a full set of 3 months / 30 GB of UAL data. That is the first step before we can start analyzing the data (e. g., importing the data to a database, adding indexes, augmenting the data with other sources of information, running queries, building new queries based on the specific incident, etc). The process is slow and painful. I have even started catching exceptions and sending them by SMS to my personal cell phone.

Would any of you know of a more suitable way of gathering heaps of UAL data? Note: not downloading a copy of the whole data (e. g., running queries manually through the Security & Compliance Center) is out of the question, partly for preservation / legal reasons.

Thanks! :)

Regards,

Simon

2 Replies

Exchange Remote PowerShell is definitely not the best tool to work with such amounts of data. 

Take a look at the Management activity APIs instead: https://docs.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api...

 

@Vasil MichevThis will not work for us as it would require too much setup plus we cannot predict which of our customers have Active Directory LDAP, Azure Active Directory (AAD) or neither.