SOLVED

OMS Query for exporting results in excel more than 10000 results

Brass Contributor

Hi,

I have a simple query to fetch complete computer name based on AD Site code and tried running below queries.


But certain AD sites has more that 10000 computer and only results first 10000 computername in Advanced Analytics and 5000 in Log Search.

 

I can't figure out how to export next 10,000 computer name from Advanced Analytics in Azure Log Analytics - there must be a simple way - but how? Can we do this and advice me in the right direction for the query to run export fetch the next 10,000 computer name from Advanced Analytics?

 

Sample Queries:

UAApp | project Computer, ComputerID, AppName, AppCategory , AppLanguage, Importance | where Computer contains "*LOCATIONCODE*"

 

UAApp | project Computer, ComputerID, AppName, AppCategory , AppLanguage, Importance | where Computer startswith "*LOCATIONCODE*"

 

search *
| where ( Type == "UAApp" ) | project Computer, ComputerID, AppName, AppCategory , AppLanguage, Importance | where Computer contains "*LOCATIONCODE*"

8 Replies
best response confirmed by Balasubramanian Murugesan (Brass Contributor)
Solution

Hi,

Indeed there is a 10K cap on the result set size in the UI, but there are a number of ways to handle larger result sets.

1. You can transform your results into an array, which can hold much more than 10K values. See this example, where over 40K values are put into a single array, that you can later export to excel. That would of mean you need to use excel formulas if you want to return to a tabular structure.

2. Reduce the size of your results - you can use "distinct Computer", "summarize by Computer" or "summarize makeset" to remove duplicate values from your results (Also, if all you need is that computer's name, project only that column).

3. Use the API - the 10K results limit is only a UI limitation, so if you use the API you'll always get the full result set. In your script, you can process the result set to create a CSV structure which excel can load.

 

HTH,

Noa

Hi,

Thanks for the points and I tried querying now by using distinct command and getting somehow the expected result.

ex:  UAComputer | project Computer | where Computer startswith "XXXX"

 

Can you advice on API as said below. Whether we have the API to get the full result set?

Hi,

 

The Rest API is documented here. To try querying your WS with your account easily, you can also use this API documentation site which has a "Try it" option next to each function.

This is a general limitation , not only for export to excel, same for PowerBI.  For large tenants like more as 100k spo sites/o365 groups or Onedrive user same  problem if you build runbooks or use out of the box modules like the o365 report module.
We need for example the possibility to export 150k items in flat query view.

Noa, could you please share the array example here in the post ? Access denied on your workspace.

Thanks a lot.

@Heiko Fuhrmann 

Heartbeat
| where Computer contains "con"
| project computer_result=strcat(Computer, ComputerIP, Category, OSName)
| summarize computer_results=makelist(computer_result, 1000000)
 
 

@Noa Kuperberg How do we convert into array if what we have is a pivot table (after using evaluate pivot()) instead normal table with normal column names?

This script does exactly what you want - export the data into a CSV. And yes, you will be able to export more than just 10.000 records. 

https://www.powershellgallery.com/packages/Invoke-AzOperationalInsightsQueryExport 

 

 

I think the most you can export is 500.000 records actually.

 

As well as call rate limits and daily quota caps, there are also limits on queries themselves:

  1. Queries cannot return more than 500,000 rows
  2. Queries cannot return more than 64,000,000 bytes (~61 MiB total data)
  3. Queries cannot run longer than 10 minutes by default. See this for details.

 

1 best response

Accepted Solutions
best response confirmed by Balasubramanian Murugesan (Brass Contributor)
Solution

Hi,

Indeed there is a 10K cap on the result set size in the UI, but there are a number of ways to handle larger result sets.

1. You can transform your results into an array, which can hold much more than 10K values. See this example, where over 40K values are put into a single array, that you can later export to excel. That would of mean you need to use excel formulas if you want to return to a tabular structure.

2. Reduce the size of your results - you can use "distinct Computer", "summarize by Computer" or "summarize makeset" to remove duplicate values from your results (Also, if all you need is that computer's name, project only that column).

3. Use the API - the 10K results limit is only a UI limitation, so if you use the API you'll always get the full result set. In your script, you can process the result set to create a CSV structure which excel can load.

 

HTH,

Noa

View solution in original post