Forum Discussion
Azure Data Explorer API Limits
I'm making REST API requests, and am getting "An existing connection was forcibly closed by the remote host" -- I can't find in your otherwise awesome docs what the API request limits are, could you ...
As long as it is a 429 response code then it should be limits being hit, otherwise it could be something else. In case it helps, query limits are outlined here:
https://docs.microsoft.com/en-us/azure/data-explorer/kusto/concepts/querylimits
Further information:
https://docs.microsoft.com/en-us/azure/data-explorer/kusto/concepts/resulttruncation
https://docs.microsoft.com/en-us/azure/data-explorer/kusto/concepts/querylimits
Further information:
https://docs.microsoft.com/en-us/azure/data-explorer/kusto/concepts/resulttruncation
Thanks David!
I'm with you on it being a query limit: that's what I first thought too. But I only have 40000 rows, and each of my queries should only return at most five rows. I'm using "project" to only call three columns, and I'm using a pretty strict "where" clause.
which makes me think it's not a query limit. I am iterating through a CSV file to do "matches" with KQL against the Data Explorer API, so I'm making 40,000 requests. I increased the server timeout to the max (60 min), but it still only makes about 20,000 requests before I bombs. So what should I be writing the rate limit to?
I'm with you on it being a query limit: that's what I first thought too. But I only have 40000 rows, and each of my queries should only return at most five rows. I'm using "project" to only call three columns, and I'm using a pretty strict "where" clause.
which makes me think it's not a query limit. I am iterating through a CSV file to do "matches" with KQL against the Data Explorer API, so I'm making 40,000 requests. I increased the server timeout to the max (60 min), but it still only makes about 20,000 requests before I bombs. So what should I be writing the rate limit to?
- Check the dataset size being returned (and in total). Sounds like the process may need to either be adjusted or broken down. If it is taking 60 minutes to return 20,000 requests I am figuring it is quite a large dataset. You can use this to play with the memory limit:
https://docs.microsoft.com/en-us/azure/data-explorer/kusto/concepts/querylimits#limit-on-memory-per-iterator
Otherwise potentially throttling the requests from the client being used to rate limit is one way to put a governor or ballast on the processing.
The following post is worth a read, towards the end it talks limitations and gives some examples of what can be done (which saves me typing 🙂 )
https://xhinker.medium.com/understand-kusto-inside-out-and-why-kusto-is-so-fast-54697e6648d7
Hopefully that helps somewhat.