Azure Data Explorer and Azure Monitor Integration | Short Survey

Hi Deepak (DeepakGargMsft)

Have you checked out the "Insights" blade? If not, you can read more about ADX Insights in the doc and the post.
ADX Insights can help you assess the health of your cluster and detect the top CPU and memory consumers (users and applications), as well as other important aspects like the hot cache usage per table, batching ingestion monitoring, metalized views usage, and more.
ADX Insights is based on metrics and diagnostic logs that you should stream to a log analytics workspace. Once you enable the diagnostic logs, you can query the data using KQL. ADX Insights also surfaces Advisor recommendations.
The cluster memory is currently not exposed as a metric, but you can use Insights and the logs to monitor the memory consumption of your queries.
Let me know if there are any other questions. We can also set up a quick call to discuss these topics. 

Hi Guy,

I am trying to build a service health score card for our program. Within in our program, we are using multiple azure services and Azure Data Explorer is the one we are using heavily. I want to understand that Is there a way where I can query the availability state of the kusto cluster using KSQL. I could see that there is a way to know the kusto service health using resource health (preview) feature.

Also, Is there a direct way where I can query the cluster memory usage and do you recommend any threshold which I should watch for any issue respect to performance of our cluster.

We have seen multiple issues in past where some of our customers were running repeated long queries which was exhausting memory thus impacting our cluster performance. 

Thanks & Regards,

Deepak

DeepakGargMsft, the ability to export the data of “.show queries”, “.show commands” to Log Analytics will be available in the following weeks. We will publish an article when it is available.

We are also working on new built-in monitoring experiences, including:

• Resource Health integration - 'Resource Health' determines the health of your Azure Data Explorer resource by examining various health status checks such as resource availability, query failures, low ingestion success rate and high update policy failures rate. 
• Azure Advisor (in preview for internal MS users) - Analyzes your configurations and usage telemetry and offers personalized, actionable recommendations to help you optimize your Azure resources (including Azure Data Explorer) and help you improve availability and performance, achieve operational excellence, and reduce costs. 
  • Our preview recommendations include a recommendation to change machine SKU or instance count (based on parameters like data capacity and CPU utilization). Another recommendation is to consider deleting or suspending empty or unused clusters. We are now working on a new cache optimization recommendations.
• For troubleshooting, you can use the 'Diagnose and solve problems' blade. Currently, you can see manual troubleshooting steps for Kusto queries time-out. We are working on adding more troubleshooting capabilities to this blade.
• Geneva/Jarvis and Azure Support Center are MS-internal tools. You can use Geneva to monitor your cluster. Soon, we will integrate with Azure Support Center as well. This tool will provide dynamic troubleshooting insights and diagnostic tools.
  

Thanks,
Guy Reginiano
Azure Data Explorer (Kusto)

Guy_Reginiano - Thanks for the reply. Can you please let me know when by this feature would be available? 

Also, can you please recommend what we can do to monitor the Kusto cluster Health? Is there any integration available with other system (Geneva/Jarvis) where we can check health of our kusto cluster?

DeepakGargMsft, thanks for your question. Based on the survey results, we just started to work on this new integration (it is not available yet). With this new integration, you will be able to query “.show queries”, “.show commands” from Log Analytics, create alerts according to the query state (failed queries, for example), and create workbooks (visualizations). It would be great to hear more about your monitoring use-cases.

Thanks,
Guy Reginiano
Azure Data Explorer (Kusto)

Hey, we have enabled the Diagnostics settings for our ADX cluster. Is it possible to query the .show commands, .show queries and other commands from log analytics? We want to create alerts whenever a query and/or command is failing and also want to use .show diagnostics to know the health of ADX cluster and other details.

Thanks,