End to end Azure Data Explorer pipeline deployment with ARM

[vplauzon]

There are many reasons why we want to automate Azure Data Explorer solution provisioning:  ensuring parity between environments, having fast and low-risk deployments to our production environment, build environments on the fly (e.g. temporary performance test), etc.  .

With a recent feature we introduced, it is now easier than ever to build an entire environment with a single tool:  ARM template.

We can author a single ARM template to deploy:

• Azure Data Explorer cluster
• Databases
• Schema entities on databases:  tables, functions, policies, etc.
• Data Connections:  Event Hub, IoT Hub or Blob Storage connections

A key enabling feature is the ability to deploy KQL scripts.  It is now possible to do this without a storage account to store the scripts by simply passing the scripts inline in the ARM templates.

{
    "type": "Microsoft.Kusto/Clusters/Databases/Scripts",
    "apiVersion": "2022-02-01",
    "name": "myScript",
    "properties": {
        "scriptContent": ".create table Simple(Id:int)\n\n.create table Simple2(Name:string)"
    }
}

This is all possible using traditional JSON ARM templates.  The new Bicep ARM Templates makes it easy to co-locate the scripts in separate KQL files which improves the maintainability of the solution.

resource perfTestDbs 'Microsoft.Kusto/clusters/databases/scripts@2022-02-01' = {
    name: 'myScript'
    properties: {
        scriptContent: loadTextContent('script.kql')
    }
}

This means we can now have our KQL script in a file and all the deployment described in one (or many) Bicep files and deploying it in a single deployment:  no need to break the deployment into steps and run KQL scripts in between!

This will soon be available using Terraform.

This feature makes it possible to implement simpler automation around Azure Data Explorer.  Read the updated online documentation to see how to use it in your deployments.

ADX web explorer team is looking forward for your feedback in KustoWebExpFeedback@service.microsoft.com

You’re also welcome to suggest more ideas and vote for them here - https://aka.ms/adx.ideas