Automating deployment on a project is an up-front cost that pays dividends as the project evolves, generating agility. When a process is automated, the cost of running it is low: it becomes easy to destroy and rebuild an environment, to synchronize different environments, to rollback to a previous configuration, etc. .
We structured the approach in three steps:
For each step we provided a list of tools & approaches one can use. For instance, deploying infrastructure (ADX cluster, its databases, data connections, VNET configuration, etc.), can be done with ARM templates, Terraform or imperative approach (e.g. Azure CLI).
Some projects will use only some of those steps. Some will use all of them. An IT team is only responsible for the infrastructure. A project team might be responsible for the schema entities (tables, functions, policies, etc.). A more integrated DevOps team might be responsible for both and will also load data in their test environment to run automated tests and / or prepare a QA environment.
DevOps Job ResultAt the end of the article we gave a concrete example with an Azure DevOps Pipeline orchestrating those three steps. This complete pipeline deploys a cluster with a database, 2 tables and loads data into one of the tables.
We hope this guidance helps you accelerate the automation of your projects. We encourage you to go and start automating parts of your projects to gain agility. Start with low-hanging fruits, measure the benefit and continue when it makes sense.