I am trying to create a generic centralized version control for several deployments of Azure Data Factory at our customers. One thing I have noticed is that when you enable Git integration natively in ADF, is that sensitive information such as bearer tokens that are defined in datasets is saved in plain text. I have learned that keeping secrets/keys in Git is not a best practice and my company is not allowing me to do this.
This is what I'm trying to do on a high level:
- Keep a centralized ADF configuration repository in our Azure Devops Repository that is linked with a 'donor' ADF in our Azure tenant.
- Deploy a generic version of the ADF configuration to other customers via ARM deployment. (keys & urls, etc needs to be parameterized/tokenized)
- When a change is made on the centralized ADF configuration, it needs to be reflected in the customers' ADF.