Generic deployments of Data Factory by leveraging Git integration

%3CLINGO-SUB%20id%3D%22lingo-sub-733668%22%20slang%3D%22en-US%22%3EGeneric%20deployments%20of%20Data%20Factory%20by%20leveraging%20Git%20integration%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-733668%22%20slang%3D%22en-US%22%3E%3CP%3EHi!%26nbsp%3B%3C%2FP%3E%3CP%3E%3CBR%20%2F%3EI%20am%20trying%20to%20create%20a%20generic%20centralized%20version%20control%20for%20several%20deployments%20of%20Azure%20Data%20Factory%20at%20our%20customers.%20One%20thing%20I%20have%20noticed%20is%20that%20when%20you%20enable%20Git%20integration%20natively%20in%20ADF%2C%20is%20that%20sensitive%20information%20such%20as%20bearer%20tokens%20that%20are%20defined%20in%20datasets%20is%20saved%20in%20plain%20text.%20I%20have%20learned%20that%20keeping%20secrets%2Fkeys%20in%20Git%20is%20not%20a%20best%20practice%20and%20my%20company%20is%20not%20allowing%20me%20to%20do%20this.%3CBR%20%2F%3E%3CBR%20%2F%3EThis%20is%20what%20I'm%20trying%20to%20do%20on%20a%20high%20level%3A%3C%2FP%3E%3CP%3E%3CBR%20%2F%3E-%20Keep%20a%20centralized%20ADF%20configuration%20repository%20in%20our%20Azure%20Devops%20Repository%20that%20is%20linked%20with%20a%20'donor'%20ADF%20in%20our%20Azure%20tenant.%3C%2FP%3E%3CP%3E-%20Deploy%20a%20generic%20version%20of%20the%20ADF%20configuration%20to%20other%20customers%20via%20ARM%20deployment.%20(keys%20%26amp%3B%20urls%2C%20etc%20needs%20to%20be%20parameterized%2Ftokenized)%3C%2FP%3E%3CP%3E-%20When%20a%20change%20is%20made%20on%20the%20centralized%20ADF%20configuration%2C%20it%20needs%20to%20be%20reflected%20in%20the%20customers'%20ADF.%3C%2FP%3E%3CP%3E%3CBR%20%2F%3EDoes%20anyone%20have%20experience%20in%20this%3F%3CBR%20%2F%3E%3CBR%20%2F%3EThanks!%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-733668%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAutomation%20%26amp%3B%20Control%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EData%20%2B%20Storage%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EDevOps%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Highlighted
New Contributor

Hi! 


I am trying to create a generic centralized version control for several deployments of Azure Data Factory at our customers. One thing I have noticed is that when you enable Git integration natively in ADF, is that sensitive information such as bearer tokens that are defined in datasets is saved in plain text. I have learned that keeping secrets/keys in Git is not a best practice and my company is not allowing me to do this.

This is what I'm trying to do on a high level:


- Keep a centralized ADF configuration repository in our Azure Devops Repository that is linked with a 'donor' ADF in our Azure tenant.

- Deploy a generic version of the ADF configuration to other customers via ARM deployment. (keys & urls, etc needs to be parameterized/tokenized)

- When a change is made on the centralized ADF configuration, it needs to be reflected in the customers' ADF.


Does anyone have experience in this?

Thanks!

 

0 Replies