Home
%3CLINGO-SUB%20id%3D%22lingo-sub-1151492%22%20slang%3D%22en-US%22%3ENew%20data%20flow%20features%3A%20Import%20schema%20and%20test%20connection%20from%20debug%20cluster%2C%20custom%20sink%20ordering%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1151492%22%20slang%3D%22en-US%22%3E%3CP%3ESeveral%20new%20features%20were%20added%20to%20mapping%20data%20flows%20this%20past%20week.%20Here%20are%20some%20of%20the%20highlights%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EImport%20Schema%20from%20debug%20cluster%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20can%20now%20use%20an%20active%20debug%20cluster%20to%20create%20a%20schema%20projection%20in%20your%20data%20flow%20source.%26nbsp%3B%3C%2FP%3E%0A%3CDIV%3E%0A%3CDIV%3E%3CSPAN%3EAvailable%26nbsp%3Bin%26nbsp%3Bevery%26nbsp%3Bsource%26nbsp%3Btype%2C%26nbsp%3Bimporting%26nbsp%3Bthe%26nbsp%3Bschema%20will%26nbsp%3Boverride%26nbsp%3Bthe%26nbsp%3Bprojection%26nbsp%3Bdefined%26nbsp%3Bin%26nbsp%3Bthe%20dataset.%26nbsp%3BThe%26nbsp%3Bdataset%26nbsp%3Bobject%26nbsp%3Bwill%26nbsp%3Bnot%26nbsp%3Bbe%26nbsp%3Bchanged.%20All%20previously%20existing%20methods%20of%20creating%20and%20modifying%20schemas%20are%20still%20valid%20and%20compatible.%3C%2FSPAN%3E%3C%2FDIV%3E%0A%3C%2FDIV%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20697px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F169057iCB5D061C27F7EB32%2Fimage-dimensions%2F697x338%3Fv%3D1.0%22%20width%3D%22697%22%20height%3D%22338%22%20alt%3D%22clipboard_image_0.png%22%20title%3D%22clipboard_image_0.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20more%20information%20on%20the%20projection%20tab%2C%20see%20the%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fdata-flow-source%23projection%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Edata%20flow%20source%20documentation%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ETest%20connection%20on%20Spark%20Cluster%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20can%20use%20an%20active%20debug%20cluster%20to%20verify%20data%20factory%20can%20connect%20to%20your%20linked%20service%20when%20using%20Spark%20in%20data%20flows.%20This%20is%20useful%20as%20a%20sanity%20check%20to%20ensure%20your%20dataset%20and%20linked%20service%20are%20valid%20configurations%20when%20used%20in%20data%20flows.%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ECustom%20sink%20ordering%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CDIV%3EIf%20you%20have%20multiple%20destinations%20in%20your%20data%20flow%2C%20you%20can%20now%20specify%20the%20write%20order%20of%20your%20data%20flow.%20Nondeterministic%20by%20default%2C%20enabling%20custom%20sink%20ordering%20allows%20for%20sequential%20writes%20of%20your%20data%20flow%20sinks.%3C%2FDIV%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20509px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F169058i17D54AD47A7FA546%2Fimage-dimensions%2F509x515%3Fv%3D1.0%22%20width%3D%22509%22%20height%3D%22515%22%20alt%3D%22clipboard_image_1.png%22%20title%3D%22clipboard_image_1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20more%20information%2C%20see%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fdata-flow-sink%23custom-sink-ordering%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ecustom%20sink%20ordering%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1151492%22%20slang%3D%22en-US%22%3E%3CP%3EMapping%20data%20flow%20releases%20new%20features!%20Now%20you%20can%20use%20your%20interactive%20debug%20cluster%20to%20verify%20your%20connection%20credentials%20on%20Spark%20and%20import%20schemas%20for%20your%20data%20sources.%20You%20can%20also%20specify%20your%20sink%20write%20order%20if%20subsequents%20writes%20are%20desired.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1151492%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Data%20Integration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20ETL%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMapping%20Data%20Flows%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Microsoft

Several new features were added to mapping data flows this past week. Here are some of the highlights:

 

Import Schema from debug cluster

 

You can now use an active debug cluster to create a schema projection in your data flow source. 

Available in every source type, importing the schema will override the projection defined in the dataset. The dataset object will not be changed. All previously existing methods of creating and modifying schemas are still valid and compatible.

 

clipboard_image_0.png

 

For more information on the projection tab, see the data flow source documentation 

 

Test connection on Spark Cluster

 

You can use an active debug cluster to verify data factory can connect to your linked service when using Spark in data flows. This is useful as a sanity check to ensure your dataset and linked service are valid configurations when used in data flows. 

 

Custom sink ordering

 

If you have multiple destinations in your data flow, you can now specify the write order of your data flow. Nondeterministic by default, enabling custom sink ordering allows for sequential writes of your data flow sinks.

 

clipboard_image_1.png

 

For more information, see custom sink ordering