%3CLINGO-SUB%20id%3D%22lingo-sub-1529012%22%20slang%3D%22en-US%22%3EAzure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1529012%22%20slang%3D%22en-US%22%3E%3CP%3EWe%20are%20glad%20to%20announce%20that%20now%20in%20Azure%20Data%20Factory%2C%20you%20can%20extract%20data%20from%20XML%20files%20by%20using%20copy%20activity%20and%20mapping%20data%20flow.%20With%20such%20capability%2C%20you%20can%20either%20directly%20load%20XML%20data%20to%20another%20data%20store%2Ffile%20format%2C%20or%20%3CSPAN%3Etransform%20your%20XML%20data%20and%20then%20store%20the%20results%20in%20the%20lake%20or%20database%3C%2FSPAN%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EXML%20format%20is%20supported%20on%20all%20the%20file-based%20connectors%20as%20source.%20You%20can%20find%20the%20new%20%22XML%22%20option%20in%20the%20available%20format%20list%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Linda_Wang_0-1594989922662.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F206050i0094C09BF56A822D%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22Linda_Wang_0-1594989922662.png%22%20alt%3D%22Linda_Wang_0-1594989922662.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhen%20reading%20data%20from%20XML%20files%2C%20you%20have%20the%20flexibility%20to%20define%20whether%20to%20apply%20schema%20validation%2C%20to%20specify%20how%20to%20handle%20namespaces%2C%20and%20to%20map%20selective%20fields%20or%20reshape%20the%20hierarchy%20to%20match%20downstream%20data%20need.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CEM%3ECopy%20activity%20source%20example%3A%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Linda_Wang_1-1594989936019.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F206052iE48A94E418C68D7C%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22Linda_Wang_1-1594989936019.png%22%20alt%3D%22Linda_Wang_1-1594989936019.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20mapping%20data%20flow%2C%20you%20can%20use%20XML%20as%20source%20either%20via%20an%20XML%20dataset%20or%20via%20inline%20dataset.%3C%2FP%3E%0A%3CP%3E%3CEM%3EData%20flow%20source%20example%20with%20XML%20dataset%3A%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Linda_Wang_2-1594989956975.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F206053i138F5EEBAD1D1AE9%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22Linda_Wang_2-1594989956975.png%22%20alt%3D%22Linda_Wang_2-1594989956975.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETo%20learn%20more%20about%20XML%20support%20in%20Azure%20Data%20Factory%2C%20visit%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fformat-xml%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EXML%20format%3C%2FA%3E%26nbsp%3Bin%20the%20documentation%20site.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1529012%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20Data%20Factory%20now%20supports%20reading%20data%20from%20XML%20files%20in%20copy%20activity%20and%20mapping%20data%20flow.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1529012%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Integration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ECopy%20Activity%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMapping%20Data%20Flows%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1529442%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1529442%22%20slang%3D%22en-US%22%3E%3CP%3Ethis%20is%20a%20great%20addition%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1533021%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1533021%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20for%20the%20good%20news.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EFew%20questions%20everybody%20would%20like%20to%20know%20the%20answers%3A%3C%2FP%3E%3CUL%3E%3CLI%3EWhat%20XML%20processor%20is%20in%20use%20behind%20the%20scenes%3F%3CBR%20%2F%3EIs%20it%20.Net%20Framework%20or%20something%20else%3F%3C%2FLI%3E%3CLI%3EWhen%20%3CSTRONG%3Esink%3C%2FSTRONG%3E%20support%20will%20be%20introduced%3F%3C%2FLI%3E%3CLI%3EAny%20future%20XSLT%20support%3F%3C%2FLI%3E%3CLI%3EAny%20future%20XQuery%20support%3F%3C%2FLI%3E%3CLI%3EExcerpt%20from%20the%20documentation%3A%20%3CEM%3E%22...When%20using%20XSD%20or%20DTD%20to%20validate%20XML%20files%2C%20the%20XSD%2FDTD%20must%20be%20referred%20inside%20the%20XML%20files%20through%20relative%20path...%22%3C%2FEM%3E.%3CBR%20%2F%3EIt%20is%20a%20very%20serious%20limitation.%20Are%20you%20working%20to%20remove%20such%20dependency%3F%3C%2FLI%3E%3CLI%3EIs%20the%20ADF%20XSD%20Validation%20conformant%20with%20the%20%3CSTRONG%3EXSD%201.1%3C%2FSTRONG%3E%20standard%3F%3CBR%20%2F%3EXSD%201.1%20became%20a%20W3C%20Recommendation%20in%20April%205%2C%202012.%3C%2FLI%3E%3CLI%3EDoes%20the%20XSD%20Validation%20support%20master%20XSD%20with%20%3CSTRONG%3Eimported%3C%2FSTRONG%3E%20or%20%3CSTRONG%3Eincluded%3C%2FSTRONG%3E%20XSDs%3F%3CBR%20%2F%3EIt%20is%20a%20very%20common%20scenario.%3C%2FLI%3E%3C%2FUL%3E%3CP%3EFor%20example%2C%3C%2FP%3E%3CPRE%3E%26lt%3Bxs%3Aschema%20xmlns%3Axs%3D%22http%3A%2F%2Fwww.w3.org%2F2001%2FXMLSchema%22%26gt%3B%0A%26lt%3Bxs%3Ainclude%20schemaLocation%3D%22includeMe.xsd%22%2F%26gt%3B%0A...%0A%26lt%3B%2Fxs%3Aschema%26gt%3B%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1534011%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1534011%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20is%20great%20news.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAny%20future%20plan%20to%20have%20sink%20support%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1534711%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1534711%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20is%20a%20great%20addition.%20Thanks!!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1534929%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1534929%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F734250%22%20target%3D%22_blank%22%3E%40PuranJoshi%3C%2FA%3E%26nbsp%3BThanks%20for%20the%20feedback.%20For%20now%20we%20don't%20have%20plan%20to%20support%20XML%20as%20sink.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1537256%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1537256%22%20slang%3D%22en-US%22%3E%3CP%3EYeah!%26nbsp%3BI%20expected%20a%20lot%20for%20this!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1538421%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1538421%22%20slang%3D%22en-US%22%3E%3CP%3EExcellent%2Ci%20was%20trying%20to%20convert%20an%20xml%20file%20to%20csv%20through%20a%20Copy%20Activity%20pipe%20line%20but%20getting%20an%20error%20%2C'%20Could%20not%20find%20the%20schema%20information%20for%20the%20element'.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAny%20idea%20about%20this%2Ccan%20some%20one%20help%20me.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ETHank%20you%20%3A)%3C%2Fimg%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1540268%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1540268%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F736418%22%20target%3D%22_blank%22%3E%40BijuNambiarC%3C%2FA%3E%26nbsp%3BThe%20error%20message%20seems%20indicating%20XSD%20not%20found.%26nbsp%3BDid%20you%20configure%20XSD%20schema%20validation%20in%20ADF%3F%20If%20so%2C%20please%20turn%20off%20the%20validation%20and%20try%20again.%20And%20note%20the%20XML%20connector%20behavior%20on%20schema%20validation%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fformat-xml%23xml-connector-behavior%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fformat-xml%23xml-connector-behavior%3C%2FA%3E.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1544876%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1544876%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3BI%20am%20retrieving%20an%20XML%20file%20from%20an%20HTTP%20connection%20in%20Data%20Factory%20(from%20a%20SOAP%20API).%20What%20is%20the%20supported%20method%20to%20sink%2Fsave%20this%20data%20to%20Data%20Lake%20in%20order%20to%20run%20Data%20Flow%20transformations%20on%20the%20data%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1548889%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1548889%22%20slang%3D%22en-US%22%3E%3CP%3EDoes%20anyone%20have%20a%20method%20to%20save%2Fsink%20this%20retrieved%20XML%20data%20from%20the%20HTTP%20connection%3F%20Binary%20isn't%20an%20option%2C%20needs%20to%20be%20something%20that%20can%20be%20parsed%20out%20into%20a%20database.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1549299%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1549299%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3BDo%20you%20have%20an%20example%20of%20this%3F%20I%20can't%20find%20a%20file%20type%20that%20works%20with%20saving%20the%20XML%20to%20the%20Sink%20source.%20I%20need%20it%20to%20be%20something%20that%20I%20can%20parse%20through.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1550049%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1550049%22%20slang%3D%22en-US%22%3E%3CP%3EAs%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3B%20described%2C%20you%20can%20copy%20the%20results%20of%20the%20http%20to%20data%20lake.%20I%20got%20this%20to%20work%20with%20binary%20and%20XML.%20For%20complex%20XML%20support%20I%20was%20able%20to%20copy%20it%20to%20a%20SQL%20server%20instance%20where%20you%20can%20use%20TSQL%20XML%20functions%20to%20parse%20the%20results%2C%20I%20used%20binary%20dataset%20and%20copy%20the%20results%20of%20data%20lake%20and%20copy%20it%20to%20a%20string%20in%20a%20table%2C%20but%20XML%20should%20work%20too%20with%20an%20XML%20data%20type.%20I%20haven't%20tried%20that%20yet%2C%20but%20that%20may%20be%20a%20good%20solution%20for%20complex%20XML%20parsing%20and%20processing%20in%20line%20instead%20of%20saving%20it%20to%20a%20table.%20In%20my%20case%20I%20had%20to%20create%20a%20loop%20because%20there%20is%20a%20limit%20of%20rows%20returned%20on%20the%20soap%20envelope.%20But%20that%20is%20another%20story.%20Hope%20this%20helps.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22WillT985_2-1595944648855.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F208443i13D8BCDE6A130703%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22WillT985_2-1595944648855.png%22%20alt%3D%22WillT985_2-1595944648855.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22WillT985_0-1595944372367.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F208441i83A5B5A9F3819C7F%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22WillT985_0-1595944372367.png%22%20alt%3D%22WillT985_0-1595944372367.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22WillT985_1-1595944595267.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F208442iFE134EB394E667F1%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22WillT985_1-1595944595267.png%22%20alt%3D%22WillT985_1-1595944595267.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1553454%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1553454%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3B%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%20%26nbsp%3BIs%20there%20any%20limitation%20to%20convert%20the%20xml%20files%20to%20csv%20using%20the%20copy%20activity%3F%201%20GB%20file%20gets%20converted%20within%202%20hour%20without%20any%20parallelism%2FDIU%2Fblocks%2C%20but%20a%202GB%20file%20fails%20with%20a%20memory%20exception.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAny%20suggestion%20to%20process%20a%20large%20xml%20file.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThank%20you%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1554515%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1554515%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F732057%22%20target%3D%22_blank%22%3E%40WillT985%3C%2FA%3E%26nbsp%3BI%20must%20be%20missing%20something.%20This%20is%20what%20I%20have%20so%20far.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E1.%20Linked%20Service%3A%20HTTP%20type%2C%20base%20URL%20is%20my%20external%20API.%20Basic%20auth%2C%20username%2C%20password%3C%2FP%3E%3CP%3E2.%20A%20dataset%2C%20type%20HTTP%2FXML%20type%2C%20connected%20to%20my%20Linked%20Service%20above.%20I%20have%20entered%20a%20base%20URL.%20It%20connects%20without%20issue%3C%2FP%3E%3CP%3E3.%20Pipeline%20activity%20of%20%22Copy%20data%22%3C%2FP%3E%3CUL%3E%3CLI%3E%3CFONT%20size%3D%222%22%3ESource%20dataset%3A%20the%20HTTP%2FXML%20dataset%20from%20above%3C%2FFONT%3E%3C%2FLI%3E%3CLI%3E%3CFONT%20size%3D%222%22%3ERequest%20method%3A%20POST%3C%2FFONT%3E%3C%2FLI%3E%3CLI%3E%3CFONT%20size%3D%222%22%3EAdditional%20headers%3A%20Content-Type%3A%20application%2Fsoap%2Bxml%3Bcharset%3DUTF-8%3Baction%3D%22%3CA%20href%3D%22http%3A%2F%2Foutsidesoft.com%2Fwebservices%2FAPI%2FAuthenticate%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttp%3A%2F%2Foutsidesoft.com%2Fwebservices%2FAPI%2FAuthenticate%3C%2FA%3E%22%3C%2FFONT%3E%3C%2FLI%3E%3CLI%3E%3CFONT%20size%3D%222%22%3ERequest%20body%3A%20%3CABC123....%3E%26lt%3B%26gt%3B....%3CAUTHENTICATE%3E...etc%20etc%3C%2FAUTHENTICATE%3E%3C%2FABC123....%3E%3C%2FFONT%3E%3C%2FLI%3E%3C%2FUL%3E%3CP%3E%3CFONT%20size%3D%224%22%3EWhat%20are%20the%20next%20steps%3F%20When%20I%20try%20to%20set%20up%20a%20Sink%20dataset%2C%20I%20choose%20my%20data%20lake%2C%20but%20then%20when%20to%20choose%20a%20format%20is%20when%20I%20run%20into%20a%20problem.%20I%20want%20to%20save%20the%20file%20in%20a%20format%20I%20can%20parse.%20If%20I%20choose%20CSV%2C%20then%20run%20a%20debug%2C%20I%20get%20the%20following%20error%3A%3C%2FFONT%3E%3C%2FP%3E%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CFONT%20size%3D%223%22%3E%3CSTRONG%3EErrorCode%3DSchemaMappingFailedInHierarchicalToTabularStage%2C'Type%3DMicrosoft.DataTransfer.Common.Shared.HybridDeliveryException%2CMessage%3DFailed%20to%20process%20hierarchical%20to%20tabular%20stage%2C%20error%20message%3A%20One%20or%20more%20errors%20occurred.%2CSource%3DMicrosoft.DataTransfer.ClientLibrary%2C'%22%2C%3C%2FSTRONG%3E%3C%2FFONT%3E%3C%2FP%3E%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CFONT%20size%3D%224%22%3EWhat%20are%20the%20proper%20steps%20to%20take%20to%20get%20a%20sink%20working%3F%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%3CFONT%20size%3D%224%22%3EMy%20future%20plans%20are%20to%20do%20another%20%22Copy%20data%22%2C%20and%20set%20up%20the%20variable%20to%20get%20the%20token%20as%20you%20demo'd%20above%2C%20and%20then%20make%20subsequent%20API%20calls%20with%20the%20token%20and%20save%20that%20newly%20retrieved%20data%20to%20the%20data%20lake%20in%20their%20own%20tables.%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CFONT%20size%3D%224%22%3EThank%20you%20all%3C%2FFONT%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1554571%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1554571%22%20slang%3D%22en-US%22%3E%3CP%3ELinda%20mentioned%20configuring%20the%20Sink%20and%20Source%20as%20XML%20for%20Data%20Lake%2C%20but%20this%20is%20not%20an%20option.%20Only%20options%20are%3A%26nbsp%3BAvro%2C%26nbsp%3BBinary%2C%20DelimitedText%2C%20Json%2C%20ORC%2C%20Parquet.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAlso%2C%20Linda%2C%20my%20Sink%20is%20an%20HTTP%2FXML%20dataset%20connected%20to%20a%20Linked%20Source%20type%20of%20HTTP.%20Do%20you%20have%20a%20demo%20of%20how%20to%20setup%20a%20proper%20copy%20that%20uses%20an%20HTTP%20linked%20service%20source%2C%20to%20map%20fields%20from%20a%20SOAP%20API%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIf%20I%20select%20DelimitedText%2FCSV%20for%20Data%20Link%20source's%20format%20and%20debug%20the%20pipeline%2C%20I%20get%20the%20following%20error%3A%3C%2FP%3E%3CP%3E%3CSTRONG%3EErrorCode%3DSchemaMappingFailedInHierarchicalToTabularStage%2C'Type%3DMicrosoft.DataTransfer.Common.Shared.HybridDeliveryException%2CMessage%3DFailed%20to%20process%20hierarchical%20to%20tabular%20stage%2C%20error%20message%3A%20One%20or%20more%20errors%20occurred.%2CSource%3DMicrosoft.DataTransfer.ClientLibrary%2C'%22%2C%3C%2FSTRONG%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1558627%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1558627%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F739439%22%20target%3D%22_blank%22%3E%40scossever%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIn%20my%20case%2C%20First%20I%20created%20a%20copy%20activity%20that%20uses%20a%20binary%20dataset%20source%20type%20to%20call%20the%20Marketing%20Cloud%20SOAP%20Api%20and%20puts%20the%20results(XML%20envelope)%20in%20Data%20lake%20as%20Binary%20Dataset%20Sink%20in%20Data%20lake.%20My%20binary%20source%20dataset%20uses%20an%20http%20linked%20service%2C%20under%20the%20Base%20URL%20you%20would%20use%20your%20API%20url%20from%20your%20MC.%20but%20again%20the%20result%20will%20get%20dumped%20to%20data%20lake%20as%20binary%20sink.%20You%20are%20correct%20that%20there%20is%20no%20SINK%20XML%20so%20you%20will%20need%20to%20use%20a%20binary%20dataset%20for%20sink%20that%20points%20to%20a%20location%2Ffolder%20in%20ADL.%3C%2FP%3E%3CP%3EThen%2C%20I%20created%20a%20new%20XML%20source%20dataset%20that%20points%20to%20the%20binary%20sink%20from%20step%20above.%20I%20then%20use%20this%20source%20in%20a%20new%20copy%20activity%20as%20source%20dataset%2C%20for%20my%20sink%20in%20my%20case%20I%20use%20Azure%20DB%20table%20that%20has%20the%20columns%20that%20I%20want%20to%20map.%20Then%20under%20mappings%20you%20can%20import%20schemas%20and%20you%20will%20see%20XML%20structure.%20Here%20is%20where%20it%20can%20get%20complex%2C%20if%20its%20a%20simple%20collection%20reference%20then%20you%20should%20be%20able%20to%20map%20it.%20Here%20is%20how%20my%20mapping%20looks.%3C%2FP%3E%3CDIV%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-right%22%20image-alt%3D%22xml%20mapping.PNG%22%20style%3D%22width%3A%20845px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F209317i892D24FF42FFF6D6%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22xml%20mapping.PNG%22%20alt%3D%22xml%20mapping.PNG%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1562444%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1562444%22%20slang%3D%22en-US%22%3E%3CP%3EGreat%20addition.%20Does%20this%20also%20include%20XML%20data%20type%3F%20I%20have%20json%20files%20and%20one%20of%20the%20columns%20is%20XML.%20I%20had%20a%20quick%20look%20and%20I'm%20still%20not%20sure%20this%20is%20supported%20for%20either%20copy%20activity%20or%20mapping%20data%20flows%3F%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1564214%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1564214%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F743738%22%20target%3D%22_blank%22%3E%40sandeepthachan%3C%2FA%3E%26nbsp%3Bplease%20file%20a%20support%20ticket%20for%20the%20copy%20activity%20perf%20and%26nbsp%3B%3CSPAN%3Ememory%20exception%20issue%2C%20engineer%20can%20look%20into%20your%20particular%20case%20and%20we%20may%20need%20more%20info%20on%20your%20exact%20data%20shape.%20Meanwhile%2C%20you%20can%20give%20mapping%20data%20flow%20a%20try%20for%20super%20large%20XML%20file.%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1564223%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1564223%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F417921%22%20target%3D%22_blank%22%3E%40DC_07%3C%2FA%3E%26nbsp%3Bthanks%20for%20sharing%20the%20use%20case.%20This%20XML%20format%20support%20is%20orthogonal%20to%20XML%20data%20type%20in%20data.%20The%20XML%20field%20in%20your%20case%20will%20be%20treated%20as%20string.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1549196%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1549196%22%20slang%3D%22en-US%22%3E%3CDIV%20class%3D%22lia-message-author-with-avatar%22%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F739439%22%20target%3D%22_blank%22%3E%40scossever%3C%2FA%3E%20as%20it's%20SOAP%20API%2C%20you%20can%20use%20copy%20activity%20with%20XML%20format%20on%20top%20of%20HTTP%20connector%20as%20source%2C%20in%20copy%20activity%20%22mapping%22%20to%20map%20the%20fields%20directly%20to%20sink%20database.%20If%20you%20need%20complex%20transformation%20in-between%2C%20where%20Data%20Flow%20fits%2C%20you%20can%20use%20copy%20activity%20to%20copy%20to%20lake%20then%20apply%20Data%20Flow-%20config%20both%20copy%20source%20and%20sink%20with%20%3CSTRIKE%3EXML%3C%2FSTRIKE%3E%26nbsp%3BBinary%20format%2C%20one%20for%20HTTP%20connector%20one%20for%20data%20lake%20connector.%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1564736%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1564736%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3Bthanks%20for%20the%20reply.%20I%20have%20tested%20with%20copy%20activity%20the%20XML%20column%20mapped%20as%20string%20and%20the%20error%20is%20same%20as%20previous%20to%20this%20release%20%3CIMG%20class%3D%22lia-deferred-image%20lia-image-emoji%22%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Fhtml%2Fimages%2Femoticons%2Fsad_40x40_1.gif%22%20alt%3D%22%3Asad%3A%22%20title%3D%22%3Asad%3A%22%20%2F%3E%20The%20XML%20column%20from%20the%20JSON%20file%20is%20attempting%20to%20insert%20into%20an%20XML%20datatype%20in%20Azure%20SQL%20DB%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%3EOperation%20on%20target%20Copy%20data1%20failed%3A%20ErrorCode%3DSqlOperationFailed%2C'Type%3DMicrosoft.DataTransfer.Common.Shared.HybridDeliveryException%2CMessage%3DA%20database%20operation%20failed.%20Please%20search%20error%20to%20get%20more%20details.%2CSource%3DMicrosoft.DataTransfer.ClientLibrary%2C''Type%3DSystem.Data.SqlClient.SqlException%2CMessage%3DXML%20parsing%3A%20line%201%2C%20character%2055%2C%20unable%20to%20switch%20the%20encoding%2CSource%3D.Net%20SqlClient%20Data%20Provider%2CSqlErrorNumber%3D9402%2CClass%3D16%2CErrorCode%3D-2146232060%2CState%3D1%2CErrors%3D%5B%7BClass%3D16%2CNumber%3D9402%2CState%3D1%2CMessage%3DXML%20parsing%3A%20line%201%2C%20character%2055%2C%20unable%20to%20switch%20the%20encoding%2C%7D%2C%5D%2C'%3C%2FPRE%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1602189%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1602189%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F739439%22%20target%3D%22_blank%22%3E%40scossever%3C%2FA%3E%26nbsp%3B%20not%20sure%20if%20you%20got%20it%20working.%20If%20you%20use%20a%20delimited%20csv%20sink%2C%20it%20should%20work%20and%20you%20should%20get%20back%20the%20XML%20string%20in%20a%20row%20and%20column%2C%20so%20just%20map%20the%20one%20column.%20here%20is%20my%20sink%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22WillT985_0-1597947395668.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F213497iDB35D35E67B4ED45%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22WillT985_0-1597947395668.png%22%20alt%3D%22WillT985_0-1597947395668.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22WillT985_1-1597947437992.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F213498i4ED184C12B7388F1%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22WillT985_1-1597947437992.png%22%20alt%3D%22WillT985_1-1597947437992.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22WillT985_2-1597947456103.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F213499i2E3FF2C3EAE1BC96%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22WillT985_2-1597947456103.png%22%20alt%3D%22WillT985_2-1597947456103.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1607186%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1607186%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI'm%20trying%20with%20a%20Dataflow%20and%20which%20is%20success%20for%201%20GB%20or%20less%20files%20but%20whenever%20its%202%20GB%20or%20more%20failing%20with%20memory%20error%20%2CCan%20you%20help%20me%20to%20solve%20this%20issue.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThank%20you%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1608759%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1608759%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F768606%22%20target%3D%22_blank%22%3E%40BijuC%3C%2FA%3E%26nbsp%3BThank%20you%20for%20trying%20it%20out%20and%20raising%20the%20issue.%20Please%20file%20a%20support%20ticket%20on%20the%20memory%20error%2C%20we%20would%20need%20to%20collect%20some%20context%20of%20your%20runs%20there%20for%20investigation.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1609773%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1609773%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3BThank%20you%20for%20the%20reply%2Ccould%20you%20please%20provide%20me%20the%20URL%20to%20raise%20the%20support%20ticket.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1610102%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1610102%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F768606%22%20target%3D%22_blank%22%3E%40BijuC%3C%2FA%3E%26nbsp%3BYou%20can%20go%20to%20Azure%20portal%20-%26gt%3B%20navigate%20to%20your%20data%20factory%20-%26gt%3B%20on%20the%20left%20panel%20you%20can%20find%20%22New%20support%20request%22%20under%20%22Support%20%2B%20troubleshooting%22.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1613197%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1613197%22%20slang%3D%22en-US%22%3E%3CBLOCKQUOTE%3E%3CP%3EHey..Hi%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ECould%20you%20please%20tell%20me%20how%20to%20configure%20xsd%20validation.%20All%20i%20see%20here%20it%20needs%20to%20be%20mentioned%20inside%20the%20XML%20files%20using%20relative%20path.%20Could%20you%20please%20show%20one%20with%20an%20example.%20Would%20be%20really%20helpful.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20and%20Regards%3C%2FP%3E%3CP%3EAbhishek%3C%2FP%3E%3C%2FBLOCKQUOTE%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1651690%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1651690%22%20slang%3D%22en-US%22%3E%3CP%3EHey%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMay%20i%20know%20how%20i%20can%20load%20data%20into%20an%20XML%20file%20in%20ADF.%20If%20sink%20isn't%20an%20option%2C%20what%20are%20my%20other%20alternatives%20to%20handle%20this%20scenario.%3C%2FP%3E%3CP%3EDo%20respond%20if%20anyone%20has%20any%20ideas.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1862368%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1862368%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20have%20a%20copy%20activity%20in%20a%20pipeline%20where%20the%20source%20is%20XML%20file%20and%20the%20sink%20is%20SQL%20table.%26nbsp%3B%20I%20have%20some%20issues%20with%20mapping%20the%20XML%20file%20due%20its%20hierarchical%20structure.%20I%20wonder%20if%20is%20possible%20to%20use%20XPath%20in%20the%20mapping%20to%20pick%20the%20xml%20nodes%20%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3EDid%20anyone%20use%20the%20XPath%20in%20this%20circumstance%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%26nbsp%3B%3CBR%20%2F%3EJohn%20Balta%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1895402%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20adds%20support%20for%20XML%20format%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1895402%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%20rel%3D%22noopener%22%3E%40Linda_Wang%3C%2FA%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20XML%20validation%20seems%20to%20be%20very%20restrictive%20...%20%3CEM%3EWhen%20using%20XSD%20or%20DTD%20to%20validate%20XML%20files%2C%20the%20XSD%2FDTD%20must%20be%20referred%20inside%20the%20XML%20files%20through%20relative%20path...%22%3C%2FEM%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3EThe%20Stackoverflow%20response%20(%3CFONT%3E%3CA%20href%3D%22https%3A%2F%2Fstackoverflow.com%2Fquestions%2F63923010%2Fxml-validation-in-azure-data-factory%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fstackoverflow.com%2Fquestions%2F63923010%2Fxml-validation-in-azure-data-factory%3C%2FA%3E%3C%2FFONT%3E)%20shows%20the%20use%20of%20the%26nbsp%3B%3CSPAN%3Exsi%3AnoNamespaceSchemaLocation%3D%22order.xsd%22%3C%2FSPAN%3E%20to%20specify%20the%20xsd%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3EBut%20how%20can%20this%20be%20done%20for%20an%20external%20source%2C%20where%20we%20don't%20have%20any%20control%20of%20the%20XML%20content%2C%20so%20there%20is%20no%20way%20we%20can%20add%20a%20%22%3CSPAN%20class%3D%22hljs-attr%22%3Exsi%3AnoNamespaceSchemaLocation%22%20attribute%20%3F%3C%2FSPAN%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

We are glad to announce that now in Azure Data Factory, you can extract data from XML files by using copy activity and mapping data flow. With such capability, you can either directly load XML data to another data store/file format, or transform your XML data and then store the results in the lake or database.

 

XML format is supported on all the file-based connectors as source. You can find the new "XML" option in the available format list:

Linda_Wang_0-1594989922662.png

 

When reading data from XML files, you have the flexibility to define whether to apply schema validation, to specify how to handle namespaces, and to map selective fields or reshape the hierarchy to match downstream data need.

 

Copy activity source example:

Linda_Wang_1-1594989936019.png

 

In mapping data flow, you can use XML as source either via an XML dataset or via inline dataset.

Data flow source example with XML dataset:

Linda_Wang_2-1594989956975.png

 

To learn more about XML support in Azure Data Factory, visit XML format in the documentation site.

30 Comments
Senior Member

this is a great addition

New Contributor

Hi @Linda_Wang,

 

Thanks for the good news.

 

Few questions everybody would like to know the answers:

  • What XML processor is in use behind the scenes?
    Is it .Net Framework or something else?
  • When sink support will be introduced?
  • Any future XSLT support?
  • Any future XQuery support?
  • Excerpt from the documentation: "...When using XSD or DTD to validate XML files, the XSD/DTD must be referred inside the XML files through relative path...".
    It is a very serious limitation. Are you working to remove such dependency?
  • Is the ADF XSD Validation conformant with the XSD 1.1 standard?
    XSD 1.1 became a W3C Recommendation in April 5, 2012.
  • Does the XSD Validation support master XSD with imported or included XSDs?
    It is a very common scenario.

For example,

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:include schemaLocation="includeMe.xsd"/>
...
</xs:schema>

 

Visitor

This is great news.

 

Any future plan to have sink support?

Regular Visitor

This is a great addition. Thanks!!

Microsoft

@PuranJoshi Thanks for the feedback. For now we don't have plan to support XML as sink.

Occasional Visitor

Yeah! I expected a lot for this!

Occasional Visitor

Excellent,i was trying to convert an xml file to csv through a Copy Activity pipe line but getting an error ,' Could not find the schema information for the element'.

 

Any idea about this,can some one help me.

 

THank you :) 

Microsoft

@BijuNambiarC The error message seems indicating XSD not found. Did you configure XSD schema validation in ADF? If so, please turn off the validation and try again. And note the XML connector behavior on schema validation: https://docs.microsoft.com/en-us/azure/data-factory/format-xml#xml-connector-behavior.

Frequent Visitor

@Linda_Wang I am retrieving an XML file from an HTTP connection in Data Factory (from a SOAP API). What is the supported method to sink/save this data to Data Lake in order to run Data Flow transformations on the data?

Frequent Visitor

Does anyone have a method to save/sink this retrieved XML data from the HTTP connection? Binary isn't an option, needs to be something that can be parsed out into a database.

Microsoft
@scossever as it's SOAP API, you can use copy activity with XML format on top of HTTP connector as source, in copy activity "mapping" to map the fields directly to sink database. If you need complex transformation in-between, where Data Flow fits, you can use copy activity to copy to lake then apply Data Flow- config both copy source and sink with XML Binary format, one for HTTP connector one for data lake connector.
Frequent Visitor

@Linda_Wang Do you have an example of this? I can't find a file type that works with saving the XML to the Sink source. I need it to be something that I can parse through.

Senior Member

As @Linda_Wang  described, you can copy the results of the http to data lake. I got this to work with binary and XML. For complex XML support I was able to copy it to a SQL server instance where you can use TSQL XML functions to parse the results, I used binary dataset and copy the results of data lake and copy it to a string in a table, but XML should work too with an XML data type. I haven't tried that yet, but that may be a good solution for complex XML parsing and processing in line instead of saving it to a table. In my case I had to create a loop because there is a limit of rows returned on the soap envelope. But that is another story. Hope this helps.

 

WillT985_2-1595944648855.png

 

WillT985_0-1595944372367.png

WillT985_1-1595944595267.png

 

 

Hi @Linda_Wang ,

 

   Is there any limitation to convert the xml files to csv using the copy activity? 1 GB file gets converted within 2 hour without any parallelism/DIU/blocks, but a 2GB file fails with a memory exception.

 

Any suggestion to process a large xml file.

 

Thank you

Frequent Visitor

@WillT985 I must be missing something. This is what I have so far.

 

1. Linked Service: HTTP type, base URL is my external API. Basic auth, username, password

2. A dataset, type HTTP/XML type, connected to my Linked Service above. I have entered a base URL. It connects without issue

3. Pipeline activity of "Copy data"

  • Source dataset: the HTTP/XML dataset from above
  • Request method: POST
  • Additional headers: Content-Type: application/soap+xml;charset=UTF-8;action="http://outsidesoft.com/webservices/API/Authenticate"
  • Request body: <soap:abc123....><soap:/Header/>....<soap:Body><api:Authenticate>...etc etc

What are the next steps? When I try to set up a Sink dataset, I choose my data lake, but then when to choose a format is when I run into a problem. I want to save the file in a format I can parse. If I choose CSV, then run a debug, I get the following error:

ErrorCode=SchemaMappingFailedInHierarchicalToTabularStage,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to process hierarchical to tabular stage, error message: One or more errors occurred.,Source=Microsoft.DataTransfer.ClientLibrary,'",

 

What are the proper steps to take to get a sink working?

My future plans are to do another "Copy data", and set up the variable to get the token as you demo'd above, and then make subsequent API calls with the token and save that newly retrieved data to the data lake in their own tables.

 

Thank you all

Frequent Visitor

Linda mentioned configuring the Sink and Source as XML for Data Lake, but this is not an option. Only options are: Avro, Binary, DelimitedText, Json, ORC, Parquet.

 

Also, Linda, my Sink is an HTTP/XML dataset connected to a Linked Source type of HTTP. Do you have a demo of how to setup a proper copy that uses an HTTP linked service source, to map fields from a SOAP API?

 

If I select DelimitedText/CSV for Data Link source's format and debug the pipeline, I get the following error:

ErrorCode=SchemaMappingFailedInHierarchicalToTabularStage,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to process hierarchical to tabular stage, error message: One or more errors occurred.,Source=Microsoft.DataTransfer.ClientLibrary,'",

Senior Member

@scossever 

In my case, First I created a copy activity that uses a binary dataset source type to call the Marketing Cloud SOAP Api and puts the results(XML envelope) in Data lake as Binary Dataset Sink in Data lake. My binary source dataset uses an http linked service, under the Base URL you would use your API url from your MC. but again the result will get dumped to data lake as binary sink. You are correct that there is no SINK XML so you will need to use a binary dataset for sink that points to a location/folder in ADL.

Then, I created a new XML source dataset that points to the binary sink from step above. I then use this source in a new copy activity as source dataset, for my sink in my case I use Azure DB table that has the columns that I want to map. Then under mappings you can import schemas and you will see XML structure. Here is where it can get complex, if its a simple collection reference then you should be able to map it. Here is how my mapping looks.

 
 

xml mapping.PNG

Visitor

Great addition. Does this also include XML data type? I have json files and one of the columns is XML. I had a quick look and I'm still not sure this is supported for either copy activity or mapping data flows? 

 

Microsoft

@sandeepthachan please file a support ticket for the copy activity perf and memory exception issue, engineer can look into your particular case and we may need more info on your exact data shape. Meanwhile, you can give mapping data flow a try for super large XML file.

Microsoft

@DC_07 thanks for sharing the use case. This XML format support is orthogonal to XML data type in data. The XML field in your case will be treated as string.

Visitor

@Linda_Wang thanks for the reply. I have tested with copy activity the XML column mapped as string and the error is same as previous to this release :sad: The XML column from the JSON file is attempting to insert into an XML datatype in Azure SQL DB

 

Operation on target Copy data1 failed: ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed. Please search error to get more details.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=XML parsing: line 1, character 55, unable to switch the encoding,Source=.Net SqlClient Data Provider,SqlErrorNumber=9402,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=9402,State=1,Message=XML parsing: line 1, character 55, unable to switch the encoding,},],'
Senior Member

@scossever  not sure if you got it working. If you use a delimited csv sink, it should work and you should get back the XML string in a row and column, so just map the one column. here is my sink 

WillT985_0-1597947395668.png

 

WillT985_1-1597947437992.png

 

WillT985_2-1597947456103.png

 

Occasional Visitor

@Linda_Wang 

 

I'm trying with a Dataflow and which is success for 1 GB or less files but whenever its 2 GB or more failing with memory error ,Can you help me to solve this issue.

 

Thank you

Microsoft

@BijuC Thank you for trying it out and raising the issue. Please file a support ticket on the memory error, we would need to collect some context of your runs there for investigation.

Occasional Visitor

@Linda_Wang Thank you for the reply,could you please provide me the URL to raise the support ticket.

Microsoft

@BijuC You can go to Azure portal -> navigate to your data factory -> on the left panel you can find "New support request" under "Support + troubleshooting".

Senior Member

Hey..Hi

 

Could you please tell me how to configure xsd validation. All i see here it needs to be mentioned inside the XML files using relative path. Could you please show one with an example. Would be really helpful.

 

Thanks and Regards

Abhishek

Senior Member

Hey 

 

May i know how i can load data into an XML file in ADF. If sink isn't an option, what are my other alternatives to handle this scenario.

Do respond if anyone has any ideas.

Visitor

Hi @Linda_Wang 

 

I have a copy activity in a pipeline where the source is XML file and the sink is SQL table.  I have some issues with mapping the XML file due its hierarchical structure. I wonder if is possible to use XPath in the mapping to pick the xml nodes   

Did anyone use the XPath in this circumstance?

 

Thanks 
John Balta

Occasional Visitor

Hi @Linda_Wang

 

The XML validation seems to be very restrictive ... When using XSD or DTD to validate XML files, the XSD/DTD must be referred inside the XML files through relative path..."

 

The Stackoverflow response (https://stackoverflow.com/questions/63923010/xml-validation-in-azure-data-factory) shows the use of the xsi:noNamespaceSchemaLocation="order.xsd" to specify the xsd

 

But how can this be done for an external source, where we don't have any control of the XML content, so there is no way we can add a "xsi:noNamespaceSchemaLocation" attribute ?