ADF connector updates: Azure Database for PostgreSQL, REST and Excel

Published Dec 11 2020 05:01 AM 2,803 Views
Microsoft

Azure Data Factory has enhanced the Azure Database for PostgreSQL, REST and Excel connectors to fulfill your emerging data integration need:

 

Azure Database for PostgreSQL in Data Flow

Now you can use Azure Database for PostgreSQL connector in Data Flow to build powerful ETL processes. You can read data directly from Azure Database for PostgreSQL and apply data transformations or join with other data, or post transformation you can perform inserts, updates, deletes, and upserts to publish the analytics result set into Azure Database for PostgreSQL. Learn more about Azure Database for PostgreSQL connector.

Linda_Wang_0-1607690931447.png

 

REST connector as sink

Generic REST connector is one of the most commonly used connectors in ADF to integrate with various SaaS applications or your own applications. It helps you easily extend and reach a great number of data sources than the built-in ones. Now using copy activity, not only you can use REST connector to retrieve data, ADF newly enables REST connector as sink so you can also write data via REST APIs. Learn more about REST connector.

Linda_Wang_1-1607690931455.png

 

Excel format - worksheet index

When parsing Excel files using copy activity or data flow, you have a new option to point to a worksheet by index in addition to by name. For example, if you just want to read the only worksheet in your Excel files and ignore its name, you can simply specify index as 0. Learn more about Excel format.

Linda_Wang_2-1607690931461.png

 

6 Comments
%3CLINGO-SUB%20id%3D%22lingo-sub-1979486%22%20slang%3D%22en-US%22%3EADF%20connector%20updates%3A%20Azure%20Database%20for%20PostgreSQL%2C%20REST%20and%20Excel%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1979486%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20Data%20Factory%20has%20enhanced%20the%20Azure%20Database%20for%20PostgreSQL%2C%20REST%20and%20Excel%20connectors%20to%20fulfill%20your%20emerging%20data%20integration%20need%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EAzure%20Database%20for%20PostgreSQL%20in%20Data%20Flow%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3ENow%20you%20can%20use%20Azure%20Database%20for%20PostgreSQL%20connector%20in%20Data%20Flow%20to%20build%20powerful%20ETL%20processes.%20You%20can%20read%20data%20directly%20from%20Azure%20Database%20for%20PostgreSQL%20and%20apply%20data%20transformations%20or%20join%20with%20other%20data%2C%20or%20post%20transformation%20you%20can%20perform%20inserts%2C%20updates%2C%20deletes%2C%20and%20upserts%20to%20publish%20the%20analytics%20result%20set%20into%20Azure%20Database%20for%20PostgreSQL.%26nbsp%3BLearn%20more%20about%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fconnector-azure-database-for-postgresql%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%20Database%20for%20PostgreSQL%20connector%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Linda_Wang_0-1607690931447.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239845i6D2754C6985F66B2%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Linda_Wang_0-1607690931447.png%22%20alt%3D%22Linda_Wang_0-1607690931447.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EREST%20connector%20as%20sink%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EGeneric%20REST%20connector%20is%20one%20of%20the%20most%20commonly%20used%20connectors%20in%20ADF%20to%20integrate%20with%20various%20SaaS%20applications%20or%20your%20own%20applications.%20It%20helps%20you%20easily%20extend%20and%20reach%20a%20great%20number%20of%20data%20sources%20than%20the%20built-in%20ones.%20Now%20using%20copy%20activity%2C%20not%20only%20you%20can%20use%20REST%20connector%20to%20retrieve%20data%2C%20ADF%20newly%20enables%20REST%20connector%20as%20sink%20so%20you%20can%20also%20write%20data%20via%20REST%20APIs.%20Learn%20more%20about%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fconnector-rest%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EREST%20connector%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Linda_Wang_1-1607690931455.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239843i1307B6E2A4328E7D%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Linda_Wang_1-1607690931455.png%22%20alt%3D%22Linda_Wang_1-1607690931455.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EExcel%20format%20-%20worksheet%20index%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EWhen%20parsing%20Excel%20files%20using%20copy%20activity%20or%20data%20flow%2C%20you%20have%20a%20new%20option%20to%20point%20to%20a%20worksheet%20by%20index%20in%20addition%20to%20by%20name.%20For%20example%2C%20if%20you%20just%20want%20to%20read%20the%20only%20worksheet%20in%20your%20Excel%20files%20and%20ignore%20its%20name%2C%20you%20can%20simply%20specify%20index%20as%200.%20Learn%20more%20about%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fformat-excel%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EExcel%20format%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Linda_Wang_2-1607690931461.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239844iE61ECEC9D42C4A3F%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Linda_Wang_2-1607690931461.png%22%20alt%3D%22Linda_Wang_2-1607690931461.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1979486%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20Data%20Factory%20enables%20Azure%20Database%20for%20PostgreSQL%20in%20data%20flow%2C%20adds%20REST%20connector%20as%20sink%2C%20and%20provides%20new%20option%20for%20parsing%20Excel%20format.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1979486%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Data%20Integration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ECopy%20Activity%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMapping%20Data%20Flows%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1979981%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20connector%20updates%3A%20Azure%20Database%20for%20PostgreSQL%2C%20REST%20and%20Excel%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1979981%22%20slang%3D%22en-US%22%3E%3CP%3EI%20there%20was%20a%20limit%20of%202%20MB%20of%20downloading%20data%20from%20a%20REST%20source.%20Is%20there%20a%20similar%20limitation%20for%20using%20REST%20as%20a%20sink%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1982784%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20connector%20updates%3A%20Azure%20Database%20for%20PostgreSQL%2C%20REST%20and%20Excel%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1982784%22%20slang%3D%22en-US%22%3E%3CP%3EHm%20strange.%20It%20is%20not%20listed%20in%20the%20docs%20as%20limitation.%20I%20checked%20how%20did%20implement%20it%20in%20the%20end.%20REST%20we%20could%20not%20get%20working%2C%20Web%20activity%20did%20work.%20The%20repsonse%20is%20picked%20up%20in%20a%20stored%20procedure%20to%20write%20to%20a%20database.%20It%20is%20a%20general%20ADF%20limitation%20that%20the%20response%20cannot%20be%20too%20big.%20In%20our%20case%20it%20was%20sufficient.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWriting%20to%20REST%20opens%20a%20few%20scenario's%20we've%20left%20untouched%20for%20the%20moment%2C%20will%20look%20into%20that!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2178116%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20connector%20updates%3A%20Azure%20Database%20for%20PostgreSQL%2C%20REST%20and%20Excel%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2178116%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3Bare%20there%20any%20docs%20or%20guidance%20on%20how%20to%20interpret%20a%20response%20from%20the%20REST%20connector%20sink%3F%3C%2FP%3E%3CP%3Ee.g.%20if%20we%20wanted%20to%20take%20some%20action%20on%20a%20failed%20response%20or%20retrieve%20a%20value%20from%20the%20response%20(such%20as%20a%20new%20%3CEM%3Eresource%3C%2FEM%3E%20Id%20generated%20in%20the%20case%20of%20a%20POST%20request)%20to%20use%20later%20in%20the%20pipeline%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Dec 11 2020 05:01 AM
Updated by: