Blog Post

Azure Data Factory Blog
1 MIN READ

ADF connector updates: Azure Database for PostgreSQL, REST and Excel

Linda_Wang's avatar
Linda_Wang
Icon for Microsoft rankMicrosoft
Dec 11, 2020

Azure Data Factory has enhanced the Azure Database for PostgreSQL, REST and Excel connectors to fulfill your emerging data integration need:

 

Azure Database for PostgreSQL in Data Flow

Now you can use Azure Database for PostgreSQL connector in Data Flow to build powerful ETL processes. You can read data directly from Azure Database for PostgreSQL and apply data transformations or join with other data, or post transformation you can perform inserts, updates, deletes, and upserts to publish the analytics result set into Azure Database for PostgreSQL. Learn more about Azure Database for PostgreSQL connector.

 

REST connector as sink

Generic REST connector is one of the most commonly used connectors in ADF to integrate with various SaaS applications or your own applications. It helps you easily extend and reach a great number of data sources than the built-in ones. Now using copy activity, not only you can use REST connector to retrieve data, ADF newly enables REST connector as sink so you can also write data via REST APIs. Learn more about REST connector.

 

Excel format - worksheet index

When parsing Excel files using copy activity or data flow, you have a new option to point to a worksheet by index in addition to by name. For example, if you just want to read the only worksheet in your Excel files and ignore its name, you can simply specify index as 0. Learn more about Excel format.

 

Updated Dec 11, 2020
Version 1.0

6 Comments

  • brianbolger generic in ADF, you can reference the properties in activity output in subsequent activities, e.g. @{activity('Copy').output.xxx}. REST sink doesn't expose specific info now, the request response is not captured there as during one copy activity run multiple requests might be send to write data in batches; error should be there.

  • brianbolger's avatar
    brianbolger
    Copper Contributor

    Linda_Wang are there any docs or guidance on how to interpret a response from the REST connector sink?

    e.g. if we wanted to take some action on a failed response or retrieve a value from the response (such as a new resource Id generated in the case of a POST request) to use later in the pipeline?

  • Johannes_Vink based on your description it's actually reversed - there is an output size limit for Web activity which feed data back to ADF control flow for subsequent activity to consume, whereas no limit for REST connector as source in copy activity which copy the data to sink data store.

  • Johannes_Vink's avatar
    Johannes_Vink
    Brass Contributor

    Hm strange. It is not listed in the docs as limitation. I checked how did implement it in the end. REST we could not get working, Web activity did work. The repsonse is picked up in a stored procedure to write to a database. It is a general ADF limitation that the response cannot be too big. In our case it was sufficient.

     

    Writing to REST opens a few scenario's we've left untouched for the moment, will look into that!

  • Johannes_Vink  I remember there isn't a size limit in REST source. Where do you see the limit? And no limit for REST sink, it writes data in batches.

  • Johannes_Vink's avatar
    Johannes_Vink
    Brass Contributor

    I there was a limit of 2 MB of downloading data from a REST source. Is there a similar limitation for using REST as a sink?