Web activity response timeout improvement

Published Mar 17 2022 10:18 AM 2,405 Views
Microsoft

Web activity is helpful when invoking an external endpoint from within an ADF pipeline. While Azure Data Factory/ Synapse pipelines offer various orchestration activities, web activity helps provide extensibility when building data pipelines that span across different services. 

 

Web activities work exceptionally well with APIs that follow the asynchronous request-reply pattern, a suggested approach for building scalable web APIs/ services. In such scenarios, it automatically polls for the status endpoint (when it receives HTTP 202) till the API returns HTTP 200. It behaves synchronously and waits till the response is HTTP 200 before kicking off the down-stream activities. 

 

async-request.png

Most Azure-based APIs are async request-reply pattern and gels with the existing web activity behavior. 

 

Problem

But in case you have API endpoints that do not follow the above pattern and expect the caller to wait for a response till the processing/ actions are done. As default action, the web activity will timeout within 1 minute if it does not receive any response.  

 

Improvement

We are introducing a response timeout property 'httpRequestTimeout' in Web activity that will let you provide larger timeouts for such synchronous APIs.  

 

Screenshot 2022-03-17 at 9.33.41 AM.png

 

Reference: 

Web Activity - Azure Data Factory & Azure Synapse | Microsoft Docs

2 Comments
%3CLINGO-SUB%20id%3D%22lingo-sub-3260307%22%20slang%3D%22en-US%22%3EWeb%20activity%20response%20timeout%20improvement%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3260307%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20data-preserver-spaces%3D%22true%22%3EWeb%20activity%20is%20helpful%20when%20invoking%20an%20external%20endpoint%20from%20within%20an%20ADF%20pipeline.%20While%20Azure%20Data%20Factory%2F%20Synapse%20pipelines%20offer%20various%20orchestration%20activities%2C%20web%20activity%20helps%20provide%20extensibility%20when%20building%20data%20pipelines%20that%20span%20across%20different%20services.%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-preserver-spaces%3D%22true%22%3EWeb%20activities%20work%20exceptionally%20well%20with%20APIs%20that%20follow%20the%26nbsp%3B%3C%2FSPAN%3E%3CA%20class%3D%22editor-rtfLink%22%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Farchitecture%2Fpatterns%2Fasync-request-reply%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3E%3CSPAN%20data-preserver-spaces%3D%22true%22%3Easynchronous%20request-reply%20pattern%3C%2FSPAN%3E%3C%2FA%3E%3CSPAN%20data-preserver-spaces%3D%22true%22%3E%2C%20a%20suggested%20approach%20for%20building%20scalable%20web%20APIs%2F%20services.%20In%20such%20scenarios%2C%20it%20automatically%20polls%20for%20the%20status%20endpoint%20(when%20it%20receives%20HTTP%20202)%20till%20the%20API%20returns%20HTTP%20200.%20It%20behaves%20synchronously%20and%20waits%20till%20the%20response%20is%20HTTP%20200%20before%20kicking%20off%20the%20down-stream%20activities.%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CDIV%20id%3D%22tinyMceEditorAbhishekNarain_0%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22async-request.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F356642iB1B626912E71723D%2Fimage-size%2Fmedium%3Fv%3Dv2%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22async-request.png%22%20alt%3D%22async-request.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%E2%80%83%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-preserver-spaces%3D%22true%22%3EMost%20Azure-based%20APIs%20are%20async%20request-reply%20pattern%20and%20gels%20with%20the%20existing%20web%20activity%20behavior.%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EProblem%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-preserver-spaces%3D%22true%22%3EBut%20in%20case%20you%20have%20API%20endpoints%20that%20do%20not%20follow%20the%20above%20pattern%20and%20expect%20the%20caller%20to%20wait%20for%20a%20response%20till%20the%20processing%2F%20actions%20are%20done.%20As%20default%20action%2C%20the%20web%20activity%20will%20timeout%20within%201%20minute%20if%20it%20does%20not%20receive%20any%20response.%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EImprovement%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-preserver-spaces%3D%22true%22%3EWe%20are%20introducing%20a%20response%20timeout%20property%20'httpRequestTimeout'%20in%20Web%20activity%20that%20will%20let%20you%20provide%20larger%20timeouts%20for%20such%20synchronous%20APIs.%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Screenshot%202022-03-17%20at%209.33.41%20AM.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F356644i56AFC691BF348E89%2Fimage-size%2Fmedium%3Fv%3Dv2%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Screenshot%202022-03-17%20at%209.33.41%20AM.png%22%20alt%3D%22Screenshot%202022-03-17%20at%209.33.41%20AM.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%E2%80%83%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EReference%3A%26nbsp%3B%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-preserver-spaces%3D%22true%22%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fdata-factory%2Fcontrol-flow-web-activity%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EWeb%20Activity%20-%20Azure%20Data%20Factory%20%26amp%3B%20Azure%20Synapse%20%7C%20Microsoft%20Docs%3C%2FA%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-3260307%22%20slang%3D%22en-US%22%3E%3CP%3EWeb%20activity%20response%20timeout%20improvement%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-3260307%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Data%20Integration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20ETL%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EBig%20Data%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-3290530%22%20slang%3D%22en-US%22%3ERe%3A%20Web%20activity%20response%20timeout%20improvement%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3290530%22%20slang%3D%22en-US%22%3E%3CP%3EThank%20you%20Abhishek%20for%20sharing%20this.%20Its%20really%20useful.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-3268343%22%20slang%3D%22en-US%22%3ERe%3A%20Web%20activity%20response%20timeout%20improvement%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3268343%22%20slang%3D%22en-US%22%3E%3CP%3EReally%20like%20this%20feature%20and%20it%20resolves%20high%20voted%20feature%20on%20Idea%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ffeedback.azure.com%2Fd365community%2Fidea%2F1360f3f9-7026-ec11-b6e6-000d3a4f032c%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EPlease%20add%20support%20to%20specify%20longer%20timeout%20for%20Web%20Activity%20%C2%B7%20Community%20(azure.com)%3C%2FA%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E
Co-Authors
Version history
Last update:
‎Mar 17 2022 10:18 AM
Updated by: