Azure Data Factory topics

Web activity failure due to Invoking endpoint failed with HttpStatusCode - 403 -- help?

AzureNewbie1 — Wed, 25 Feb 2026 17:21:42 GMT

Hi,
I have an Azure Data Factory (ADF) instance that I am using to create a Pipeline to ingest external (cloud based) 3rd party data into my Azure SQL Server database. I am a novice with ADF and have only used it to ingest some external SQL data into my SQL database - it did work.
The external source I'm attempting to extract from uses an OAuth 2.0 API and an API is something I've not used before.

Using Postman (never used this software before this attempt), I have passed the external source's base_url, client_id, and client_secret, and in return successfully received an access token. This tells me that the base_url, client_id, and client_secret values I passed are correct and accepted by the target source/application.

Feeling encourage to implement the same values into ADF, I first created a Linked Service which with a successful test connection returned - see below. This Linked Service uses the same values as the Postman entry which granted an access token.

I then created a Pipeline with a Web activity object within it. The General and User Properties don't have any configuration, only the Settings tab does which shown below. Again, the URL, Client ID and Client Secret configured here are the same as those used in Postman (and the Linked Service).

I execute the Web object and it returns with a failure - see below.

The error states the endpoint refused the request (for an access token). Is this accurate as I was able to receive an access token via Postman when using the same credentials? I don't understand why via Postman I can received an access token but via ADF it errors. I'm wondering if I've completed the ADF parts incorrectly, or if there is more needed just to received an access token, or if it's something else?
Are you able to advise what's taking place here?
Thanks.

Unable to INSERT rec into Table.

DRN_SDE — Mon, 16 Feb 2026 10:38:02 GMT

I am using the ADF . Execute Pipeline activity.
My PL is parametrized.

My stage table have the below rec:

network_org_name	partner_name	parent_partner_name	CorPart_number	pipeline_run_id	activity_run_id
Se Network Producing	UB SA	S Reine Company Ltd	1226133	ebf961f9-3afc-4abc-9ee0-111330c9f1eb	858c1b5c-a699-4e73-84d8-3a42cab115d2

WITH CodeSourceTableCTE AS ( SELECT DISTINCT p.[partner_number] ,n.[network_org_number] FROM [utl].[stg_partner_network_org] stg LEFT JOIN [utl].[partner] p1 ON stg.[parent_partner_name] = p1.[partner_name] INNER JOIN [utl].[partner] p ON stg.[partner_name] = p.[partner_name] AND ISNULL(p.[delete_ind], '') <> 'Y' AND ISNULL(p.[parent_partner_number], '') = ISNULL(p1.[partner_number], '') AND ISNULL(p1.[delete_ind], '') <> 'Y' AND ISNULL(stg.[corpart_number], '') = ISNULL(p.[external_id], '') INNER JOIN [utl].[network_org] n ON stg.[network_org_name] = n.[network_org_name] WHERE stg.[partner_name] IS NOT NULL ) MERGE [utl].[partner_network_org] T USING CodeSourceTableCTE S ON T.[partner_number] = S.[partner_number] AND T.[network_org_number] = S.[network_org_number] WHEN MATCHED AND (ISNULL(T.[delete_ind], '') <> 'N') THEN UPDATE SET T.[delete_ind] = 'N', T.[last_modified_date] = GETUTCDATE(), T.[last_modified_by] = '@{pipeline().parameters.Source_File_Name}' WHEN NOT MATCHED BY TARGET THEN INSERT ([partner_number], [network_org_number], [created_date], [created_by], [last_modified_date], [last_modified_by], [delete_ind]) VALUES (S.[partner_number], S.[network_org_number], GETUTCDATE(), '@{pipeline().parameters.Source_File_Name}', GETUTCDATE(), '@{pipeline().parameters.Source_File_Name}', 'N'); SELECT COUNT(*) FROM [utl].[partner_network_org] WHERE [created_by] = '@{pipeline().parameters.Source_File_Name}'

But when I am trying to check using below

select * from utl.partner_network_org where partner_number in (select partner_number from utl.partner where last_modified_by like '%corpart%')

I don't see records any help is appreciated

Azure Data Factory - help needed to ingest data, that uses an API, into a SQL Server instance

AzureNewbie1 — Wed, 28 Jan 2026 15:22:42 GMT

Hi,
I need to ingest 3rd-party data, that uses an API, into my Azure SQL Server instance (ASSi) using Azure Data Factory (ADF).
I'm a report developer so this area is unfamiliar to me, although I have previously integrated external, on-premise SQL Server data into our ASSi using ADF so I do have some exposure to the tool (just not API connections).

The 3rd-party data belongs to a company named 'iLevel' (in case this is relevant). iLevel have provided some API documentation which is targeted for an experienced data engineer that understands API connections. This documentation has just a few sections before the connection focused details end. I'll list them below:
1) It mentions to download 'Postman Collection' and mentions no more than this. I've never heard of Postman Collection and I don't know why it's needed. Through limited exposure online, I don't understand its purpose, certainly not in my scenario.

2) It has the title 'Access to the API' and then lists four URLs which are the endpoints (I don't know which to use and will need to ask iLevel of this but, I guess, any will do for testing purposes).

3) Authentication and Authorization
a) Generate a 'client id' and 'client secret' by logging into iLevel and generating these values with some clicks of a button. I've successfully generated both these values.
b) Obtain an Access Token - it'll be easier to screenshot the instructions for this (I've blanked part of the URL for confidentiality).

These are all the instructions on connection to the 3rd-party data. Unfortuntely for me, my lack of experience in this area means these instrcutions don't help me. I don't believe I'm any closer to connecting to the 3rd-party source data.

Taking into consideration the above instructions but choosing to try and error in ADF, a tool I'm a little bit more familiar with, I've performed the following steps:
1) Created a Linked Service.
I understand the iLevel solution is in the cloud and therefore the 'AutoResolveIntegrationRuntime' option has been selected as the 'Connect via integration runtime' value. For the 'Base URL' I've entered one of the four URL endpoints that were listed in the documentation (again, I will need to confirm which endpoint to use).

The 'Test Connection' returns a successful result but I think it means nothing because if I were to placed 'xxx' at the end of the Base URL and test the connection, it stills returns successful when I know the URL with the 'xxx' post-fix isn't legit.

2) Create an ADF Pipeline containing 'Web activity' and 'Set variable' objects.
The only configuration under the Web activity is the 'Settings' pane which has:

The 'Body' property has (the client id and client secret taken from the iLevel solution are included in the body but blanked out):

If the Web activity is successful then the Pipeline's next object (the Set variable) should assign the access token to a variable - as I understand this is what the Web activity is doing:

The 'Value' property has:

This is as far as I've got into my efforts of this integration task because the Web activity object fails when executed. The error message does state it is to do with an invalid 'client id' or 'client secret' - see below:

You may direct to focus on the incorrect client id or client secret, however I don't have any confidence that I understand how to configure ADF to obtain an access token, and I'm maybe missing something if I see no need for Postman Collection use.

What is Postman Collection and do I need it for what I'm trying to achieve? If yes, can anyone provide training material that suits my need?
Have I configured ADF correctly and it is indeed an issue with the client id or client server, or is the error message received just a by-product of an incorrect ADF configuration?

Your help will be most appreciated. Many thanks.

ADF unable to ingest partitioned Delta data from Azure Synapse Link (Dataverse/FnO)

Raheelislam — Fri, 09 Jan 2026 07:10:44 GMT

We are ingesting Dynamics 365 Finance & Operations (FnO) data into ADLS Gen2 using Azure Synapse Link for Dataverse, and then attempting to load that data into Azure SQL Database using Azure Data Factory (ADF).

This is part of a migration effort as Export to Data Lake is being deprecated.

Source Details
Source: ADLS Gen2
Data generated by: Azure Synapse Link for Dataverse (FnO)
Format on lake:
Delta / Parquet
Partitioned folder structure (e.g. PartitionId=xxxx)
Destination:
Azure SQL Database
Issue Observed in ADF

When configuring ADF pipelines:

Using ADLS Gen2 dataset with:
Delta / Parquet
Recursive folder traversal
Wildcard paths
We encounter:
No data returned in Data Preview
Or runtime error such as:
“No partitions information found in metadata file”
Despite this:
The data is present in ADLS
The same data can be successfully queried using Synapse serverless SQL
Key Question for ADF / Synapse Engineers

What is the recommended and supported ADF ingestion pattern for:

Partitioned Delta/Parquet data produced by Azure Synapse Link for Dataverse

Specifically:

Should ADF:
Read Delta tables directly, or
Use Synapse serverless SQL external tables/views as an intermediate layer?
Is there a reference architecture for:
Synapse Link → ADLS → ADF → Azure SQL
Are there ADF limitations when consuming Synapse Link–generated Delta tables?

Many customers are now forced to migrate due to Export to Data Lake deprecation, but current ADF documentation does not clearly explain how to replace existing ingestion pipelines when using Synapse Link for FnO.

Any guidance, patterns, or official documentation would be greatly appreciated.

Copy Data Activity Failed with Unreasonable Cause

AJ81 — Wed, 03 Dec 2025 02:59:45 GMT

It is a simple set up but it has baffled me a lot. I'd like to copy data to a data lake via API. Here are the steps I've taken:

Created a HTTP linked service as below:

Created a dataset with a HTTP Binary data format as below:

Created a pipeline with a Copy Data activity only as shown below:

Made sure linked service and dataset all working fine as below:

Created a Sink dataset with 3 parameters as shown below:

Passed parameters from pipeline to Sink dataset as below:

That's all. Simple, right? But the pipeline failed with a clear message "usually this is caused by invalid credentials." as below:

Summary: No need to worry about the Sink side of parameters etc. which I have used same thing for years on other pipelines and all succeeded. This time the API failed to reach a data lake from source side as said "invalid credentials". In Step 4 above one could see the linked service and dataset connections were succeeded, ie. credentials have been checked and passed already. How come it failed in data copy activity complaining an invalid credentials? Pretty weird. Any advice and suggestions will be welcomed.

User Properties of Activities in ADF: How to add dynamic content in it?

manuj — Thu, 13 Nov 2025 12:17:10 GMT

On ADF, I am using a for each loop in which I am using an Execute Pipeline Activity which is getting executed for different iterations as per the values of the items provided to the For-Each Loop.

I am stuck on a scenario which requires me to add the Dynamic Content Expression in the User Properties of individual activities of ADF. Specific to my case, I want to add the Dynamic Content Expression in the User Properties of Execute Pipeline Activity so that I get to individual runs of these activities on Azure Monitor with a specific label attached to it through its User Properties.

The necessity to add the Dynamic Content Expression in the User Properties is due to the reason that each execution in respective iterations of these activities corresponds to a particular Step from a set of Steps configured for the Data Load Job as a whole, which has been orchestrated through ADF. To identify the association with the respective Job-Step, I require to add Dynamic Content Expression in its User Properties.

Any sort of response regarding this is highly appreciated.

Thank You!

Data flow sink to Blob storage not writing to subfolder

DuncanKing — Tue, 14 Oct 2025 10:21:58 GMT

Hi Everybody

This seems like it should be straightforward, but it just doesn't seem to work... I have a file containing JSON data, one document per line, with many different types of data. Each type is identified by a field named "OBJ", which tells me what kind of data it contains. I want to split this file into separate files in Blob storage for each object type prior to doing some downstream processing. So, I have a very simple data flow - a source which loads the whole file, and a sink which writes the data back to separate files. In the sink settings, I've set the "File name option" setting to "Name file as column data" and selected my OBJ column for the "Column Data", and this basically works - it writes out a separate file for each OBJ value, containing the right data. So far, so good.

However, what doesn't seem to work is the very simplest thing - I want to write the output files to a folder in my Blob storage container, but the sink seems to completely ignore the "Folder path" setting and just writes them into the root of the container. I can write my output files to a different container, but not to a subfolder inside the same container. It even creates the folder if it's not there already, but doesn't use it.

Am I missing something obvious, or does the "Folder path" setting just not work when naming files from column data? Is there a way around this?

Problem with Linked Service to SQL Managed Instance

alexp01482 — Fri, 26 Sep 2025 06:31:27 GMT

I'm trying to create a linked Service to a SQL Managed Instance. The Managed Instance is configured with a Vnet_local endpoint

If I try to connect with an autoresolve IR or a SHIR I get the following error

The value of the property '' is invalid: 'The remote name could not be resolved: 'SQL01.public.ec9fbc2870dd.database.windows.net''.
The remote name could not be resolved: 'SQL01.public.ec9fbc2870dd.database.windows.net'

Is there a way to connect to it without resorting to a private endpoint?

Cheers

Alex

ADF connection issue with Cassandra

panksume — Wed, 10 Sep 2025 22:00:34 GMT

Hi,

I am trying to connect a cassandra DB hosted in azure cosmos db. I created the linked service but getting below error on test connection.

Already checked the cassandra DB and its public network access is set to all networks.

Google suggested enabling SSL but there is no such option in linked service.

Please help.

Failed to connect to the connector. Error code: 'Unknown', message: 'Failed to connect to Cassandra server due to, ErrorCode: InternalError'
Failed to connect to the connector. Error code: 'InternalError', message: 'Failed to connect to Cassandra server due to, ErrorCode: InternalError'
Failed to connect to Cassandra server due to, ErrorCode: InternalError
All hosts tried for query failed (tried 51.107.58.67:10350: SocketException 'A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied')

User configuration issue

Aviavi-123 — Tue, 19 Aug 2025 12:27:01 GMT

Hi,

I am getting below error
"Execution fail against sql server. Please contact SQL Server team if you need further support. Sql error number: 3930. Error Message: The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction"
I never faced this kind of error.
Could anyone please let me know what i can do and i needs to do.
I am beginner, please explain me.

Thanks.

Help with Partial MongoDB Update via Azure Data Factory Data Flow

leopoldinoex — Thu, 14 Aug 2025 16:29:48 GMT

Hello, everyone!

I have a complex question about how to perform a partial update on a MongoDB collection using Data Flow in Azure Data Factory. My goal is to modify only some nested fields without overwriting the entire document.
My flow reads JSON files with the following structure:

{
"_id": {
"$oid": "1xp3232to"
},
"root_field": "root_value",
"main_array": [
{
"array_id": "id001",
"status": "PENDING",
"nested_array": []
}
],
"numeric_value": {
"$numberDecimal": "10.99"
}
}

I need Data Flow to make two changes in a single run:

Change the status field from "PENDING" to "SENT".
Add a new object to the nested_array with the following data:
event: "SENT"
description: "FILE GENERATED"
timestamp: (current date and time)
system: "Sis Test"

I've tried some expressions with update and append in the Derived Column transformation, but I can't get the syntax right to make both changes at the same time.

My biggest concern is with the MongoDB Sink: how to configure it so that Data Flow performs a partial update and doesn't overwrite the entire document, losing root_field, numeric_value, etc.?

My questions are:
What is the correct expression for the Derived Column that makes these two nested modifications in a single step?

How should I configure the MongoDB Sink to ensure the update is partial, using _id as the key?

I really appreciate the community's help!

Getting an Oauth2 API access token using client_id and client_secret - help

AzureNewbie1 — Thu, 14 Aug 2025 15:42:48 GMT

Hi,
I'm attempting to integrate external data into our SQL Server. The third-party data is from a solution called iLevel. They use token based Oauth2 APIs for access. The integration tool is ADF Pipelines.
I'm not a data engineer but it has fallen upon me to complete this exercise. What I've attempted so far is failing and I don't know why. I would like your help on this. I'll explain what I've configured so far in the order I configured it.

1) To generate a client_id and client_secret, I logged on to the iLevel solution itself and generated the same for my account (call me 'Joe' account) and the Team account (call it 'Data team' account). I've recorded the client_id and client_secret for both users/accounts in notepad for reference.
2) I logged in Azure Data Factory using my 'Joe Admin' admin account (this is the account I need to log in with for any ADF development).
3) I created a Linked Service with the following configuration. Note how the Test connection was successful. I guess this means our ADF instance can connect to iLevel's Base URL.

4) I then created a dataset for iLevel. I configured this based on an online example I was following which I can't get working, so this configuration may be incorrect.

5) I then created a Pipeline which contains a 'Web' activity and a 'Set variable' activity.
The Pipeline has a variable as shown below.

The 'Web' activity has the following configuration:

URL = is iLevel's token URL (it is different from the Base URL used in the Linked Service).
Body = I've blocked out the client_id and client_secret (I'm using the client_id and client_secret generated for the 'Data team' account - remember I'm logged into ADF using the 'Joe Admin' account - not sure if this makes a difference) but have placed red brackets around where the start and end of each values is. I'm wrapping the values in any single or double quotes - not sure if I'm meant to.

I'm not sure if I have configured the Body correctly. The ilevel documentation states to use an Authorization header, Content-Type header and Body - it states to the following is needed to obtain an access token, but it doesn't state exactly how to submit the information (i.e. how to format it). Notice how, in my configuration, I haven't used an Authorization header - this partly because an online example I've followed doesn't use one. If iLevel state to use one then I think I should but I don't know how to format it - any ideas?

The 'Set variable' activity has the following activity. The idea is the access token is retrieved from the 'Web' activity and placed in the 'Set variable' "iLevel access token" variable.

At this point I validate all and it comes back with no errors found. I then Debug it to see if it does indeed work but it returns an error stating the request contains an invalid client_id or client_secret. The client_id and client_secret values used are the exact same I generated from within the iLevel solution just a few hours ago.

Is anyone able to point out to me why this isn't working?
Have I populated all that I need to (as mentioned, iLevel say to use an Authorization header which I haven't but I don't know how to format it if I were to use one)?
What can I do to get this working?

I'm just trying to get the access token at the moment. I've not even attempted to extract the iLevel data and can't until I get a working token. iLevel's token have a 1 hour time-to-live so the Pipeline needs to generate a new token each time it's executed.

You help will be most appreciated. Thanks.

Dynamics AX connector stops getting records after amount of time

boydVos — Thu, 31 Jul 2025 10:44:40 GMT

Hello everyone,

I am using the Dynamics AX connector to get data out of Finance. After a certain amount of time it suddenly doesnt get any new records anymore and it keeps running until it reaches the general timeout.

It gets 290,000 records in like 01:30:00 and then keeps running and doesn't get new records anymore. Sometimes it gets stuck earlier or later.

Sometimes it also gives me this error:

Failure happened on 'Source' side. ErrorCode=ODataRequestTimeout,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Fail to get response from odata service in a expected time.,Source=Microsoft.DataTransfer.Runtime.ODataConnector,''Type=System.Threading.Tasks.TaskCanceledException,Message=A task was canceled.,Source=mscorlib,'

This is my pipeline JSON:

{

"name": "HICT - Init Sync SalesOrders",

"properties": {

"activities": [

{

"name": "Get FO SalesOrders",

"type": "Copy",

"dependsOn": [],

"policy": {

"timeout": "0.23:00:00",

"retry": 0,

"retryIntervalInSeconds": 30,

"secureOutput": false,

"secureInput": false

"userProperties": [],

"typeProperties": {

"source": {

"type": "DynamicsAXSource",

"query": "$filter=FM_InterCompanyOrder eq Microsoft.Dynamics.DataEntities.NoYes'No' and dataAreaId eq 'prev'&$select=SalesOrderNumber,SalesOrderName,IsDeliveryAddressPrivate,FormattedInvoiceAddress,FormattedDeliveryAddress,ArePricesIncludingSalesTax,RequestedReceiptDate,QuotationNumber,PriceCustomerGroupCode,PBS_PreferredInvoiceDate,PaymentTermsBaseDate,OrderTotalTaxAmount,OrderTotalChargesAmount,OrderTotalAmount,TotalDiscountAmount,IsInvoiceAddressPrivate,InvoiceBuildingCompliment,InvoiceAddressZipCode,LanguageId,IsDeliveryAddressOrderSpecific,IsOneTimeCustomer,InvoiceAddressStreetNumber,InvoiceAddressStreet,InvoiceAddressStateId,InvoiceAddressPostBox,InvoiceAddressLongitude,InvoiceAddressLatitude,InvoiceAddressDistrictName,InvoiceAddressCountyId,InvoiceAddressCountryRegionISOCode,InvoiceAddressCity,FM_Deadline,Email,DeliveryTermsCode,DeliveryModeCode,DeliveryBuildingCompliment,DeliveryAddressCountryRegionISOCode,DeliveryAddressZipCode,DeliveryAddressStreetNumber,SalesOrderStatus,DeliveryAddressStreet,DeliveryAddressStateId,SalesOrderPromisingMethod,DeliveryAddressPostBox,DeliveryAddressName,DeliveryAddressLongitude,DeliveryAddressLocationId,DeliveryAddressLatitude,DeliveryAddressDunsNumber,DeliveryAddressDistrictName,DeliveryAddressDescription,DeliveryAddressCountyId,DeliveryAddressCity,CustomersOrderReference,IsSalesProcessingStopped,CustomerRequisitionNumber,SalesOrderProcessingStatus,CurrencyCode,ConfirmedShippingDate,ConfirmedReceiptDate,SalesOrderOriginCode,URL,OrderingCustomerAccountNumber,InvoiceCustomerAccountNumber,ContactPersonId,FM_WorkerSalesTaker,FM_SalesResponsible,PaymentTermsName,DefaultShippingSiteId,DefaultShippingWarehouseId,DeliveryModeCode,dataAreaId,FM_InterCompanyOrder&cross-company=true",

"httpRequestTimeout": "00:15:00",

"additionalHeaders": {

"Prefer": "odata.maxpagesize=1000"

"retrieveEnumValuesAsString": true

"sink": {

"type": "JsonSink",

"storeSettings": {

"type": "AzureBlobStorageWriteSettings",

"copyBehavior": "FlattenHierarchy"

"formatSettings": {

"type": "JsonWriteSettings"

}

"enableStaging": false,

"enableSkipIncompatibleRow": true,

"logSettings": {

"enableCopyActivityLog": true,

"copyActivityLogSettings": {

"logLevel": "Warning",

"enableReliableLogging": false

"logLocationSettings": {

"linkedServiceName": {

"referenceName": "AzureBlobStorage",

"type": "LinkedServiceReference"

"path": "ceexports"

}

"inputs": [

{

"referenceName": "AX_SalesOrders_Dynamics_365_FO_ACC",

"type": "DatasetReference"

}

"outputs": [

{

"referenceName": "Orders_FO_D365_Data_JSON",

"type": "DatasetReference"

}

]

{

"name": "Get_All_CE_Table_Data",

"type": "ForEach",

"dependsOn": [

{

"activity": "Get FO SalesOrders",

"dependencyConditions": [

"Completed"

]

}

"userProperties": [],

"typeProperties": {

"items": {

"value": "@pipeline().parameters.CE_Tables",

"type": "Expression"

"activities": [

{

"name": "Copy_CE_TableData",

"type": "Copy",

"dependsOn": [],

"policy": {

"timeout": "0.12:00:00",

"retry": 0,

"retryIntervalInSeconds": 30,

"secureOutput": false,

"secureInput": false

"userProperties": [],

"typeProperties": {

"source": {

"type": "CommonDataServiceForAppsSource"

"sink": {

"type": "DelimitedTextSink",

"storeSettings": {

"type": "AzureBlobStorageWriteSettings",

"copyBehavior": "FlattenHierarchy"

"formatSettings": {

"type": "DelimitedTextWriteSettings",

"quoteAllText": true,

"fileExtension": ".txt"

}

"enableStaging": false

"inputs": [

{

"referenceName": "CE_Look_Up_Tables",

"type": "DatasetReference",

"parameters": {

"entiryName": "@item().sourceDataset"

}

"outputs": [

{

"referenceName": "CE_GenericBlobSink",

"type": "DatasetReference",

"parameters": {

"sinkPath": {

"value": "@item().sinkPath",

"type": "Expression"

}

]

}

]

}

{

"name": "Transform_Create_CE_JSON",

"type": "ExecuteDataFlow",

"dependsOn": [

{

"activity": "Get_All_CE_Table_Data",

"dependencyConditions": [

"Succeeded"

]

}

"policy": {

"timeout": "0.12:00:00",

"retry": 0,

"retryIntervalInSeconds": 30,

"secureOutput": false,

"secureInput": false

"userProperties": [],

"typeProperties": {

"dataflow": {

"referenceName": "FO_Transform_CE_Select",

"type": "DataFlowReference"

"compute": {

"coreCount": 16,

"computeType": "General"

"traceLevel": "Fine"

}

"parameters": {

"CE_Tables": {

"type": "array",

"defaultValue": [

{

"name": "D365_CE_ACC_AccountRelations",

"sourceDataset": "crmp_accountrelation",

"sinkPath": "ce-exports/D365_CE_ACC_AccountRelations.json"

{

"name": "D365_CE_ACC_ContactRelations",

"sourceDataset": "crmp_contactrelation",

"sinkPath": "ce-exports/D365_CE_ACC_ContactRelations.json"

{

"name": "D365_CE_ACC_PriceCustomerGroup",

"sourceDataset": "msdyn_pricecustomergroup",

"sinkPath": "ce-exports/D365_CE_ACC_PriceCustomerGroup.json"

{

"name": "D365_CE_ACC_SalesOrderOrigin",

"sourceDataset": "odin_salesorderorigin",

"sinkPath": "ce-exports/D365_CE_ACC_SalesOrderOrigin.json"

{

"name": "D365_CE_ACC_ShipVia",

"sourceDataset": "msdyn_shipvia",

"sinkPath": "ce-exports/D365_CE_ACC_ShipVia.json"

{

"name": "D365_CE_ACC_SystemUser",

"sourceDataset": "systemuser",

"sinkPath": "ce-exports/D365_CE_ACC_SystemUser.json"

{

"name": "D365_CE_ACC_TermsOfDelivery",

"sourceDataset": "msdyn_termsofdelivery",

"sinkPath": "ce-exports/D365_CE_ACC_TermsOfDelivery.json"

{

"name": "D365_CE_ACC_Worker",

"sourceDataset": "cdm_worker",

"sinkPath": "ce-exports/D365_CE_ACC_Worker.json"

{

"name": "D365_CE_ACC_TransactionCurrency",

"sourceDataset": "transactioncurrency",

"sinkPath": "ce-exports/D365_CE_ACC_TransactionCurrency.json"

{

"name": "D365_CE_ACC_Warehouse",

"sourceDataset": "msdyn_warehouse",

"sinkPath": "ce-exports/D365_CE_ACC_Warehouse.json"

{

"name": "D365_CE_ACC_OperationalSite",

"sourceDataset": "msdyn_operationalsite",

"sinkPath": "ce-exports/D365_CE_ACC_OperationalSite.json"

{

"name": "D365_CE_ACC_PaymentTerms",

"sourceDataset": "odin_paymentterms",

"sinkPath": "ce-exports/D365_CE_ACC_PaymentTerms.json"

}

]

}

"annotations": [],

"lastPublishTime": "2025-07-30T12:55:32Z"

"type": "Microsoft.DataFactory/factories/pipelines"

}

Azure Data Factory ForEach Loop Fails Despite Inner Activity Error Handling - Seeking Best Practices

vijaybandari — Sun, 20 Jul 2025 09:29:40 GMT

Hello Azure Data Factory Community,

I'm encountering a persistent issue with my ADF pipeline where a ForEach loop is failing, even though I've implemented error handling for the inner activities. I'm looking for insights and best practices on how to prevent internal activity failures from propagating up and causing the entire ForEach loop (and subsequently the pipeline) to fail, while still logging all outcomes.

My Setup:

My pipeline processes records using a ForEach loop. Inside the loop, I have a Web activity (Sample_put_record) that calls an external API. This API call can either succeed or fail for individual records.

My current error handling within the ForEach iteration is structured as follows:

1.Sample_put_record (Web Activity): Makes the API call.

2.Conditional Logic: I've tried two main approaches:

•Approach A (Direct Success/Failure Paths): The Sample_put_record activity has a green arrow (on success) leading to a Log Success Items (Script activity) and a red arrow (on failure) leading to a Log Failed Items (Script activity). Both logging activities are followed by Wait activities (Dummy Wait For Success/Failure).

•Approach B (If Condition Wrapper): I've wrapped the Sample_put_record activity and its success/failure logging within an If Condition activity. The If Condition's expression is @equals(activity('Sample_put_record').status, 'Succeeded'). The True branch contains the success logging, and the False branch contains the failure logging. The intention here was for the If Condition to always report success, regardless of the Sample_put_record outcome, to prevent the ForEach from failing.

The Problem:

Despite these error handling attempts, the ForEach loop (and thus the overall pipeline) still fails when an Sample_put_record activity fails. The error message I typically see for the ForEach activity is "Activity failed because an inner activity failed." When using the If Condition wrapper, the If Condition itself sometimes fails with the same error, indicating that an activity within its True or False branch is still causing a hard failure.

For example, a common failure for Sample_put_record is: "valid":false,"message":"WARNING: There was no xxxxxxxxxxxxxxxxxxxxxxxxx scheduled..." (a user configuration/data issue). Even when my Log Failed Items script attempts to capture this, the ForEach still breaks.

What I've Ensured/Considered:

•Wait Activity Configuration: Wait activities are configured with reasonable durations and do not appear to be the direct cause of failure.

•No Unhandled Exceptions: I'm trying to ensure no unhandled exceptions are propagating from my error handling activities.

•Pipeline Status Goal: My ultimate goal is for the overall pipeline status to be Succeeded as long as the pipeline completes its execution, even if some Sample_put_record calls fail and are logged. I need to rely on the logs to identify actual failures, not the pipeline status.

My Questions to the Community:

1.What is the definitive best practice in Azure Data Factory to ensure a ForEach loop never fails due to an inner activity failure, assuming the inner activity's failure is properly logged and handled within that iteration?

2.Are there specific nuances or common pitfalls with If Condition activities or Script activities within ForEach loops that could still cause failure propagation, even with try-catch and success exits?

3.How do you typically structure your ADF pipelines to achieve this level of resilience where internal failures are logged but don't impact the overall pipeline success status?

4.Are there any specific configurations on the ForEach activity itself (e.g., Continue on error setting, if it exists for ForEach?) or other activities that I might be overlooking?

Any detailed examples, architectural patterns, or debugging tips would be greatly appreciated. Thank you in advance for your help!

Copy Activity Successful, But Times Out

istock-ewh — Fri, 11 Jul 2025 20:14:56 GMT

This appears to be an edge case, but I wanted to share.

A copy activity is successful, but times out. Duration is 1:58:55. Times out at 2:00:12. Runs a second time time and is successful, loading duplicate records. The duplicate records is the undesired result.

Copy Activity

General
- Timeout: 0.02:00:00
- Retry: 2
Source
- mySQL Parameterized
- SQL Parameterized
Sink
- Synapse SQL Pool Parameterized
- Copy method: COPY command
Settings
- Use V2 Hiearchy storage for staging

General

Synapse/ADF Managed Network

Advice requested: how to capture full SQL CDC changes using Dataflow and ADLS gen2

AnnejanBarelds — Thu, 03 Jul 2025 16:02:49 GMT

Hi,

I'm working on a fairly simple ETL process using Dataflow in Azure Data Factory, where I want to capture the changes in a CDC-enabled SQL table, and store those in Delta Lake format in a ADLS gen2 sink. The resulting dataset will be further processed, but for me this is the end of the line. I don't have an expert understanding of all the details of the Delta Lake format, but I do know that I can use it to store changes to my data over time. So in the sink, I enabled all Update methods (Insert, Delete, Upsert, Update), since my CDC source should be able to figure out the correct row transformation. Key columns are set to the primary key columns in SQL.

All this works fine as long as I configure my source to use CDC with 'netChanges: true'. That yields a single change row for each record, which is correctly stored in the sink.

But I want to capture all changes since the previous run, so I want to set the source to netChanges: false. That yields rows for every change since the previous time the dataflow ran. But for every table that actually has records with more than one change, the dataflow fails saying "Cannot perform Merge as multiple source rows matched and attempted to modify the same target row in the Delta table in possibly conflicting ways."

I take that to mean that my dataflow is, as it is, not smart enough to loop through all changes in the source, and apply them to the sink in order. So apparently something else has to be done. My intuition says that, since CDC actually provides all the metadata to make this possible, there's probably an out-of-the-box way to achieve what I want. But I can't readily find that magic box I should tick 😉.

I can probably build it out 'by hand', by somehow looping over all changes and applying them in order, but before I go down that route, I came here to learn from the experts whether this is indeed the only way, or, preferably, that there is a neat trick I missed to get this done easily.

Thanks so much for your advice!

Solution: Handling Concurrency in Azure Data Factory with Marker Files and Web Activities

mkoralage — Sun, 29 Jun 2025 03:45:17 GMT

Hi everyone,

I wanted to share a concurrency issue we encountered in Azure Data Factory (ADF) and how we resolved it using a small but effective enhancement—one that might be useful if you're working with shared Blob Storage across multiple environments (like Dev, Test, and Prod).

Background: Shared Blob Storage & Marker Files

In our ADF pipelines, we extract data from various sources (e.g., SharePoint, Oracle) and store them in Azure Blob Storage. That Blob container is shared across multiple environments.

To prevent duplicate extractions, we use marker files:

started.marker → created when a copy begins
completed.marker → created when the copy finishes successfully

If both markers exist, pipelines reuse the existing file (caching logic). This mechanism was already in place and worked well under normal conditions.

The Issue: Race Conditions

We observed that simultaneous executions from multiple environments sometimes led to:

Overlapping attempts to create the same started.marker
Duplicate copy activities
Corrupted Blob files

This became a serious concern because the Blob file was later loaded into Azure SQL Server, and any corruption led to failed loads.

The Fix: Web Activity + REST API

To solve this, we modified only the creation of started.marker by:

Replacing Copy Activity with a Web Activity that calls the Azure Storage REST API
The API uses Azure Blob Storage's conditional header If-None-Match: * to safely create the file only if it doesn't exist

If the file already exists, the API returns "BlobAlreadyExists", which the pipeline handles by skipping.

The Copy Activity is still used to copy the data and create the completed.marker—no changes needed there.

Updated Flow

Check marker files:
- If both exist (started and completed) → use cached file
- If only started.marker → wait and retry
- If none → continue to step 2
Web Activity calls REST API to create started.marker
- Success → proceed with copy in step 3
- Failure → another run already started → skip/retry
Copy Activity performs the data extract
Copy Activity creates completed.marker

Benefits

Atomic creation of started.marker → no race conditions
Minimal change to existing pipeline logic with marker files
Reliable downstream loads into Azure SQL Server
Preserves existing architecture (no full redesign)

Would love to hear:

Have you used similar marker-based patterns in ADF?
Any other approaches to concurrency control that worked for your team?

Thanks for reading! Hope this helps someone facing similar issues.

Blob Storage Event Trigger Disappears

RobDuMo — Tue, 24 Jun 2025 13:13:21 GMT

Yesterday I ran into an odd situation where there was a resource lock and I was unable to rename pipelines or drop/create storage event triggers. An admin cleared the lock and I was able to remove and clean up the triggers and pipelines. Today, when I try to recreate the blob storage trigger to process a file when it appears in a container, the trigger creates just fine but on refresh, it disappears. If I try to recreate it again with the same name as the one that went away ADF UI says it already exists. I cannot assign it to a pipeline because the UI does not see it. Any insight as to where it is, how I can see it, or even what logs would have such activity recorded to give a clue as to what is going on. This seems like a bug.

Best practice to integrate to Azure DevOps?

alwaysLearner — Sun, 08 Jun 2025 17:43:27 GMT

Different sources suggesting different recommendations regarding ADF and ADO integration. Some say to use 'adf_publish' branch, while some suggest to use 'main' branch to be source for triggering yaml pipelines and disabling 'Publish' function in ADF. I guess practices are changing and setup could be different. The problem is finding all this information on the Internet makes it so confusing.

So, the question is what is the best practice now (taking into account all the latest changes in ADO) regarding branches? How you set up your ADF and ADO integrations?

Parameterization of Linked Services

alwaysLearner — Sun, 08 Jun 2025 17:28:04 GMT

I am trying to parameterize Linked Service in ADF. Probably got confused, and hope someone will make it clear.

Two questions:

I have two parameters: 'url' and 'secretName'. However, in ARM template I only see 'url' parameter, but not 'secretName'. Why 'secretName' is not parameterized?
How do I supply a value for the 'url' parameter when I will deploy ARM template to another environment (let's say 'Test' environment)?

These are files:

Linked Service:

{ "name": "LS_DynamicParam", "properties": { "parameters": { "SA_URL": { "type": "String", "defaultValue": "https://saforrisma.dfs.core.windows.net/" }, "SecretName": { "type": "String", "defaultValue": "MySecretInKeyVault" } }, "annotations": [], "type": "AzureBlobFS", "typeProperties": { "url": "@{linkedService().SA_URL}", "accountKey": { "type": "AzureKeyVaultSecret", "store": { "referenceName": "LS_AKV", "type": "LinkedServiceReference" }, "secretName": { "value": "@linkedService().SecretName", "type": "Expression" } } } } }

ARMTemplatePArametersForFactory.json

{ "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#", "contentVersion": "1.0.0.0", "parameters": { "factoryName": { "value": "ADF-Dev" }, "LS_AKV_properties_typeProperties_baseUrl": { "value": "https://kv-forrisma.vault.azure.net/" }, "LS_MAINStorage_properties_typeProperties_connectionString_secretName": { "value": "storageaccount-adf-dev" }, "LS_DynamicParam_properties_typeProperties_url": { "value": "@{linkedService().SA_URL}" } } }