Forum Widgets
Latest Discussions
Clarification on Staging Directory Usage for SAP CDC Connector in Azure Data Factory
Hi! I'm currently working on a project where we are ingesting data from SAP using the SAP CDC connector in Azure Data Factory(Data flow). The source is S4HAHA CDS views. We are using a staging directory for the data flow with a checkpoint mechanism, similar to described here: https://learn.microsoft.com/en-us/azure/data-factory/connector-sap-change-data-capture My question is: Does the staging directory only act as a temporary storage location during ingestion from sap? If i understand correctly its used for retries, but no real usage once the deltas have been ingested. After the data has been loaded to the destination(in our case container inside of ADLS), is the data needed for maintaining delta states? Can the data be safely deleted(from the staging container) without impacting the subsequent load runs? We were thinking of implementing a 7 day retention policy on the staging container so we can manage storage efficiently. Thank you in advance for any information regarding this.
Executing Oracle Stored Procedure in Azure Data Factory
Hello There, I want to execute(call) an Oracle Stored Procedure from Azure Data Factory (ADF). The Stored Procedure resides in an On-prem oracle server and I was able to establish the connectivity via the Self Hosted Integration Run Time. The oracle stored procedure has 1 input parameter (user defined datatype of object) and 1 output parameter (Varchar2 datatype). Can you please navigate me in terms of which activity to use in ADF to call the Oracle Stored Procedure and show how it is done in terms of variable declarations and the call command. A walk through example with screenshots would be really helpful. Also, I need to capture the Oracle Stored Procedure Output parameter value in ADF so that I can use that to control the pipeline flow in the subsequent steps. Thank You
What is the way to use OUTPUT parameter for an Oracle Stored procedure in ADF pipelines?
I have a oracle database package and i am trying to call a stored procedure inside that package. The procedure has a OUT parameter which we want to use in the activities further in ADF pipelines. But ADF pipelines does not have a way to get the OUT parameter values and use it in pipeline. This is a very important feature.
Using "Expect Unique" assert type on ADF flow with additional streams
Hello, I'm trying to use ADF functionality assert with an additional stream but it seems it doesn't work. It recognizes the column name and I can select it but when I try a data preview, I get a "Resolved attribute(s) [...] missing from (list of attributes coming from main source) in operator !Project (list of attributes from main source + that attribute)" The attribute does exist in my source, I can see it in the data preview, and use it in lookup. I tried other attributes with same result. The documentation is not very clear on whether or not this scenario is possible. I'm guessing not but would like confirmation. Thank you.
Writing data to a lookup field via alternative keys
Hi Folks, I have 2 table, 1. Contact with a secondary key 2. Policyholder with a lookup field to contact and a column that contain the contact secondary key guid as a string I want to take the column named : contactSecondaryKeyString in policyholder which contain the secondary key string(guid) and push it to my lookup field policyholder in the table policy holder. As per microsoft it is possible :Copy and transform data in Dynamics 365 (Microsoft Dataverse) or Dynamics CRM - Azure Data Factory & Azure Synapse | Microsoft Learn Here is my mapping code : and I get the error : "ErrorCode=DynamicsOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=,Source=,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Dynamics operation failed with error code: -2147217149, error message: 'prsc_phplan' entity doesn't contain attribute with Name = 'prsc_policyholder@prsc_systemid' and NameMapping = 'Logical' I tried many variation and still can't figure it out...
What Synapse Serverless SQL pool authentication type for ADF Linked Service?
Hi, I'm relatively new to Azure Data Factory and require your guidance on how to successfully create/test a Linked Service to the Azure Synapse Analytics Serverless SQL pool. In the past, I've successfully created a Linked Service to a third-party (outside our domain) on-premises SQL Server through creating a self-hosted integration runtime on their box and then creating a Linked Service to use that. The Server Name, Database Name, Windows authentication, my username and password all configured by the third-party is what I entered into the Linked Service configuration boxes. All successfully tested. This third-party data was extracted and imported, via ADF Pipelines, into an Azure SQL Server database within our domain. Now I need to extract data from our own (hosted in our domain) Azure Synapse Analytics Serverless SQL pool database. My attempt is this, and it fails: 1) I create a 'Azure Synapse Analytics' Data Store Linked Service. 2) I select the 'AutoResolveIntegrationRuntime' as the runtime to use - I'm thinking this is correct as the Synapse source is within our domain (we're fully MS cloud based). 3) I select 'Enter manually' under the 'Account selection method'. 4) I've got the Azure Synapse Analytics Serverless SQL endpoint - which I place into the 'Fully qualified domain name' field. 5) I entered the data SQL Database name found under the 'SQL database' node/section present on the Data >> Workspace screen in Synapse. 6) I choose 'System-assigned managed identity' as the Authentication type - this is a guess and I was hoping it would recognised my username/account that I am building the Linked Service with, as that account also can query Synapse too and so has Synapse access. 7) I check the 'Trust server certification' box. All else is default. When I click test connection, it fails with the following message: "Cannot connect to SQL Database. Please contact SQL server team for further support. Server: 'xxxxxxxxxxxx-ondemand.sql.azuresynapse.net', Database: 'Synapse_Dynamics_data', User: ''. Check the linked service configuration is correct, and make sure the SQL Database firewall allows the integration runtime to access. Login failed for user '<token-identified principal>'." I've reached out to our I.T. (who are novices with Synapse, ADF, etc.. even though they did install them in our domain) and they don't know how to help me. I'm hoping you can help. 1) Is choosing the 'Azure Synapse Analytics' the correct Data Store to chose when looking extract data from an Azure Synapse Serverless SQL pool SQL database? 2) Is using the AutoResolveIntegrationRuntime correct if Synapse is held within our domain? I've previously confirmed this runtime works (and still does) as when importing the third-party data I had to use that runtime to load the data to our Azure SQL Server database. 3) Have I populated the correct values for the 'Fully qualified domain name' and 'Database name' fields by entering the Azure Synapse Analytics Serverless SQL endpoint and subsequent SQL Database name, respectively? 4) Is choosing 'System-assigned managed identity' as the Authentication type correct? I'm guessing this could be the issue. I selected this as when loading the mentioned third-party data into the Azure SQL Server database, within our domain, this was the authentication type that was used (and works) and so I'm assuming it somehow recognises the user logged in and, through the magic of cloud authentication, says this user has the correct privileges (as I should have the correct privileges so say I.T.) so allow the Linked Service to work. Any guidance you can provide me will be much appreciated. Thanks.Send files from Blob Storage to Vendor API
Hello, In an Azure Blob container in our tenant we have several thousand .json files that need to be ingest by a vendor search api. The vendor example is to send as PUT /v2/indices/{index public key}/documents/{document id} to their api. My background with ADF is copyfiles from a local fileshare to blob. I just copy with source and sink - works Never having done this before and using Copilot it suggested creating a pipeline, using the Get Metadata activity. I did that. The setting of Get Metadata point to a Dataset. (see attachments for images - not the blob setting show successful connection and preview) at this point I just tried to debug it and got this message: Error code 2001 Failure typeUser configuration issue Details The length of execution output is over limit (around 4MB currently) Activity ID b74f3802-804f-4b04-91c2-3f68546a40a5 Each files is about 20KB, but I suspect it is trying to get all the files as one. If this is the case, how do I get it to iterate one by one? Copilot said to use a Filter activity, but that is AFTER the Get Metadata statement. Any help on how to proceed OR troubleshoot this better? Thanks, V
ServiceNow Connection - data request URL too long for pagination
Hi, So we've encountered an issue after setting up a connection between data-factory and ServiceNow. Our team has been trying to query a really big table (alm_asset) from our ServiceNow instance - and when we try to add Pagination to be anyhting but empty, for some reason DataFactory lists all of the columns to be queried. Now that column query list we couldn't find, and our REST request could not be executed because of the too long url, so pagination could not fit. The API request to ServiceNow failed. Request Url : -- removed -- , Status Code: BadRequest, Error message: {"error":{"message":"Pagination not supported","detail":"The requested query is too long to build the response pagination header URLs. Please do one of the following: shorten the sysparm_query, or query without pagination by setting the parameter 'sysparm_suppress_pagination_header' to true, or set 'sysparm_limit' with a value larger then 4182 to bypass the need for pagination."},"status":"failure"} This 4182 is just a on a sub-production instance, on produciton instance we have significantly more data. Can somebody help how to edit the params sent for the REST API through that connector?
Azure Data Factory Web API call to Azure REST API Slow
I can reproduce this easily. Call to Azure REST API to publish data to Event Hub using SAS authorization takes few milliseconds from any REST client. Create Event hub namespace URI and Event Hub Name Copy the URI and RootManageSharedAccessKey Create authorization for REST API https://learn.microsoft.com/en-us/rest/api/eventhub/generate-sas-token#code-try-8 Send data to Event Hub via REST API using Authorization with Postman or Thunder client https://learn.microsoft.com/en-us/rest/api/eventhub/send-event Create Azure Data Factory Create foreach activity containing web activity and iterate over an array of four items in parallel Web activity task takes over 3 to 4 seconds when called from Azure Data Factory Web activity. Any ideas are welcome.
