Latest Discussions
Send files from Blob Storage to Vendor API
Hello, In an Azure Blob container in our tenant we have several thousand .json files that need to be ingest by a vendor search api. The vendor example is to send as PUT /v2/indices/{index public key}/documents/{document id} to their api. My background with ADF is copyfiles from a local fileshare to blob. I just copy with source and sink - works Never having done this before and using Copilot it suggested creating a pipeline, using the Get Metadata activity. I did that. The setting of Get Metadata point to a Dataset. (see attachments for images - not the blob setting show successful connection and preview) at this point I just tried to debug it and got this message: Error code 2001 Failure typeUser configuration issue Details The length of execution output is over limit (around 4MB currently) Activity ID b74f3802-804f-4b04-91c2-3f68546a40a5 Each files is about 20KB, but I suspect it is trying to get all the files as one. If this is the case, how do I get it to iterate one by one? Copilot said to use a Filter activity, but that is AFTER the Get Metadata statement. Any help on how to proceed OR troubleshoot this better? Thanks, V
ServiceNow Connection - data request URL too long for pagination
Hi, So we've encountered an issue after setting up a connection between data-factory and ServiceNow. Our team has been trying to query a really big table (alm_asset) from our ServiceNow instance - and when we try to add Pagination to be anyhting but empty, for some reason DataFactory lists all of the columns to be queried. Now that column query list we couldn't find, and our REST request could not be executed because of the too long url, so pagination could not fit. The API request to ServiceNow failed. Request Url : -- removed -- , Status Code: BadRequest, Error message: {"error":{"message":"Pagination not supported","detail":"The requested query is too long to build the response pagination header URLs. Please do one of the following: shorten the sysparm_query, or query without pagination by setting the parameter 'sysparm_suppress_pagination_header' to true, or set 'sysparm_limit' with a value larger then 4182 to bypass the need for pagination."},"status":"failure"} This 4182 is just a on a sub-production instance, on produciton instance we have significantly more data. Can somebody help how to edit the params sent for the REST API through that connector?
Azure Data Factory Web API call to Azure REST API Slow
I can reproduce this easily. Call to Azure REST API to publish data to Event Hub using SAS authorization takes few milliseconds from any REST client. Create Event hub namespace URI and Event Hub Name Copy the URI and RootManageSharedAccessKey Create authorization for REST API https://learn.microsoft.com/en-us/rest/api/eventhub/generate-sas-token#code-try-8 Send data to Event Hub via REST API using Authorization with Postman or Thunder client https://learn.microsoft.com/en-us/rest/api/eventhub/send-event Create Azure Data Factory Create foreach activity containing web activity and iterate over an array of four items in parallel Web activity task takes over 3 to 4 seconds when called from Azure Data Factory Web activity. Any ideas are welcome.
Failure of azure data factory integration runtime with Vnet enabled
I had been using Data Factory's integration runtime with VNet successfully, but it recently stopped connecting to Cosmos DB with the MongoDB API (which is also within a VNet). After setting up a new integration runtime with VNet enabled and selecting the region as 'Auto Resolve,' the pipeline ran successfully with this new runtime. Could you help me understand why the previous integration runtime—configured with VNet enabled and the region set to match that of Azure Data Factory—worked for over a month but then suddenly failed? The new integration runtime with VNet and 'Auto Resolve' region worked, but I'm uncertain if the 'Auto Resolve' region contributed to the success or if something else allowed it to connect. Error:Failure happened on 'Source' side. ErrorCode=MongoDbConnectionTimeout,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=>Connection to MongoDB server is timeout.,Source=Microsoft.DataTransfer.Runtime.MongoDbAtlasConnector,''Type=System.TimeoutException,Message=A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/cosmontiv01u.mongo.cosmos.azure.com:10255" }", EndPoint:
REST API and Pagination
So,I'm having some issues getting all the data from a Rest API intoAzure Data Factory. Currently I have everything setup correctly the mapping the sourceetc.The REST API returns only 100 results and there are pagination rules. I'm not 100% sure how-tosetupthese rules in Azure. Currently the Copy Data activity only getsthe first page of 100 and that'sit. }, "pagination": { "endCursor": "965125e2-b907-493e-b0fd-c938531f60e2", "hasNextPage": true } } Each page ends with this Json response. Where endCursor needs to be added to the relative URL like this ?after=965125e2-b907-493e-b0fd-c938531f60e2 For Example, the first request would be "https://api.samsara.com/fleet/reports/vehicles/fuel-energy?startDate=2023-12-29T00%3A00%3A00.00-06%3A00&endDate=2023-12-29T00%3A00%3A00.00-06%3A00" Then the next request "https://api.samsara.com/fleet/reports/vehicles/fuel-energy?startDate=2023-12-29T00%3A00%3A00.00-06%3A00&endDate=2023-12-29T00%3A00%3A00.00-06%3A00?after=965125e2-b907-493e-b0fd-c938531f60e2" In the source dataset the relative URL is also set as dynamic content. @{concat('/fleet/reports/vehicles/fuel-energy?startDate=', encodeURIComponent(formatDateTime(subtractFromTime(utcnow(), 96, 'Hour'), 'yyyy-MM-ddTHH:mm:ss.fffffffZ')),'&endDate=', encodeURIComponent(formatDateTime(subtractFromTime(utcnow(), 72, 'Hour'), 'yyyy-MM-ddTHH:mm:ss.fffffffZ')))} How would I define this requirement here? Or am I looking at this totally wrong?
Copy data from SQL and post it into Hubspot API (for company) using ADF
Hi everyone, I am new to ADF and seeking help in a scenario where I need to copy data from SQL database and feed it into Hubspot for company. Below is the query I am using to convert it into Json and the dump into Blob. SELECT top 10 name AS [properties.name], new_businesswebsite AS [properties.domain], new_city AS [properties.city], new_emergencycontactmobile AS [properties.phone] FROM [zdev].CRM_Accounts FOR JSON PATH, WITHOUT_ARRAY_WRAPPER; This is the preview of data Preview data Linked service:DATA_WAREHOUSE Object:JSON_F52E2B61-18A1-11d1-B105-00805F49916B {"properties":{"name":"Al Duaa Tours Limited (Accept No Bookings)","domain":"www.alatours.com","city":"Renfrew","phone":"07921005"}},{"properties":{"name":"TEST","city":"Jersy","phone":"01239"}},{"properties":{"name":"American Express"}},{"properties":{"name":"Reizen De Cauwer","city":"Sint Niklaas"}},{"properties":{"name":"TEST","city":"Jersy"}},{"properties":{"name":"SPORTSCORP","city":"BOLTON"}},{"properties":{"name":"HIS CHARIOT TRAVELS ","city":"AJA"}},{"properties":{"name":"TEST","city":"Jersy","phone":"0156789"}},{"properties":{"name":"Cruise Village (Accept No Bookings)","domain":"https:\/\/www.thecruisevillage.com","city":"Blackpool","phone":"07976926998"}},{"properties":{"name":"Riotravel Turismo 5661 ","city":"Rio de Janeiro"}} In the next step I am trying to use foreach loop activity to parse data into hubspot with the relevant columns. Settings Items @activity('Customer Data Copy').output Activities: Web For Web activity below are settings. URL: is not a problem Method: Post Body: { "properties": { "name": "@{item().properties.name}", "domain": "@{item().properties.domain}", "city": "@{item().properties.city}", "phone": "@{item().properties.phone}" } } Error: Failure type: User configuration issue Details: The function 'length' expects its parameter to be an array or a string. The provided value is of type 'Object'. Please suggest how to resolve this issue, or provide alternative optimized way to achieve this goal. There are around 50k rows that should be imported and api can't be feed for much records at once.
Issue with Azure Data Factory Pipeline Connection to SQL Server
Hi everyone, I’m encountering an issue with my Azure Data Factory pipeline that uses a copy activity to transfer data from MariaDB. The connection is established using a self-hosted integration runtime, and everything seems to be configured correctly. The test connection is valid, and I can preview the data without any issues. However, when I run the pipeline, I receive the following error: ``` ErrorCode=SqlFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot connect to SQL Database. Please contact SQL server team for further support. Check the linked service configuration is correct, and make sure the SQL Database firewall allows the integration runtime to access. 'Type=System.Data.SqlClient.SqlException,Message=A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server),' ``` The pipeline used to work fine before, and I haven't made any changes to the linked service configuration. I have also confirmed that the IP address used by the self-hosted IR has been added to the firewall rules of the SQL server, but I’m not an admin, so I can’t verify it fully. Has anyone encountered a similar issue or have suggestions on how to troubleshoot further? Thanks in advance!API fetching data from GIT loading to Storage Account in Parquet
I have a api that i am calling via ADF the data that i am bringing is 28 days and i want to build historical data in the incoming data there is a column called "day" which holds date. i want to reference that and make the adf pipeline so it writes incrementally what would be the approach?Column in Common Data Model source always returning null values in ADF
I have a pipeline in ADF where I am fetching data from a Common Data Model (CDM) and this pipeline has been working fine for a long time. We updated one of the Entities in the CDM with a new column of type string which I am now trying to fetch. When I import the new schema into ADF Source I find the new column but all data fetched for that column only returns null. I have confirmed the following: All other columns are working as before and returns data. The CSV file referred to by the CDM has values that are not null in the new column, so data exists. Data type in the schema matches the data type specified in the CDM for the new column (string) Does anyone have an idea of what could be the issue? I saw this post with similar issue but the data type in that case is date:https://learn.microsoft.com/en-us/answers/questions/942876/column-is-always-getting-null-values-in-adf-data-f
