Forum Widgets
Latest Discussions
Importing an ARM template into ADF DevOps
Hi everyone, requesting help in ADF and I am new to ADF & ADO. I have 2 ADF workspaces A & B; A is enabled with Git and B is in Live mode. All my resources like pipelines, data flows, datasets, linked services, credentials & triggers are present in Workspace B. I need to import these into collaboration branch of Workspace A. I tried copying the ARM Template to the git repo into the collaboration branch manually, but this doesn't show up anything in the Workspace A. Am I missing anything here? Any help is much appreciated! Thanks ln
ServiceNow Connection - data request URL too long for pagination
Hi, So we've encountered an issue after setting up a connection between data-factory and ServiceNow. Our team has been trying to query a really big table (alm_asset) from our ServiceNow instance - and when we try to add Pagination to be anyhting but empty, for some reason DataFactory lists all of the columns to be queried. Now that column query list we couldn't find, and our REST request could not be executed because of the too long url, so pagination could not fit. The API request to ServiceNow failed. Request Url : -- removed -- , Status Code: BadRequest, Error message: {"error":{"message":"Pagination not supported","detail":"The requested query is too long to build the response pagination header URLs. Please do one of the following: shorten the sysparm_query, or query without pagination by setting the parameter 'sysparm_suppress_pagination_header' to true, or set 'sysparm_limit' with a value larger then 4182 to bypass the need for pagination."},"status":"failure"} This 4182 is just a on a sub-production instance, on produciton instance we have significantly more data. Can somebody help how to edit the params sent for the REST API through that connector?
OData Connector for Dynamics Business Central
Hey Guys, I'm trying to connect Dynamics Business Central OData API in ADF but I'm not sure what I'm doing wrong here because the same Endpoint is returning data on Postman but returning an error in ADF LinkedService. https://api.businesscentral.dynamics.com/v2.0/{tenant-id}/Sandbox-UAT/ODataV4/Company('company-name')/Chart_of_Accounts
Manage Pipelines (Start/Stop/Monitoring)
I cannot find a way to manage many pipelines for ETL. For example, in case of multiple pipelines execution, if i want to disable the execution of any pipelines - how can this be done ? Is there a tool by Microsoft or any third party tool, which can help manage, execution and monitoring of pipelines in ADF ? Also, are there any best practices or patterns to manage multiple pipelines ?How do I unpivot with schema drift enabled?
I have a source without a pre-defined schema. I derive each column name using a column pattern expression: Data preview shows what I expect (which is a file in a blob container): I then have a Select step that selects each column and renames 'Marker name' to 'Marker_name22': Data preview again shows what I expect (same columns with 'Marker name' renamed). Now in the unpivot step, I would like to ungroup by the 'Marker_name22' column and unpivot all other columns, but the 'Marker_name22' column is not available: I am unsure how to proceed from here. Thanks in advance for the help.
How do I create a flow that adapts to new columns dynamically?
Hello, I have files landing in a blob storage container that I'd like to copy to a SQL database table. The column headers of these files are date markers, so each time a new file is uploaded, a new date will appear as a new column. How can I handle this in a pipeline? I think I'll need to dynamically accept the schema and then use an unpivot transformation to normalize the data structure for SQL, but I am unsure how to execute this plan. Thanks!Documentation Generator for Azure Data Factory
Hi I couldnot find a document generator in ADF ( unless I am missing something) , So I have written a utility script in Python for generating documentation ( sort of a quick reference ) from the ARM export. I am still in the process of refining it and adding more artifacts, but you can modify and extend it as required. I hope someone will find it useful, to get a quick overview of all the objects in the ADF. https://github.com/sanjayganvkar/AzureDataFactory---Documentation-Generator Regards Sanjay
