Recent Discussions
Region-access support requests for PostgreSQL Flexible Server are stuck in an AI triage loop
As an Azure MVP, I want to flag a real gap I just hit firsthand. I opened a support ticket (Issue type: Service and subscription limits (quotas), Quota type: Region access) because a subscription of ours couldn't provision Azure Database for PostgreSQL Flexible Server in East US — az postgres flexible-server list-skus --location eastus returns "Provisioning is restricted in this region... open a support request with Issue type of 'Service and subscription limits'", exactly as documented. Over 9 days, I got three AI-generated responses (case #2608260040010469), all repeating the exact same generic instructions I had already followed and explicitly confirmed I'd followed — twice. No human engineer was ever assigned (confirmed via the Support API: supportEngineer: {} the entire time). The case was even auto-flagged for closure citing "no response," despite an active reply from me on record before that message was sent. This isn't a one-off complaint about response speed — it's a structural problem: region-access requests for this service appear to route through an automation loop that can't recognize "I already did what you're asking" and doesn't have a reliable escalation path to a human, even after an explicit "I'm still having an issue" signal. Has anyone else hit this on region-access requests specifically? And is there a known way to get these routed to an actual engineer instead of restarting the same script every 24-48h?
OData Connector for Dynamics Business Central
Hey Guys, I'm trying to connect Dynamics Business Central OData API in ADF but I'm not sure what I'm doing wrong here because the same Endpoint is returning data on Postman but returning an error in ADF LinkedService. https://api.businesscentral.dynamics.com/v2.0/{tenant-id}/Sandbox-UAT/ODataV4/Company('company-name')/Chart_of_Accounts
Architectural Feedback: Is SPFx + Azure Database a viable choice for an internal ticketing system?
Hi everyone, I am currently evaluating the architecture for a new interdepartmental ticketing system and would love to get your feedback, real-world experiences, or best practices regarding our proposed approach. Our goal is to centralize and streamline requests, tracking, and communication across various departments within the organization. We want to provide an intuitive interface that integrates seamlessly into our existing workplace (SharePoint Online / Microsoft Teams). Proposed Architecture Frontend / UI: Custom web parts developed using SPFx (SharePoint Framework), embedded into SharePoint pages and surfaced in Microsoft Teams. Backend / Database: Azure Database for PostgreSQL for managing relational data, audit logs, complex ticket statuses, and SLA metrics. Integration Layer: Custom REST APIs (hosted via Azure Functions / App Service) to handle transactions securely between SPFx and PostgreSQL Rationale Behind This Choice User Experience: SPFx provides a native look and feel within Microsoft 365, minimizing user onboarding friction. Relational Data Complexity: We prefer PostgreSQL over standard SharePoint Lists due to data volume projections, complex relational data models, granular access control requirements per department, and advanced analytics needs. Questions Architecture Viability: Do you consider using SPFx purely as a frontend consumer for external REST APIs connected to PostgreSQL a clean and sustainable pattern long-term? SharePoint Lists / Dataverse vs. External DB: Is the added overhead of maintaining a custom API + PostgreSQL in Azure justified, or would you strongly favor Dataverse or native SharePoint Lists for this specific use case? Authentication & Security: Any specific tips when securing SPFx calls to Azure API/Functions using Microsoft Entra ID (Azure AD) to ensure proper department-level data segregation? Maintenance & Governance: Are there any major maintenance, lifecycle, or performance bottlenecks we should prepare for when maintaining a hybrid SPFx + external DB solution? Thanks in advance for any insights, lessons learned, or potential red flags!
Kusto: summarize consecutive segments of the same value per batch
Problem statement (Azure Data Explorer / Kusto) I have a table in Azure Data Explorer (Kusto) with the following columns: batch_number (string) team (string) datetime (datetime) Each row represents work done by a team on a batch at a given time. The data is chronological per batch, but teams can change over time and may come back later. Example data (ordered by batch_number, then datetime): batch_number | datetime | team 001 | 2024-01-01 08:00:00 | A 001 | 2024-01-01 08:01:00 | A 001 | 2024-01-01 08:02:00 | A 001 | 2024-01-01 08:03:00 | B 001 | 2024-01-01 08:04:00 | B 001 | 2024-01-01 08:05:00 | A 001 | 2024-01-01 08:06:00 | A 002 | 2024-01-01 08:00:00 | A 002 | 2024-01-01 08:01:00 | A 002 | 2024-01-01 08:02:00 | B 002 | 2024-01-01 08:03:00 | C Goal For each batch, I want to group consecutive rows where the team stays the same and compute: start_time = min(datetime) end_time = max(datetime) The result must preserve team sequences, not just distinct teams. Expected result: batch_number | team | start_time | end_time 001 | A | 2024-01-01 08:00:00 | 2024-01-01 08:02:00 001 | B | 2024-01-01 08:03:00 | 2024-01-01 08:04:00 001 | A | 2024-01-01 08:05:00 | 2024-01-01 08:06:00 002 | A | 2024-01-01 08:00:00 | 2024-01-01 08:01:00 002 | B | 2024-01-01 08:02:00 | 2024-01-01 08:02:00 002 | C | 2024-01-01 08:03:00 | 2024-01-01 08:03:00 Notes Team A appears twice for batch 001 because it appears in two separate consecutive segments. The grouping must be done per batch, not across batches. Important constraints A simple summarize by batch_number and team is not correct because it merges non-consecutive segments. I can easily implement this logic in pandas using a cumulative sequence identifier, but I have not found a reliable equivalent in Kusto. Question What is the correct and reliable Kusto query to compute these consecutive team segments per batch?
Backup and restore the Azure Database for MySQL flexible server
Hi everyone, I’m running into an issue while performing a geo-restore for an Azure MySQL Flexible Server using Azure CLI and Portal. When executing the az mysql flexible-server geo-restore command, the restore process fails with the error “InvalidAdminLoginPassword: The AdministratorLoginPassword cannot be empty or null”. I don’t see any administrator password configuration option in the Azure Portal, and there is also no auto-populated argument in the CLI. According to the official documentation, geo-restore uses the source server as the restore point, so the administrator login configuration is expected to be derived from the source server. Are there any recommended best practices for handling admin credentials securely during geo-restore? Thanks in advance for your insights. I used this official doc as a reference, https://learn.microsoft.com/en-us/azure/mysql/flexible-server/how-to-restore-server-portal
Microsoft Integration Runtime Download 5.68.9705.1
There is a new Runtime version, 5.68.9705, released today, however it looks like the release notes are protected and can not be opened once downloaded: https://www.microsoft.com/en-gb/download/details.aspx?id=39717&msockid=3edfc848eac467411552dd76ebf56680
ADF - REST API Copy Data Activity - Best Practices
Hey everyone, I'm relatively new to Azure and am using Azure Data Factory (ADF) to extract data from Vonigo via their API. Everything is working, but I'm looking for ways to improve efficiency. Right now it takes 30+ minutes to pull all franchise data for a given report. The challenge is that Vonigo only allows reports to be run for one franchise at a time, and switching franchises requires a separate API call. My current process is: Select a franchise from a list Run all required reports for that franchise Save the results to Blob Storage Move to the next franchise and repeat (We'll eventually ingest the files into our silver layer for transformation.) Speed Issue One thing that significantly slows the pipeline down is that many activities are running sequentially. If I disable sequential execution, I've seen cases where data gets written to the wrong destination or associated with the wrong franchise. Has anyone successfully parallelized a similar process while maintaining data integrity? Are there specific points in the workflow where parallel execution would be safe? Pagination / Loop Issue Originally, I used a Lookup activity to inspect the most recently created file. An If activity would then determine whether the file contained any records: If records existed, increment the page number and continue. If no records existed, end the loop. This worked, but the Lookup activity added noticeable overhead. To improve performance, I changed the logic to use the Copy Activity output instead. Specifically, I'm checking the amount of data read from the last API call. Pages with no records appear to consistently return the same data-read value, so I use that to determine when to stop paging. This approach is much faster, but it feels more fragile since it's relying on an indirect indicator rather than the actual record count. Would you trust the Copy Activity output in this scenario, stick with the Lookup approach, or recommend a different pattern altogether? Thanks for any suggestions.
Migrating ~100 SSIS Packages to Azure: Acceleration Strategies & ADF vs. Fabric Guidance
Hi everyone, Our team is currently planning to migrate approximately 100 SSIS packages (including several large and highly complex workflows) to Azure. We are evaluating two primary paths: a lift-and-shift using Azure Data Factory (ADF) SSIS Integration Runtime (SSIS-IR) vs. a full rewrite. Current Architecture / Tech Stack: Azure, VMs, Azure SQL Managed Instance (SQL MI) We would appreciate insights on the following: Accelerated Rewrite Options: If we decide to rewrite, are there automation tools, frameworks, or best practices to expedite the transition rather than manually rebuilding every activity in ADF? ADF vs. Microsoft Fabric: Given Microsoft’s strategic push toward Fabric, is ADF still the ideal migration target, or should we consider targeting Fabric Data Pipelines directly? Thank you in advance for any inputs, advice, or shared experiences!
Azure SQL Firewall / Locks
Hi there, I have 2 environments. I'm more of admin on Azure environment (recently made as subscription admin) after which Dev issue - Azure SQL I'm having difficulty to remove IP from Azure SQL Firewall. (Earlier i was able to) today my manager granted me subscription admin and as SQL Security Manager and it still not able to remove grayed out IPs.Problem with Linked Service to SQL Managed Instance
Hi I'm trying to create a linked Service to a SQL Managed Instance. The Managed Instance is configured with a Vnet_local endpoint If I try to connect with an autoresolve IR or a SHIR I get the following error The value of the property '' is invalid: 'The remote name could not be resolved: 'SQL01.public.ec9fbc2870dd.database.windows.net''. The remote name could not be resolved: 'SQL01.public.ec9fbc2870dd.database.windows.net' Is there a way to connect to it without resorting to a private endpoint? Cheers Alex
Azure Data Factory ForEach Loop Fails Despite Inner Activity Error Handling - Seeking Best Practices
Hello Azure Data Factory Community, I'm encountering a persistent issue with my ADF pipeline where a ForEach loop is failing, even though I've implemented error handling for the inner activities. I'm looking for insights and best practices on how to prevent internal activity failures from propagating up and causing the entire ForEach loop (and subsequently the pipeline) to fail, while still logging all outcomes. My Setup: My pipeline processes records using a ForEach loop. Inside the loop, I have a Web activity (Sample_put_record) that calls an external API. This API call can either succeed or fail for individual records. My current error handling within the ForEach iteration is structured as follows: 1.Sample_put_record (Web Activity): Makes the API call. 2.Conditional Logic: I've tried two main approaches: •Approach A (Direct Success/Failure Paths): The Sample_put_record activity has a green arrow (on success) leading to a Log Success Items (Script activity) and a red arrow (on failure) leading to a Log Failed Items (Script activity). Both logging activities are followed by Wait activities (Dummy Wait For Success/Failure). •Approach B (If Condition Wrapper): I've wrapped the Sample_put_record activity and its success/failure logging within an If Condition activity. The If Condition's expression is @equals(activity('Sample_put_record').status, 'Succeeded'). The True branch contains the success logging, and the False branch contains the failure logging. The intention here was for the If Condition to always report success, regardless of the Sample_put_record outcome, to prevent the ForEach from failing. The Problem: Despite these error handling attempts, the ForEach loop (and thus the overall pipeline) still fails when an Sample_put_record activity fails. The error message I typically see for the ForEach activity is "Activity failed because an inner activity failed." When using the If Condition wrapper, the If Condition itself sometimes fails with the same error, indicating that an activity within its True or False branch is still causing a hard failure. For example, a common failure for Sample_put_record is: "valid":false,"message":"WARNING: There was no xxxxxxxxxxxxxxxxxxxxxxxxx scheduled..." (a user configuration/data issue). Even when my Log Failed Items script attempts to capture this, the ForEach still breaks. What I've Ensured/Considered: •Wait Activity Configuration: Wait activities are configured with reasonable durations and do not appear to be the direct cause of failure. •No Unhandled Exceptions: I'm trying to ensure no unhandled exceptions are propagating from my error handling activities. •Pipeline Status Goal: My ultimate goal is for the overall pipeline status to be Succeeded as long as the pipeline completes its execution, even if some Sample_put_record calls fail and are logged. I need to rely on the logs to identify actual failures, not the pipeline status. My Questions to the Community: 1.What is the definitive best practice in Azure Data Factory to ensure a ForEach loop never fails due to an inner activity failure, assuming the inner activity's failure is properly logged and handled within that iteration? 2.Are there specific nuances or common pitfalls with If Condition activities or Script activities within ForEach loops that could still cause failure propagation, even with try-catch and success exits? 3.How do you typically structure your ADF pipelines to achieve this level of resilience where internal failures are logged but don't impact the overall pipeline success status? 4.Are there any specific configurations on the ForEach activity itself (e.g., Continue on error setting, if it exists for ForEach?) or other activities that I might be overlooking? Any detailed examples, architectural patterns, or debugging tips would be greatly appreciated. Thank you in advance for your help!
Copy Data Activity Failed with Unreasonable Cause
It is a simple set up but it has baffled me a lot. I'd like to copy data to a data lake via API. Here are the steps I've taken: Created a HTTP linked service as below: Created a dataset with a HTTP Binary data format as below: Created a pipeline with a Copy Data activity only as shown below: Made sure linked service and dataset all working fine as below: Created a Sink dataset with 3 parameters as shown below: Passed parameters from pipeline to Sink dataset as below: That's all. Simple, right? But the pipeline failed with a clear message "usually this is caused by invalid credentials." as below: Summary: No need to worry about the Sink side of parameters etc. which I have used same thing for years on other pipelines and all succeeded. This time the API failed to reach a data lake from source side as said "invalid credentials". In Step 4 above one could see the linked service and dataset connections were succeeded, ie. credentials have been checked and passed already. How come it failed in data copy activity complaining an invalid credentials? Pretty weird. Any advice and suggestions will be welcomed.Data flow sink to Blob storage not writing to subfolder
Hi Everybody This seems like it should be straightforward, but it just doesn't seem to work... I have a file containing JSON data, one document per line, with many different types of data. Each type is identified by a field named "OBJ", which tells me what kind of data it contains. I want to split this file into separate files in Blob storage for each object type prior to doing some downstream processing. So, I have a very simple data flow - a source which loads the whole file, and a sink which writes the data back to separate files. In the sink settings, I've set the "File name option" setting to "Name file as column data" and selected my OBJ column for the "Column Data", and this basically works - it writes out a separate file for each OBJ value, containing the right data. So far, so good. However, what doesn't seem to work is the very simplest thing - I want to write the output files to a folder in my Blob storage container, but the sink seems to completely ignore the "Folder path" setting and just writes them into the root of the container. I can write my output files to a different container, but not to a subfolder inside the same container. It even creates the folder if it's not there already, but doesn't use it. Am I missing something obvious, or does the "Folder path" setting just not work when naming files from column data? Is there a way around this?
User Properties of Activities in ADF: How to add dynamic content in it?
On ADF, I am using a for each loop in which I am using an Execute Pipeline Activity which is getting executed for different iterations as per the values of the items provided to the For-Each Loop. I am stuck on a scenario which requires me to add the Dynamic Content Expression in the User Properties of individual activities of ADF. Specific to my case, I want to add the Dynamic Content Expression in the User Properties of Execute Pipeline Activity so that I get to individual runs of these activities on Azure Monitor with a specific label attached to it through its User Properties. The necessity to add the Dynamic Content Expression in the User Properties is due to the reason that each execution in respective iterations of these activities corresponds to a particular Step from a set of Steps configured for the Data Load Job as a whole, which has been orchestrated through ADF. To identify the association with the respective Job-Step, I require to add Dynamic Content Expression in its User Properties. Any sort of response regarding this is highly appreciated. Thank You!Azure Data Factory - help needed to ingest data, that uses an API, into a SQL Server instance
Hi, I need to ingest 3rd-party data, that uses an API, into my Azure SQL Server instance (ASSi) using Azure Data Factory (ADF). I'm a report developer so this area is unfamiliar to me, although I have previously integrated external, on-premise SQL Server data into our ASSi using ADF so I do have some exposure to the tool (just not API connections). The 3rd-party data belongs to a company named 'iLevel' (in case this is relevant). iLevel have provided some API documentation which is targeted for an experienced data engineer that understands API connections. This documentation has just a few sections before the connection focused details end. I'll list them below: 1) It mentions to download 'Postman Collection' and mentions no more than this. I've never heard of Postman Collection and I don't know why it's needed. Through limited exposure online, I don't understand its purpose, certainly not in my scenario. 2) It has the title 'Access to the API' and then lists four URLs which are the endpoints (I don't know which to use and will need to ask iLevel of this but, I guess, any will do for testing purposes). 3) Authentication and Authorization a) Generate a 'client id' and 'client secret' by logging into iLevel and generating these values with some clicks of a button. I've successfully generated both these values. b) Obtain an Access Token - it'll be easier to screenshot the instructions for this (I've blanked part of the URL for confidentiality). These are all the instructions on connection to the 3rd-party data. Unfortuntely for me, my lack of experience in this area means these instrcutions don't help me. I don't believe I'm any closer to connecting to the 3rd-party source data. Taking into consideration the above instructions but choosing to try and error in ADF, a tool I'm a little bit more familiar with, I've performed the following steps: 1) Created a Linked Service. I understand the iLevel solution is in the cloud and therefore the 'AutoResolveIntegrationRuntime' option has been selected as the 'Connect via integration runtime' value. For the 'Base URL' I've entered one of the four URL endpoints that were listed in the documentation (again, I will need to confirm which endpoint to use). The 'Test Connection' returns a successful result but I think it means nothing because if I were to placed 'xxx' at the end of the Base URL and test the connection, it stills returns successful when I know the URL with the 'xxx' post-fix isn't legit. 2) Create an ADF Pipeline containing 'Web activity' and 'Set variable' objects. The only configuration under the Web activity is the 'Settings' pane which has: The 'Body' property has (the client id and client secret taken from the iLevel solution are included in the body but blanked out): If the Web activity is successful then the Pipeline's next object (the Set variable) should assign the access token to a variable - as I understand this is what the Web activity is doing: The 'Value' property has: This is as far as I've got into my efforts of this integration task because the Web activity object fails when executed. The error message does state it is to do with an invalid 'client id' or 'client secret' - see below: You may direct to focus on the incorrect client id or client secret, however I don't have any confidence that I understand how to configure ADF to obtain an access token, and I'm maybe missing something if I see no need for Postman Collection use. What is Postman Collection and do I need it for what I'm trying to achieve? If yes, can anyone provide training material that suits my need? Have I configured ADF correctly and it is indeed an issue with the client id or client server, or is the error message received just a by-product of an incorrect ADF configuration? Your help will be most appreciated. Many thanks.ADF - Support for Federated Identity Credentials
Our team is working on migrating existing pipelines to Azure Data Factory (ADF). As part of this effort, we are evaluating ADF for executing Kusto commands and would like to use an existing Microsoft Entra application registration authenticated through Federated Identity Credentials (OIDC workload identity federation). Today, ADF appears to primarily support Managed Identity-based authentication patterns. Is there any planned support for using a customer-owned App Registration with Federated Identity Credentials (FIC)? If so, is there any public roadmap or timeline that can be shared? We are actively moving away from secret- and certificate-based authentication. Supporting FIC would allow us to continue using our existing application identities, which already have the required permissions on source and destination datasets, and would avoid the need to introduce additional User Assigned Managed Identities and request new permission grants from dataset owners as part of the migration.Unable to INSERT rec into Table.
I am using the ADF . Execute Pipeline activity. My PL is parametrized. My stage table have the below rec: network_org_name partner_name parent_partner_name CorPart_number pipeline_run_id activity_run_id Se Network Producing UB SA S Reine Company Ltd 1226133 ebf961f9-3afc-4abc-9ee0-111330c9f1eb 858c1b5c-a699-4e73-84d8-3a42cab115d2 WITH CodeSourceTableCTE AS ( SELECT DISTINCT p.[partner_number] ,n.[network_org_number] FROM [utl].[stg_partner_network_org] stg LEFT JOIN [utl].[partner] p1 ON stg.[parent_partner_name] = p1.[partner_name] INNER JOIN [utl].[partner] p ON stg.[partner_name] = p.[partner_name] AND ISNULL(p.[delete_ind], '') <> 'Y' AND ISNULL(p.[parent_partner_number], '') = ISNULL(p1.[partner_number], '') AND ISNULL(p1.[delete_ind], '') <> 'Y' AND ISNULL(stg.[corpart_number], '') = ISNULL(p.[external_id], '') INNER JOIN [utl].[network_org] n ON stg.[network_org_name] = n.[network_org_name] WHERE stg.[partner_name] IS NOT NULL ) MERGE [utl].[partner_network_org] T USING CodeSourceTableCTE S ON T.[partner_number] = S.[partner_number] AND T.[network_org_number] = S.[network_org_number] WHEN MATCHED AND (ISNULL(T.[delete_ind], '') <> 'N') THEN UPDATE SET T.[delete_ind] = 'N', T.[last_modified_date] = GETUTCDATE(), T.[last_modified_by] = '@{pipeline().parameters.Source_File_Name}' WHEN NOT MATCHED BY TARGET THEN INSERT ([partner_number], [network_org_number], [created_date], [created_by], [last_modified_date], [last_modified_by], [delete_ind]) VALUES (S.[partner_number], S.[network_org_number], GETUTCDATE(), '@{pipeline().parameters.Source_File_Name}', GETUTCDATE(), '@{pipeline().parameters.Source_File_Name}', 'N'); SELECT COUNT(*) FROM [utl].[partner_network_org] WHERE [created_by] = '@{pipeline().parameters.Source_File_Name}' But when I am trying to check using below select * from utl.partner_network_org where partner_number in (select partner_number from utl.partner where last_modified_by like '%corpart%') I don't see records any help is appreciated
ADF unable to ingest partitioned Delta data from Azure Synapse Link (Dataverse/FnO)
We are ingesting Dynamics 365 Finance & Operations (FnO) data into ADLS Gen2 using Azure Synapse Link for Dataverse, and then attempting to load that data into Azure SQL Database using Azure Data Factory (ADF). This is part of a migration effort as Export to Data Lake is being deprecated. Source Details Source: ADLS Gen2 Data generated by: Azure Synapse Link for Dataverse (FnO) Format on lake: Delta / Parquet Partitioned folder structure (e.g. PartitionId=xxxx) Destination: Azure SQL Database Issue Observed in ADF When configuring ADF pipelines: Using ADLS Gen2 dataset with: Delta / Parquet Recursive folder traversal Wildcard paths We encounter: No data returned in Data Preview Or runtime error such as: “No partitions information found in metadata file” Despite this: The data is present in ADLS The same data can be successfully queried using Synapse serverless SQL Key Question for ADF / Synapse Engineers What is the recommended and supported ADF ingestion pattern for: Partitioned Delta/Parquet data produced by Azure Synapse Link for Dataverse Specifically: Should ADF: Read Delta tables directly, or Use Synapse serverless SQL external tables/views as an intermediate layer? Is there a reference architecture for: Synapse Link → ADLS → ADF → Azure SQL Are there ADF limitations when consuming Synapse Link–generated Delta tables? Many customers are now forced to migrate due to Export to Data Lake deprecation, but current ADF documentation does not clearly explain how to replace existing ingestion pipelines when using Synapse Link for FnO. Any guidance, patterns, or official documentation would be greatly appreciated.
Recent Blogs
- 4 MIN READA key benefit of Azure HorizonDB comes from its smart storage layer, which works seamlessly with Postgres and takes over durability and high availability work typically done by Postgres itself. Thi...Sep 04, 2026
- Problem Statement Customer trying to mograte SQL Server database to Azure SQL Database by restoring a .bak file to a jump server, exporting it to a .bacpac, and importing that .bacpac to Azure SQL ...Sep 04, 2026
