Forum Discussion
SQL Server issue but don't know what - please help!
Hi,
I'm facing a SQL Server focused issue that I don't understand why it's occurring and would like your help to identify and resolve, please. I will provide an in-depth breakdown of the scenario.
Two years ago, I created a Azure Data Factory (ADF) Pipeline to take data from Azure Synapse to Azure SQL Server, and two other Pipelines to take data from an On-Premises Sage server to the same Azure SQL Server. These Pipelines were working perfectly up until two days ago (11/03/26) when the Pipelines still always complete successfully but the duration have increased greatly. The below screenshot shows the log of these three Pipeline for the past week. Note how prior to the 12/03/26 the Synapse Pipeline took about 6 minutes to complete and the two Sage Pipelines took around 25 seconds. Also note how on the 12/03/26 the Synapse Pipeline took over 3 hours but the two Sage Pipelines continued with their normal 25 seconds. Notice today (13/03/26) the Synapse Pipeline was still slow but now the two Sage Pipelines are taking over an hour. I'll note here that the Pipelines still complete successfully (so the config must be correct as it has been for two years).
Each Pipeline contains a single (for Sage) or several (for Synapse) 'Copy data' objects. The objects have their source configuration to simply extract from the source (so either Synapse or Sage) and then the sink configuration has a 'Pre-Copy script' which simply Truncates the target SQL Server table, before loading to the SQL Server table. The screenshot below is one example - each Pre-Copy script is populated the same but just with the tables being different.
When I look at the log for the Pipelines, I see a common theme for each and that is the 'Pre-Copy script' (the Truncate) is consuming 99% of the time. The screenshot below shows this common theme.
So at this point, I ask the question why, after two years of all working well and completing very quickly, are the Pipelines taking so long to complete?
This also seems to be an intermittent problem as I have performed several manual executions which will take a long while, then revert back to the quick several minutes again (so good), then revert once more to taking a long while again. It is intermittent. See the screenshot below. Notice how the same Pipeline have different durations. It's intermittent.
The reason why I think it is a SQL Server focused issue for the following reasons:
1) Synapse - Performing simple commands (Select Top) returns data within seconds.
2) SQL Server - Performing simple commands (Select Top) usually takes seconds but during the past two days is often taking half hour.
3) SQL Server - When amending a SQL View it would take a split second to complete. During the past two days, it goes over ten minutes and still doesn't complete (I was amending the View for testing purposes).
4) SQL Server - When looking at the Views node under a database, during the past two days, it intermittently doesn't show the View. Sometimes, it will work if I log out and then back into SQL Server.
5) Power BI - Refreshing a Power BI report, whether from the Power BI Service or Desktop, call SQL Server Views. These SQL Server Views read from several SQL Server tables. These report refreshes are failing due to an IDbCommand interface error. These Power BI report data refreshes simply read from a SQL Server table via a SQL View. They don't reference or consider at all the any ADF Pipelines. The Pipelines execute between 2am and 3am each morning. The Power BI report refresh their data around 12pm to 5pm.
As SQL Server seems to be problematic at the point of the 'Pre-Copy script', which is one of the end to end process, and then problematic at the very other end where Power BI reports consumes SQL Views - leads me to believe the issue is with SQL Server.
I am a report developer and not an ADF expert. I've built the ADF process on the side. I haven't changed any development/configuration/etc... between this all working and failing. Our I.T. department have advised they have made no changes to contribute to this issue.
Please can someone advise on what's happening here and why this issue has arose when for two years all was fine?
Thanks.