Forum Widgets
Latest Discussions
Using a connection string to connect to Blob storage
Hi This might be a simple one but I've searched around a lot for an answer on this and have come up with nothing so i'm hoping someone here can help. In Synapse I just want to create a linked service to Blob storage using it's connection string stored in Key Vault - it allows me to create it and 'test connection' succeeds but when I move to the Data tab in Synapse and try to view it there it show an error It seems to think the account name is blank, note there is nothing in the parenthesis: I also get this error when i hover over the red X: This does work when i enter the storage details manually and use key vault to get the access key, rather than the whole connection string so it's definitely not an issue with permissions on the key vault. I wondered the connection string needs to be in any particular format to work - I just copied it directly from the storage account in the portal and it looks correct. I've tried this with different storage accounts on different tenants and its the same. What am I missing? Thanks :-)Synapse Webhook Action with Private Logic App
Hi all, I have a Synapse workspace with public access disabled and using all private endpoints, both for inbound and outbound access from the managed vnet. I also have a Logic App with private endpoints. Both Synapse and Logic App are in separate virtual networks but peered together at a central hub site. Each have access to private DNS zones with records to resolve to each resource. When I disabled public network access on the Logic App, I can no longer use a Webhook activity from a Synapse pipeline with callback URI. A Web action works just fine, but with the Webhook activity, I get a response from the Logic App of 403 Forbidden. Ordinarily this looks like a permission issue, but when public network is enabled, the Logic App workflow works fine. When the Webhook action fails to runs, there is no activity run logged on the Logic App. There's something that the Webhook action is not getting back from the Logic App when public network access is disabled. I've been trying to find a solution (including sending back a 202 response to Synapse from the Logic App), but it continues to baffle me. Has any one else successfully configured Synapse Webhook action to call a workflow in a Standard Logic App over private endpoints? Any ideas or suggestions to troubleshoot this?
Copy Activity - JSON Mapping
Hello, I have created a copy activity in Azure synapse Analytics. I have a JSON file as an input and would like to unpack and save it as a csv file. I have tried several times but can not get the data in the correct output. The below is my input file: { "status": "success", "requestTime": "2025-06-26 15:23:41", "data": [ "Monday", "Tuesday", "Wednesday" ] } I would like to save it in the following output. status requestTime Data success 26/06/2025 15:23 Monday success 26/06/2025 15:23 Tuesday success 26/06/2025 15:23 Wednesday I am struggling to configure the mapping section correctly. I can not understand how to unpack the data array. The $['data'][0] gives me the first element. I would like to extract all elements in the format above. Any help would be appreciated.
Partitioning in Azure Synapse
Hello, Im currently working on an optimization project, which as led me down a rabbithole of technical differences between the regular MSSQL and the dedicated SQL pool that is Azure PDW. I noticed, that when checking the distributions of partitions, when creating a table, for lets say splitting data by YEAR([datefield]) with ranges for each year '20230101','20240101' etc, the sys partitions view claims that all partitions have equal amount of rows. Also from the query plans, i can not see any impact in the way the query is executed, even though partition elimination should be the first move, when querying with Where [datefield] = '20230505'. Any info and advice would be greatly appreciated.
Access dedicated SQL pool from notebook
I have some notebooks where I use the com.microsoft.spark.sqlanalytics library to fetch the data from the dedicated SQL pool. Everything was working fine until a couple of days when we started getting the errors which are not very helpful. The error is like this: Py4JJavaError: An error occurred while calling o4062.count. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 0.0 failed 4 times, most recent failure: Lost task 2.3 in stage 0.0 (TID 21) (vm-00321611 executor 2): java.lang.IllegalArgumentException: For input string: 'null' The code was working without issues up until a couple of days and there were no new deployments prior to that. The error occurs when the data is being accessed. Here is an excerpt: dfData = spark.read.option(Constants.DATABASE, "sql_db_pool1").synapsesql(query) cnt = dfData.count() The error is coming deep from the library and there is no way to determine what argument is null. Anybody ran into an issue like this? Regards
Synapse workspace cost reduction
I have a Cosmos DB where I have one container that contains different documents. One document is a main document that has another related document. Both documents are related to each other by using a partition key. There will be one main document and multiple event documents with the same partition key. The main document has fields like date, country, and categories, which the event document does not have, while the event document has fields like event type, event dateandtime etc. To filter how many events happened for a particular category on a particular day, we have to use the main document. The events can be repetitive on a single day. My requirement is to create a Power BI report to display how many events happened on a particular day and for which country in the last 2 months (each event should display only one time per category, per country in a day). I want to get this data from Synapse and load it into Power BI for the last 2 months. I used the Synapse view and implemented incremental dataset refresh in a power BI. In a Synapse view, I created a main view that loads data for a main document, and in another view, I get those partition keys from the main view and then load the data for an event document. There are 2 dates in a main document: created date and change date. I cannot use the change date in incremental dataset refresh as it creates duplicate records, so I used the created date and then used the data to detect changes for the last 30 days (this is the time period where the main document can change). It works well, but the problem here is that it takes a lot of time to execute the query, which is causing more cost for data processing in Synapse. Is there any suggestion to reduce the cost consumption of Synapse as well as query execution time/dataset refresh time in Power BI?
IP whitelist for Synapse Spark Pools to reach external endpoint with firewall?
I am trying to reach an external vendor SFTP site from my Synapse Spark notebook. The site is behind a firewall. I want to get the IP range for all of our Spark pools to the vendor so they can whitelist them. Struggling to get a clear idea of that list. Closest I found so far was "Azure Cloud East US", which is rather broad. Any advice/ideas how to get a refined list/range of IPs?
Rest api call with multiple pages
Hello, I need to migrate an SSIS package that make API calls and return multiple pages. The call goes to https://{{HOSTNAME}}/personnel/v1/person-details, and I can manually do https://{{HOSTNAME}}/personnel/v1/person-details?page=1&per_page=200, my issue is that I do not get any metadata to create the rules on the foreach loop on the first call, and I am not sure how to go about it I get something like this, no metadata [ { "additionalName1": null, "additionalName2": null, "nationality1": null, "nationality2": null, "nationality3": null, }, { "additionalName1": null, "additionalName2": null, "nationality1": null, "nationality2": null, "nationality3": null, }, { "additionalName1": null, "additionalName2": null, "nationality1": null, "nationality2": null, "nationality3": null, } ]Accessing serverless sql pool tables from dedicated sql pool
I'm trying to access the tables available in the synapse serverless sql pool from the dedicated sql pool. I'd like to create some simple stored procedures to import data from delta/parquet tables that are mapped as external tables in the serverless sql pool and load them into some dedicated sql pool tables. Is there a simple way to do this without having to define external tables in the dedicated sql pool too? I tried this and there seem to be many limitations (delta not supported, etc.).
