Recent Discussions
Kusto: summarize consecutive segments of the same value per batch
Problem statement (Azure Data Explorer / Kusto) I have a table in Azure Data Explorer (Kusto) with the following columns: batch_number (string) team (string) datetime (datetime) Each row represents work done by a team on a batch at a given time. The data is chronological per batch, but teams can change over time and may come back later. Example data (ordered by batch_number, then datetime): batch_number | datetime | team 001 | 2024-01-01 08:00:00 | A 001 | 2024-01-01 08:01:00 | A 001 | 2024-01-01 08:02:00 | A 001 | 2024-01-01 08:03:00 | B 001 | 2024-01-01 08:04:00 | B 001 | 2024-01-01 08:05:00 | A 001 | 2024-01-01 08:06:00 | A 002 | 2024-01-01 08:00:00 | A 002 | 2024-01-01 08:01:00 | A 002 | 2024-01-01 08:02:00 | B 002 | 2024-01-01 08:03:00 | C Goal For each batch, I want to group consecutive rows where the team stays the same and compute: start_time = min(datetime) end_time = max(datetime) The result must preserve team sequences, not just distinct teams. Expected result: batch_number | team | start_time | end_time 001 | A | 2024-01-01 08:00:00 | 2024-01-01 08:02:00 001 | B | 2024-01-01 08:03:00 | 2024-01-01 08:04:00 001 | A | 2024-01-01 08:05:00 | 2024-01-01 08:06:00 002 | A | 2024-01-01 08:00:00 | 2024-01-01 08:01:00 002 | B | 2024-01-01 08:02:00 | 2024-01-01 08:02:00 002 | C | 2024-01-01 08:03:00 | 2024-01-01 08:03:00 Notes Team A appears twice for batch 001 because it appears in two separate consecutive segments. The grouping must be done per batch, not across batches. Important constraints A simple summarize by batch_number and team is not correct because it merges non-consecutive segments. I can easily implement this logic in pandas using a cumulative sequence identifier, but I have not found a reliable equivalent in Kusto. Question What is the correct and reliable Kusto query to compute these consecutive team segments per batch?
Azure Data Factory - help needed to ingest data, that uses an API, into a SQL Server instance
Hi, I need to ingest 3rd-party data, that uses an API, into my Azure SQL Server instance (ASSi) using Azure Data Factory (ADF). I'm a report developer so this area is unfamiliar to me, although I have previously integrated external, on-premise SQL Server data into our ASSi using ADF so I do have some exposure to the tool (just not API connections). The 3rd-party data belongs to a company named 'iLevel' (in case this is relevant). iLevel have provided some API documentation which is targeted for an experienced data engineer that understands API connections. This documentation has just a few sections before the connection focused details end. I'll list them below: 1) It mentions to download 'Postman Collection' and mentions no more than this. I've never heard of Postman Collection and I don't know why it's needed. Through limited exposure online, I don't understand its purpose, certainly not in my scenario. 2) It has the title 'Access to the API' and then lists four URLs which are the endpoints (I don't know which to use and will need to ask iLevel of this but, I guess, any will do for testing purposes). 3) Authentication and Authorization a) Generate a 'client id' and 'client secret' by logging into iLevel and generating these values with some clicks of a button. I've successfully generated both these values. b) Obtain an Access Token - it'll be easier to screenshot the instructions for this (I've blanked part of the URL for confidentiality). These are all the instructions on connection to the 3rd-party data. Unfortuntely for me, my lack of experience in this area means these instrcutions don't help me. I don't believe I'm any closer to connecting to the 3rd-party source data. Taking into consideration the above instructions but choosing to try and error in ADF, a tool I'm a little bit more familiar with, I've performed the following steps: 1) Created a Linked Service. I understand the iLevel solution is in the cloud and therefore the 'AutoResolveIntegrationRuntime' option has been selected as the 'Connect via integration runtime' value. For the 'Base URL' I've entered one of the four URL endpoints that were listed in the documentation (again, I will need to confirm which endpoint to use). The 'Test Connection' returns a successful result but I think it means nothing because if I were to placed 'xxx' at the end of the Base URL and test the connection, it stills returns successful when I know the URL with the 'xxx' post-fix isn't legit. 2) Create an ADF Pipeline containing 'Web activity' and 'Set variable' objects. The only configuration under the Web activity is the 'Settings' pane which has: The 'Body' property has (the client id and client secret taken from the iLevel solution are included in the body but blanked out): If the Web activity is successful then the Pipeline's next object (the Set variable) should assign the access token to a variable - as I understand this is what the Web activity is doing: The 'Value' property has: This is as far as I've got into my efforts of this integration task because the Web activity object fails when executed. The error message does state it is to do with an invalid 'client id' or 'client secret' - see below: You may direct to focus on the incorrect client id or client secret, however I don't have any confidence that I understand how to configure ADF to obtain an access token, and I'm maybe missing something if I see no need for Postman Collection use. What is Postman Collection and do I need it for what I'm trying to achieve? If yes, can anyone provide training material that suits my need? Have I configured ADF correctly and it is indeed an issue with the client id or client server, or is the error message received just a by-product of an incorrect ADF configuration? Your help will be most appreciated. Many thanks.
Backup and restore the Azure Database for MySQL flexible server
Hi everyone, Iโm running into an issue while performing a geo-restore for an Azure MySQL Flexible Server using Azure CLI and Portal. When executing the az mysql flexible-server geo-restore command, the restore process fails with the error โInvalidAdminLoginPassword: The AdministratorLoginPassword cannot be empty or nullโ. I donโt see any administrator password configuration option in the Azure Portal, and there is also no auto-populated argument in the CLI. According to the official documentation, geo-restore uses the source server as the restore point, so the administrator login configuration is expected to be derived from the source server. Are there any recommended best practices for handling admin credentials securely during geo-restore? Thanks in advance for your insights. I used this official doc as a reference, https://learn.microsoft.com/en-us/azure/mysql/flexible-server/how-to-restore-server-portal
Switch Azure SQL Provisioned to Serverless and back to Provisioned
Hi Can we switch Azure SQL from "Provisioned" to "Serverless" and back to "Provisioned" without any interruptions? Wanted to observe how serverless works for a week. Data should never be lost while switching the mode (NOT a transactional data). Its more of nightly refresh and nothing much happening during day time. Its a retail dataWH. Currently im connecting to the below Azure SQL which is a provisioned instance. We are scaling up or down per our needs / work load and paying $X. sql-xxxx-xxx-xxx-dev.database.windows.net Nightly process runs at higher cores and then scales back to the lowest 2 core. When someone queries tables, joins during day time, it takes 10-30 mins or so depending on the table size. I'm wondering if i switch to serverless i would save money even if billed by seconds since nightly process runs 1hr and not much of querying every day during day time. i would like to test that.
ADF unable to ingest partitioned Delta data from Azure Synapse Link (Dataverse/FnO)
We are ingesting Dynamics 365 Finance & Operations (FnO) data into ADLS Gen2 using Azure Synapse Link for Dataverse, and then attempting to load that data into Azure SQL Database using Azure Data Factory (ADF). This is part of a migration effort as Export to Data Lake is being deprecated. Source Details Source: ADLS Gen2 Data generated by: Azure Synapse Link for Dataverse (FnO) Format on lake: Delta / Parquet Partitioned folder structure (e.g. PartitionId=xxxx) Destination: Azure SQL Database Issue Observed in ADF When configuring ADF pipelines: Using ADLS Gen2 dataset with: Delta / Parquet Recursive folder traversal Wildcard paths We encounter: No data returned in Data Preview Or runtime error such as: โNo partitions information found in metadata fileโ Despite this: The data is present in ADLS The same data can be successfully queried using Synapse serverless SQL Key Question for ADF / Synapse Engineers What is the recommended and supported ADF ingestion pattern for: Partitioned Delta/Parquet data produced by Azure Synapse Link for Dataverse Specifically: Should ADF: Read Delta tables directly, or Use Synapse serverless SQL external tables/views as an intermediate layer? Is there a reference architecture for: Synapse Link โ ADLS โ ADF โ Azure SQL Are there ADF limitations when consuming Synapse Linkโgenerated Delta tables? Many customers are now forced to migrate due to Export to Data Lake deprecation, but current ADF documentation does not clearly explain how to replace existing ingestion pipelines when using Synapse Link for FnO. Any guidance, patterns, or official documentation would be greatly appreciated.PostgreSQL 18 is now GA onย Azure Database for PostgreSQL
PostgreSQL 18 is now GA on Azure Database for PostgreSQL Excited to announce that Flexible Server now offers full general availability of #PostgreSQL18 - the fastest GA weโve ever shipped after community release. This means: ๐ธ๐ฐ๐ณ๐ญ๐ฅ๐ธ๐ช๐ฅ๐ฆ ๐ณ๐ฆ๐จ๐ช๐ฐ๐ฏ ๐ด๐ถ๐ฑ๐ฑ๐ฐ๐ณ๐ต, ๐ช๐ฏ-๐ฑ๐ญ๐ข๐ค๐ฆ ๐ฎ๐ข๐ซ๐ฐ๐ณ-๐ท๐ฆ๐ณ๐ด๐ช๐ฐ๐ฏ ๐ถ๐ฑ๐จ๐ณ๐ข๐ฅ๐ฆ๐ด (๐๐11-๐๐17 โ ๐๐18), ๐๐ช๐ค๐ณ๐ฐ๐ด๐ฐ๐ง๐ต ๐๐ฏ๐ต๐ณ๐ข ๐๐ ๐ข๐ถ๐ต๐ฉ๐ฆ๐ฏ๐ต๐ช๐ค๐ข๐ต๐ช๐ฐ๐ฏ, and ๐๐ถ๐ฆ๐ณ๐บ ๐๐ต๐ฐ๐ณ๐ฆ ๐ธ๐ช๐ต๐ฉ ๐๐ฏ๐ฅ๐ฆ๐น ๐๐ถ๐ฏ๐ช๐ฏ๐จ. Check out the full blog for a deep dive ๐https://techcommunity.microsoft.com/blog/adforpostgresql/postgresql-18-now-ga-on-azure-postgres-flexible-server/4469802 #Microsoft #Azure #Cloud #Database #Postgres #PG18
Copy Data Activity Failed with Unreasonable Cause
It is a simple set up but it has baffled me a lot. I'd like to copy data to a data lake via API. Here are the steps I've taken: Created a HTTP linked service as below: Created a dataset with a HTTP Binary data format as below: Created a pipeline with a Copy Data activity only as shown below: Made sure linked service and dataset all working fine as below: Created a Sink dataset with 3 parameters as shown below: Passed parameters from pipeline to Sink dataset as below: That's all. Simple, right? But the pipeline failed with a clear message "usually this is caused by invalid credentials." as below: Summary: No need to worry about the Sink side of parameters etc. which I have used same thing for years on other pipelines and all succeeded. This time the API failed to reach a data lake from source side as said "invalid credentials". In Step 4 above one could see the linked service and dataset connections were succeeded, ie. credentials have been checked and passed already. How come it failed in data copy activity complaining an invalid credentials? Pretty weird. Any advice and suggestions will be welcomed.User Properties of Activities in ADF: How to add dynamic content in it?
On ADF, I am using a for each loop in which I am using an Execute Pipeline Activity which is getting executed for different iterations as per the values of the items provided to the For-Each Loop. I am stuck on a scenario which requires me to add the Dynamic Content Expression in the User Properties of individual activities of ADF. Specific to my case, I want to add the Dynamic Content Expression in the User Properties of Execute Pipeline Activity so that I get to individual runs of these activities on Azure Monitor with a specific label attached to it through its User Properties. The necessity to add the Dynamic Content Expression in the User Properties is due to the reason that each execution in respective iterations of these activities corresponds to a particular Step from a set of Steps configured for the Data Load Job as a whole, which has been orchestrated through ADF. To identify the association with the respective Job-Step, I require to add Dynamic Content Expression in its User Properties. Any sort of response regarding this is highly appreciated. Thank You!
Dataflow snowflake connection issue
I'm trying to set up a sink to snowflake in dataflow, but when I test the connection it doesn't work It just comes out JDBC driver communication error, tried searching online and looking at documentation link but couldn't find anything about this issue. But this same dataset works fine outside of the dataflow, I can preview the data in same dataset: There seems to be issue with dataflow. Even when I try to execute the dataflow through pipeline the same error message comes up: Does anyone know how to solve this problem with dataflows? Also with the sink settings option if I select recreate table will it create the table in snowflake if it doesn't already exist? Trying to find an easy way to copy a lot of tables into snowflake without explicitly having to create the table first especially if the metadata is only known at runtime, the pipeline copy job doesn't work as the table has to exist before it can insert data into it, but dataflow seems promising if the connection actually works.
Oracle 2.0 Upgrade Woes with Self-Hosted Integration Runtime
This past weekend my ADF instance finally got the prompt to upgrade linked services that use the Oracle 1.0 connector, so I thought, "no problem!" and got to work upgrading my self-hosted integration runtime to 5.50.9171.1 Most of my connection use service_name during authentication, so https://learn.microsoft.com/en-us/azure/data-factory/connector-oracle?tabs=data-factory, I should be able to connect using the Easy Connect (Plus) Naming convention. When I do, I encounter this error: Test connection operation failed. Failed to open the Oracle database connection. ORA-50201: Oracle Communication: Failed to connect to server or failed to parse connect string ORA-12650: No common encryption or data integrity algorithm https://docs.oracle.com/error-help/db/ora-12650/ I did some digging on this error code, and the troubleshooting doc suggests that I reach out to my Oracle DBA to update Oracle server settings. Which, I did, but I have zero confidence the DBA will take any action. https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-oracle Then I happened across this documentation about the upgraded connector. https://learn.microsoft.com/en-us/azure/data-factory/connector-oracle?tabs=data-factory#upgrade-the-oracle-connector Is this for real? ADF won't be able to connect to old versions of Oracle? If so I'm effed because my company is so so legacy and all of our Oracle servers at 11g. I also tried adding additional connection properties in my linked service connection like this, but I have honestly no idea what I'm doing: Encryption client: accepted Encryption types client: AES128, AES192, AES256, 3DES112, 3DES168 Crypto checksum client: accepted Crypto checksum types client: SHA1, SHA256, SHA384, SHA512 But no matter what, the issue persists. :( Am I missing something stupid? Are there ways to handle the encryption type mismatch client-side from the VM that runs the self-hosted integration runtime? I would hate to be in the business of managing an Oracle environment and tsanames.ora files, but I also don't want to re-engineer almost 100 pipelines because of a connector incompatability.Prevent Accidental Deletion of an Instance in Azure Postgres
Did you know that accidental deletion of database servers is a leading source of support tickets? Read this blog post to learn how you can safeguard your Azure Database for PostgreSQL Flexible Server instances using ARMโs CanNotDelete lock โ an easy best-practice that helps prevent accidental deletions while keeping regular operations seamless. ๐ Prevent Accidental Deletion of an Instance in Azure Postgres
Increasing the number of scripts for Bicep deployments
We were quite surprised to run into a limit of 50 scripts per ADX cluster limit... Considering there is a limit of 10,000 databases, having a limit of 50 scripts seems wrong... Is this a bug? Anyway to increase it? Our use-case would really require that we can provision certain assets alongside our Bicep infra code. We give each customer their own database and provision certain defaults tables and table mapping assets there. We started to hit this 50 script limits just after 4 customers...Data flow sink to Blob storage not writing to subfolder
Hi Everybody This seems like it should be straightforward, but it just doesn't seem to work... I have a file containing JSON data, one document per line, with many different types of data. Each type is identified by a field named "OBJ", which tells me what kind of data it contains. I want to split this file into separate files in Blob storage for each object type prior to doing some downstream processing. So, I have a very simple data flow - a source which loads the whole file, and a sink which writes the data back to separate files. In the sink settings, I've set the "File name option" setting to "Name file as column data" and selected my OBJ column for the "Column Data", and this basically works - it writes out a separate file for each OBJ value, containing the right data. So far, so good. However, what doesn't seem to work is the very simplest thing - I want to write the output files to a folder in my Blob storage container, but the sink seems to completely ignore the "Folder path" setting and just writes them into the root of the container. I can write my output files to a different container, but not to a subfolder inside the same container. It even creates the folder if it's not there already, but doesn't use it. Am I missing something obvious, or does the "Folder path" setting just not work when naming files from column data? Is there a way around this?September 2025 Recap: Whatโs New with Azure Database for PostgreSQL
September 2025 Recap for Azure Database for PostgreSQL September was a big month for Azure Postgres! From the public preview of PostgreSQL 18 (launched same day as the community!) to the GA of Azure Confidential Computing and Near Zero Downtime scaling for HA, this update is packed with new capabilities that make PostgreSQL on Azure more secure, performant, and developer-friendly. ๐ก Hereโs a quick peek at whatโs inside: PostgreSQL 18 (Preview) โ early access to the latest community release on Azure Near Zero Downtime Scaling (GA) โ compute scaling in under 30 seconds for HA servers Azure Confidential Computing (GA) โ hardware-backed data-in-use protection PostgreSQL Discovery & Assessment in Azure Migrate (Preview) โ plan your migration smarter LlamaIndex Integration โ build AI apps and vector search using Azure Postgres VS Code Extension Enhancements โ new Server Dashboard + Copilot Chat integration Catch all the highlights and hands-on guides in the full recap ๐ #PostgreSQL #AzureDatabase #AzurePostgres #CloudDatabases #AI #OpenSource #MicrosoftPostgreSQL 18 Preview on Azure Database for PostgreSQL
PostgreSQL 18 Preview on Azure Postgres Flexible Server Weโre excited to bring the latest Postgres innovations directly into Azure. With PG18 Preview, you can already test: ๐น Asynchronous I/O (AIO) โ faster queries & lower latency ๐น Vacuuming enhancements โ less bloat, fewer replication conflicts ๐น UUIDv7 support โ better indexing & sort locality ๐น B-Tree skip scan โ more efficient use of multi-column indexes ๐น Improved logical replication & DDL โ easier schema evolution across replicas And thatโs just the start โ PG18 includes hundreds of community contributions, with 496 from Microsoft engineers alone ๐ช ๐ Try it out today on Azure Postgres Flexible Server (initially in East Asia), share your feedback, and help shape GA.Problem with Linked Service to SQL Managed Instance
Hi I'm trying to create a linked Service to a SQL Managed Instance. The Managed Instance is configured with a Vnet_local endpoint If I try to connect with an autoresolve IR or a SHIR I get the following error The value of the property '' is invalid: 'The remote name could not be resolved: 'SQL01.public.ec9fbc2870dd.database.windows.net''. The remote name could not be resolved: 'SQL01.public.ec9fbc2870dd.database.windows.net' Is there a way to connect to it without resorting to a private endpoint? Cheers AlexAzure SQL server rollback itself?
We have an Azure SQL server. It is a datasource of a Power App canvas app. Today I connected to it with SSMS v19. First, I ran 'Begin tran' twice (is it a mistake?). Then 'Delete From dbo.table1 where ID=30' and another row with ID=31. Then I verified these 2 rows are deleted by 'Select * from dbo.table1' Finally, I ran 'Commit tran' I verified again above 2 rows are deleted by 'Select * from dbo.table1' However, there is no change in the Power App. So I reopen the SSMS and connect to the DB again. This time when I ran 'Select * from dbo.table1', the 2 rows are showing up. What could be the problem? Is it a bug in old version SSMS?
Recent Blogs
- 5 MIN READThe Azure SQL Migration extension for Azure Data Studio is being deprecated and will be retired by February 28, 2026. As part of our unified and streamlined migration strategy for Azure SQL, we are c...Feb 03, 2026
- While configuring Auditing settings for Azure SQL, the deployment may fail and following error is encountered: Failed to save Auditing settings for server. The client has permission to perf...Feb 02, 2026
