KQL Query output limit of 5 lakh rows
Hi , i have a kusto table which has more than 5 lakh rows and i want to pull that into power bi. When i run the kql query it gives error due to the 5 lakh row limit but when i use set notruncation before the query then i do not get this row limit error on power bi desktop but get this error in power bi service after applying incremental refresh on that table. My question is that will set notruncation always work and i will not face any error further for millions of rows and is this the only limit or there are other limits on ADE due to which i may face error due to huge volume of data or i should export the data from kusto table to azure blob storage and pull the data from blob storage to power bi. Which will be the best way to do it?
Azure ADX - UpdatePolicy fails to insert data
Hi Everyone, I believe everyone is doing good and safe. I am facing challenge with ADX. Please find the problem details below. Problem statement: We are unable to insert a result data into a target table from source table using an UpdatePolicy. Description: We have written an UpdatePolicy on a table. This UpdatePolicy will accept query parameters as an ADX function. This function returns output result in the form of table. Further, This table output result received should be inserted into target table. Additional Details: UpdatePolicy helps to update the data into a target table from source table dynamically/automatically. UpdatePolicy is almost equivalent to Triggers in SQL Server to do dynamic insert into a target table. Syntax of UpdatePolicy .alter table TargetTable policy update ``` [ { "IsEnabled": true, "Source": "SourceTable", "Query": "SourceTable | extend Result = G3MS_ClearAlarm(Id, CountryCode, OccuredTime) | project AlarmId = Result.AlarmId, ClearAlarmId = Result.ClearAlarmId, ClearTime = Result.ClearTime", "IsTransactional": true, "PropagateIngestionProperties": false } ] ``` Error Received when executed Error during execution of a policy operation: Request is invalid and cannot be processed: Semantic error: SEM0085: Tabular expression is not expected in the current context. If anyone has any suggestions/thoughts on this will be very beneficial to complete the requirement.
External Table in ADX
Hi, I'm trying to create an external table in ADX which uses a Synapse Analytics (SA) database view (called undelivered). The undelivered view itself is query data from a Cosmos analytical store I've create a user defined idenity Added the identiy to the ADX cluster, SA and Cosmos Updated the ADX database: .alter-merge cluster policy managed_identity[ { "ObjectId": "a3d7ddcd-d625-4715-be6f-c099c56e1567", "AllowedUsages": "ExternalTable" } ] Created the database users in SA -- Create a database user for the ADX Managed Identity CREATE USER [adx-synapse-identity] FROM EXTERNAL PROVIDER; -- Grant read permissions ALTER ROLE db_datareader ADD MEMBER [adx-synapse-identity]; GRANT SELECT ON OBJECT::undelivered TO [adx-synapse-identity]; From within SA I can "SELECT * FROM undelivered" and the correct information is returned But when I come to create the external table in ADX: .create-or-alter external table MyExternalTable ( Status: string ) kind=sql table=undelivered ( h@'Server=tcp:synapse-xxxxx.sql.azuresynapse.net,1433;Database="Registration";ManagedIdentityClientId=<key>;Authentication=Active Directory Managed Identity;' ) with ( managed_identity = "<key>" ) I get the error: Managed Identity 'system' is not allowed by the managed_identity policy for usage: ExternalTable So even with me specifying the managed identity I want to use it is still trying to use the system one. How can I get the external table created with the correct managed identity? Any questions please just ask Thanks
KQL Query to summerize session counts vertically
I'm trying to find a "good" way to achieve what I think is a simple task but cannot think of a simple solution. I have logs with session information, one entry per session StartTime(datetime), EndTime(datetime), Duration(in seconds), Computer(string) I want to count how many sessions are active for each 5 minute interval and graph that. Keep in mind, the sessions will overlap. I included a graphic of what I'm trying to do. With the result below. The black boxes represent an entry with a starttime and an endtime. Which should return: Time + 5, 1 Time + 10, 3 Time + 15, 3 Time + 20, 2 I have found many similar examples but they all depend on the the thing I'm trying to bin, or group, to be a single point in time, but my problem is each entry has an active range, a start time and an end time per record.
[New post] Microsoft Fabric real-time analytics exploration: Managed Grafana integration
The goal for today is to render real-time ingested Eventstream IoT data available in a Managed Grafana instance. https://sandervandevelde.wordpress.com/2023/12/05/microsoft-fabric-real-time-analytics-exploration-managed-grafana-integration/[New post] Microsoft Fabric real-time analytics exploration: Eventstream Custom apps integration
Learn how customs apps can be integrated in Microsoft Fabric using EventHub-compatible public endpoints living in Fabric Eventstream, both for ingress and egress. These are part eight and nine in a series of posts about Fabric for IoT Engineers. Check out Ingress https://sandervandevelde.wordpress.com/2023/12/02/microsoft-fabric-real-time-analytics-exploration-eventstream-source-for-custom-app-ingress/%20 and Egress https://sandervandevelde.wordpress.com/2023/12/02/microsoft-fabric-real-time-analytics-exploration-eventstream-destination-for-custom-app-egress/#mvpbuzz[New blog post] Microsoft Fabric real-time analytics exploration: KQL Database mirroring
Integrate KQL Database mirroring into Microsoft Fabric Real-Time Analytics. Create a OneLake shortcut for IoT data ingested in a KQL database table and query it in Lakehouse using SQL. This is part seven of a blog post series about Microsoft Fabric RTA for IoT Developers. Read the full story at https://sandervandevelde.wordpress.com/2023/11/27/microsoft-fabric-real-time-analytics-exploration-kql-database-mirroring/. #mvpbuzz[blog series] Introduction to Microsoft Fabric Real-Time Analytics for IoT developers
Are you interested in combining Microsoft Fabric Real-Time Analytics with Azure IoT? In my series of five blog posts, learn how to combine Eventstreams, KQL Databases, Reflect, and Data Activator for powerful queries, insightful reports, and condition-based alerts. Learn about working with (streaming) sample data and actual IoT data ingested from the industrial router via an Azure IoT Hub. For those who are just exploring Microsoft Fabric, learn how to pause your Capacity automatically using an Azure Function and save some money. Just start here https://sandervandevelde.wordpress.com/2023/11/10/microsoft-fabric-real-time-analytics-exploration-for-iot-developers/ https://sandervandevelde.wordpress.com/2023/11/12/microsoft-fabric-real-time-analytics-exploration-storing-and-reporting-time-series-data/ https://sandervandevelde.wordpress.com/2023/11/13/microsoft-fabric-real-time-analytics-exploration-ingesting-streams-of-sample-events/ https://sandervandevelde.wordpress.com/2023/11/14/microsoft-fabric-real-time-analytics-exploration-ingesting-iot-device-events/ https://sandervandevelde.wordpress.com/2023/11/15/microsoft-fabric-real-time-analytics-exploration-activating-iot-device-reflex-data/[New blog post] Plotting the Azure Digital Twins graph in Azure Data Explorer
See how your Azure Digital Twins graph evolves over time. Add extra information to your relationship popups. Learn how to use @@plotly graph visualizations in Azure Data Explorer in addition to Azure Digital Twins data history Read the full story at https://sandervandevelde.wordpress.com/2023/10/08/plotting-the-azure-digital-twins-graph-in-azure-data-explorer/[New blog post] Putting the SenseCAP T1000 GPS Tracker on the map using Azure Data Explorer
Check out my blog post on how to putt the Seeed Studio SenseCAP T1000 GPS Tracker on the map using Azure Data Explorer. Realtime positions, formatted and forwarded by the LoRaWAN The Things Network backend, are transformed using the Kusto Query Language and represented in an ADX Dashboard. https://sandervandevelde.wordpress.com/2023/10/03/putting-the-sensecap-t1000-gps-tracker-on-the-map-using-azure-data-explorer/
