Forum Widgets
Latest Discussions
Partial query failure: Low memory condition Kusto
I am getting below Error Message while executing query in Kusto "Partial query failure: Low memory condition (E_LOW_MEMORY_CONDITION). (message: 'bad allocation', details: ''). [0]Kusto.Data.Exceptions.KustoDataStreamException: Query execution has resulted in error (0x80DA0007): Partial query failure: Low memory condition (E_LOW_MEMORY_CONDITION). (message: 'bad allocation', details: '')" How to handle ?
Most effecient way to identify duplicates in data?
We're moving data analytics towards Kusto and one feature we would like to have is to sanity-check our data and find duplicates of data in a dataset. The problem I want to solve is to make sure we don't accidentally ingest duplicates and then report too high sales numers. Our plan now is to introduce an extra column with a sha1 hash of the row and do something like "summarize count() by sha1 | where count_ > 1" ... but that would need to touch every column in the data set. I realize that's in the nature of the problem, but I'd just like to investigate strategies what would make this as effecient as possible. Strategies I've thought of would be to first limit the dataset to a certain timewindow or perhaps by customerId. I also know about the ingestion tags but I don't want to impact extents too much since this is more of a sanity-check. What other strategies could we use to make this operation as efficient as possible?
Kusto Query between TimeGenerated
Hi there, I want to be able to look into a Kusto query in the Perf table for Virtual Machines and I want the TimeGenerated to both be between 3 weeks ago - but also only the events in TimeGenerated between 7:00am (12:00PM UTC) -> 10:00PM (3:00AM UTC) for each of those days. I cannot figure out how to get this to work, is this even possible? Thanks!Welcome to Azure Data Explorer (Kusto) Space
Welcome to the Azure Data Explorer (Kusto) space @ TechCommunity. Join us to share questions, thoughts or ideas about Kusto and receive answers from the diverse Azure Data Explorer community. Our community is here to assist you with any question or challenge such as creating a new Data Explorer cluster, database or table, ingesting data or performing a complex query. Learn more about Data Explorer (Kusto): Azure Data Explorer Documentation Course – Basics of KQL Query explorer Azure Portal User Voice End to End Lab Azure Data Explorer Blog Investigate your data with Azure Data Explorer (Kusto). Question, comment or request? Post it here. BR, Azure Data Explorer product teamGet a permanent URL for Azure Storage Blob?
I have images in Azure Storage Blob. I am trying to get a permanent URL for the images from Azure Storage Explorer. I can right-click on the actual blob container, select "Get Shared Access Signature" but it has a start and expiration time. I know I can set it to expire in 2050 or something like that, but is there a way to just have a URL that I can use with no start/expire date? The URL I get has the start/expire dates in it as shown below. Looking for a way to avoid that if possible. https://storageaccpimtname.blob.core.windows.net/filesareheresomewhere/" & ThisItem.ItemNumber & ".jpg?st=2019-11-22T18%3A16%3A00Z&se=2051-01-01T07%3A59%3A00Z&sp=rl&sv=2018-03-28&sr=c&sig=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" Apologies if not the right forum. Couldn't find one specific to Azure Storage Blobs.Harnessing the Power of Left-Anti Joins in the Kusto Query Language
The Kusto query language supports a variety of joins. Left-anti might not be among the most common ones used, but it can be one of the most powerful. The docs state that a left-anti join “returns all records from the left side that do not match any record from the right side.” Let’s walk through two ways that this can be used in your processing pipeline. Late-Arriving Data Let’s say that we have an incoming stream if time-series data that we want to process. We have a function called ProcessData(startTime:datetime, endTime:datetime) that periodically gets executed and written to a table called Output via .set-or-append commands. The function processes data between those two timestamps in the parameters. Since we don’t want to end up with duplicate rows, we can’t rerun with the same time window. We can, however, catch the late arriving data for that time window by implementing ProcessData in such a way that it reprocesses all the data in the previous day and then does a left-anti join against the Output table to only return the results haven’t been recorded yet. Anything new gets written to the Output table by the set-or-append command and the duplicates get thrown away. .create-or-alter function with (folder = "demo", skipvalidation = "true") ProcessData (startTime:datetime, endTime:datetime) { let lookback = 1d; let allData = SourceData | where Timestamp >= startTime - lookback and Timestamp < endTime ; OutputTable | join kind = leftanti (allData) on DeviceId, Timestamp } [Update 2019-02-21] The Kusto docs have a good document on dealing with late arriving data. Changelog Left-anti joins can also be used to create a changelog. Let’s say there is a process that is dumping 500,000 rows of data into a table. Those rows contain information about a set of devices. The table gets dropped and replaced every day. We can make a CreateChangelog() function that gets its results written to the Changelog table via set-or-append commands. We can do a left-anti join with the data we already have in Output and only write the rows that have changed. So the CreateChangelog function body would look something like this: DeviceData | where PreciseTimeStamp >= startTime and PreciseTimeStamp < endTime | project DeviceId, DimensionA | join kind = leftanti( Output | project DeviceId, DimensionA ) on DeviceId | project DeviceId, DimensionA, ProcessedTime=now() Now the Output table has a record of every time that a device was added, removed or modified.
