Forum Discussion
Best practice for tracking data source
Microsoftdo the options mentioned in this blog post allow you to achieve your intention? (add another column, at ingestion time, but in a simple and efficient manner)
https://yonileibowitz.github.io/kusto.blog/blog-posts/ingestion-time-metadata.html
Hey Yoni,
Indeed, adding columns was my first thought. However using that method I'll have one column of data and many many columns of tracing info. Thought there might be a more efficient way. perhaps using a tag per file.
Regarding the method of defining additional columns with the ingestion string, my data is ingested by an Event Hub subscription using a Column mapping (using this tutorial: https://docs.microsoft.com/en-us/azure/data-explorer/ingest-data-event-grid)
Where can I find the column mapping I created to add the additional commands?
Where can I get the source and filename from to add? Is there a list of parameters that can be used in the ingest string $source, $blob_name etc.
Thanks.
- Yoni
If your source data is formatted as JSON, a JSON mapping will allow you to specify 2 (and only these 2) special transformations: SourceLocation and SourceLineNumber, which enable you to enrich your records with both the name of the file that included the record, and the line number of that record in the source file: https://docs.microsoft.com/en-us/azure/kusto/management/mappings#json-mapping
as for viewing existing mappings which have already been created, you can use this command: https://docs.microsoft.com/en-us/azure/kusto/management/tables#show-ingestion-mappings
- If the source data is TXT, is it possible to specify the SourceLocation and SourceLineNumber? I haven't been able to find a solution for this scenario.
