It is common that JSON log files contain many types of records together. In a Root Cause Analysis, we often want to explore only a specific type of records, which is just a fraction of the data.
Today, as ADX does not support filtered ingestion, we have no other choice but to repeat the following pattern:
1) Load all the records to a "raw" table. (Its bad as 90% of that data is junk)
2) Connect the row table to a derived table with an update policy and with that execute a filter which will get only the records we care about to the derived table.
3) purge the "raw" table.
4) Execute the analytics against the derived table.
Is there a plan to support filtered ingestion? Is there a better way to do this, than what we do today?
