Microsoftmco365 agreed Marius. If I would have the choice for this use case, I would also look first for a traditional database instead of ingesting this data in Kusto. The reason is simple, I believe Kusto is best in class for massive amounts of log files, IOT and sensor data. But this use case sounds more like a standard reporting scenario, so Azure SQL DB would be my first idea (assuming we stay in the Microsoft Azure world).
Of course, deduplication will always matter independent of what database we use. In our Kusto example, I would not want to do deduplication in ADF. As you already mentioned, it will be costly and assuming our data changes over time complexity will increase and so will operations effort.
I would simply ingest any data in Kusto and then have a materialized view on top of it which does the deduplication. The ingest table retention policy would be set to a few days only. This solution can be set up in a few minutes with almost no operations effort.
