ADF adds new hierarchical data handling and new flexibility for complex joins

[Mark Kromer]

Here are some new updates to announce for ADF that have already landed in the service this week and will deploy over the coming weeks in May:

1. Hierarchical data modeling in data flows has been expanded with Flatten, collect aggregation, and updated sink mapping. We've also published a new tutorial on how to use these capabilities when moving from relational to hierarchical with Azure SQL DB and CosmosDB.
   
2. Non-equi join support added to Join, Lookup, and Exists transformations. This update will land in the coming weeks. You will now be able to add non-equi joins to all ADF join types in the UI and add "and" conditions with additional equi/non-equi types.
   
3. An update that already landed in ADF for Joins, Exists, and Lookups is an update to the Optimize tab. You can now switch off native Spark data push-down to execution nodes when performing a join using the broadcast option. The default is auto and you can use "fixed" to override the default data flow behavior in Spark by broadcasting only left or right side.
   
4. One last interesting update coming to the ADF UI next month will be an update to the method used by the ADF UI to handle extreme device and browser size factors. Watch this short recording for an example of how the UI in your browser will react to small form factors. 

[Johannes_Vink]

Looking forward to the small form factors! Especially around the monitoring.