In January, we announced that we've elevated our Change Data Capture features front-and-center in ADF. In ADF, CDC processes are light-weight always-running (not batch) data processing with a latency option. And up until today, the lowest latency we were allowing for CDC processing was 15 minutes. But today, I am super-excited to announce that we have enabled the real-time option!
Now you can process your change data in seconds. Follow these instructions for setting up a CDC process in ADF and set the Latency to "real-time". That's it! You won't need to build a pipeline or set a trigger. Your CDC process will continuously look for changes on your sources until you stop it. In the monitoring of your CDC processes, you will see checkpoints occur every few seconds as ADF continues to monitor your sources for changes.
To make building your CDC processes even faster and simpler, we've also introduced auto-mapping to CDC. Now when you build change data processes, ADF will automatically map your sources to your targets without the need for column mapping. You can always move the toggler slider to turn off auto-mapping and map your columns semantically, including fuzzy lookup logic that is built into ADF. Auto mapping, as opposed to column mapping, provides support for schema drift so that ADF can account for column changes between individual polling intervals.
Microsoft
