Forum Discussion
vikas_gautam
Microsoft
MicrosoftDifferent pools for workers and driver - in ADF triggered ADB jobs
Hello All,
Azure Databricks allows usage of separate compute pools for drivers and workers when you create a job via the native Databricks workflows.
For customers using ADF as an orchestrator for ADB jobs, is there a way to achieve the same when invoking notebooks/jobs via ADF? The linked service configuration in ADF seems to allow only one instance pool.
Appreciate any pointers.
Thanks !
Unfortunately, ADF doesn’t natively support separate driver and worker pools for Databricks jobs like you can in Databricks workflows. The Linked Service only lets you specify a single cluster. However, you can work around this by manually setting up a cluster with separate driver and worker pools and then using the Databricks REST API in ADF to trigger jobs on that cluster. So, while it’s not built-in, there’s a way to achieve it with a little custom setup.
Unfortunately, ADF doesn’t natively support separate driver and worker pools for Databricks jobs like you can in Databricks workflows. The Linked Service only lets you specify a single cluster. However, you can work around this by manually setting up a cluster with separate driver and worker pools and then using the Databricks REST API in ADF to trigger jobs on that cluster. So, while it’s not built-in, there’s a way to achieve it with a little custom setup.
