Forum Discussion

vikas_gautam's avatar
vikas_gautam
Icon for Microsoft rankMicrosoft
Jan 13, 2025
Solved

Different pools for workers and driver - in ADF triggered ADB jobs

Hello All,    Azure Databricks allows usage of separate compute pools for drivers and workers when you create a job via the native Databricks workflows.  For customers using ADF as an orchestra...
azure data factory
azure databricks
  • JerryW1940's avatar
    JerryW1940
    Jan 13, 2025

    Unfortunately, ADF doesn’t natively support separate driver and worker pools for Databricks jobs like you can in Databricks workflows. The Linked Service only lets you specify a single cluster. However, you can work around this by manually setting up a cluster with separate driver and worker pools and then using the Databricks REST API in ADF to trigger jobs on that cluster. So, while it’s not built-in, there’s a way to achieve it with a little custom setup.

JerryW1940's avatar
JerryW1940
Brass Contributor
Jan 13, 2025

Unfortunately, ADF doesn’t natively support separate driver and worker pools for Databricks jobs like you can in Databricks workflows. The Linked Service only lets you specify a single cluster. However, you can work around this by manually setting up a cluster with separate driver and worker pools and then using the Databricks REST API in ADF to trigger jobs on that cluster. So, while it’s not built-in, there’s a way to achieve it with a little custom setup.

Resources