Forum Discussion

Wiliam_Rosa's avatar
Nov 21, 2026

Azure Databricks: Genie Migrate for Serverless, an Agent That Accepts Rejection

Title: Genie Migrate for Serverless: an Agent That Proposes Changes and Accepts Rejection

Migrating a workload from classic compute to serverless is, in theory, a configuration swap. In practice, it is a sequence of trial and error: a library that existed on the classic cluster does not exist in the serverless environment, a Spark configuration that used to make sense simply does not apply anymore, an incompatible access mode only shows up once the job is already running. Each individual migration consumes time disproportionate to the size of the actual problem, and that is exactly why so many classic compute fleets simply never migrate, even when everyone agrees they should.

Genie Migrate for Serverless, now in Beta, attacks this friction in a specific way: instead of documentation promising that "migration is simple," an agent does the discovery, proposes every change individually, and applies all of it to a clone that can be discarded with zero consequence if something goes wrong.

The problem that manual migration exposes

Documentation lists three recurring sources of failure in manual migration: access mode incompatibility, Spark configuration that does not apply in the serverless environment, and a library installed on the classic cluster that simply does not exist on the other side. Each individual migration takes too long precisely because these problems only surface at execution time, not at configuration time, so the common practice ends up being try, fail, adjust, try again, repeatedly, per job.

The mechanism: three stages

The agent's design follows three stated principles: no production breakage, full control over every proposed change, and a clean rollback if anything goes wrong. In practice, that translates into three phases.

First, assessment: the agent scans notebooks and jobs and classifies each one as ready now, needing admin action, or not ready yet (a workload whose source of truth lives outside the workspace, such as a Databricks Asset Bundles job or a notebook in a Git folder, cannot be migrated through this path yet).

Second, translation: every proposed change shows up as a labeled, side-by-side diff, and nothing is written until you accept it. The agent maps classic cluster configuration to a serverless compute selection, resolves the library list against the serverless runtime, carries forward Spark configuration that still applies and flags the uncertain ones for your decision instead of simply removing them, and rewrites incompatible code using Spark Connect, also as a diff.

Third, reversible application: the validated translation is applied to a clone of the job, without touching the running original. You run the clone on demand and, if satisfied, promote the clone to replace the original, keeping the same job ID, schedule, and permissions. If you do not promote it, the original stays untouched. Worth noting: today's validation is static (consistency, library resolution, clean configuration application); an isolated sandbox validation that compares the migrated job's output against the classic baseline is still planned and is not current behavior.

Hands-on: bulk migration via the CLI

For anyone who needs to migrate many jobs at once, instead of one at a time inside Genie Code, the compute-migrations command group in the Databricks CLI exposes the same agent as a script, with an explicit state flow worth understanding before automating it:

# Starts a migration for a specific job
# (requires Databricks CLI 1.24.0 or above and the "Compute Agent" preview enabled)
databricks compute-migrations create -o json --request-id "$(uuidgen)" --json '{
  "asset_type": "JOB",
  "asset_uri": "jobs/482910",
  "auto_approve": false,
  "run_validation": false
}'

# Tracks state until it reaches TRANSLATION_READY
databricks compute-migrations get <migration-id> -o json

# After reviewing the diff manually, approve the translation
databricks compute-migrations approve-migration-translation <migration-id> --etag <etag>

# Track until MIGRATION_REVIEW and approve the final execution,
# which effectively replaces the original job with the migrated clone
databricks compute-migrations approve-migration-execution <migration-id> --etag <etag>

Every rejection requires a specific reason (reject-migration-translation followed by the migration id, a reason string, and an etag), and the agent revises the clone again from that feedback. Using an etag on every command is not empty bureaucracy: it prevents approving a version of the plan that has already gone stale because someone, or the agent itself, changed the migration's state between your last read and your decision.

In practice: the point that deserves extra attention here is that the validation run (run_validation set to true) is not isolated. The clone runs against the same table, volume, checkpoint, and external system the original job uses. If the job writes to a production table or fires a notification, the validation run does that too, for real. Reserve that option only for jobs where an extra run is genuinely harmless, because this setting cannot be changed after the migration is created.

What this does not solve

The agent proposes and applies accepted changes, but it does not verify that the result is correct. Validation today is structural, not semantic: it confirms that configuration resolves and that libraries exist, not that the migrated job's data output is identical to the classic job's. This needs to be verified manually before trusting it in production, especially for workloads writing to a critical table.

There is also a real blocklist: custom images, ML Runtime variants, Databricks Runtime versions earlier than 13, JAR or Maven dependencies, and Scala or R code all stop by default and require an explicit decision to proceed anyway, accepting the risk that the unmigrated functionality breaks later. A job with more than ten migratable tasks cannot be migrated, and through the CLI only jobs are supported (standalone notebooks are not yet), and a job whose source is a remote Git repository fails migration. A workload whose source of truth is an Asset Bundle requires manually reconciling the change the agent applied back into the bundle, or the next deploy overwrites the migration.

Is it worth adopting?

Yes, especially for anyone with a large fleet of classic jobs stalled purely for lack of time to review each one manually. The real gain is not the migration itself, Spark has documented that manually for years, the gain is turning the work of "figuring out what will break" from manual investigation into reviewing an already-prepared diff. Even so, treat promoting the clone like a normal production deploy: review it, validate the data output yourself, and only then discard the original classic job.

References

  • Databricks Blog, "Introducing Genie Migrate for Serverless" (https://www.databricks.com/blog/introducing-genie-migrate-serverless)
  • Microsoft Learn, "Migrate from classic compute to serverless compute - Azure Databricks" (https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/migration)
  • Microsoft Learn, "Connect to serverless compute - Azure Databricks" (https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/)
No RepliesBe the first to reply