Azure SQL DB HS Data Load Using Microsoft JDBC Driver Instead of Apache Spark Connector for SQL

Introduction

Migration from heterogeneous database platforms such as Db2, Oracle, and Sybase to Microsoft SQL platforms has been a common modernization pattern for many years. For large-volume data migrations, customers commonly extract data from the source database in Parquet format, securely transfer it to Azure Storage using SFTP, and subsequently ingest it into Azure SQL Database Hyperscale. Data load from Azure Data Lake Storage (ADLS) Parquet files to Azure SQL Database Hyperscale can be done using Fabric Spark Notebook, Fabric Data Factory, Azure Data Factory, Azure SQL Database Hyperscale OPENROWSET function, Databricks Spark notebook etc. This blog provides guidance on using the Microsoft JDBC Driver for SQL Server within Databricks Spark workloads to load data into Azure SQL Database Hyperscale.

The Apache Spark Connector for SQL Server and Azure SQL which has been widely used is no longer maintained, doesn't support Spark 3.4 onwards, and relies on Databricks Runtime version 13.3 which is end of support in August 2026. Those factors mean that organizations should consider moving to the Microsoft supported Microsoft JDBC Driver for SQL Server. This blog explores how to make that transition with minimal modifications to existing Databricks Spark notebooks, allowing customers to continue loading data efficiently into Azure SQL Database Hyperscale while using a supported Microsoft driver.

Why Azure SQL Database Hyperscale for Azure Cloud Applications?

Azure SQL Database Hyperscale is a strong choice for cloud applications that need to grow beyond traditional database-size constraints while retaining the familiar SQL Server engine and a fully managed database service. Its independently scalable compute and storage architecture supports rapid compute scaling, fast database copy and restore operations, high availability, and multiple read-only replicas for workload isolation. Those capabilities are especially valuable for data-intensive SaaS, operational, and analytical applications whose storage and read demand can grow independently; as with every service tier, the final choice should be validated against workload, availability, latency, and cost requirements. Hyperscale is particularly attractive for migration projects because it supports rapid storage growth and large-scale data ingestion without the storage limitations associated with traditional database tiers. Read more on Azure SQL Database Hyperscale at link.

Typical pattern for a large heterogeneous database migration

The following diagram shows a common high-level pattern for migrating data from an on-premises database to Azure SQL Database Hyperscale. Multiple migration architectures are possible. This article examines the part of the flow that uses the Microsoft JDBC Driver for SQL Server to load data from an Azure Databricks Spark workload.

Figure 1. Typical large-scale migration architecture from Db2, Oracle, and Sybase to Azure SQL Database Hyperscale.

Why move from the Apache Spark connector for SQL Server and Azure SQL to Microsoft JDBC Driver for SQL Server?

The repository for Apache Spark Connector for SQL Server and Azure SQL was archived in February 2025 and is now read-only. Apache Spark Connector for SQL Server and Azure SQL git repository is not updated to support Spark version higher than 3.4. Databricks Runtime version 13.3 support for Apache Spark version 3.4.1 is ending in Aug 2026, this makes it a poor long-term dependency for customers adopting newer Spark, Scala, and Databricks Runtime versions. The supported Microsoft JDBC Driver for SQL Server is a Type 4 Java driver that connects directly over TDS and supports Azure SQL Database Hyperscale. The driver can transparently translate eligible JDBC prepared-statement batch inserts into its Bulk Copy API when useBulkCopyForBatchInsert=true.

Important differences between Apache Spark Connector for SQL and Microsoft JDBC Driver for SQL Server:

Prerequisites and one-time setup

1. Use a Databricks compute mode that permits a custom JDBC driver. On dedicated classic compute, install it from Compute > cluster > Libraries > Install new > Maven, then restart the cluster. 
2. Use a supported Microsoft JDBC Driver for SQL Server v13.x. For Java 11 or later, install Maven coordinate com.microsoft.sqlserver:mssql-jdbc:13.4.0.jre11. Confirm the current generally available version, support lifecycle, and Java compatibility before deployment.
3. Remove or exclude older mssql-jdbc JARs. If multiple copies are present, class-path resolution can load an older version even when the cluster interface shows a newer library. 
4. Provide network connectivity to <server>.database.windows.net:1433, preferably through approved private networking and DNS.
5. Use Microsoft Entra authentication. Create a contained database user for the managed identity and grant only the permissions needed on the staging table or schema.
6. Pre-create the destination table with explicit SQL types. Avoid using production overwrite mode as an implicit schema-management mechanism.

High level steps to use Microsoft SQL JDBC Driver for data load to Hyperscale from Databricks are mentioned in below diagram:

Detail setup to utilize Microsoft SQL JDBC Driver for data load to Microsoft SQL from Databricks

Grant least-privilege access to a managed identity

-- Run while connected as the Microsoft Entra administrator.

CREATE USER [your-managed-identity-name] FROM EXTERNAL PROVIDER;

GRANT CONNECT TO [your-managed-identity-name];

GRANT INSERT ON OBJECT::dbo.FactLoad_Stage TO [your-managed-identity-name];

GRANT SELECT ON OBJECT::dbo.FactLoad_Stage TO [your-managed-identity-name];

Verify which Microsoft JDBC Driver for SQL Server Spark loaded

Azure Databricks clusters can contain multiple JDBC driver versions. The cluster installation interface does not prove which JAR won class-path resolution. Run the following notebook cell after restarting the cluster:

Verify JDBC driver version:

driver_class = spark._jvm.java.lang.Class.forName(
    "com.microsoft.sqlserver.jdbc.SQLServerDriver"
)
driver_package = driver_class.getPackage()

print("Driver version:", driver_package.getImplementationVersion())
print("Driver source :", driver_class.getProtectionDomain()
      .getCodeSource().getLocation().toString())

The tests described in this article used version 13.4.0. If the result shows 11.2 or another unexpected version, resolve the duplicate JAR files before running performance tests. This check also confirms that the runtime supports the required connection properties: cacheBulkCopyMetadata requires 12.8 or later, while bulkCopyForBatchInsertBatchSize and bulkCopyForBatchInsertTableLock require 12.10 or later.

Sample PySpark notebook to load data to Azure SQL Database Hyperscale using Microsoft JDBC Driver for SQL Server

The following example reads Parquet data, controls write parallelism, uses passwordless authentication, enables the driver's bulk-copy conversion, and measures rows per second. Replace every placeholder with a value validated for the target environment.

1. Configuration

Configure load parameters:

import time

abfss_src = "abfss://<container>@<account>.dfs.core.windows.net/<path>"
server = "<logical-server-name>.database.windows.net"
database = "<database-name>"
target_table = "dbo.FactLoad_Stage"
managed_identity_client_id = "<user-assigned-managed-identity-client-id>"

# Start with measured values, then tune one dimension at a time.
target_partitions = 64
spark_batch_size = 100000
bulk_copy_batch_size = 100000

2. Read and shape the source partitions

Partition source parquet data:

df = spark.read.parquet(abfss_src)
source_partitions = df.rdd.getNumPartitions()
print(f"Source partitions: {source_partitions}")

# coalesce() only reduces partitions. repartition() is required to increase them.
if source_partitions > target_partitions:
    df = df.coalesce(target_partitions)
elif source_partitions < target_partitions:
    df = df.repartition(target_partitions)

print(f"Write partitions : {df.rdd.getNumPartitions()}")

# count() is useful for benchmark math but performs a separate full scan.
# Persist first if cluster memory/disk capacity and test design justify it.
row_count = df.count()
print(f"Row count        : {row_count:,}")

3. Build a secure JDBC URL

Build JDBC URL:

jdbc_url = (
    f"jdbc:sqlserver://{server}:1433;"
    f"databaseName={database};"
    "authentication=ActiveDirectoryManagedIdentity;"
    f"user={managed_identity_client_id};"
    "encrypt=true;"
    "trustServerCertificate=false;"
    "hostNameInCertificate=*.database.windows.net;"
    "loginTimeout=30;"
    "packetSize=32767;"
    "applicationName=Databricks-Parquet-Hyperscale-Load;"
    "useBulkCopyForBatchInsert=true;"
    f"bulkCopyForBatchInsertBatchSize={bulk_copy_batch_size};"
    "bulkCopyForBatchInsertTableLock=true;"
    "cacheBulkCopyMetadata=true;"
)

4. Execute and measure the write

Execute bulk load:

start_time = time.perf_counter()

(df.write
   .format("jdbc")
   .mode("append")
   .option("url", jdbc_url)
   .option("dbtable", target_table)
   .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
   .option("batchsize", spark_batch_size)
   .option("isolationLevel", "NONE")
   .save())

elapsed_seconds = time.perf_counter() - start_time
print(f"Elapsed seconds  : {elapsed_seconds:,.2f}")
print(f"Rows per second  : {row_count / elapsed_seconds:,.0f}")
print("SQL JDBC bulk write completed")

Note: For Apache Spark Sql connector 1) Format will be com.microsoft.sqlserver.jdbc.spark instead of jdbc.

Why options are not duplicated. 

Driver connection properties belong in the JDBC URL or JDBC properties object. Spark options such as `batchsize`, `dbtable`, and `isolationLevel` remain DataFrameWriter options. Assigning one owner to each setting avoids ambiguity during tuning.

What each performance setting does

Design the target for loading, not only querying

Even a well-tuned driver cannot compensate for an expensive target table. Every nonclustered index must be maintained, foreign keys and check constraints consume CPU, and triggers add work. For recurring large loads, use a dedicated staging table, then validate and publish the data through a controlled database-side operation.

• Prefer an empty heap or an appropriately designed staging table during ingestion.

• For a one-time load, evaluate removing unnecessary nonclustered indexes and rebuilding them in an approved maintenance window. Include the rebuild time and transaction-log cost in the comparison.

• Consider clustered columnstore for large analytical fact tables when it matches the query and update patterns.

• Avoid running many concurrent jobs against the same ascending key or allocation structures. This can cause latch contention before the transaction-log throughput limit is reached.

• Keep batches bounded. Long-running transactions can increase Hyperscale active-log usage and extend recovery or retry scope.

Tune with a controlled matrix

Do not change partitions, batch size, packet size, driver version, target indexes, and compute tier in one test. Use the same source slice and target shape, warm-up once, and repeat each configuration. A practical sequence is:

1. Upgrade to and verify driver 13.4.0 while holding existing settings constant.
2. Compare 16, 32, and 64 output partitions.
3. At the best partition count, compare Spark and Bulk Copy batches of 10K, 20K, 50K, and 100K rows.
4. Compare packet size default versus 32,767 bytes.

5. Compare runs with and without a table lock on an isolated target.

6. Run at least three measured iterations and report median throughput plus variability.

Load speed of 140+ MB/s is based on the current Hyperscale Premium Series throughput limit of 150 MB/s. As this limit upgrades the underlying load speed will also change.
Hyperscale FAQs

Measure the database, not only the notebook clock

Rows per second is comparable only when row shape remains constant. If reporting MB/s, define the numerator explicitly. Compressed Parquet size, uncompressed row size, TDS payload, allocated table space, and transaction-log volume measure different things.

Observe Azure SQL Database Hyperscale resource utilization

SELECT
    end_time,
    avg_cpu_percent,
    avg_data_io_percent,
    avg_log_write_percent,
    max_worker_percent,
    max_session_percent
FROM sys.dm_db_resource_stats
ORDER BY end_time DESC;

Inspect active load requests and waits on Azure SQL Database Hyperscale

SELECT
    r.session_id,
    r.status,
    r.command,
    r.wait_type,
    r.wait_time,
    r.cpu_time,
    r.total_elapsed_time,
    r.reads,
    r.writes
FROM sys.dm_exec_requests AS r
WHERE r.database_id = DB_ID()
ORDER BY r.total_elapsed_time DESC;

Where Do Fabric, Azure Data Factory, and OPENROWSET Fit in Azure SQL Database Hyperscale Data Loads?

Microsoft Fabric, Azure Data Factory, and `OPENROWSET` represent different ingestion architectures. JDBC is a good fit when Azure Databricks Spark already reads and transforms Parquet data and the team needs supported TDS connectivity instead of the archived Apache Spark connector.

A server-side bulk-access path using `OPENROWSET` can remove Spark from the data-movement path, but it consumes SQL compute resources that are also used for query execution. This can become the limiting factor in some environments.

For new Spark-based ingestion solutions, evaluate the supported Spark connector for SQL databases in Microsoft Fabric as an alternative architecture. 

Azure Data Factory provides low-code / no-code option for easy data copy from ADLS to Azure SQL Database Hyperscale.

Decision matrix to for Recommended Tool selection:

Key Takeaways

• Apache Spark Connector for SQL Server and Azure SQL is archived and no longer maintained.
• Microsoft JDBC Driver for SQL Server is the supported replacement.
• Existing Databricks Spark notebooks can migrate with minimal code changes.
• Bulk-copy optimization can deliver sustained ingestion rates exceeding 140 MB/s.
• Proper partitioning, batching, table design, and log utilization monitoring are critical for achieving maximum throughput.

Because the Apache Spark Connector for SQL Server and Azure SQL is archived, organizations should plan a transition to a supported alternative. The Microsoft JDBC Driver for SQL Server allows existing Azure Databricks Spark pipelines to retain their JDBC processing model while receiving ongoing support, security updates, and bulk-copy optimizations.

Performance depends on the complete ingestion path. Tune partition count, batch size, target-table design, driver properties, and database capacity through controlled tests. Measure both Spark execution and Azure SQL Database resource utilization and validate failure and recovery behavior before deploying the process in production.

Disclaimer: The guidance, recommendations, and examples provided in this blog are based on our experience with loading parquet files from ADLS to Azure SQL Database Hyperscale using Microsoft JDBC driver for SQL Server and may not be universally applicable to all environments. Every customer’s workload, configuration, and performance characteristics are unique. You should thoroughly test and validate these recommendations in your own development or staging environment before applying them in production. We do not assume any responsibility or liability for any issues, downtime, or impacts resulting from the use of the information in this blog.

Feedback and suggestions

If you have feedback or suggestions for improving this data migration asset, please send an email to Database Platform Engineering Team.