Service Broker on Ubuntu 20.04 (Docker) cryptography error

Hi,

Has anyone successfully created a multi‑instance / multi‑host SQL Server Service Broker implementation on Linux, specifically on distributions using OpenSSL 3?

I have a fully working Service Broker environment on SQL Server 2017, running in Docker on Ubuntu 18.04 (OpenSSL 1.1.1). My environment has 6 containers, arranged to simulate two datacenters:

• DC1: Primary AG replica + synchronous replica + async replica
• DC2: Same setup
• 7 databases in the Availability Group

Service Broker dialogs between databases on the same SQL instance work perfectly. This confirms:

• Database‑level dialog security works
• Certificate‑based authentication between database principals is correct
• DMKs are auto-opened and protected by SMK
• All dialogs use BEGIN DIALOG with certificates valid and present

Cross-instance messages work correctly.

I have:

• Broker endpoints using certificate‑based endpoint authentication
• Proper certificate exchange (private key on owning instance, public key on the peer)
• Routes defined correctly
• Remote Service Bindings used to force dialog security
• Certificates created dynamically with correct validity date ranges
• DMKs created, opened, and re-encrypted with SMK at runtime
• Scripts that fully automate certificate creation, export, import, user creation, RSB creation, etc.

Everything here works fine as long as I stay on Ubuntu 18.04 (OpenSSL 1.1.1).

The problem:

If I switch only the base Linux image—to Ubuntu 20.04, Ubuntu 22.04, or RHEL 8—and change nothing else, then all cross‑instance Service Broker dialogs fail with:

An exception occurred while enqueueing a message in the target queue.

Error: 9641, State: 122.

A cryptographic operation failed. This error indicates a serious problem with SQL Server.

This happens on SQL Server 2019 and 2022, both CU up-to-date, both using OpenSSL 3.

Key points:

• Intra‑instance dialog security still works
• Endpoint transport security still works (i.e., endpoints authenticate and connect)
• The failure occurs only when creating session keys for cross‑instance dialog security
• The failure happens even when using:
• encryption = off
• fresh DMKs
• fresh certificates
• certificates stored in master or user DB
• key length 2048 or 3072
• different validity periods

So far I have:

• Confirmed DMKs exist on both instances
• Confirmed DMKs are encrypted by SMK
• Confirmed DMKs auto-open (is_master_key_encrypted_by_server = 1)
• Manually opened DMKs before creating certificates
• Recreated SMK/DMK on clean containers

Everything behaves as expected.

Repeatedly rebuilt all of the following:

• Endpoint authentication certificates
• Dialog security certificates
• Remote Service Bindings
• Database principals
• Routes
• Service Broker services, queues, and message types

I’ve verified:

• All certificate subjects match
• All public keys export/import correctly
• Private keys exist where they should exist
• Thumbprints match across both sides
• No old certificates remain in master or user DBs

Modified the OpenSSL behaviour to legacy (1.1.1) behaviour, e.g.

[openssl_init]

providers = provider_sect

ssl_conf  = ssl_sect

alg_section = evp_properties

[evp_properties]

rh-allow-sha1-signatures = yes

fips_mode = no

[provider_sect]

default = default_sect

legacy  = legacy_sect

[default_sect]

activate = 1

[legacy_sect]

activate = 1

[ssl_sect]

system_default = system_default_sect

[all_policy]

rsa_pkcs1_padding_check = 0

[system_default_sect]

# Lowest policy to allow legacy algorithms (including SHA-1, 1024-bit RSA)

CipherString = DEFAULT:@SECLEVEL=0

To rule out network/transport issues, I validated:

• Endpoints authenticate each other
• Connections appear in sys.dm_broker_connections
• STATE = OPEN

Transport security appears to be working on OpenSSL 3.

I confirmed:

• Messages flow when RSB is removed and encryption=off is used
• Messages fail only when RSB is enabled, which requires certificate‑based dialog security
• The target instance claims “private key missing” even though it is present and readable

Has anyone successfully created a multi‑instance / multi‑host SQL Server Service Broker implementation on Linux, specifically on distributions using OpenSSL 3? If so, any guidance on how to resolve the issues i am facing?

Many thanks,

Andrew