Blog Post

Post Quantum Crypto Tech Blog
11 MIN READ

When Bigger PQC Keys and Signatures Break Software Assumptions

MichaelHoward-MSFT's avatar
Oct 28, 2026

Post-quantum cryptography brings much larger keys, signatures, and certificates - and those sizes can break assumptions far beyond cryptographic code. Here’s what engineers should test now to prevent failures across networks, protocols, storage, and applications.

There is an interesting backstory to my previous post about PQ hybrid TLS 1.3 and RDP, and it is the sole reason I wrote this post.

I built a tool that captures detailed TLS and process information, although “built” is generous: it was about 20% hand-written and 80% vibe-coded, to help easily identify where hybrid TLS is used and where it is not used, mostly to passively and quickly verify that a communication channel is using hybrid TLS!

Everything worked until I configured a Windows Server 2025 virtual machine in Azure to require hybrid TLS 1.3 with secp384r1-ML-KEM-1024. My tool then stopped logging packets for those connections; not corrupted or malformed packets, no packets at all.

Naturally, I assumed the TLS connection had failed, but the problem was in my code.

This was my first, and hopefully only, code regression triggered by PQC. The failure illustrates a broader lesson for software engineers: larger cryptographic objects can expose size and framing assumptions that classical algorithms rarely hit.

Who is the audience for this post? Colleagues asked me this while reviewing early drafts. The short answer: anyone who builds or maintains products that use PQC. You do not need to develop operating systems or parse network packets - the lessons apply broadly.

Before diving into the issue, let’s borrow the King of Hearts’ advice to the White Rabbit in Alice’s Adventures in Wonderland: “Let’s begin at the beginning!”

The Venerable 1,500-byte MTU

The maximum transmission unit (MTU) is the largest IP packet a network link can carry without fragmentation; on conventional Ethernet, it is typically 1,500 bytes.

MTU matters because packets that exceed this limit must be fragmented or as is more common with TCP, split across multiple segments. As we move to post-quantum cryptography (PQC), larger key-establishment payloads, public keys, ciphertexts, and signatures make this boundary much easier to cross.

The size difference can be substantial. A classic X25519 elliptic curve key share is only 32 bytes, whereas an ML-KEM-1024 encapsulation key is 1,568 bytes.

That’s almost 50x bigger!

Combining ML-KEM-1024 with P-384 produces a 1,665-byte  key_share  in a TLS 1.3  ServerHello.

Signatures have a similar size increase: an RSA-2048 signature is 256 bytes, while ML-DSA signatures range from roughly 2.4 KB to 4.6 KB, depending on the parameter set.

These size increases illustrate the change in scale that implementations and network tooling must accommodate.

The following diagram has more examples. Most of the time, the PQC algorithms you’ll come across are ML-KEM and ML-DSA, but I have added SLH-DSA for completeness. You probably won’t see much of SLH-DSA in the wild, however.

Sizes are the standardized raw cryptographic objects and exclude protocol, certificate, and container overhead.

For SLH-DSA, those that end in ‘s’ have smaller signatures and those with an ‘f’ are faster algorithms, with a size tradeoff.

The size difference is even more apparent in X.509 certificates.

The following PowerShell example compares the encoded size of an ECDSA certificate with an ML-DSA certificate, including the public key, signature, certificate metadata, and ASN.1 encoding overhead.

'ECDSA_nistP384', 'ML-DSA-65' | ForEach-Object {
    $cert = New-SelfSignedCertificate -KeyAlgorithm $_ `
        -Subject "CN=Trust Me Bro $_, O=`"Trust me, bro`"" `
        -CertStoreLocation Cert:\CurrentUser\My `
        -NotAfter (Get-Date).AddDays(7)

    Export-Certificate -Cert $cert -FilePath ".\trustmebro-$_.cer"
}

I see this:

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           10/6/2026  7:31 AM            506 trustmebro-ecdsa.cer
-a---           10/6/2026  7:31 AM           5577 trustmebro-mldsa65.cer

My Code Bug

Back to my story and my bug!

IMPORTANT: This was a bug in MY code not in the Windows TLS stack! Schannel in Windows, which handles TLS, performed flawlessly handling the larger TLS message sizes.

My code regression appeared only when the TLS 1.3 connection negotiated to use secp384r1-ML-KEM-1024; all the smaller TLS negotiated groups I tested worked correctly.

This is what GitHub Copilot told me about my code:

 

The 1,665-byte hybrid key_share produced a 1,755-byte ServerHello TLS record. On a conventional Ethernet network with a 1,500-byte MTU, one TCP segment commonly carries at most 1,460 bytes over IPv4, after the usual 20-byte IPv4 and 20-byte TCP headers, or 1,440 bytes over IPv6.

Note: IPv6 leaves 20 fewer bytes for TCP payload because its base header is 40 bytes which is twice the size of IPv4’s usual 20-byte header, largely because IPv6 addresses are 128 bits rather than 32 bits.

TCP options can reduce that payload further, while jumbo frames can increase it. In my capture, the first TCP payload contained 1,460 bytes, so the TLS record continued in a second segment.

The network stack and TLS handshake handled this segmentation normally, but the diagnostic parser assumed that every TLS record is contained in a single captured packet, and that is not the case here.

Conceptually, the old logic in my code did this:

 if (record_length > tcp_payload_length)
     return; // treated a valid, but segmented record as invalid

The fix adds bounded, sequence-aware TCP reassembly before parsing TLS. It retains the first part of the record, identifies subsequent data by connection tuple and TCP sequence number, ignores retransmitted bytes, and parses the  ServerHello  only after all 1,755 declared bytes have been collected.

This is the broader lesson: PQC’s larger objects can reveal hidden size and framing assumptions in capture tools, telemetry pipelines, proxies, and custom protocol parsers. Thankfully, my code just dropped the data, but other code might lead to a memory safety issue!

But this kind of issue is not unique to TLS. Below are some examples where a larger PQC packet can lead to issues.

QUIC

QUIC (RFC 9000) is a UDP-based, encrypted transport protocol that integrates TLS 1.3 directly into its handshake to establish secure connections in as little as one round trip, or zero with resumption. It provides multiplexed, independently flow-controlled streams that avoid TCP's head-of-line blocking, along with connection migration across network changes, and it serves as the transport for HTTP/3.

QUIC originated at Google and was first deployed in Chrome. Because Microsoft Edge is built on the Chromium engine, both Chrome and Edge support the standardized IETF QUIC used by HTTP/3.

QUIC's anti-amplification limit (RFC 9000 §8.1) creates another constraint.

Until a server has validated the client's address, it may send no more than three times the number of bytes it has received from that address. Because the client's first Initial datagram must be padded to at least 1,200 bytes, the server's initial budget is roughly 3,600 bytes. That budget must cover the entire first flight: the Initial packet carrying ServerHello, plus the Handshake packets carrying EncryptedExtensions, Certificate, CertificateVerify, and Finished.

Classical ECDSA chains fit comfortably within that limit. ML-DSA does not. An ML-DSA-65 certificate carries a 1,952-byte public key and a 3,309-byte signature, so it is roughly 5.3 KB per certificate. A leaf-plus-intermediate chain combined with the CertificateVerify signature comes to about 14 KB, which is several times over budget. A hybrid key share such as X25519-ML-KEM-768 adds about 1.1 KB to ServerHello, although the client's correspondingly larger ClientHello often spans two datagrams, which raises the budget to around 7,200 bytes.

When the server hits the limit, it must stall until it receives more bytes from the client, either ACKs or Probe Timeout (PTO) packets. This adds one or more round trips to the handshake and erodes QUIC's latency advantage.

So far, we have focused on low-level TCP issues, but what about HTTP[S]?

Cookies and Query Strings

PQC signatures and payloads become a problem as soon as you put them in a token that travels in a cookie or URL: a JWT/JWS session cookie, a signed URL (CDN or pre-signed object style), a signed redirect or state parameter, or a SAML/OIDC artifact passed in a query string. Classical signatures are small enough to fit anywhere; PQC signatures are not.

The table of sizes is in bytes, and with HTTP-type data, the data becomes larger courtesy of Base64 encoding! For example:

Algorithm

Raw signature

base64url

ECDSA P-256

64 B

86 chars

RSA-2048

256 B

342 chars

ML-DSA-44

2,420 B

3,227 chars

ML-DSA-65

3,309 B

4,412 chars

ML-DSA-87

4,627 B

6,170 chars

So, what breaks?

Cookies

RFC 6265 only guarantees about 4,096 bytes per cookie, name and attributes included. A single ML-DSA-65 signature already exceeds that. Browsers handle an oversized cookie by silently ignoring the Set-Cookie, so the server sees a missing session rather than an error. Frameworks that chunk large cookies across several cookies, such as ASP.NET Core's chunking cookie manager, avoid the per-cookie limit but just move the pressure to total header size.

Per-request amplification

Cookies go out on every request in its scope, including requests for static assets. A 3–6 KB signature then costs a few kilobytes on every request instead of about 100 bytes.

Header compression

HTTP/2 and HTTP/3 reduce repeated header overhead by replacing previously seen fields with compact references. HTTP/2 uses HPACK, while HTTP/3 uses QPACK, which adapts the same basic idea to QUIC’s out-of-order streams. Both rely on bounded dynamic tables, so unusually large or frequently changing values-such as cookies containing PQC signatures—compress poorly: they may be too large to index, may displace other useful entries, and may therefore be transmitted almost in full on every request.

HPACK's default dynamic table is 4,096 bytes (SETTINGS_HEADER_TABLE_SIZE described in RFC 9113 §6.5.2). A field larger than the table can never be indexed, so an ML-DSA-65 or ML-DSA-87 cookie is re-sent literally (Huffman-coded at best) on every request.

QPACK's dynamic table capacity defaults to 0 until the peer negotiates one. A cookie that does fit evicts most other entries, which hurts compression of every other header.

Query strings

Internet Information Services (IIS) request filtering defaults to maxQueryString = 2,048 and maxUrl = 4,096, so an ML-DSA-44 signed URL fails on a default IIS configuration with a 404.15 or 414.

Nginx, Apache, CDNs, and WAFs cap the request line at roughly 8 KB, which ML-DSA-87 or any SLH-DSA variant signature can blow through.

Side effects of putting signatures in URLs

URLs get written to access logs, proxy logs, browser history, and Referer headers. Large signatures cause truncated log lines, which breaks forensic replay and signature verification during incident response, and they inflate log storage. Size limits also apply to QR codes, email links, and anything users copy and paste.

Application Storage and Data Models

Applications that persist signatures, public keys, certificates, or signed objects can fail even when the cryptographic library works correctly. A database column sized for an ECDSA signature, perhaps binary(64), varbinary(128), or a short text field containing Base64, cannot hold a 2,420- to 4,627-byte ML-DSA signature, much less a 7,856-byte or larger SLH-DSA signature.

The constraint may also be hidden outside the table definition. Object-relational mappings, API schemas, validation rules, fixed-size structures, message queues, cache entries, audit records, and log pipelines often repeat the original size assumption. Larger PQC objects can increase row size, push data out of row, reduce rows per page, enlarge indexes and backups, and increase memory allocation and serialization costs. Migration testing should therefore trace each cryptographic object end to end, creation, encoding, transport, persistence, retrieval, logging, replication, and verification, and record both the algorithm identifier and the actual object length rather than treating every “signature” or “public key” as interchangeable.

Also, at 7,856 bytes, an SLH-DSA-SHA2-128s signature is already close to an 8 KiB database page. Larger signatures may require row-overflow or LOB storage, limiting indexing and increasing I/O, transaction-log, backup, and replication costs.

Here’s a wrap-up of where you might hit some application-layer issues:

  • Fixed-size buffers and structures: Arrays, stack buffers, IPC records, and native structures sized for ECDSA or RSA may truncate PQC objects or introduce memory-safety defects.
  • API and serialization schemas: OpenAPI definitions, JSON validators, ASN.1 schemas, Protocol Buffers, and generated client models may enforce outdated length limits.
  • Message queues and event systems: Larger signatures can exceed message, property, or envelope limits, especially after Base64 encoding adds roughly one-third overhead.
  • Caches and session stores: Authentication tickets, signed session state, and distributed-cache entries may exceed item limits or increase eviction and memory pressure.
  • File parsers: Fixed-width signature fields, small length fields, or single-read assumptions may reject, truncate, or incorrectly parse ML-DSA objects.

I could keep going, but you can see this is a potentially large issue.

Engineering for PQC Size Growth

PQC migration is not just an algorithm replacement. Larger key shares, public keys, ciphertexts, signatures, and certificate chains can cross boundaries that classical cryptography rarely reached. The resulting failures may appear far from the cryptographic code: segmented TLS records that parsers drop, QUIC handshakes delayed by amplification limits, cookies and URLs rejected by intermediaries, headers that no longer compress efficiently, and application objects truncated by fixed buffers, schemas, queues, caches, logs, or file formats. Encoding and protocol overhead amplify the increase further, and hybrid deployments may temporarily carry both classical and post-quantum material.

The safest engineering assumption is that cryptographic objects are variable-length, potentially much larger than today, and may arrive or be processed incrementally. Treat every boundary between components as a size-sensitive interface and test the complete path rather than only the cryptographic operation.

Practical Mitigations

The increased PQC-data size risks described above are manageable, but only if engineering teams treat size and interoperability as explicit design constraints rather than edge cases. The following practices help expose brittle assumptions early and ensure systems can accommodate larger and variable-length cryptographic objects throughout the full data path.

  • Inventory size assumptions: Search for fixed-size arrays, narrow length fields, maximum-length validators, database and schema limits, and code that assumes one packet, read, frame, or message contains a complete object.
  • Use length-aware, bounded processing: Allocate from validated lengths, use sufficiently wide length types, enforce explicit upper bounds, and reject oversized or malformed input safely rather than truncating it.
  • Support streaming and reassembly: Design parsers for partial reads, segmentation, multiple records, retransmission, and out-of-order delivery where the protocol permits it. Transport boundaries != message boundaries.
  • Keep large signatures out of repeated request metadata: Avoid placing PQC signatures directly in cookies, query strings, or frequently repeated headers. Prefer compact opaque references to server-side state when the trust and deployment model allow it.
  • Budget for encoding and container overhead: Test raw, DER, Base64 or base64url, JSON, certificate-chain, CMS, and protocol framing sizes. Size for the complete deployed object, not the primitive’s raw byte count.
  • Test realistic, worst cases end to end: Include the largest approved algorithm and parameter set, hybrid modes, full certificate chains, constrained MTUs, packet loss, proxies, CDNs, WAFs, queues, caches, logs, and older clients. Add boundary tests just below, at, and above every configured limit.
  • Measure and observe: Record negotiated algorithms and object lengths, surface truncation and size-limit failures explicitly, and monitor handshake bytes, packet counts, retries, latency, rejection rates, queue failures, and log loss.
  • Build crypto agility into interfaces: Store the algorithm identifier with the object, avoid algorithm-specific field widths, centralize policy and limits, and make algorithms and parameter sets replaceable without redesigning storage, protocols, or APIs.

For software engineers, the immediate action is simple: start testing with larger PQC objects your application may support, trace them through every encode, transport, parse, cache, store, and log boundary, and remove any assumption that was safe only because RSA and ECC objects were small. Focus on what is currently the pathological case of P-384 + ML-KEM-1024 and ML-DSA-87.

Wrap-Up

The essential engineering lesson here is to treat cryptographic objects as variable-length (potentially much larger) data and remove assumptions inherited from smaller RSA and ECC objects.

Test your solutions with the ML-KEM and ML-DSA end-to-end to make sure all intermediate steps can handle the larger payload sizes inherent in post-quantum algorithms.

Thanks!

Many thanks to my Microsoft colleagues whose feedback helped strengthen this post.

Andrei Popov – Windows Security

Raul Garcia – Microsoft Crypto Board

Published Oct 28, 2026
Version 1.0
No CommentsBe the first to comment