About HTTPS, SChannel, TLS, CAPI, SSL Certificates and their keys

So you want your site to be available over secure HTTP and you already know that you need an SSL certificate. Still, many things can go wrong. I’m not going to document a specific error; rather I’d say how SChannel, TLS, keys and CAPI fit in, and where to look for problems.

Because we set the HTTPS binding and certificate for a site in IIS Manager, we might be tempted to think that IIS is managing the Secure Channel, or SChannel, through which HTTP traffic takes place. But that’s wrong. IIS Manager is going to configure the HTTP.SYS driver, specifying the certificates to be used for HTTPS traffic. And then, HTTP.SYS is going to take care of the TLS channel for HTTPS communication. Look:

 

IIS sits on top of HTTP.SYS, which in turn relies on SChannel provider for HTTPS communication

 

Before any HTTPS traffic can happen, a TLS SChannel has to be established. Then, through that channel, normal HTTP requests and responses would travel. The secure channel is initiated by HTTP.SYS, which sits below IIS. If the SChannel fails, then requests don’t reach to IIS, they will not show up in IIS logs. With SChannel failures, HTTPS requests don’t even make it to HTTP.SYS. Why?­

The HTTP.SYS driver is going to call the SChannel provider to create the secure tunnel. SChannel will use the CAPI, the Cryptographic API of Windows. The certificate’s public and private keys are going to be used by CAPI2. Notice my highlight: the private key of the certificate is needed for the SChannel. If that private key is missing or inaccessible due to ACL (Access Control List), then “game over” – no tunnel can be created, HTTPS traffic would fail. And there are other reasons too for which the certificate may be rejected by the SChannel provider; see later in this article.

A certificate that is paired with its private key

 

The SChannel provider is logging into the Windows Events – look inside the System log with the Event Viewer, looking for source SChannel. Here are some common SChannel events and SSL/TLS protocol alerts.

The logging of the Crypto API is not turned on by default. If you want to troubleshoot, you’ll need to turn on the CAPI2 logging in the Windows Events Viewer.

 

 

 

The TLS handshake

 

Let’s see the steps of the TLS negotiation, how the secure channel is being established between the client and the server before any HTTPS traffic can happen.

1. TCP handshake is performed between client and server, then...
2. Client Hello: The client sends to server the versions of TLS protocol supported, as well as the suite of ciphers (encryption/decryption algorithms it supports for key exchange, data encryption and handshake security);
3. Server Hello: After the server compares what the client sends with its own supported TLS versions and ciphers, the server decides which ones will be used and sends the selection to the client.
4. Server certificate: The server has to authenticate itself to the client, so it sends its certificate. The keys will be used for encryption.
5. Client switches to encrypted; all following messages from client will be encrypted from this point.
6. Server switches to encrypted, too. From this point, normal HTTP requests and response can travel via the encrypted channel.

In fact, the TLS handshake is a bit more complicated, so I recommend the Pluralsight training Analyzing and Decrypting TLS Traffic for details. But the above steps are enough to illustrate what could go wrong and where to look for problems.

 

 

 

TCP handshake

 

Condition: The client (browser) has to be able to resolve the name of a site (the hostname) to an IP address. For that, needed DNS records must be in place. Do you have a DNS A or CNAME record in place? Can the IP address of the hosting environment be determined for the site/host name specified in the browser’s address bar? You can check with nslookup command-line tool:

C:> nslookup duckduckgo.com

Checking name resolution to IP addresses with nslookup

 

 

 

Client Hello and Server Hello

 

The client is sending to the server its supported TLS versions and cipher suites. The server compares the list with its own “records”. If the server does not support any of the TLS versions specified by client: fail. If or none of server's cipher suites matches one from the client’s list, boom! Look into Windows Events, in the System log, for entries sourced SChannel.

A network trace capturing TLS negotiation, Client Hello

If you suspect problems with the encryption, then that’s the domain of CAPI2 - Cryptographic API logging must be enabled, then events must be studied.

 

 

 

Server Certificate

 

The server authenticates first; then, optionally, the client. But before the server sends its certificate, that must be usable. The SChannel provider performs some validation on the certificate, before using it.

The private key of the certificate must be present on the server and accessible to SChannel provider. There is common misconception that the private key of a certificate stays within the certificate. That’s wrong; the private key belongs to a certificate, but it is stored separately. Certificates are held in Registry, while their private keys have various repositories, depending on where the certificate is found in the Certificate Manager: read key storage and retrieval and certificate locations in Windows.

The issuing authority of the certificate must be trusted by the server. The certificate of the issuing authority must be present in the Computer’s Trusted Root Certification Authorities. If there are intermediate certification authorities, their certificates must be present in the Computer’s Intermediate Certification Authorities. The certificate chain must be trusted.
 

Both client and server should trust the certification chain. Issuing root and intermediate certification autorities

The certificate must not be revoked. And its Key Usage flags must allow for Digital Signature and Key Encipherment. Working with SSL certificates using IIS Manager helps avoid many of the pitfalls commonly seen when setting up HTTPS.
 

Key usage flags needed for HTTPS

 

 

 

Encrypted communication

 

Finally, before the communication is being switched to encrypted – Change Cipher Spec – the client has to accept the server’s certificate. Some browsers, like Internet Explorer, would display warnings for sites presenting improper certificates - but would still allow you to navigate; after all, technically the encryption of the tunnel can still happen. But other browsers or other client applications might be more… stubborn, should I say? They would not allow continuing navigation or sending requests if the certificate is “smelly”.

If the certificate is expired, not valid anymore: reject. If it was revoked by its issuing authority: reject. If the site/server/host name of the URL does not accommodate the certificate’s subject (issued to field): reject. If the client does not trust the issuing authority of a certificate: reject (on Windows, the certificate of the issuing authority must be present in the Computer’s Trusted Root Certification Authorities / Intermediate Certification Authorities).

In all these cases, the encryption of the secure channel would work, technically; but the clients/browsers simply don’t like the certificate presented by the server - the server is not authenticated.
 

When a client checks a certificate. It also checks for revocation.

On a side note, only the key exchange relies on the computationally expensive asymmetric encryption, during the TLS handshake (let’s say with the certificate’s public and private key). Once the client and the server have established the secure channel, the data exchange will rely on symmetric encryption. This is why the certificate’s Key Usage flag must include Key Encipherment – it’s for the key exchange.

 

 

 

Handling certificates and their private keys

 

I keep saying about certificate private keys because their absence is one of the most common issues. Failing to have the private key of a certificate leads to frustration: IIS Manager would not even bother to display a server certificate if its private key is not present. So, in the HTTPS binding, one cannot pick a certificate if its private key is missing. Further on, if the private key is accidentally deleted, corrupted or becomes inaccessible due to rights, SChannel issues immediately appear. Same happens if the certificate chain is not trusted (somehow the certificate of the issuing authority is missing from trusted, or that certificate is expired).

So how do we handle this private key?

The “birth” of a certificate starts with a Certificate Request, a CSR. One creates a CSR, then a certification authority (CA) approves it and generates a certificate file.

When creating CSR, a key pair is generated: public and private key. The public key is, well, public, anyone could see it. But the private key remains securely stored on the machine where the CSR was generated.

Once the CA issues the certificate, we could install that certificate on a different machine. But wait, where is the private key? Oh, the certificate must be paired with its private key on the machine where the CSR was created. Once this pairing is done, both certificate and its private key can be exported in a PFX container. The PFX file may contain the certificate, its private key, and all the trust chain – certificate of the root authority and any intermediate authorities. Then the PFX can be imported on other machines if needed.
 

A certificate needs its private key to be usable for HTTPS

 

I recommend using IIS Manager only for creating certificates to be used for sites and web applications. IIS Manager ensures that the private key is exportable once the certificate was paired with it and it sets the proper key usage flags. It also ensures that the certificate is placed in the proper repository – Computer’s certificates. When exporting a certificate from IIS Manager, the entire trust chain of certificates is included in the PFX too.