This is the second part of our series of articles about troubleshooting TLS / SSL communications problems when you make Http Web Request or WCF queries from your ASP.NET applications to SSL endpoints.
In our first scenario, we troubleshooted a "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel" error message. We made some basic tests, such as a "browser test" and found that the certificate was not valid.
Now we have faced another problem but the browser test did not show any problem so the certificate seems valid in our browser test, and, we are still seeing the very same error message telling us that the existing connection is forcibly closed by the remote host. Remember that our ASP.NET web application is making an HTTP web request to https://iis85.buggybits.com/ URL so our ASP.NET web application is acting as a client here.
About "browser tests"...
Although a browser can be used to test the communication, this may not be the safest way to test each and every scenario because this test will use current user's certificate store. When ASP.NET application is used it will use the computer account's store. This difference may cause different behaviors.
Also, when testing with browser, we rely on the browser TLS negotiation settings and its choices during the SSL handshake. When it is an ASP.NET application that runs as a client, depending on the .NET configuration, TLS negotiation in the SSL handshake may behave differently.
Since we don't have a good information in our browser test, we move to the second step of our troubleshooting which is to check if the TCP and SSL handshake is done without a problem. Note that the TCP and SSL handshake happens whenever an HTTPS request made.
A few words about SSL handshake
SSL handshake is greatly explained here, so I will not dive into all of the details. I suggest you to read it if your unfamiliar with it.
Here is what happens after a TCP handshake as a summary:
- Client sends a CLIENT HELLO package to the server and it includes the SSL / TLS versions and the cipher suites it supports.
- Then the server responds with a SERVER HELLO package which includes the SSL / TLS versions and the cipher suits that it supports.
- If the client sends a TLS version lower than the server supports the negotiation fails.
- If the server responds with a lower TLS version and if the client supports that TLS version, SSL handshake continues with that TLS version. This is called TLS fallback. For example, if the client supports both TLS 1.0 and TLS 1.2, and the server supports only TLS 1.0, the SSL handshake may start with TLS 1.2 by client, and then it may actually happen in TLS 1.0 when server replies with "I support TLS 1.0 and let's continue with that" message.
- Cipher suite negotiation also happens here.
Now it is a good time to capture a network trace from the machine where our client, ASP.net application, is hosted to check TCP and SSL handshake.
Note that it is always suggested to capture the network trace from both client and the server (even from the intermediate devices or servers, such as proxies, when possible), because capturing network traces from different nodes (e.g.: client and the server) could help analyzing the issue more detail. That is because you can check if the same traffic / package sent from the client arrives at the remote server without a problem. In our case it is OK to capture and take a look at the trace from client machine as we are currently trying to understand if TCP and SSL handshake is done sucessfully.
I personally choose Network Monitor but some of you may prefer Wireshark over Network Monitor. It is up to you to choose the tool to capture and analyze a network trace.
You can see a screenshot of the network trace I captured while I reproduce the same problem:
(Please note that the TMG is just the name of the server, and not the "Threat Management Gateway", sorry for the confusion)
Trace above shows that the TCP handshake is successfull (the first three frames). Then the client sends the "Client Hello" in the fourth frame (frame # 818) but instead of a "Server Hello" reply it receives a RESET package from the server as seen in the sixth frame (frame # 822) – you can see the R flag, which stands for TCP RST (reset) is indicated on the frame. So SSL handshake is failing here.
We know that the TLS and cipher suites are negotiated in the SSL handshake so those are the first things to check. If we look at the Client Hello details in the trace we see that the client sends TLS 1.0 request to the server:
This is interesting because the machine runs the ASP.NET application supports TLS 1.2 and we can confirm that from the SChannel registry at the following key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols (ref.: https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/manage-ssl-protocols-in-ad...:(
As seen in the screenshot above, TLS 1.2 is enabled as both client and server. If we check the other TLS protocols we confirm that TLS 1.0, 1.1 and 1.2 are all enabled.
Here is a theory comes at this point: our ASP.NET client tries communicating via TLS 1.0 but the remote server does not support it. How can we confirm our theory?
Remember our first troubleshooting step - browser test. It was working fine if we browse the same URL in a browser from our client (where our ASP.NET application runs). So we have a working scenario and let's take a look at the network trace taken for our browser test and see what the client requests and what the server responds (again, the TMG is just the name of the server, and not the "Threat Management Gateway", sorry for the confusion):
As seen above, there is Client Hello and server response with Server Hello and the certificate without its private key then the handshake suceeds. Let's take a closer look at the Client Hello (frame # 256) first:
This time, in sucessfull scenario with Ineternet Explorer, we are seeing that the client starts negotiation with TLS 1.2, actually it is the strongest version of the TLS supported on the machine. Now let's take a look at what server responds within the Server Hello:
Bingo! Server supports TLS 1.2. Then it is expected to fail if ASP.NET client tries to negotiate over TLS 1.0 which is not supported by the server.
IE just works fine because it uses the same TLS version with server but it is interesting to see that the ASP.NET client goes with TLS 1.0, instead of stronger version of TLS 1.2, even both IE and the ASP.NET client runs on the same server.
A quick word about Internet Explorer's TLS settings
If you open Internet Explorer's advanced settings you can configure which SSL and TLS versions that Internet Explorer can use:
So Internet Explorer uses the strongest version of the protocol. Then why does not ASP.NET use the strongest one? Here comes a security update in the picture:
Microsoft Security Advisory 2960358 - Update for Disabling RC4 in .NET TLS
This update disables the RC4 and SSL 3.0 for .NET Framework and there are updates for different .NET Framework versions. As described in the article you can use the following registry key to force the usage of the strongest TLS version:
For 32-bit applications on 32-bit systems and 64-bit applications on x64-based systems:
For 32-bit applications on x64-based systems:
Since we are using .NET 4.6.x, instead of setting the registry key we can just set targetFramework=4.6 in the web.config file to force using the strongest TLS version:
<compilation targetFramework="4.6" /> <httpRuntime targetFramework="4.6" />
Note: Please make sure you read and understand the following article:
Transport Layer Security (TLS) best practices with the .NET Framework
After setting the targetFramework in my application the TLS version negotiation started to work as expected, as confirmed in the network trace screenshot below:
(Once again, please note that the TMG is just the name of the server, and not the "Threat Management Gateway", sorry for the confusion)
Now if you check the Client Hello and Server Hello details you would see that TLS negotiation is suceeded as both are using TLS 1.2:
I would like to say that "we nailed it" but no, not at all. It is still not working. If you look at closely to the network trace, this time client is sending a FIN package and closing the TCP connection (frame # 185). Why is that? Let's analyze it in our last scenario.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.