SSL/TLS connection issue troubleshooting guide
You may experience exceptions or errors when establishing TLS connections with Azure services. Exceptions are vary dramatically depending on the client and server types. A typical ones such as "Could not create SSL/TLS secure channel." "SSL Handshake Failed", etc. In this article we will discuss common causes of TLS related issue and troubleshooting steps.
Troubleshooting Azure Redis Connectivity Issues
Scenario: You want to test the connectivity of REDIS endpoint from your machine using the non-SSL port 6379 or SSL port 6380 as per requirement. Actions: There are different options available to test the connectivity of your Azure Redis cache endpoint from your machine. Let’s check on few of them. 1. Test the connectivity using REDIS CLI and STUNNEL You can test the connectivity for REDIS for your machine on non-SSL port 6379 and SSL port 6380 using REDIS CLI tool. To test the connectivity to non-SSL port, kindly use the below syntax: redis-cli.exe -h <YOUR CACHE ENDPOINT> -p 6379 -a <ACCESS KEY> To test the connectivity on SSL port 6380, you need to make use of STUNNEL and make an entry in the configuration file as depicted below. Please reload the configurations after making the entry. In case the entry is missing or the configuration isn’t loaded correctly, you may receive errors like “No connection could be made because the target machine actively refused it”, “Connection Reset by Peer” or “Unknown Error” etc. Once configuration has been loaded successfully, try connecting to REDIS using the below syntax: redis-cli.exe -p 6380 -a <ACCESS KEY> In both the above tests, we made of simple PING command and received a PONG back. 2. Test the connectivity using PSPING You can test whether you are getting response from the REDIS end point on the desired port or not by making use of PSPING. The tool can be downloaded from the below link: https://docs.microsoft.com/en-us/sysinternals/downloads/psping Please try the below syntax to test the connectivity and check on the number of packets being sent and received as part of testing. psping -q <YOUR CACHE ENDPOINT>:<Port Number> If the number of sent packets are not equal to the received packets, it points to drop in connectivity. 3. Test the connectivity using Port Query tool You can also make use of the Port Query tool in order to test the connectivity and the tool can be downloaded from the below link: https://www.microsoft.com/en-in/download/details.aspx?id=24009. You need to mention the REDIS endpoint in the destination section and provide the port for which the connectivity needs to be tested. If the port is open, you will be getting the status as ‘LISTENING’ as shown below and if the port is blocked, the status will be shown as ‘FILTERED’. The below 2 screenshots depicts how the status will get reflecting depending upon the port status. If you are not getting successful response from any of the above methods, please try checking on the below parameters to isolate the issue further. 4. Checking the Firewall Rule Check if there is any firewall rule configured over the REDIS. This can be checked from the Azure Portal under the Settings blade. Below is the screenshot for the same: If there is/are any rule configured, the connection to REDIS will only be allowed from the mentioned IP or the IP ranges. In case you try to connect from an IP outside the firewall rule mentioned, you will get an error. Below is an example of a connectivity test using REDIS CLI from an IP outside the firewall rule. 5. Checking the VNET configurations and NSG rules In case you are using a Premium cache which has a VNET configuration assigned to it, the most common test can be done from the REDIS console. Incase there are any missing configurations, you will get and error like below : In this case, you can try validating the below pointers: The client application/source and the REDIS are under same regions. The below example shows to check the location/region configuration of a VM and a Azure REDIS instance The client application/source and the REDIS are under same VNET. The below example shows to check the Virtual Network configuration of a VM and a Azure REDIS instance. You can check if a VNET is assigned to your cache or not from the “Virtual Network” section under the Settings blade from the Azure Portal . In case the client application/source are under different VNET’s, both the VNET’s should have VNET peering enabled within the same Azure region. Global peering is not supported yet. All the Inbound and Outbound rules are in place as per the requirement. From isolation standpoint, the above mentioned one is a sample rule that can added be to both inbound & outbound configurations in order to allow all the ports/protocols/sources and destination and test the connectivity. Please note that the rule precedence or “Priority” should be higher for rule to get implemented. 6. Check REDIS endpoint being whitelisted Last but not the least, in case you are using a firewall or proxy in your network, please ensure that the REDIS endpoint is whitelisted with the port number i.e. *.redis.cache.windows.net should be whitelisted for the port 6379 and port 6380 as per requirement. Hope this helps!
Connect to Azure Cache for Redis using SSL Port 6380 from Linux VM
Scenario: You are using a Linux VM and you want to connect to Azure Cache for Redis using SSL Port 6380. Action: You can connect to Azure Cache for Redis using SSL Port with the help of Stunnel and Redis-cli. The steps are as follows: Step 1: Install the Redis-cli tool in your Linux machine. The command is as below: sudo apt-get update sudo apt-get install redis-tools Note: redis-tools package has redis-cli tool as well among other tools. Step 2: Since the redis-cli doesn’t support SSL port (6380), we can make use of stunnel to connect to Azure Cache for Redis using SSL port. We have version 4 of the utility, called stunnel4 which can be installed using the below command: sudo apt-get install stunnel4 Note: If you want to run the Redis using non SSL port 6379, in that case you do not need stunnel and you can directly access using the below command provided non-ssl port is open in Azure Cache for Redis: redis-cli -p 6379 -a <Your Access Key for Azure Cache for Redis> -h < yourcachename.redis.cache.windows.net> Step 3: To configure the service to start at booting, you must modify the /etc/default/stunnel4 file using the below command: sudo nano /etc/default/stunnel4 This opens a file where you have a variable ‘ENABLED’ which must be set to 1 to enable the service to start as shown below: You can save the changes with CTL+X and then pressing ENTER. Step 4: We need to configure the Azure Cache for Redis for redis-cli which must be mentioned in Redis configuration file of stunnel. Execute the below command: sudo nano /etc/stunnel/redis.conf This creates a new file where add the following entry and insert the actual name of your Azure Cache for Redis in place of yourcachename. [redis-cli] client = yes accept = 127.0.0.1:6380 connect = yourcachename.redis.cache.windows.net:6380 Save the file. Step 5: Now, we have configured the stunnel and hence need to restart the service which can be done with the help of below command: sudo systemctl restart stunnel4.service Step 6: If you check the services listening for connections on your Redis, you should see stunnel listening on port 6380 as below: sudo netstat -plunt Step 7: Now you can connect to Azure Cache for Redis using SSL port with the help of Redis-cli. Below is the command: redis-cli -p 6380 -a <Your Access Key for Azure Cache for Redis> You can see that Redis gets connected successfully and you will be able to perform operations on Azure Cache for Redis: Hope this helps!Common causes of SSL/TLS connection issues and solutions
In the TLS connection common causes and troubleshooting guide (microsoft.com) and TLS connection common causes and troubleshooting guide (microsoft.com), the mechanism of establishing SSL/TLS and tools to troubleshoot SSL/TLS connection were introduced. In this article, I would like to introduce 3 common issues that may occur when establishing SSL/TLS connection and corresponding solutions for windows, Linux, .NET and Java. TLS version mismatch Cipher suite mismatch TLS certificate is not trusted TLS version mismatch Before we jump into solutions, let me introduce how TLS version is determined. As the dataflow introduced in the first session(https://techcommunity.microsoft.com/t5/azure-paas-blog/ssl-tls-connection-issue-troubleshooting-guide/ba-p/2108065), TLS connection is always started from client end, so it is client proposes a TLS version and server only finds out if server itself supports the client's TLS version. If the server supports the TLS version, then they can continue the conversation, if server does not support, the conversation is ended. Detection You may test with the tools introduced in this blog(TLS connection common causes and troubleshooting guide (microsoft.com)) to verify if TLS connection issue was caused by TLS version mismatch. If capturing network packet, you can also view TLS version specified in Client Hello. If connection terminated without Server Hello, it could be either TLS version mismatch or Ciphersuite mismatch. Solution Different types of clients have their own mechanism to determine TLS version. For example, Web browsers - IE, Edge, Chrome, Firefox have their own set of TLS versions. Applications have their own library to define TLS version. Operating system level like windows also supports to define TLS version. Web browser In the latest Edge and Chrome, TLS 1.0 and TLS 1.1 are deprecated. TLS 1.2 is the default TLS version for these 2 browsers. Below are the steps of setting TLS version in Internet Explorer and Firefox and are working in Window 10. Internet Explorer Search Internet Options Find the setting in the Advanced tab. Firefox Open Firefox, type about:config in the address bar. Type tls in the search bar, find the setting of security.tls.version.min and security.tls.version.max. The value is the range of supported tls version. 1 is for tls 1.0, 2 is for tls 1.1, 3 is for tls 1.2, 4 is for tls 1.3. Windows System Different windows OS versions have different default TLS versions. The default TLS version can be override by adding/editing DWORD registry values ‘Enabled’ and ‘DisabledByDefault’. These registry values are configured separately for the protocol client and server roles under the registry subkeys named using the following format: <SSL/TLS/DTLS> <major version number>.<minor version number><Client\Server> For example, below is the registry paths with version-specific subkeys: Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client For the details, please refer to Transport Layer Security (TLS) registry settings | Microsoft Learn. Application that running with .NET framework The application uses OS level configuration by default. For a quick test for http requests, you can add the below line to specify the TLS version in your application before TLS connection is established. To be on a safer end, you may define it in the beginning of the project. ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 Above can be used as a quick test to verify the problem, it is always recommended to follow below document for best practices. https://docs.microsoft.com/en-us/dotnet/framework/network-programming/tls Java Application For the Java application which uses Apache HttpClient to communicate with HTTP server, you may check link How to Set TLS Version in Apache HttpClient | Baeldung about how to set TLS version in code. Cipher suite mismatch Like TLS version mismatch, CipherSuite mismatch can also be tested with the tools that introduced in previous article. Detection In the network packet, the connection is terminated after Client Hello, so if you do not see a Server Hello packet, that indicates either TLS version mismatch or ciphersuite mismatch. If server is supported public access, you can also test using SSLLab(https://www.ssllabs.com/ssltest/analyze.html) to detect all supported CipherSuite. Solution From the process of establishing SSL/TLS connections, the server has final decision of choosing which CipherSuite in the communication. Different Windows OS versions support different TLS CipherSuite and priority order. For the supported CipherSuite, please refer to Cipher Suites in TLS/SSL (Schannel SSP) - Win32 apps | Microsoft Learn for details. If a service is hosted in Windows OS. the default order could be override by below group policy to affect the logic of choosing CipherSuite to communicate. The steps are working in the Windows Server 2019. Edit group policy -> Computer Configuration > Administrative Templates > Network > SSL Configuration Settings -> SSL Cipher Suite Order. Enable the configured with the priority list for all cipher suites you want. The CipherSuites can be manipulated by command as well. Please refer to TLS Module | Microsoft Learn for details. TLS certificate is not trusted Detection Access the url from web browser. It does not matter if the page can be loaded or not. Before loading anything from the remote server, web browser tries to establish TLS connection. If you see the error below returned, it means certificate is not trusted on current machine. Solution To resolve this issue, we need to add the CA certificate into client trusted root store. The CA certificate can be got from web browser. Click warning icon -> the warning of ‘isn’t secure’ in the browser. Click ‘show certificate’ button. Export the certificate. Import the exported crt file into client system. Windows Manage computer certificates. Trusted Root Certification Authorities -> Certificates -> All Tasks -> Import. Select the exported crt file with other default setting. Ubuntu Below command is used to check current trust CA information in the system. awk -v cmd='openssl x509 -noout -subject' ' /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt If you did not see desired CA in the result, the commands below are used to add new CA certificates. $ sudo cp <exported crt file> /usr/local/share/ca-certificates $ sudo update-ca-certificates RedHat/CentOS Below command is used to check current trust CA information in the system. awk -v cmd='openssl x509 -noout -subject' ' /BEGIN/{close(cmd)};{print | cmd}' < /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem If you did not see desired CA in the result, the commands below are used to add new CA certificates. sudo cp <exported crt file> /etc/pki/ca-trust/source/anchors/ sudo update-ca-trust Java The JVM uses a trust store which contains certificates of well-known certification authorities. The trust store on the machine may not contain the new certificates that we recently started using. If this is the case, then the Java application would receive SSL failures when trying to access the storage endpoint. The errors would look like the following: Exception in thread "main" java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at org.example.App.main(App.java:54) Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309) Run the below command to import the crt file to JVM cert store. The command is working in the JDK 19.0.2. keytool -importcert -alias <alias> -keystore "<JAVA_HOME>/lib/security/cacerts" -storepass changeit -file <crt_file> Below command is used to export current certificates information in the JVM cert store. keytool -keystore " <JAVA_HOME>\lib\security\cacerts" -list -storepass changeit > cert.txt The certificate will be displayed in the cert.txt file if it was imported successfully.
