During TLS handshake, client machine makes a request to IIS server by sending Client Hello package. Then the server responds it with Server Hello.
If the server is not sending Server Hello package, the communication won’t be established and the customer is likely to see error messages below:
Internet Explorer:
This page can’t be displayed
Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try connecting to again
Chrome:
This site can’t provide a secure connection
The website sent an invalid response
ERR_SSL_PROTOCOL_ERROR
Solution
Keep in mind that the TLS protocol errors above might be misleading. However, you should start the troubleshooting by analyzing TLS negotiation first.
Here are the steps to go through:
- Make sure the website binding in IIS for 443 port has a certificate attached
- Make sure all TLS versions are enabled in Internet Explorer (This is for testing. You can later disable the unsecure versions)
- Check SCHANNEL registry keys to make sure what you set in Internet Explorer is applied at registry level
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols - Collect a network trace. Check if the client and server are agreeing on a cipher suite. If they are not, make sure the client’s cipher suite list matches with the server's list. Related Group Policy setting:
Computer Configuration > Administrative Templates > Network > SSL Configuration Settings > SSL Cipher Suite Order - If the issue sill exists, look for any network device (proxy, firewall, load balancer etc.) in between that might be intercepting or blocking TLS traffic
If you don’t see any issue with TLS protocol and network devices, continue the troubleshooting by focusing on the web server:
- Check the website binding in IIS. The certificate should be valid
- Make sure the port 443 is being listened (netstat -an -p TCP | find /I "listening")
- Change the port number to 444 and test. If it is accessible, it means there is a software blocking 443 port
- Disable Windows Firewall (If it works, you can enable it back and set up rules accordingly)
- Look for any third-party applications in the server such as Symantec Endpoint Security and Symantec Data Center Security Server Agent (Security Server Agent uses port 443). Uninstall them (Don’t just disable them. Uninstall them completely. If it works, you can install them back and configure them accordingly)
- Check if there is any Microsoft software that is using port 443. Applications like SQL Server Reporting Services (SSRS) and Windows Admin Center might interfere the port 443.