Microsoft JDBC driver 1.2: sqljdbc driver fails to establish secure connection

First published on MSDN on Dec 01, 2011

If you are using the Microsoft JDBC driver 1.2 for your NetWeaver AS Java, you can get the following error message when your application server tries to connect to SQL Server:

SEVERE: com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: The TDS protocol stream is not valid.
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSChannel.throwSSLConnectionFailed(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.loginWithoutFailover(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(Unknown Source)

This is a known issue of this version of the Microsoft JDBC driver. This version of the driver cannot use SSL if the SQL Server uses a certificate that is larger than 4KB. Even if the NetWeaver AS Java does not use SSL to connect to your SQL Server, the login information (user and password) are still encrypted.

The recommendation is to switch to a newer version of the Microsoft JDBC driver. This solution is valid for SAP systems based on NetWeaver 7.1 and higher.

This blog entry is an addendum to SAP Note 1428134 .

SAP systems based on NetWeaver 7.1 and higher

Some releases of NetWeaver 7.1 are still using the Microsoft JDBC 1.2 driver. If you get the error above, update your driver to Microsoft JDBC 2.0. For details on how to update your driver, see SAP Note 1433363

SAP systems based on NetWeaver 7.0x

Since these systems are using Java 1.4, you cannot update your Microsoft JDBC driver because the newer JDBC drivers will require at least Java 1.5.

If you configured SQL Server to use a specific certificate you can either use another certificate, or reset the configuration to the default.

If you have not configured SQL Server to use a specific certificate, SQL Server might still use an installed certificate if the certificate fulfills the requirements.

To prevent SQL Server from automatically loading an installed certificate, you can remove the certificate if you are sure that you do not need the certificate, or you can remove the access permissions on the certificate for the SQL Server service account user.

1. 
   Check if SQL Server uses an installed certificate.
   Open the SQL Server error log. If your SQL Server loads an installed certificate, it writes the following log entry:
   
   The certificate [Cert Hash(sha1) "<thumbprint>"] was successfully loaded for encryption.
   
   To locate the certificate, you need the <thumbprint> of the log entry.
   If you do not have such a log entry but an entry like
   
   A self-generated certificate was successfully loaded for encryption
   
   SQL Server uses a self-signed certificate that should not lead to this error.


2. 
   Find the certifcate.
   Open a Microsoft Managament Console (Start -> Run.. -> mmc).
   
   
   Add the certificate snap-in for the Computer Account of the Local Machine.
   
   
   Find the certificate with the thumbprint of the log entry.
   
   
   Double-click on a certificate, and select the tab "Details". You can see the thumbprint in the field "Thumbprint".
   
   
   If you are sure that you do not need this certificate, you can delete it. If you are not sure, or you need the certificate, proceed with the next step.


3. 
   Remove access permission on the private key file.
   If you need the certificate, you can also remove access permissions on the private key of the certificate.
   The private key is a file on the hard drive of your machine. You find it with the following PowerShell command (in one line):
   
   $env:ALLUSERSPROFILE + "\Application Data\Microsoft\Crypto\RSA\MachineKeys\" + (((gci cert:\LocalMachine\my | ? {$_.thumbprint –like "<thumbprint>"}).PrivateKey).CspKeyContainerInfo).UniqueKeyContainerName
   
   
   
   Open the folder that contains the private key file and right-click on the file and select "Properties".
   
   Select the "Security" tab and click on “Edit”.
   
   
   Add the SQL Server service account.
   
   
   Select "Deny Full Control".
   
   
   Close the dialogs with the OK button.
   To activate the changes, restart the SQL Server service.


4. 
   Check again
   Check if SQL Server now creates a self-signed certificate by checking the error log for the following entry:
   
   A self-generated certificate was successfully loaded for encryption.