Handshake failure trying to make SSL connection to MS SQL Server using Java JRE 1.6

I had a very strange Java problem today. I am posting the description of the problem and solution here in hopes that it might help out someone else in the same arcane pickle.

This morning, Windows installed two updates on my development laptop at around 5:00 am. The next time I restarted the laptop after these updates were installed, the program I work with (Oracle Xstore) stopped working. I know that it was working at 4:30 am, because I made screen shots of my latest fix to show the client. Imagine my dismay when it failed when I tried running it again at 10 am. [See actual exception text and the Windows update ids at the bottom of the post.]

Xstore would fail within 5 seconds of starting. It quit with a fatal error because it could not open any data sources. This is the kind of thing that might happen if the SQL Server service is not running. But in this case, the service was running and none of the Xstore files had been changed since before the update was installed.

My guess is that one of the updates is forcing MS SQL Server to be more cryptographically demanding when a client makes an SSL connection.

Boring details

In the case of Xstore, that client is the Microsoft JDBC driver. The failure is not the fault of the database driver, however, because Java programs rely on the Java Runtime Environment (JRE) to provide cryptographic services. No sense making everybody implement cryptography on their own.

In fact, Java allows for multiple implementations of the Java cryptography environment (JCE) API. There is a list of 9 of them that come with the JRE in the /lib/security/java.security file.

It turns out that the latest Java JRE (Java 1.8.x) can handle the higher demands of SQL Server. I tried running Xstore under the 1.8 JRE and it had no problem handshaking with MS SQL Server and opening the data sources. So why was it failing for me?

Unfortunately, our client uses an old version of Xstore (4.5.1) that requires an old version of the JRE (1.6.x). The default Sun implementations that come with the old JRE were just not up to the job. Sun made a trade-off of speed vs cryptographic strength that was not good enough anymore after Microsoft decided to strengthen the security around SSL handshaking.

[Specifically, the old Sun JCE providers limit the maximum acceptable prime size for Diffie-Hellman primes to 1024 bits - whatever that means. See http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6521495]

How to fix
The solution was to add another implementation of JCE from bouncycastle.org (a non-profit organization headquartered in Australia chock full of people who actually understand what a Diffie-Hellman prime is). I had to download their library to /lib/ext folder (i.e. c:\xstore\windows\jre\lib\ext in my case). Then I added a line to the java.security file (which is just a text file) to add the bouncy castle library to the list of acceptable JCE providers.

1. Copy bcprov-debug-jdk15on-154.jar to /lib/ext  
2. Add a line to /lib/security/java.security as shown below.

Really boring details

Below is the list of JCE provider libraries that came with JRE 1.6 (found in /jre/lib/security/java.security). I commented out the original list. This is followed by the new list with BouncyCastleProvider included. Bouncy Castle was added as provider #2, and pushed the others down numerically on the list. It is advised to leave the Sun provider at the top where the JRE expects it.

# List of providers and their preference orders (see above):

#

#security.provider.1=sun.security.provider.Sun

#security.provider.2=sun.security.rsa.SunRsaSign

#security.provider.3=com.sun.net.ssl.internal.ssl.Provider

#security.provider.4=com.sun.crypto.provider.SunJCE

#security.provider.5=sun.security.jgss.SunProvider

#security.provider.6=com.sun.security.sasl.Provider

#security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI

#security.provider.8=sun.security.smartcardio.SunPCSC

#security.provider.9=sun.security.mscapi.SunMSCAPI

security.provider.1=sun.security.provider.Sun

security.provider.2=org.bouncycastle.jce.provider.BouncyCastleProvider

security.provider.3=sun.security.rsa.SunRsaSign

security.provider.4=com.sun.net.ssl.internal.ssl.Provider

security.provider.5=com.sun.crypto.provider.SunJCE

security.provider.6=sun.security.jgss.SunProvider

security.provider.7=com.sun.security.sasl.Provider

security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI

security.provider.9=sun.security.smartcardio.SunPCSC

security.provider.10=sun.security.mscapi.SunMSCAPI

Specific details

[The Windows updates were: KB 3163018, KB 3149135; for 64-bit Windows 10]

The root cause exception that I was seeing was:

Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 1024 (inclusive)

at com.sun.crypto.provider.DHKeyPairGenerator.initialize(DashoA13*..)

at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:627)

at com.sun.net.ssl.internal.ssl.DHCrypt.(DHCrypt.java:100)

... 42 more