Wednesday, June 15, 2016

Handshake failure trying to make SSL connection to MS SQL Server using Java JRE 1.6

I had a very strange Java problem today. I am posting the description of the problem and solution here in hopes that it might help out someone else in the same arcane pickle.

This morning, Windows installed two updates on my development laptop at around 5:00 am. The next time I restarted the laptop after these updates were installed, the program I work with (Oracle Xstore) stopped working. I know that it was working at 4:30 am, because I made screen shots of my latest fix to show the client. Imagine my dismay when it failed when I tried running it again at 10 am. [See actual exception text and the Windows update ids at the bottom of the post.]


Xstore would fail within 5 seconds of starting. It quit with a fatal error because it could not open any data sources. This is the kind of thing that might happen if the SQL Server service is not running. But in this case, the service was running and none of the Xstore files had been changed since before the update was installed.

My guess is that one of the updates is forcing MS SQL Server to be more cryptographically demanding when a client makes an SSL connection.

Boring details
In the case of Xstore, that client is the Microsoft JDBC driver. The failure is not the fault of the database driver, however, because Java programs rely on the Java Runtime Environment (JRE) to provide cryptographic services. No sense making everybody implement cryptography on their own.

In fact, Java allows for multiple implementations of the Java cryptography environment (JCE) API. There is a list of 9 of them that come with the JRE in the /lib/security/java.security file.

It turns out that the latest Java JRE (Java 1.8.x) can handle the higher demands of SQL Server. I tried running Xstore under the 1.8 JRE and it had no problem handshaking with MS SQL Server and opening the data sources. So why was it failing for me?

Unfortunately, our client uses an old version of Xstore (4.5.1) that requires an old version of the JRE (1.6.x). The default Sun implementations that come with the old JRE were just not up to the job. Sun made a trade-off of speed vs cryptographic strength that was not good enough anymore after Microsoft decided to strengthen the security around SSL handshaking.

[Specifically, the old Sun JCE providers limit the maximum acceptable prime size for Diffie-Hellman primes to 1024 bits - whatever that means. See http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6521495]

How to fix
The solution was to add another implementation of JCE from bouncycastle.org (a non-profit organization headquartered in Australia chock full of people who actually understand what a Diffie-Hellman prime is). I had to download their library to
/lib/ext folder (i.e. c:\xstore\windows\jre\lib\ext in my case). Then I added a line to the java.security file (which is just a text file) to add the bouncy castle library to the list of acceptable JCE providers.


  1. Copy bcprov-debug-jdk15on-154.jar to /lib/ext  
  2. Add a line to /lib/security/java.security as shown below.

Really boring details
Below is the list of JCE provider libraries that came with JRE 1.6 (found in /jre/lib/security/java.security). I commented out the original list. This is followed by the new list with BouncyCastleProvider included. Bouncy Castle was added as provider #2, and pushed the others down numerically on the list. It is advised to leave the Sun provider at the top where the JRE expects it.

# List of providers and their preference orders (see above):
#
#security.provider.1=sun.security.provider.Sun
#security.provider.2=sun.security.rsa.SunRsaSign
#security.provider.3=com.sun.net.ssl.internal.ssl.Provider
#security.provider.4=com.sun.crypto.provider.SunJCE
#security.provider.5=sun.security.jgss.SunProvider
#security.provider.6=com.sun.security.sasl.Provider
#security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI
#security.provider.8=sun.security.smartcardio.SunPCSC
#security.provider.9=sun.security.mscapi.SunMSCAPI

security.provider.1=sun.security.provider.Sun
security.provider.2=org.bouncycastle.jce.provider.BouncyCastleProvider
security.provider.3=sun.security.rsa.SunRsaSign
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
security.provider.10=sun.security.mscapi.SunMSCAPI

Specific details
[The Windows updates were: KB 3163018, KB 3149135; for 64-bit Windows 10]

The root cause exception that I was seeing was:
Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 1024 (inclusive)
at com.sun.crypto.provider.DHKeyPairGenerator.initialize(DashoA13*..)
at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:627)
at com.sun.net.ssl.internal.ssl.DHCrypt.(DHCrypt.java:100)
... 42 more

7 comments:

Anonymous said...

Thanks for posting this - very helpful!

Xstore is not the only application out there that has this issue with the recent Windows 10 security update.

Anonymous said...

Thanks for posting this - very helpful!

Xstore is not the only app out there having problems with this Win 10 security update.

Jimhead said...

A recent Windows update (found on 17/07/17) broke the JDBC connection again with the error "Unsupported curveId: 29". I implemented the following fix to work around it: https://github.com/tobszarny/ssl-provider-jvm16/releases​

shiva chethan said...

After using JCE provider libraries getting an error "Unsupported curveId: 29". Did anyone faced this error? Above link provided by Jimhead doesn't exist. Can any of you please assist to resolve this error.

milkchaser said...

Shiva, that link worked for me. You might want to try it again.

Jimhead said...

Yep - still works for me. I recently tried getting SQL Server 2017 working using JRE1.6. I forget the JDBC error created but I gave up trying to resolve it after an hour. Luckily the POS system I implement/support (funny that I found this original post on a retail tech post) is pretty much database agnostic so I went back to SQL2012 or SQL2008 if I remember correctly.

milkchaser said...

I support POS, too. Xstore. What system do you support?