Basic socket communication
Writing a client/server application communicating over sockets is pretty easy in Java. The server creates an instance of the aptly named ServerSocket class, which waits for incoming connections. The client creates a Socket instance, which then connects to the server (thus creating another Socket on the server side via the accept method of the ServerSocket), and that’s it. Although Sockets are normally blocking read operations, it is easy to emulate a single threaded non-blocking communication with simply checking the available bytes on the socket, and reading only that much.
Blocking secure socket communication, the problem
Switching to a secure socket connection is very easy: First, you need to create some keystores, then some initialization in the code and then you can use some factory classes to create the secure sockets instead of the original constructor calls. Everything else is almost the same.. except..
Getting the number of available bytes are not reliable any more. The reason is simple: As your security and application data arrives in the same stream, the security engine is not able to tell the amount of available application bytes without trying to decrypt everything, but that is a complex operation. Probably more complex and requires more effort than you would like to have in an infinite loop.
Because of this, our workaround for blocking communication is not usable anymore. For a simple application with just one or a handful of clients it is acceptable to have a separate thread for all of the connected clients on the server side, but we need another solution for high user counts.
SocketChannels
Java NIO has a nonblocking socket solution. SocketChannels are designed for this. Because of the fundamentally different usage and the additional classes, such as Selectors, most people never uses them, but they are not overly complex, and with a little practice can be used easily. For beginners, I’d recommend reading this article: Rox Java NIO Tutorial. It’s probably not the easiest to understand, but contains very useful tips and explains typical pitfalls.
Using this, we solve the issue with the blocking channels, but security is still lacking. The bad news is: It is pretty hard to make SocketChannels secure. This is your second chance to turn back, and use the blocking solution, they say a few thousand threads are not really a problem for a good server 
Securing SocketChannels
Ok, you are desperate. You want ‘em non-blocking channels communicate securely. Here is what you need to add to the code:
- You don’t need to do anything with the ServerSocketChannel, and the way it accepts connections.
- I prefer doing the reading and writing in a separate class, which “owns” the connection and the SSLEngine. (See Below)
- Initialize your security.
- On the server side, for every new socketchannel, create a new SSLEngine.
- On the client side, do the same for your one channel.
- Initiate the handshake.
- After the handshake, wrap everything you want to send and unwrap everything that you receive on both sides.
So far it is not too complex, right? Let’s dig deeper.
Initializing security:
private void setupSecurity() {
SecureRandom secureRandom = new SecureRandom();
secureRandom.nextInt();
KeyStore clientKeyStore = KeyStore.getInstance("JKS");
clientKeyStore.load(new FileInputStream("client.jks"), "KeyStorePassword".toCharArray());
TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
tmf.init(clientKeyStore);
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(clientKeyStore, "KeyInKeystorePassword".toCharArray());
sslContext = SSLContext.getInstance("TLS");
sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), secureRandom);
}
Couldn’t be more straightforward. Creating an SSLEngine is just as easy:
SSLEngine engine = sslContext.createSSLEngine();
And now the handshake.. You can find a lot of explanations here (it’s a bit dry, but very useful). The scaffolding for the code can be found in the same article, but you are better off if you check the code here: [jdk home]\sample\nio\server\ChannelIOSecure.java
It is very complicated, and there is a hidden “flaw”: It’s meant to be used with blocking channels. (See below). Also, some structures are a bit more complex than I’d usually like it, for example the 3 levels deep switch constructs. Anyway, for now you should just try to copy it as closely as you can, and if you can make it work, then you can improve the code as much as you want.
The problems I faced
Obviously, it didn’t work for me the first time I tried. Here are some tips:
- Don’t give up. It’s complex and ugly, but it can be done.
- If the handshake doesn’t start, and/or the server complains about plaintext connection, insert the engine.beginHandshake() line before the handshake process. Sometimes the only issue that you see is that the clients sends the messages and the server simply absorbs them without any apparent effect.
- If you get repeated buffer underflow errors on the server side, and using non-blocking communication, it’s time to add a new check for zero length reads. Trying to unwrap zero length ByteBuffers can cause this issue.
- In case of unsupported record version problems or sequence violation (during the handshake) try to empty your buffers before you read/unwrap into them and after you have sent them to the other party. Leftover bytes can cause this issue. (Unsupported record version Unknown) (Handshake message sequence violation)
- If the server/client complains about invalid handshake mac, check your flushing code. You may loose some important bytes. (bad handshake record MAC)
- Google is your friend: There are a lot of people out there who solved the same issues and wrote an article about it.
- It’s a good idea to add some logging to the code to see what is happening. It helps you to see the sequence of events on both sides.
- There are some issues and solutions here I didn’t face. Still, you may see them.
I hope this will help some of you to develop some great, secure applications.
