SSL support (Kafka 0.9+) revisit

[jabbors]

Hi,

SSL support it suppose to be working according to issue #581.

However, I've spent two days trying to get it working with Kafka 0.9.0. I can connect to to my kafka cluster with openssl s_client -connect domain.com:9093 -tls1 so I know it is configured correctly.

But when I try connecting with sarama I get errors:

[Sarama] 2016/04/20 11:16:39 client/metadata retrying after 250ms... (1 attempts remaining)
[Sarama] 2016/04/20 11:16:39 client/metadata fetching metadata for all topics from broker domain.com:9093
[Sarama] 2016/04/20 11:16:39 Failed to connect to broker domain.com:9093: tls: first record does not look like a TLS handshake
[Sarama] 2016/04/20 11:16:39 client/metadata got error from broker while fetching metadata: tls: first record does not look like a TLS handshake
[Sarama] 2016/04/20 11:16:39 client/metadata no available broker to send metadata request to
[Sarama] 2016/04/20 11:16:39 client/brokers resurrecting 1 dead seed brokers
[Sarama] 2016/04/20 11:16:39 Closing Client
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

In the kafka logs I see the following errors

[2016-04-20 11:16:39,073] DEBUG Connection with a.b.c.d/a.b.c.d disconnected (org.apache.kafka.common.network.Selector)
javax.net.ssl.SSLHandshakeException: no cipher suites in common
  at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1348)
  at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519)
  at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1200)
  at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1172)
  at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
  at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:377)
  at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:242)
  at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:68)
  at org.apache.kafka.common.network.Selector.poll(Selector.java:281)
  at kafka.network.Processor.run(SocketServer.scala:413)
  at java.lang.Thread.run(Thread.java:745)

Further investigation using Wireshark indicates that the Kafka server does not respond with a ServerHello in the TLS handshake. Which is all due to Sarama and Kafka couldn't agree on common cipher.

This issue is probably not in Sarama, more correctly in Go TLS or in Java, but I'd like to know how people claims this is working.

[eapache]

The ciphers available in go and Java are fairly standard and have quite a bit of overlap by default. If they can't find one in common then I suspect one or the other is misconfigured. Since I don't know of a way to change the available ciphers for go (short of recompiling the go toolchain from scratch) I must assume that something is configured oddly about your Java installation. By default both openjdk and oracle have plenty of common cipher suites available.

[tom-pang]

As a note, we've been using the SSL support in production from sarama for months now and it's been flawless, so I too suspect a JVM config issue.

[jabbors]

Yeah, I ran this java code http://stackoverflow.com/questions/20000020/enabled-ciphers-on-ubuntu-openjdk-7 to print the available ciphers in the JVM (both openjdk and oracle) and the list was long, with many common ciphers supported by the Go configuration. However, when I ran a sslscan against the Kafka broker the list was much shorter and none of them where common with those in Go.

So I too suspect a JVM config issue. And since I'm not a Java entusiast nor expert, I've chosen the go with default settings when installing anything Java related. So the "working" out-of-the-box Java installation I was hoping for is not a fact. Guess I have to start digging and debugging the JVM.

[jabbors]

@tcrayford: Which platform do run Kafka on, which Java and version, and what are the supported ciphers if you run a sslscan against a broker? Also which Go version and platform do you run Sarama on?

I have installed Kafka on a Ubuntu 14.04 cloud machine, tested with openjdk7 and oracle8. sslscan lists the following ciphers for openjdk7 (even less supported ciphers in oracle8):

  Supported Server Cipher(s):
Preferred TLSv1.2  256 bits  DHE-DSS-AES256-SHA256         DHE 1024 bits
Accepted  TLSv1.2  256 bits  DHE-DSS-AES256-SHA            DHE 1024 bits
Accepted  TLSv1.2  128 bits  DHE-DSS-AES128-SHA256         DHE 1024 bits
Accepted  TLSv1.2  128 bits  DHE-DSS-AES128-SHA            DHE 1024 bits
Accepted  TLSv1.2  112 bits  EDH-DSS-DES-CBC3-SHA          DHE 1024 bits
Preferred TLSv1.1  256 bits  DHE-DSS-AES256-SHA            DHE 1024 bits
Accepted  TLSv1.1  128 bits  DHE-DSS-AES128-SHA            DHE 1024 bits
Accepted  TLSv1.1  112 bits  EDH-DSS-DES-CBC3-SHA          DHE 1024 bits
Preferred TLSv1.0  256 bits  DHE-DSS-AES256-SHA            DHE 1024 bits
Accepted  TLSv1.0  128 bits  DHE-DSS-AES128-SHA            DHE 1024 bits
Accepted  TLSv1.0  112 bits  EDH-DSS-DES-CBC3-SHA          DHE 1024 bits

Tested with Go 1.6 on both OSX and Ubuntu 14.04.

[elvarb]

Might be interested in this ticket https://issues.apache.org/jira/browse/KAFKA-3647

[elvarb]

I have been testing Kafka with lots of different versions of Java but the supported server ciphers never change.

[tom-pang]

@jabbors we use Ubuntu and oracle8 and we've never had to tweak anything.

[jabbors]

@elvarb, thanks for this link, I did install the JCE as suggested but still no go. I'll post to the kafka issue and let them know I also experience the same issue.

@tcrayford, do you mind sharing your kafka configuration? Could be a trivial setup I'm overseeing as I don't get it to work.

This is currently my ssl config.

security.protocol=SSL
security.inter.broker.protocol=SSL
ssl.keystore.location=/etc/kafka/server.keystore.jks
ssl.keystore.password=*****
ssl.key.password=*****
ssl.truststore.location=/etc/kafka/server.truststore.jks
ssl.truststore.password=*****
#ssl.client.auth=none
#ssl.client.auth=requested
ssl.client.auth=required
#ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
#ssl.cipher.suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256

[tom-pang]

@jabbors we don't do much more than the standard config does, just add an SSL listener and a key/truststore

[elvarb]

@tcrayford it's currently pointing to a problem with the jks / certs that results in only the DSS ciphers being used, do you think it might be something related to how the certs are created?

[elvarb]

When you create the keystore use -keyalg RSA, then you will get the right ciphers.

[jabbors]

Thanks @elvarb, got it working with the keying RSA parameter.