Hi, I have been testing upgrade to 4.0, I started out with a cluster with 3.0.15 and server encryption enabled. Due to some issues in my environment I did upgrade one of the nodes to 3.11.3, I think this turned out to be a good thing since I could observer the behaviour of upgrading from both 3.0.15 and 3.11.3 at the same time.
At first I didn't have much success at all, it look like found multiple issues mostly with server encryption so I decided to simplify thing and disabled server encryption. So with server encryption disabled the upgrade was working ok, what I did notice was exceptions in the 3.0.15 and 3.11.3 nodes once the first 4.0 node started. 3.0.15 exception: 2018-10-22T11:05:38.883+0200 ERROR [MessagingService-Incoming-/10.216.193.244] CassandraDaemon.java:223 Exception in thread Thread[MessagingService-Incoming-/10.216.193.244,5,main] java.lang.RuntimeException: Unknown column coordinator_port during deserialization at org.apache.cassandra.db.Columns$Serializer.deserialize(Columns.java:433) ~[apache-cassandra-3.0.15.jar:3.0.15] at org.apache.cassandra.db.filter.ColumnFilter$Serializer.deserialize(ColumnFilter.java:447) ~[apache-cassandra-3.0.15.jar:3.0.15] at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:647) ~[apache-cassandra-3.0.15.jar:3.0.15] at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:584) ~[apache-cassandra-3.0.15.jar:3.0.15] at org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50) ~[apache-cassandra-3.0.15.jar:3.0.15] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[apache-cassandra-3.0.15.jar:3.0.15] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) ~[apache-cassandra-3.0.15.jar:3.0.15] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.0.15.jar:3.0.15] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.0.15.jar:3.0.15] 3.11.3 exception: 2018-10-22T11:12:05.060+0200 ERROR [MessagingService-Incoming-/10.216.193.244] CassandraDaemon.java:228 Exception in thread Thread[MessagingService-Incoming-/10.216.193.244,5,main] java.lang.RuntimeException: Unknown column coordinator_port during deserialization at org.apache.cassandra.db.Columns$Serializer.deserialize(Columns.java:452) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.db.filter.ColumnFilter$Serializer.deserialize(ColumnFilter.java:482) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:760) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:697) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94) ~[apache-cassandra-3.11.3.jar:3.11.3] I got one or two of these exceptions every minute, I could not see that this was causing any problems but I don't really like exceptions in the log. Should we expect exceptions like this during an upgrade or is this a problem we should look in to? So now I enabled server encryption again, with this configuration: storage_port: 12700 ssl_storage_port: 12701 server_encryption_options: # set to true for allowing secure incoming connections enabled: true # If enabled and optional are both set to true, encrypted and unencrypted connections are handled on the storage_port optional: false # if enabled, will open up an encrypted listening socket on ssl_storage_port. Should be used # during upgrade to 4.0; otherwise, set to false. enable_legacy_ssl_storage_port: true # on outbound connections, determine which type of peers to securely connect to. 'enabled' must be set to true. internode_encryption: all keystore: /usr/share/cassandra/.ssl/cil-intern/server/keystore.jks keystore_password: '*********' truststore: /usr/share/cassandra/.ssl/cil-intern/server/truststore.jks truststore_password: '**********' # More advanced defaults below: protocol: TLSv1.2 # store_type: JKS cipher_suites: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA] # require_client_auth: false # require_endpoint_verification: false As far as I understand this configuration is correct but the 4.0 node can't contact the old nodes. I have tried with the default protocol setting also but it makes no difference. The behaviour is different in 3.0.15 and 3.11.3, in 3.0.15 I get this exception: 2018-10-22T11:57:44.366+0200 ERROR [MessagingService-NettyInbound-Thread-3-8] InboundHandshakeHandler.java:300 Failed to properly handshake with peer /10.216.193.246:58296. Closing the channel. io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: SSLv2Hello is disabled at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:417) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:317) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLHandshakeException: SSLv2Hello is disabled at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:637) at sun.security.ssl.InputRecord.read(InputRecord.java:527) at sun.security.ssl.EngineInputRecord.read(EngineInputRecord.java:382) at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:962) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1275) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ... 14 common frames omitted and in the 4.0 debug log I see this related to the 3.0.15 node: 2018-10-22T13:25:34.414+0200 DEBUG [ScheduledFastTasks:1] OutboundMessagingConnection.java:259 connection attempt 788 to 10.216.193.246:12700 (GOSSIP) 2018-10-22T13:25:34.414+0200 DEBUG [ScheduledFastTasks:1] NettyFactory.java:321 creating outbound bootstrap to peer 10.216.193.246:12700, compression: true, encryption: enabled (jdk), coalesce: DISABLED, protocolV ersion: 12 2018-10-22T13:25:34.526+0200 DEBUG [GossipTasks:1] OutboundMessagingConnection.java:259 connection attempt 789 to 10.216.193.246:12700 (GOSSIP) 2018-10-22T13:25:34.527+0200 DEBUG [GossipTasks:1] NettyFactory.java:321 creating outbound bootstrap to peer 10.216.193.246:12700, compression: true, encryption: enabled (jdk), coalesce: DISABLED, protocolVersion: 12 In the 3.11.3 node I don't see any errors, I do see this: 2018-10-22T13:20:36.270+0200 INFO [HANDSHAKE-/10.216.193.244] OutboundTcpConnection.java:561 Handshaking version with /10.216.193.244 and in the 4.0 node I find this: 2018-10-22T13:20:36.270+0200 INFO [MessagingService-NettyInbound-Thread-3-1] InboundHandshakeHandler.java:146 connection from peer /10.216.193.243:54262, protocol = TLSv1.2, cipher suite = TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 2018-10-22T13:20:41.147+0200 DEBUG [main] OutboundMessagingConnection.java:259 connection attempt 0 to 10.216.193.243:12700 (GOSSIP) 2018-10-22T13:20:41.147+0200 DEBUG [main] NettyFactory.java:321 creating outbound bootstrap to peer 10.216.193.243:12700, compression: true, encryption: enabled (jdk), coalesce: DISABLED, protocolVersion: 11 2018-10-22T13:20:41.248+0200 DEBUG [ScheduledFastTasks:1] OutboundMessagingConnection.java:259 connection attempt 1 to 10.216.193.243:12700 (GOSSIP) 2018-10-22T13:20:41.248+0200 DEBUG [ScheduledFastTasks:1] NettyFactory.java:321 creating outbound bootstrap to peer 10.216.193.243:12700, compression: true, encryption: enabled (jdk), coalesce: DISABLED, protocolV ersion: 11 It looks like the 4.0 node doesn’t accept incoming messages from the old nodes on the ssl_storage_port and continues to try to contact the old nodes on the storage_port that are disabled in the old nodes. I don't really know what to make of this, any ideas or suggestions? Regards, Tommy --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
