Cassandra performance and read/write latency
Greetings Cassandra Developers! We've been trying to benchmark Cassandra performance and have developed a test client written in C++ that uses multiple threads to send out a large number of write and read requests (as fast as the server can handle them). One of the results we're seeing is a bit surprising, and I'm hoping someone here can help shed some light on the topic - as far as I can tell, it hasn't been discuseed on the mailing list. Most of the requests return in a reasonable amount of time (10s or 100s of milliseconds), but every once in a while, the server seems to just "stop" for up to several seconds. During this time, all the reads and writes will take several seconds to complete and network traffic in an out of the system drops off to nearly zero. When plotted on a graph, these appear as very larges spikes every few minutes. (Though without any particular pattern to how often those spikes occur). Even though the average response time is very good (and therefore we get a reasonable number of requests/sec) these occasional outliers are a showstopper for our potential applications. We've experimented with a number of different machines of different capabilities including a range of physical machines, and clusters of machines on Amazon's EC2. We've also used different numbers of nodes in the cluster and different values for ReplicationFactor. All are qualitatively similar, though the numbers vary as expected (i.e. fast machines improve both the average and maximum numbers, but the max values are still on the order of seconds) I know Cassandra has lots of configuration parameters that can be tweaked, but most of the other parameters are left at the default values of Cassandara-0.6.2 or 0.6.3. Has anyone else seen nodes "hang" for several seconds like this? I'm not sure if this is a Java VM issue (e.g. garbage collection) or something specific to the Cassandra application. I'll be happy to share more details of our experiments either on the mailing list, or with interested parties offline. But I thought I'd start with a brief description and see how consistent it is with other experiences. I'm sort of expecting to see "Well, of course you'll see that kind of behavior because you didn't change..." I'm also interested in comparing notes with anyone else that has been doing read/write throughput benchmarks with Cassandara. Thanks in advance for any information or suggestions you may have! -- Peter Fales Alcatel-Lucent Member of Technical Staff 1960 Lucent Lane Room: 9H-505 Naperville, IL 60566-7033 Email: peter.fa...@alcatel-lucent.com Phone: 630 979 8031
Re: Cassandra nodes on EC2 in two different regions not communicating
It's pretty easy to do if you're willing/able to make some small changes to the Cassandra code. Is that an option? On Thu, Feb 24, 2011 at 06:52:24AM -0600, Himanshi Sharma wrote: > Hi All, > > I am trying to configure a cassandra cluster of 2 nodes in different > regions of amazon ec2. > > Nodes are running fine but when i try nodetool command with ring option i > get single node. > > And both the nodes are able to ping each other, telnet each other. Not > able to figure out what is the problem. > > Plz help > > Thanks, > Himanshi Sharma > > =-=-= > Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > -- Peter Fales Alcatel-Lucent Member of Technical Staff 1960 Lucent Lane Room: 9H-505 Naperville, IL 60566-7033 Email: peter.fa...@alcatel-lucent.com Phone: 630 979 8031
Re: Cassandra nodes on EC2 in two different regions not communicating
I sent a note mentioning this yesterday and referring to my post from last September. See: http://www.mail-archive.com/user@cassandra.apache.org/msg05692.html On Thu, Feb 24, 2011 at 07:04:03AM -0600, Himanshi Sharma wrote: > Peter, > > Please tell me the options. I m new to cassandra and definitely want to > try out. > > Thanks, > Himanshi > > > > > From: > Peter Fales > To: > "dev@cassandra.apache.org" > Date: > 02/24/2011 06:30 PM > Subject: > Re: Cassandra nodes on EC2 in two different regions not communicating > > > > It's pretty easy to do if you're willing/able to make some small changes > to the Cassandra code. Is that an option? > > On Thu, Feb 24, 2011 at 06:52:24AM -0600, Himanshi Sharma wrote: > > Hi All, > > > > I am trying to configure a cassandra cluster of 2 nodes in different > > regions of amazon ec2. > > > > Nodes are running fine but when i try nodetool command with ring option > i > > get single node. > > > > And both the nodes are able to ping each other, telnet each other. Not > > able to figure out what is the problem. > > > > Plz help > > > > Thanks, > > Himanshi Sharma > > > > =-=-= > > Notice: The information contained in this e-mail > > message and/or attachments to it may contain > > confidential or privileged information. If you are > > not the intended recipient, any dissemination, use, > > review, distribution, printing or copying of the > > information contained in this e-mail message > > and/or attachments to it are strictly prohibited. If > > you have received this communication in error, > > please notify us by reply e-mail or telephone and > > immediately and permanently delete the message > > and any attachments. Thank you > > > > -- > Peter Fales > Alcatel-Lucent > Member of Technical Staff > 1960 Lucent Lane > Room: 9H-505 > Naperville, IL 60566-7033 > Email: peter.fa...@alcatel-lucent.com > Phone: 630 979 8031 > > > =-=-= > Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > -- Peter Fales Alcatel-Lucent Member of Technical Staff 1960 Lucent Lane Room: 9H-505 Naperville, IL 60566-7033 Email: peter.fa...@alcatel-lucent.com Phone: 630 979 8031
Re: Cassandra nodes on EC2 in two different regions not communicating
Are you sure you actually re-built Cassandra after making those source code changes? I believe "Cannot assign requested address" is the error you get with the official Cassandra because the external IP address is not the actual address of an adapter. The patches change the code so that it listens on all interfaces (so it should not actually be requesting a specific address) On Mon, Feb 28, 2011 at 06:23:53AM -0600, Himanshi Sharma wrote: > Hi Peter, > > I made changes as per ur patch. Still its nt working. Using external ip in > Listen addr gives this exception. > > ERROR 11:47:14,092 Exception encountered during startup. > java.net.BindException: Cannot assign requested address > at sun.nio.ch.Net.bind(Native Method) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137) > at > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77) > at > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70) > at > org.apache.cassandra.net.MessagingService.listen(MessagingService.java:138) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:319) > at > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:99) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177) > Exception encountered during startup. > java.net.BindException: Cannot assign requested address > at sun.nio.ch.Net.bind(Native Method) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137) > at > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77) > at > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70) > at > org.apache.cassandra.net.MessagingService.listen(MessagingService.java:138) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:319) > at > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:99) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177) > > So i kept them blank. Cassandra runs. Bt still not showing nodes of other > regions in nodetool cmd, whereas i am able to ping and telnet. > > Thanks, > Himanshi Sharma > > > > > From: > Peter Fales > To: > "dev@cassandra.apache.org" > Date: > 02/24/2011 06:44 PM > Subject: > Re: Cassandra nodes on EC2 in two different regions not communicating > > > > I sent a note mentioning this yesterday and referring to my post from > last September. See: > http://www.mail-archive.com/user@cassandra.apache.org/msg05692.html > > On Thu, Feb 24, 2011 at 07:04:03AM -0600, Himanshi Sharma wrote: > > Peter, > > > > Please tell me the options. I m new to cassandra and definitely want to > > try out. > > > > Thanks, > > Himanshi > > > > > > > > > > From: > > Peter Fales > > To: > > "dev@cassandra.apache.org" > > Date: > > 02/24/2011 06:30 PM > > Subject: > > Re: Cassandra nodes on EC2 in two different regions not communicating > > > > > > > > It's pretty easy to do if you're willing/able to make some small changes > > to the Cassandra code. Is that an option? > > > > On Thu, Feb 24, 2011 at 06:52:24AM -0600, Himanshi Sharma wrote: > > > Hi All, > > > > > > I am trying to configure a cassandra cluster of 2 nodes in different > > > regions of amazon ec2. > > > > > > Nodes are running fine but when i try nodetool command with ring > option > > i > > > get single node. > > > > > > And both the nodes are able to ping each other, telnet each other. > Not > > > able to figure out what is the problem. > > > > > > Plz help > > > > > > Thanks, > > > Himanshi Sharma > > > > > > =-=-= > > > Notice: The information contained in this e-mail > > > message and/or attachments to it may contain > > > confidential or privileged information. If you are > > > not the intended recipient, any dissemination, use, > > > review, distribution, printing or copying of the > > > information contained in this e-mail message > > > and/or attachments to it are strictly prohibited. If > > > you have received this communication in error, > > > please notify us by reply e-mail or telephone and > > > immediately and permanently delete the message > > &
Re: Cassandra nodes on EC2 in two different regions not communicating
Are you running nodetool on the host, or on a remote machine? If local, does "netstat -n" show that you are listening on the JMX port? If remote, have you added the JMX port to your security group? On Tue, Mar 01, 2011 at 12:54:13AM -0600, Himanshi Sharma wrote: > Peter, > > I build Cassandra after making changes as per ur patch. Nw when try > nodetool i get this exception. > > > [root@ip-10-170-153-71 bin]# ./nodetool -h 175.41.143.192 ring > > Error connection to remote JMX agent! > java.rmi.ConnectException: Connection refused to host: 10.130.62.177; > nested exception is: > java.net.ConnectException: Connection timed out > at > sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) > at > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198) > at > sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110) > at > javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) > at > javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327) > at > javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279) > at > javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248) > at > org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:110) > at org.apache.cassandra.tools.NodeProbe.(NodeProbe.java:86) > at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:476) > Caused by: java.net.ConnectException: Connection timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > at > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > at java.net.Socket.connect(Socket.java:529) > at java.net.Socket.connect(Socket.java:478) > at java.net.Socket.(Socket.java:375) > at java.net.Socket.(Socket.java:189) > at > sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22) > at > sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128) > at > sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) > ... 10 more > > > Himanshi Sharma > > > > > From: > Peter Fales > To: > "dev@cassandra.apache.org" > Date: > 02/28/2011 06:25 PM > Subject: > Re: Cassandra nodes on EC2 in two different regions not communicating > > > > Are you sure you actually re-built Cassandra after making those source > code changes? I believe "Cannot assign requested address" is the error > you > get with the official Cassandra because the external IP address is > not the actual address of an adapter. The patches change the code > so that it listens on all interfaces (so it should not actually be > requesting a specific address) > > On Mon, Feb 28, 2011 at 06:23:53AM -0600, Himanshi Sharma wrote: > > Hi Peter, > > > > I made changes as per ur patch. Still its nt working. Using external ip > in > > Listen addr gives this exception. > > > > ERROR 11:47:14,092 Exception encountered during startup. > > java.net.BindException: Cannot assign requested address > > at sun.nio.ch.Net.bind(Native Method) > > at > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137) > > at > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77) > > at > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70) > > at > > > org.apache.cassandra.net.MessagingService.listen(MessagingService.java:138) > > at > > > org.apache.cassandra.service.StorageService.initServer(StorageService.java:319) > > at > > > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:99) > > at > > > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177) > > Exception encountered during startup. > > java.net.BindException: Cannot assign requested address > > at sun.nio.ch.Net.bind(Native Method) > > at > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137) > > at > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77) > > at > > sun.nio.ch.ServerSocketAdapto
