Re: Heavy writes ok for single node, but failed for cluster

2011-04-29 Thread aaron morton
Can you try turning up the logging to TRACE, if you know how turn it up for the org.apache.cassandra.gms.FailureDetector and org.apache.cassandra.gms.Gossiper loggers You'll see a lot of messages. Some of the interesting ones will be... "FailureDetector.java (line 128) reporting /127.0.0.2" -

Re: Heavy writes ok for single node, but failed for cluster

2011-04-28 Thread Sheng Chen
Thank you for your patch. I believe the latter version I used (the latest 0.7 branch) includes the patch, but the problem remains. Is there anything else that may block this heartbeat, like gc? Here are some logs during heartbeat failure. INFO [GossipTasks:1] 2011-04-29 07:25:09,716 Gossiper.jav

Re: Heavy writes ok for single node, but failed for cluster

2011-04-28 Thread Jonathan Ellis
This means a node was too busy with something else to send out its heartbeat. Sometimes this is STW GC. Other times it is a bug (one was fixed for 0.7.6 in https://issues.apache.org/jira/browse/CASSANDRA-2554). On Thu, Apr 28, 2011 at 3:57 AM, Sheng Chen wrote: > Thank you for your advice. Rf>=2

Re: Heavy writes ok for single node, but failed for cluster

2011-04-28 Thread Sheng Chen
Thank you for your advice. Rf>=2 is a good work around. I was using 0.7.4 and have updated to the latest 0.7 branch, which includes 2554 patch. But it doesn't help. I still get lots of UnavailableException after the following logs, INFO [GossipTasks:1] 2011-04-28 16:12:17,661 Gossiper.java (line

Re: Heavy writes ok for single node, but failed for cluster

2011-04-27 Thread Sylvain Lebresne
On Wed, Apr 27, 2011 at 10:32 AM, Sheng Chen wrote: > I succeeded to insert 1 billion records into a single node cassandra, >>> bin/stress -d cas01 -o insert -n 10 -c 5 -S 34 -C5 -t 20 > Inserts finished in about 14 hours at a speed of 20k/sec. > But when I added another node, tests always

Heavy writes ok for single node, but failed for cluster

2011-04-27 Thread Sheng Chen
I succeeded to insert 1 billion records into a single node cassandra, >> bin/stress -d cas01 -o insert -n 10 -c 5 -S 34 -C5 -t 20 Inserts finished in about 14 hours at a speed of 20k/sec. But when I added another node, tests always failed with UnavailableException in an hour. >> bin/stress