Can you turn the logging up to DEBUG level and look for a message from
CassandraServer that says "... timed out" ?
Also check the thread pool stats "nodetool tpstats" to see if the node is
keeping up.
Aaron
On 7 Apr 2011, at 13:43, Sheng Chen wrote:
> Thank you Aaron.
>
> It does not seem
Thank you Aaron.
It does not seem to be an overload problem.
I have 16 cores and 48G ram on the single node, and I reduced the concurrent
threads to be 1.
Still, it just suddenly dies of a timeout, while the cpu, ram, disk load are
below 10% and write latency is about 0.5ms for the past 10 minute
TimedOutException means that the less than CL number of nodes responded to the
coordinator before the rpc_timeout.
So it was overloaded. Which makes sense when you say it only happens with
secondary indexes. Consider things like
- reducing the throughput
- reducing the number of clients
- ensuri
I used py_stress module to insert 10m test data with a secondary index.
I got the following exceptions.
# python stress.py -d xxx -o insert -n 1000 -c 5 -s 34 -C 5 -x keys
total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
265322,26532,26541,0.00186140829433,10
630300,36497,3650
