> I've tested again with recording LiveSSTableCount and MemtableDataSize
> via jmx. I guess this result supports my suspect on memtable
> performance because I cannot find Full GC this time.
> This is a result in smaller data size (160million records on
> cassandra) on different disk configuration
Look at the graph again. Especially from the first posting.
The records/second read (by the client) goes down as disk reads goes down.
Looks like something is fishy with the memtables.
Terje
On Tue, Nov 23, 2010 at 1:54 AM, Edward Capriolo wrote:
> On Mon, Nov 22, 2010 at 8:28 AM, Shotaro Kamio
On Mon, Nov 22, 2010 at 8:28 AM, Shotaro Kamio wrote:
> Hi Peter,
>
> I've tested again with recording LiveSSTableCount and MemtableDataSize
> via jmx. I guess this result supports my suspect on memtable
> performance because I cannot find Full GC this time.
> This is a result in smaller data size
> After a memtable flush, you see minimum cpu and maximum read
> throughput both in term of disk and cassandra records read.
> As memtable increase in size, cpu goes up and read drops.
> If this is because of memtable or GC performance issue, this is the
> big question.
>
> As each memtable is just
Thanks, Peter.
I missed an important point. Each time the performance resets, there
is a memtable flush. The memtable is flushed every hour. It is about
128MB. Each time the memtable flushed, there is a GC spike.
After a memtable flush, you see minimum cpu and maximum read
throughput both in term
> - About 300 million records are stored in single cassandra node. (16
> cores, 32GB memory, SSD disk)
> - One record is avg. 1KB.
So about 300 GB of disk space in total?
> - Multi-thread client on different server does read and write:
> - 120 reader threads continuously read random records from
