Can you paste the output of cfstats and cfhistograms?
Also try and get histo at 2 diff points 1) when it looks good 2) when it
gets slow
http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html
Look for jmap -histo
On Thu, Jun 20, 2013 at 12:27 AM, Joel Samuelsson wrote:
> 12.3 GB
12.3 GB data per node (only one ńode).
16GB RAM.
In virtual environment with the CPU specified as "8 cores", average CPU use
is close to 0% (basically no load, around 12 requests / sec, mostly from
OpsCenter).
Average memory use is 4GB. Around 1GB heap used by Cassandra (out of 4GB).
2013/6/19 Mo
How much data do you have per node?
How much RAM per node?
How much CPU per node?
What is the avg CPU and memory usage?
On Wed, Jun 19, 2013 at 12:16 AM, Joel Samuelsson wrote:
> My Cassandra ps info:
>
> root 26791 1 0 07:14 ?00:00:00 /usr/bin/jsvc -user
> cassandra -home /opt
2013/6/19 Takenori Sato :
> GC options are not set. You should see the followings.
>
> -XX:+PrintGCDateStamps -XX:+PrintPromotionFailure
> -Xloggc:/var/log/cassandra/gc-1371603607.log
>
>> Is it normal to have two processes like this?
>
> No. You are running two processes.
It's "normal" as this i
Right, after getting the GC logging information I tested upgrading to 1.2.
Didn't help but I forgot to reenable the GC options.
> No. You are running two processes.
Ok, that's weird. I am using an unmodified version of a startup script in
/etc/init.d/cassandra from the Debian package. Here's some
GC options are not set. You should see the followings.
-XX:+PrintGCDateStamps -XX:+PrintPromotionFailure
-Xloggc:/var/log/cassandra/gc-1371603607.log
> Is it normal to have two processes like this?
No. You are running two processes.
On Wed, Jun 19, 2013 at 4:16 PM, Joel Samuelsson
wrote:
> M
My Cassandra ps info:
root 26791 1 0 07:14 ?00:00:00 /usr/bin/jsvc -user
cassandra -home /opt/java/64/jre1.6.0_32/bin/../ -pidfile
/var/run/cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log
-cp
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4
Is your young generation size set to 4GB? Can you paste the output of ps
-ef|grep cassandra ?
On Tue, Jun 18, 2013 at 8:48 AM, Joel Samuelsson
wrote:
> Yes, like I said, the only relevant output from that file was:
> 2013-06-17T08:11:22.300+: 2551.288: [GC 870971K->216494K(4018176K),
> 145.18
Yes, like I said, the only relevant output from that file was:
2013-06-17T08:11:22.300+: 2551.288: [GC 870971K->216494K(4018176K),
145.1887460 secs]
2013/6/18 Takenori Sato
> GC logging is not in system.log. But in the following file.
>
> JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-`d
GC logging is not in system.log. But in the following file.
JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-`date +%s`.log"
At least, no GC logs are shown in your post.
On Tue, Jun 18, 2013 at 5:05 PM, Joel Samuelsson
wrote:
> Can't find any promotion failure.
>
> In system.log this is what
Can't find any promotion failure.
In system.log this is what I get:
INFO [ScheduledTasks:1] 2013-06-17 08:13:47,490 GCInspector.java (line
122) GC for ParNew: 145189 ms for 1 collections, 225905072 used; max is
4114612224
INFO [ScheduledTasks:1] 2013-06-17 08:13:47,490 StatusLogger.java (line
57
Find "promotion failure". Bingo if it happened at the time.
Otherwise, post the relevant portion of the log here. Someone may find a
hint.
On Mon, Jun 17, 2013 at 5:51 PM, Joel Samuelsson
wrote:
> Just got a very long GC again. What am I to look for in the logging I just
> enabled?
>
>
> 2013/6
Just got a very long GC again. What am I to look for in the logging I just
enabled?
2013/6/17 Joel Samuelsson
> > If you are talking about 1.2.x then I also have memory problems on the
> idle cluster: java memory constantly slow grows up to limit, then spend
> long time for GC. I never seen suc
> If you are talking about 1.2.x then I also have memory problems on the
idle cluster: java memory constantly slow grows up to limit, then spend
long time for GC. I never seen such behaviour for 1.0.x and 1.1.x, where
on idle cluster java memory stay on the same value.
No I am running Cassandra 1
> Also can you take a heap dump at 2 diff points so that we can compare it?
Also note that a promotion failure won't happen by a particular object, but
by a fragmentation in Old Generation space. So I am not sure if you can't
tell by a heap dump comparison.
On Sun, Jun 16, 2013 at 4:44 AM, Mohit
Uncomment the followings in "cassandra-env.sh".
JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps"
JVM_OPTS="$JVM_OPTS -XX:+PrintPromotionFailure"
JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-`date +%s`.log"
*> *Also can you take a heap dump at 2 diff points so that we can compare it?
No, I'm afr
Can you paste you gc config? Also can you take a heap dump at 2 diff points so
that we can compare it?
Quick thing to do would be to do a histo live at 2 points and compare
Sent from my iPhone
On Jun 15, 2013, at 6:57 AM, Takenori Sato wrote:
> > INFO [ScheduledTasks:1] 2013-04-15 14:00:02,74
> INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line
122) GC for ParNew: 338798 ms for 1 collections, 592212416 used; max is
1046937600
This says GC for New Generation took so long. And this is usually unlikely.
The only situation I am aware of is when a fairly large object is
On Fri, Jun 7, 2013 at 12:42 PM, Igor wrote:
> If you are talking about 1.2.x then I also have memory problems on the idle
> cluster: java memory constantly slow grows up to limit, then spend long time
> for GC. I never seen such behaviour for 1.0.x and 1.1.x, where on idle
> cluster java memory s
on this e-mail. If you have
received this message in error, please contact the sender
immediately and irrevocably delete this message and any copies.
>
> From: Joel Samuelsson [mailto:samuelsson.j...@gmail.com
<mailto:samuelsson.j...@gmail.com>]
> Sent: Tue
, LT-01112 Vilnius, Lithuania
> > Follow us on Twitter: @adforminsider
> > Take a ride with Adform's Rich Media Suite
> >
> >
> >
> > Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the na
of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in e
bute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
> *From:* Joel Samuelsson [mailto:samuelsson.j...@gmail.com]
> *Sent:* Tuesday, April 16, 2013 12:52
>
this message and any copies.
From: Joel Samuelsson [mailto:samuelsson.j...@gmail.com]
Sent: Tuesday, April 16, 2013 12:52
To: user@cassandra.apache.org
Subject: Re: Reduce Cassandra GC
How do you calculate the heap / data size ratio? Is this a linear ratio?
Each node has slightly more than 12 GB
opy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
> *From:* Joel Samuelsson [mailto:samuelsson.j...@gmail.com]
> *Sent:* Tuesday, April 16, 2013 10:47
> *
Tuesday, April 16, 2013 10:47
To: user@cassandra.apache.org
Subject: Reduce Cassandra GC
Hi,
We have a small production cluster with two nodes. The load on the nodes is
very small, around 20 reads / sec and about the same for writes. There are
around 2.5 million keys in the cluster and a RF of 2.
About
Hi,
We have a small production cluster with two nodes. The load on the nodes is
very small, around 20 reads / sec and about the same for writes. There are
around 2.5 million keys in the cluster and a RF of 2.
About 2.4 million of the rows are skinny (6 columns) and around 3kb in size
(each). Curr
27 matches
Mail list logo
