Re: memory utilization

2011-03-12 Thread Jonathan Ellis
Probably relevant: we only use mmap'd I/O for single-row reads. When we are paging through entire files like we do for compaction or AES we do buffered i/o to avoid the complexity of having to manage multiple mmap segments (Java limits us to 2GB per segment). On Sat, Mar 12, 2011 at 7:06 PM, Peter

Re: memory utilization

2011-03-12 Thread Peter Schuller
>> Nothing happens, because it _doesn't have to be resident_. >> > > Hm, but why in my case top show RSS 10g, when max HEAP_SIZE is 6G? The point is that it is a result of how the kernel manages memory and how it is reported in top. It is not reflective of actual memory "use", the way users normal

Re: memory utilization

2011-03-12 Thread ruslan usifov
2011/3/12 Jonathan Ellis > Nothing happens, because it _doesn't have to be resident_. > > Hm, but why in my case top show RSS 10g, when max HEAP_SIZE is 6G?? PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 27650 cassandr 20 0 14.9g 10g 3.8g S 51 86.6 370:15.82 jsvc 205

Re: memory utilization

2011-03-12 Thread Jonathan Ellis
Nothing happens, because it _doesn't have to be resident_. On Sat, Mar 12, 2011 at 1:47 PM, ruslan usifov wrote: > > > 2011/3/11 Chris Burroughs >> >> Is there an more or less constant amount of resident memory, or is it >> growing over a period of days? > > As said in cassandra wiki: > The

Re: memory utilization

2011-03-12 Thread ruslan usifov
2011/3/11 Chris Burroughs > Is there an more or less constant amount of resident memory, or is it > growing over a period of days? > As said in cassandra wiki: >>>The main argument for using mmap() instead of standard I/O is the fact that reading entails just touching memory - in the case of th

Re: memory utilization

2011-03-11 Thread Chris Burroughs
On 03/10/2011 09:26 PM, Bill Hastings wrote: > Hi All > > Memory utilization reported by JCOnsole for Cassandra seems to be much > lesser than that reported by top ("RES" memory). Can someone explain this? > Maybe off topic but would appreciate a response. > Is th

Re: memory utilization

2011-03-10 Thread Jonathan Ellis
http://wiki.apache.org/cassandra/FAQ#mmap On Thu, Mar 10, 2011 at 8:26 PM, Bill Hastings wrote: > Hi All > > Memory utilization reported by JCOnsole for Cassandra seems to be much > lesser than that reported by top ("RES" memory). Can someone explain this? > Maybe off t

memory utilization

2011-03-10 Thread Bill Hastings
Hi All Memory utilization reported by JCOnsole for Cassandra seems to be much lesser than that reported by top ("RES" memory). Can someone explain this? Maybe off topic but would appreciate a response. -- Cheers Bill

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-20 Thread Zhu Han
t;> > >> >> committed: 34308096 >> > >> >> init: 24313856 >> > >> >> max: 226492416 >> > >> >> used: 21475376 >> > >> >> >> > >> >> If anybody is interested in it, I can provide more diagnostic >

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-19 Thread Victor Kabdebon
Hello Peter, So more information on that problem : Yes I am using this node with very few data, it is used to design requests so I don't need a very large dataset. I am running Apache Cassandra 0.6.6 on a Debian Stable, with java version "1.6.0_22". I recently restarted cassandra, thus I have thi

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-19 Thread Peter Schuller
> vic...@:~$ sudo ps aux | grep "cassandra" > cassandra     11034  0.2 22.9 1107772 462764 ?  Sl   Dec17   6:13 > /usr/bin/java -ea -Xms128M -Xmx512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOc

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-18 Thread Victor Kabdebon
t;>> > >> >> init: 1073741824 >>> > >> >> max: 1065025536 >>> > >> >> used: 18295328 >>> > >> >> >>> > >> >> $java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.jar - >>> > loc

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-18 Thread Zhu Han
mitted: 34308096 >> > >> >> init: 24313856 >> > >> >> max: 226492416 >> > >> >> used: 21475376 >> > >> >> >> > >> >> If anybody is interested in it, I can provide more diagnostic >> >

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-18 Thread Zhu Han
Zhu Han > > wrote: > > >> >> > > >> >>> After investigating it deeper, I suspect it's native memory leak > of > > >> JVM. > > >> >>> The large anonymous map on lower address space should be the > native > > >

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-18 Thread Daniel Korndorfer
; The large anonymous map on lower address space should be the native >>> heap of >>> >>> JVM, but not java object heap. Has anybody met it before? >>> >>> >>> >>> I'll try to upgrade the JVM tonight. >>> >>> >>>

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-17 Thread Zhu Han
upgrade the JVM tonight. >> >>> >> >>> best regards, >> >>> hanzhu >> >>> >> >>> >> >>> >> >>> On Thu, Dec 16, 2010 at 10:50 AM, Zhu Han >> wrote: >> >>> >> >>&g

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-16 Thread Peter Schuller
> Sorry for spam again. :-) No, thanks a lot for tracking that down and reporting details! Presumably a significant amount of users are on that version of Ubuntu running with openjdk. -- / Peter Schuller

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-15 Thread Zhu Han
.8.2) (6b18-1.8.2-4ubuntu2) >>> OpenJDK 64-Bit Server VM (build 16.0-b13, mixed mode) >>> >>> This is the memory settings: >>> >>> "/usr/bin/java -ea -Xms1G -Xmx1G ..." >>> >>> And the ondisk footprint of sstables is very smal

Re: Very high memory utilization (not caused by mmap on sstables)

2010-12-15 Thread Zhu Han
ry settings: >> >> "/usr/bin/java -ea -Xms1G -Xmx1G ..." >> >> And the ondisk footprint of sstables is very small: >> >> "#du -sh data/ >> "9.8Mdata/" >> >> The node was infrequently accessed in the last three weeks

Re: Very high memory utilization (not caused by mmap on sstables)

2010-12-15 Thread Zhu Han
Mdata/" > > The node was infrequently accessed in the last three weeks. After that, I > observe the abnormal memory utilization by top: > > PID USER PR NI *VIRT* *RES* SHR S %CPU %MEMTIME+ > COMMAND > > 7836 root 15 0 *3300m* *2.4g* 13m S0

Very high memory utilization (not caused by mmap on sstables)

2010-12-15 Thread Zhu Han
b13, mixed mode) This is the memory settings: "/usr/bin/java -ea -Xms1G -Xmx1G ..." And the ondisk footprint of sstables is very small: "#du -sh data/ "9.8Mdata/" The node was infrequently accessed in the last three weeks. After that, I observe the abnormal mem