RE: FW: Solr and Garbage Collection

2009-09-25 Thread Fuad Efendi
> Usually, fragmentation is dealt with using a mark-compact collector (or > IBM has used a mark-sweep-compact collector). > Copying collectors are not only super efficient at collecting young > spaces, but they are also great for fragmentation - when you copy > everything to the new space, you can

Re: FW: Solr and Garbage Collection

2009-09-25 Thread Mark Miller
live_objects_ (and >>> wait/lock till such objects get unlocked to be moved...) - obviously more >>> memory helps... >>> >>> 11% is extremely high. >>> >>> >>> -Fuad >>> http://www.linkedin.com/in/liferay >>> >>> >

Re: FW: Solr and Garbage Collection

2009-09-25 Thread Mark Miller
;> >> 11% is extremely high. >> >> >> -Fuad >> http://www.linkedin.com/in/liferay >> >> >> >>> -Original Message- >>> From: Jonathan Ariel [mailto:ionat...@gmail.com] >>> Sent: September-25-09 3:36 PM >>

Re: FW: Solr and Garbage Collection

2009-09-25 Thread Jonathan Ariel
> -Fuad > http://www.linkedin.com/in/liferay > > > > -Original Message- > > From: Jonathan Ariel [mailto:ionat...@gmail.com] > > Sent: September-25-09 3:36 PM > > To: solr-user@lucene.apache.org > > Subject: Re: FW: Solr and Garbage Collection > &

Re: FW: Solr and Garbage Collection

2009-09-25 Thread Yonik Seeley
On Fri, Sep 25, 2009 at 2:52 PM, Fuad Efendi wrote: > Lowering heap helps GC? Yes. In general, lowering the heap can help or hurt. Hurt: if one is running very low on memory, GC will be working harder all of the time trying to find more memory and the % of time that GC takes can go up. Help: i

RE: FW: Solr and Garbage Collection

2009-09-25 Thread Fuad Efendi
- obviously more memory helps... 11% is extremely high. -Fuad http://www.linkedin.com/in/liferay > -Original Message- > From: Jonathan Ariel [mailto:ionat...@gmail.com] > Sent: September-25-09 3:36 PM > To: solr-user@lucene.apache.org > Subject: Re: FW: Solr and Garbage

Re: FW: Solr and Garbage Collection

2009-09-25 Thread Jonathan Ariel
I'm not planning on lowering the heap. I just want to lower the time "wasted" on GC, which is 11% right now.So what I'll try is changing the GC to -XX:+UseConcMarkSweepGC On Fri, Sep 25, 2009 at 4:17 PM, Fuad Efendi wrote: > Mark, > > what if piece of code needs 10 contiguous Kb to load a docume

RE: FW: Solr and Garbage Collection

2009-09-25 Thread Fuad Efendi
Mark, what if piece of code needs 10 contiguous Kb to load a document field? How locked memory pieces are optimized/moved (putting on hold almost whole application)? Lowering heap is _bad_ idea; we will have extremely frequent GC (optimize of live objects!!!) even if RAM is (theoretically) enough.

RE: FW: Solr and Garbage Collection

2009-09-25 Thread Fuad Efendi
> He is not having a problem with OOM. He got the OOM because he lowered > the heap to try and help GC. That is very confusing!!! Lowering heap helps GC? Someone mentioned it in this thread, but my viewpoint is completely opposite. 1. Some RAM is needed to_be_reserved for FieldCache (it will be

Re: FW: Solr and Garbage Collection

2009-09-25 Thread Mark Miller
Faud, you didn't read the thread right. He is not having a problem with OOM. He got the OOM because he lowered the heap to try and help GC. He normally runs with a heap that can handle his FC. Please re-read the thread. You are confusing the tread. - Mark Fuad Efendi wrote: > Guys, thanks for

FW: Solr and Garbage Collection

2009-09-25 Thread Fuad Efendi
Guys, thanks for GC discussion; but the root of a problem is FieldCache internals. Not enough RAM for FieldCache will cause unpredictable OOM, and it does not depend on GC. How much RAM FieldCache needs in case of 2 different values for a Field, 200 bytes each (Unicode), and 100M documents? Wh