Master-Slave replica: new caches will be warmed&prepopulated _before_ making
new IndexReader available for _new_ requests and _before_ discarding old one
- it means that theoretical sizing for FieldCache (which is defined by
number of docs in an index and cardinality of a field) should be doubled..
> I read pretty much all posts on this thread (before and after this one).
Looks
> like the main suggestion from you and others is to keep max heap size
(-Xmx)
> as small as possible (as long as you don't see OOM exception).
I suggested absolute opposite; please note also that "as small as possi
;>> MaxGCPauseMillis doesnt work with UseConcMarkSweepGC - its for use with
>>>>>>> the Parallel collector. That also doesnt look like a good
>>>>>>>
>>>>>>>
>>>>>>>
>>>> survivorratio.
>>>>
>>>>
>>>>
>>>>>>&g
't your
>>>>> suggestion (of lowering heap limit) going against that?
>>>>>
>>>>>
>>>>>
>>>>>> Leaving RAM for the FileSystem cache is also very important. But you
>>>>>> should
>>>> * Looks like Solr caching made its way into tenure-generation on heap,
>>>>>>
>>>>>>
>>>> that's good. But why they get GC'ed eventually?? I did a quick check of
>>>>
>> Solr
>>
>>
mory management strategy (or just my ignorance about Solr?). I
> >> mean, wouldn't this be the "right" way of doing it -- you allow user to
> >> specify the cache size in solrconfig.xml, then user can set up heap
> limit in
> >> JAVA_OPTS accordingly, and no need to use WeakRef
you see concurrent mode failure when looking at your gc logs? ie:
>>>
>>> 174.445: [GC 174.446: [ParNew: 66408K->66408K(66416K), 0.618
>>> secs]174.446: [CMS (concurrent mode failure): 161928K->162118K(175104K),
>>> 4.0975124 secs] 228336K->162118K(24152
still getting major collections with CMS, and you
> > don't want that. You might try kicking GC off earlier with something
> > like: -XX:CMSInitiatingOccupancyFraction=50
> >
> >> * Right now I have a single Tomcat hosting Solr and other applications.
> I gues
8K(175104K),
> 4.0975124 secs] 228336K->162118K(241520K)
>
> That means you have still getting major collections with CMS, and you
> don't want that. You might try kicking GC off earlier with something
> like: -XX:CMSInitiatingOccupancyFraction=50
>
>> * Right now I
with CMS, and you
don't want that. You might try kicking GC off earlier with something
like: -XX:CMSInitiatingOccupancyFraction=50
> * Right now I have a single Tomcat hosting Solr and other applications. I
> guess now it's better to have Solr on its own Tomcat, given that i
> From: wun...@wunderwood.org
> To: solr-user@lucene.apache.org
> Subject: RE: Solr and Garbage Collection
> Date: Fri, 25 Sep 2009 09:51:29 -0700
>
> 30ms is not better or worse than 1s until you look at the service
> requirements. For many applications, it is worth dedicatin
> Actually the CPU usage of the solr servers is almost insignificant (it was
> like that before).
>>The time spent on collecting memory dropped from 11% to 3.81%
I even think that 3.81% from 5% is nothing (suspecting that SOLR uses 5%
CPU, mostly loading large field values in memory) :)))
(woul
One way to track expensive is to look at the query time, QTime, in the solr
log.
There are a couple of tools for analyzing gc logs:
http://www.tagtraum.com/gcviewer.html
https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER
They will give you frequency and duratio
apache.org
>> Sent: Monday, September 28, 2009 4:49:03 PM
>> Subject: Re: Solr and Garbage Collection
>>
>> How do you track major collections? Even better, how do you log your GC
>> behavior with details? Right now I just log total time spent on collections,
>
|-verbose:gc
|
|[GC 325407K->83000K(776768K), 0.2300771 secs]
[GC 325816K->83372K(776768K), 0.2454258 secs]
[Full GC 267628K->83769K(776768K), 1.8479984 secs]|
Additional details with: |-XX:+PrintGCDetails|
|[GC [DefNew: 64575K->959K(64576K), 0.0457646 secs] 196016K->133633K(261184K),
0.045906
rom: Jonathan Ariel
> To: solr-user@lucene.apache.org
> Sent: Monday, September 28, 2009 4:49:03 PM
> Subject: Re: Solr and Garbage Collection
>
> How do you track major collections? Even better, how do you log your GC
> behavior with details? Right now I just log total time spent on
How do you track major collections? Even better, how do you log your GC
behavior with details? Right now I just log total time spent on collections,
but I don't really know on which collections.Regard application performance
with the ConcMarkSweepGC, I think I didn't experience any impact for now.
Do you have your GC logs? Are you still seeing major collections?
Where is the time spent?
Hard to say without some of that info.
The goal of the low pause collector is to finish collecting before the
tenured space is filled - if it doesn't, a standard major collection occurs.
The collector wil
Ok... good news! Upgrading to the newest version of JVM 6 (update 6) seems
to solve this ugly bug. With the upgraded JVM I could run the solr servers
for more than 12 hours on the production environment with the GC mentioned
in the previous e-mails. The results are really amazing. The time spent on
You are running a very old version of Java 6 (update 6). The latest is
update 16. You should definitely upgrade. There is a bug in Java 6
starting with update 4 that may result in a corrupted Lucene/Solr index:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6707044
https://issues.apache.org/
0s large
> >> documents (in parallel! 1000s concurrent users), can you predict memory
> >> requirements and GC options without application-specific knowledge? What
> >> about new SOLR-Caches warming up?
> >>
> >>
> >> -Fuad
> >>
> >>
need to load 1000s large
>> documents (in parallel! 1000s concurrent users), can you predict memory
>> requirements and GC options without application-specific knowledge? What
>> about new SOLR-Caches warming up?
>>
>>
>> -Fuad
>>
>>
>>
bout new SOLR-Caches warming up?
>
>
> -Fuad
>
>
>
>> -Original Message-
>> From: Mark Miller [mailto:markrmil...@gmail.com]
>> Sent: September-27-09 2:46 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr and Garbage Collection
&
n you predict memory
> requirements and GC options without application-specific knowledge? What
> about new SOLR-Caches warming up?
>
>
> -Fuad
>
>
> > -----Original Message-----
> > From: Mark Miller [mailto:markrmil...@gmail.com]
> > Sent: September-27-09 2
46 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr and Garbage Collection
>
> If he needed double the RAM, he'd likely know by now :) The JVM likes to
> throw OOM exceptions when you need more RAM. Until it does - thats an
> odd path to focus on. There has been no
If he needed double the RAM, he'd likely know by now :) The JVM likes to
throw OOM exceptions when you need more RAM. Until it does - thats an
odd path to focus on. There has been no indication he has ever seen an
OOM with his over 10 GB heap. It sounds like he has run Solr in his
environment for
>> Ok. After the server ran for more than 12 hours, the time spent on GC
>> decreased from 11% to 3,4%, but 5 hours later it crashed.
All this 'black-hat' GC tuning and 'fast' object moving (especially objects
accessing by some thread during GC-defragmentation)
- try to use multithreaded load-str
Yes, it seems like a bug. I will update my JVM, try again and let you
know the results :)
On 9/26/09, Mark Miller wrote:
> Jonathan Ariel wrote:
>> Ok. After the server ran for more than 12 hours, the time spent on GC
>> decreased from 11% to 3,4%, but 5 hours later it crashed. This is the
>> thr
w very little about the IBM JVM, so I wasn't really
> commenting. But from the info I gleaned here and on a couple quick web
> searches, I'm not too impressed by it's GC.
>
>> wunder
>>
>> -Original Message-
>> From: Mark Miller [mailto:mar
Also, in case the info might help track something down:
Its pretty darn odd that both your survivor spaces are full. I've never
seen that ever in one of these dumps. Always one is empty. When one is
filled, its moved to the other. Then back. And forth. For a certain
number of times until its moved
Jonathan Ariel wrote:
> Ok. After the server ran for more than 12 hours, the time spent on GC
> decreased from 11% to 3,4%, but 5 hours later it crashed. This is the thread
> dump, maybe you can help identify what happened?
>
Well thats a tough ;) My guess is its a bug :)
Your two survivor spac
Ok. After the server ran for more than 12 hours, the time spent on GC
decreased from 11% to 3,4%, but 5 hours later it crashed. This is the thread
dump, maybe you can help identify what happened?
#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x2
Jonathan Ariel wrote:
> I have around 8M documents.
>
Thats actually not so bad - I take it you are faceting/sorting on quite
a few unique fields?
> I set up my server to use a different collector and it seems like it
> decreased from 11% to 4%, of course I need to wait a bit more because it is
Sorry for OFF-topic:
Create dummy "Hello, World!" JSP, use Tomcat, execute load-stress
simulator(s) from separate machine(s), and measure... don't forget to
allocate necessary thread pools in Tomcat (if you have to)...
Although such JSP doesn't use any memory, you will see how easy one can go
with
I have around 8M documents.
I set up my server to use a different collector and it seems like it
decreased from 11% to 4%, of course I need to wait a bit more because it is
just a 1 hour old log. But it seems like it is much better now.
I will tell you on Monday the results :)
On Fri, Sep 25, 2009
One more point and I'll stop - I've hit my email quota for the day ;)
While its a pain to have to juggle GC params and tune - when you require
a heap thats more than a gig or two, I personally believe its essential
to do so for good performance. The (default settings / ergonomics with
throughput)
Thats a good point too - if you can reduce your need for such a large
heap, by all means, do so.
However, considering you already need at least 10GB or you get OOM, you
have a long way to go with that approach. Good luck :)
How many docs do you have ? I'm guessing its mostly FieldCache type
stuff
Mark Miller wrote:
> Jonathan Ariel wrote:
>
>> How can I check which is the GC that it is being used? If I'm right JVM
>> Ergonomics should use the Throughput GC, but I'm not 100% sure. Do you have
>> any recommendation on this?
>>
>>
>>
> Just to straighten out this one too - Ergonomic
Jonathan Ariel wrote:
> How can I check which is the GC that it is being used? If I'm right JVM
> Ergonomics should use the Throughput GC, but I'm not 100% sure. Do you have
> any recommendation on this?
>
>
Just to straighten out this one too - Ergonomics doesn't use throughput
- throughput is
> Usually, fragmentation is dealt with using a mark-compact collector (or
> IBM has used a mark-sweep-compact collector).
> Copying collectors are not only super efficient at collecting young
> spaces, but they are also great for fragmentation - when you copy
> everything to the new space, you can
On Sep 25, 2009, at 9:30 AM, Jonathan Ariel wrote:
Hi to all!
Lately my solr servers seem to stop responding once in a while. I'm
using
solr 1.3.
Of course I'm having more traffic on the servers.
So I logged the Garbage Collection activity to check if it's because
of
that. It seems like 11
live_objects_ (and
>>> wait/lock till such objects get unlocked to be moved...) - obviously more
>>> memory helps...
>>>
>>> 11% is extremely high.
>>>
>>>
>>> -Fuad
>>> http://www.linkedin.com/in/liferay
>>>
>>>
>
ps...
>>>
>>> 11% is extremely high.
>>>
>>>
>>> -Fuad
>>> http://www.linkedin.com/in/liferay
>>>
>>>
>>>
>>>> -Original Message-
>>>> From: Jonathan Ariel [mailto:ionat...@gmail.com]
>&
;>
>> 11% is extremely high.
>>
>>
>> -Fuad
>> http://www.linkedin.com/in/liferay
>>
>>
>>
>>> -Original Message-
>>> From: Jonathan Ariel [mailto:ionat...@gmail.com]
>>> Sent: September-25-09 3:36 PM
>>
> -Fuad
> http://www.linkedin.com/in/liferay
>
>
> > -Original Message-
> > From: Jonathan Ariel [mailto:ionat...@gmail.com]
> > Sent: September-25-09 3:36 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: FW: Solr and Garbage Collection
> &
On Fri, Sep 25, 2009 at 2:52 PM, Fuad Efendi wrote:
> Lowering heap helps GC?
Yes. In general, lowering the heap can help or hurt.
Hurt: if one is running very low on memory, GC will be working harder
all of the time trying to find more memory and the % of time that GC
takes can go up.
Help: i
- obviously more
memory helps...
11% is extremely high.
-Fuad
http://www.linkedin.com/in/liferay
> -Original Message-
> From: Jonathan Ariel [mailto:ionat...@gmail.com]
> Sent: September-25-09 3:36 PM
> To: solr-user@lucene.apache.org
> Subject: Re: FW: Solr and Garbage
I'm not planning on lowering the heap. I just want to lower the time
"wasted" on GC, which is 11% right now.So what I'll try is changing the GC
to -XX:+UseConcMarkSweepGC
On Fri, Sep 25, 2009 at 4:17 PM, Fuad Efendi wrote:
> Mark,
>
> what if piece of code needs 10 contiguous Kb to load a docume
Mark,
what if piece of code needs 10 contiguous Kb to load a document field? How
locked memory pieces are optimized/moved (putting on hold almost whole
application)?
Lowering heap is _bad_ idea; we will have extremely frequent GC (optimize of
live objects!!!) even if RAM is (theoretically) enough.
> He is not having a problem with OOM. He got the OOM because he lowered
> the heap to try and help GC.
That is very confusing!!!
Lowering heap helps GC? Someone mentioned it in this thread, but my
viewpoint is completely opposite.
1. Some RAM is needed to_be_reserved for FieldCache (it will be
ying 'transaction')...
>
> GC will frequently happen even if RAM is more than enough: in case if it is
> heavily sparse... so that have even more RAM!
>
>
>
> -Original Message-
> From: Fuad Efendi [mailto:f...@efendi.ca]
> Sent: September-25-09 12:17
appen even if RAM is more than enough: in case if it is
heavily sparse... so that have even more RAM!
-Original Message-
From: Fuad Efendi [mailto:f...@efendi.ca]
Sent: September-25-09 12:17 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr and Garbage Collection
> You are saying
t;
> Yeah, I actually know very little about the IBM JVM, so I wasn't really
> commenting. But from the info I gleaned here and on a couple quick web
> searches, I'm not too impressed by it's GC.
>
>> wunder
>>
>> -Original Message-
>> From:
ow very little about the IBM JVM, so I wasn't really
commenting. But from the info I gleaned here and on a couple quick web
searches, I'm not too impressed by it's GC.
> wunder
>
> -Original Message-
> From: Mark Miller [mailto:markrmil...@gmail.com]
> Sent
]
Sent: Friday, September 25, 2009 10:31 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr and Garbage Collection
My bad - later, it looks as if your giving general advice, and thats
what I took issue with.
Any Collector that is not doing generational collection is essentially
from the dark ages a
arge
> _h.html
>
> wunder
>
> -Original Message-
> From: Mark Miller [mailto:markrmil...@gmail.com]
> Sent: Friday, September 25, 2009 10:03 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr and Garbage Collection
>
> Walter Underwood wrote:
>
&g
er [mailto:markrmil...@gmail.com]
> Sent: Friday, September 25, 2009 10:03 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr and Garbage Collection
>
> Walter Underwood wrote:
> > 30ms is not better or worse than 1s until you look at the service
> > requirements. Fo
on_mccurry/tuning_the_ibm_jvm_for_large
_h.html
wunder
-Original Message-
From: Mark Miller [mailto:markrmil...@gmail.com]
Sent: Friday, September 25, 2009 10:03 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr and Garbage Collection
Walter Underwood wrote:
> 30ms is not better or worse than 1s unti
ou have two
> live Searcher objects with all their caches. Make sure you have headroom for
> a commit.
>
> If you want to test the tenured space usage, you must test with real world
> queries. Those are the only way to get accurate cache eviction rates.
>
> wunder
>
queries. Those are the only way to get accurate cache eviction rates.
wunder
-Original Message-
From: Jonathan Ariel [mailto:ionat...@gmail.com]
Sent: Friday, September 25, 2009 9:34 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr and Garbage Collection
BTW why making them equal
>-server option of JVM is 'native CPU code', I remember WebLogic 7 console
>with SUN JVM 1.3 not showing any GC (just horizontal line).
Not sure what that is all about either. -server and -client are just two
different versions of hotspot.
The -server version is optimized for long running applicat
It won't really - it will just keep the JVM from wasting time resizing
the heap on you. Since you know you need so much RAM anyway, no reason
not to just pin it at what you need.
Not going to help you much with GC though.
Jonathan Ariel wrote:
> BTW why making them equal will lower the frequency o
BTW why making them equal will lower the frequency of GC?
On 9/25/09, Fuad Efendi wrote:
>> Bigger heaps lead to bigger GC pauses in general.
>
> Opposite viewpoint:
> 1sec GC happening once an hour is MUCH BETTER than 30ms GC once-per-second.
>
> To lower frequency of GC: -Xms4096m -Xmx4096m (ma
I can't really understand how increasing the heap will decrease the
11% dedicated to GC
On 9/25/09, Fuad Efendi wrote:
>> You are saying that I should give more memory than 12GB?
>
>
> Yes. Look at this:
>
>> > SEVERE: java.lang.OutOfMemoryError: Java heap space
>>
> org.apache.lucene.search.Fiel
I would look at the JVM. Have you tried switching to the concurrent low
pause collector ?
Colin.
-Original Message-
From: Jonathan Ariel [mailto:ionat...@gmail.com]
Sent: Friday, September 25, 2009 12:07 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr and Garbage Collection
You
for just this instance of FieldCache.
>>
>> GC does not play any role. FieldCache won't be GC-collected.
>>
>>
>> -Fuad
>> http://www.linkedin.com/in/liferay
>>
>>
>>
>>
>>> -----Original Message-
>>> From: Jonath
> You are saying that I should give more memory than 12GB?
Yes. Look at this:
> > SEVERE: java.lang.OutOfMemoryError: Java heap space
>
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:3
> 61
> > )
It can't find few (!!!) contiguous bytes for .createValue(...)
It ca
>
>
> > -Original Message-
> > From: Jonathan Ariel [mailto:ionat...@gmail.com]
> > Sent: September-25-09 11:37 AM
> > To: solr-user@lucene.apache.org; yo...@lucidimagination.com
> > Subject: Re: Solr and Garbage Collection
> >
> > Right, now I'
il.com]
> Sent: September-25-09 11:37 AM
> To: solr-user@lucene.apache.org; yo...@lucidimagination.com
> Subject: Re: Solr and Garbage Collection
>
> Right, now I'm giving it 12GB of heap memory.
> If I give it less (10GB) it throws the following e
> Bigger heaps lead to bigger GC pauses in general.
Opposite viewpoint:
1sec GC happening once an hour is MUCH BETTER than 30ms GC once-per-second.
To lower frequency of GC: -Xms4096m -Xmx4096m (make it equal!)
Use -server option.
-server option of JVM is 'native CPU code', I remember WebLogic
I've got the start of a Garbage Collection article here:
http://www.lucidimagination.com/blog/2009/09/19/java-garbage-collection-boot-camp-draft/
I plan to tie it more into Lucene/Solr and add some more about the
theory/methods in the final version.
With so much RAM, I take it you prob have a han
should help with the long
pauses.
Colin.
-Original Message-
From: Jonathan Ariel [mailto:ionat...@gmail.com]
Sent: Friday, September 25, 2009 11:37 AM
To: solr-user@lucene.apache.org; yo...@lucidimagination.com
Subject: Re: Solr and Garbage Collection
Right, now I'm giving it 12GB of
Right, now I'm giving it 12GB of heap memory.
If I give it less (10GB) it throws the following exception:
Sep 5, 2009 7:18:32 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCache
On Fri, Sep 25, 2009 at 9:30 AM, Jonathan Ariel wrote:
> Hi to all!
> Lately my solr servers seem to stop responding once in a while. I'm using
> solr 1.3.
> Of course I'm having more traffic on the servers.
> So I logged the Garbage Collection activity to check if it's because of
> that. It seems
Hi to all!
Lately my solr servers seem to stop responding once in a while. I'm using
solr 1.3.
Of course I'm having more traffic on the servers.
So I logged the Garbage Collection activity to check if it's because of
that. It seems like 11% of the time the application runs, it is stopped
because of
75 matches
Mail list logo
