Jonathan Ariel wrote: > Well.. it is strange that when I use the default GC I don't get any errors. > Not so strange - it's different code. The bug is Likely in the low pause collector and not the serial collector. > If I'm so close to run out of memory I should see those OOM exceptions as > well with the standard GC. Those? Your not seeing any that you mentioned unless you lower your heap? > BTW I'm faceting on around 13 fields and my total > number of unique values is around 30000. > One of the fields with the biggest amount of unique values has almost 16000 > unique values. > > > On Sun, Sep 27, 2009 at 4:32 PM, Fuad Efendi <f...@efendi.ca> wrote: > > >> Mark, >> >> >> Nothing against orange-hat :) >> >> Nothing against GC tuning; but if SOLR needs application-specific settings >> it should be well-documented. >> >> GC-tuning: for instance, we need it for 'realtime' Online Trading >> applications. However, even Online Banking doesn't need; primary reason - >> GC >> must happen 'outside of current transaction', GC 'must be predictable', and >> (for instance) Oracle/BEA JRockit has specific 'realtime' version for >> that... Does SOLR need that? >> >> >> Having load-stress simulator (multithreaded!!!) will definitely help to >> predict any possible bottleneck... it's even better to write it from >> scratch >> (depends on schema!), by sending random requests to SOLR in-parallel... >> instead of waiting when FieldCache tries to add new FieldImpl to cache >> (unpredictable!) >> >> >> Tomcat is multithreaded; what if end-users need to load 1000s large >> documents (in parallel! 1000s concurrent users), can you predict memory >> requirements and GC options without application-specific knowledge? What >> about new SOLR-Caches warming up? >> >> >> -Fuad >> >> >> >>> -----Original Message----- >>> From: Mark Miller [mailto:markrmil...@gmail.com] >>> Sent: September-27-09 2:46 PM >>> To: solr-user@lucene.apache.org >>> Subject: Re: Solr and Garbage Collection >>> >>> If he needed double the RAM, he'd likely know by now :) The JVM likes to >>> throw OOM exceptions when you need more RAM. Until it does - thats an >>> odd path to focus on. There has been no indication he has ever seen an >>> OOM with his over 10 GB heap. It sounds like he has run Solr in his >>> environment for quite a long time - after running for that long, until >>> he gets an OOM, its about as good as chasing ghost to worry about it. >>> >>> I like to think of GC tuning as orange-hat. Mostly because I like the >>> color orange. >>> >>> Fuad Efendi wrote: >>> >>>>>> Ok. After the server ran for more than 12 hours, the time spent on GC >>>>>> decreased from 11% to 3,4%, but 5 hours later it crashed. >>>>>> >>>>>> >>>> All this 'black-hat' GC tuning and 'fast' object moving (especially >>>> >> objects >> >>>> accessing by some thread during GC-defragmentation) >>>> >>>> - try to use multithreaded load-stress tools (at least 100 requests >>>> in-parallel) and see that you need at least double memory if 12Gb is >>>> threshold for your FieldCache (largest objects) >>>> >>>> >>>> Also, don't trust this counters: >>>> >>>> >>>>> So I logged the Garbage Collection activity to check if it's because >>>>> >> of >> >>>>> that. It seems like 11% of the time the application runs, it is >>>>> >> stopped >> >>>>> because of GC. >>>>> >>>>> >>>> Stopped? Of course, locking/unlocking in order to move objects >>>> >> currently >> >>>> accessesd in multiuser-multithreaded Tomcat... you can easily create >>>> >> crash >> >>>> scenario proving that latest-greatest JVMs are buggy too. >>>> >>>> >>>> >>>> Don't forget: Tomcat is multithreaded, and if 'core' needs 10Gb in >>>> >> order >> to >> >>>> avoid OOM, you need to double it (in order to warm new cash instances >>>> >> on >> >>>> index replica / update). >>>> >>>> >>>> http://www.linkedin.com/in/liferay >>>> >>>> >>>> >>>> >>> -- >>> - Mark >>> >>> http://www.lucidimagination.com >>> >>> >>> >> >> >> > >
-- - Mark http://www.lucidimagination.com
