Did you find any more clues? I have this problem in my machines as well.. On Fri, Jun 29, 2012 at 6:04 AM, Bernd Fehling < bernd.fehl...@uni-bielefeld.de> wrote:
> Hi list, > > while monitoring my solr 3.6.1 installation I recognized an increase of > memory usage > in OldGen JVM heap on my slave. I decided to force Full GC from jvisualvm > and > send optimize to the already optimized slave index. Normally this helps > because > I have monitored this issue over the past. But not this time. The Full GC > didn't free any memory. So I decided to take a heap dump and see what > MemoryAnalyzer > is showing. The heap dump is about 23 GB in size. > > 1.) > Report Top consumers - Biggest Objects: > Total: 12.3 GB > org.apache.lucene.search.FieldCacheImpl : 8.1 GB > class java.lang.ref.Finalizer : 2.1 GB > org.apache.solr.util.ConcurrentLRUCache : 1.5 GB > org.apache.lucene.index.ReadOnlySegmentReader : 622.5 MB > ... > > As you can see, Finalizer has already reached 2.1 GB!!! > > * java.util.concurrent.ConcurrentHashMap$Segment[16] @ 0x37b056fd0 > * segments java.util.concurrent.ConcurrentHashMap @ 0x39b02d268 > * map org.apache.solr.util.ConcurrentLRUCache @ 0x398f33c30 > * referent java.lang.ref.Finalizer @ 0x37affa810 > * next java.lang.ref.Finalizer @ 0x37affa838 > ... > > Seams to be org.apache.solr.util.ConcurrentLRUCache > The attributes are: > > Type | Name | Value > ----------------------------- > boolean| isDestroyed | true > ----------------------------- > ref | cleanupThread | null > ---------------------------- > ref | evictionListener | null > ------------------------------- > long | oldestEntry | 0 > ---------------------------------- > int | acceptableWaterMark | 9500 > -------------------------------------------------------------------------- > ref | stats | org.apache.solr.util.ConcurrentLRUCache$Stats > @ 0x37b074dc8 > ------------------------ > boolean| islive | true > ------------------------------------- > boolean| newThreadForCleanup | false > ---------------------------- > boolean| isCleaning | false > > -------------------------------------------------------------------------------- > ref | markAndSweepLock | java.util.concurrent.locks.ReentrantLock @ > 0x39bf63978 > ----------------------------- > int | lowerWaterMark | 9000 > ----------------------------- > int | upperWaterMark | 10000 > ----------------------------- > ref | map | java.util.concurrent.ConcurrentHashMap @ > 0x39b02d268 > ------------------------------------------------------------------ > > > > > 2.) > While searching for open files and their references I noticed that there > are references to > index files which are already deleted from disk. > E.g. recent index files are "data/index/_2iqw.frq" and > "data/index/_2iqx.frq". > But I also see references to "data/index/_2hid.frq" which are quite old > and are deleted way back > from earlier replications. > I have to analyze this a bit deeper. > > > So far my report, I go on analyzing this huge heap dump. > If you need any other info or even the heap dump, let me know. > > > Regards > Bernd > >
