Jonathan,
Here is the JVM argument for logging GC activity:
-Xloggc:<file> log GC status to a file with time stamps
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: Jonathan Ariel <ionat...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Monday, September 28, 2009 4:49:03 PM
> Subject: Re: Solr and Garbage Collection
>
> How do you track major collections? Even better, how do you log your GC
> behavior with details? Right now I just log total time spent on collections,
> but I don't really know on which collections.Regard application performance
> with the ConcMarkSweepGC, I think I didn't experience any impact for now.
> Actually the CPU usage of the solr servers is almost insignificant (it was
> like that before).
> BTW, do you know a good way to track the N most expensive solr queries? I
> would like to measure that on 2 different solr servers with different GC.
>
> On Mon, Sep 28, 2009 at 4:42 PM, Mark Miller wrote:
>
> > Do you have your GC logs? Are you still seeing major collections?
> >
> > Where is the time spent?
> >
> > Hard to say without some of that info.
> >
> > The goal of the low pause collector is to finish collecting before the
> > tenured space is filled - if it doesn't, a standard major collection
> > occurs.
> >
> > The collector will use recent stats it records to try and pick a good
> > time to start - as a fail safe though, it will trigger no matter what at
> > a certain percentage. With Java 1.5, it was 68% full that it triggered.
> > With 1.6, its 92%.
> >
> > If your still getting major collections, you might want to see if
> > lowering that helps (-XX:CMSInitiatingOccupancyFraction=). If not,
> > you might be near optimal settings.
> >
> > There is likely not anything else you should mess with - unless using
> > the extra thread to collect while your app is running affects your apps
> > performance - in that case you might want to look into turning on the
> > incremental mode. But you havn't mentioned that, so I doubt it.
> >
> >
> >
> > --
> > - Mark
> >
> > http://www.lucidimagination.com
> >
> >
> >
> > Jonathan Ariel wrote:
> > > Ok... good news! Upgrading to the newest version of JVM 6 (update 6)
> > seems
> > > to solve this ugly bug. With the upgraded JVM I could run the solr
> > servers
> > > for more than 12 hours on the production environment with the GC
> > mentioned
> > > in the previous e-mails. The results are really amazing. The time spent
> > on
> > > collecting memory dropped from 11% to 3.81%Do you think there is more to
> > > tune there?
> > >
> > > Thanks!
> > >
> > > Jonathan
> > >
> > > On Sun, Sep 27, 2009 at 8:39 PM, Bill Au wrote:
> > >
> > >
> > >> You are running a very old version of Java 6 (update 6). The latest is
> > >> update 16. You should definitely upgrade. There is a bug in Java 6
> > >> starting with update 4 that may result in a corrupted Lucene/Solr index:
> > >> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6707044
> > >> https://issues.apache.org/jira/browse/LUCENE-1282
> > >>
> > >> The JVM crash occurred in the gc thread. So it looks like a bug in the
> > JVM
> > >> itself. Upgrading to the latest release might help. Switching to a
> > >> different garbage collector should help.
> > >>
> > >> Bill
> > >>
> > >> On Sat, Sep 26, 2009 at 4:31 PM, Mark Miller
> > >> wrote:
> > >>
> > >>
> > >>> Jonathan Ariel wrote:
> > >>>
> > >>>> Ok. After the server ran for more than 12 hours, the time spent on GC
> > >>>> decreased from 11% to 3,4%, but 5 hours later it crashed. This is the
> > >>>>
> > >>> thread
> > >>>
> > >>>> dump, maybe you can help identify what happened?
> > >>>>
> > >>>>
> > >>> Well thats a tough ;) My guess is its a bug :)
> > >>>
> > >>> Your two survivor spaces are filled, so it was likely about to move
> > >>> objects into the tenured space, which still has plenty of room for them
> > >>> (barring horrible fragmentation). Any issues with that type of thing
> > >>> should generate an OOM anyway though. You can find people that have run
> > >>> into similar issues in the past, but a lot of times unreproducible.
> > >>> Usually, their bugs are closed and they are told to try a newer JVM.
> > >>>
> > >>> Your JVM appears to be quite a few versions back. There have been many
> > >>> garbage collection bugs fixed in the 7 or so updates since your
> > version,
> > >>> a good handful of them related to CMS.
> > >>>
> > >>> If you can, my best suggestion at the moment is to upgrade to the
> > latest
> > >>> and see how that fairs.
> > >>>
> > >>> If not, you might see if going back to the throughput collector and
> > >>> turning on the parallel tenured space collector might meet your needs
> > >>> instead. You can work with other params to get that going better if you
> > >>> have to as well.
> > >>>
> > >>> Also, adjusting other settings with the low pause collector might
> > >>> trigger something to side step the bug. Not a great option there though
> > >>>
> > >> ;)
> > >>
> > >>> How many unique fields are you sorting/faceting on? It must be a lot if
> > >>> you need 10 gig for 8 million documents. Its kind of rough to have to
> > >>> work at such a close limit to your total heap available as a min mem
> > >>> requirement.
> > >>>
> > >>> --
> > >>> - Mark
> > >>>
> > >>> http://www.lucidimagination.com
> > >>>
> > >>>
> > >>>
> > >>>> #
> > >>>> # An unexpected error has been detected by Java Runtime Environment:
> > >>>> #
> > >>>> # SIGSEGV (0xb) at pc=0x00002b4e0f69ea2a, pid=32224, tid=1103812928
> > >>>> #
> > >>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode
> > >>>> linux-amd64)
> > >>>> # Problematic frame:
> > >>>> # V [libjvm.so+0x265a2a]
> > >>>> #
> > >>>> # If you would like to submit a bug report, please visit:
> > >>>> # http://java.sun.com/webapps/bugreport/crash.jsp
> > >>>> #
> > >>>>
> > >>>> --------------- T H R E A D ---------------
> > >>>>
> > >>>> Current thread (0x000000005be47400): VMThread [stack:
> > >>>> 0x0000000041bad000,0x0000000041cae000] [id=32249]
> > >>>>
> > >>>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (),
> > >>>> si_addr=0x0000000000000000
> > >>>>
> > >>>> Registers:
> > >>>> RAX=0x00002aac929b4c70, RBX=0x0037c985003a095e,
> > RCX=0x0000000000000006,
> > >>>> RDX=0x005c49870037c996
> > >>>> RSP=0x0000000041cac550, RBP=0x0000000041cac550,
> > RSI=0x00002aac929b4c70,
> > >>>> RDI=0x0037c985003a095e
> > >>>> R8 =0x00002aadab201538, R9 =0x0000000000000005,
> > R10=0x0000000000000001,
> > >>>> R11=0x0000000000000010
> > >>>> R12=0x00002aac929b4c70, R13=0x00002aac9289cf58,
> > R14=0x00002aac9289cf40,
> > >>>> R15=0x00002aadab2015ac
> > >>>> RIP=0x00002b4e0f69ea2a, EFL=0x0000000000010206,
> > >>>>
> > >>> CSGSFS=0x0000000000000033,
> > >>>
> > >>>> ERR=0x0000000000000000
> > >>>> TRAPNO=0x000000000000000d
> > >>>>
> > >>>> Top of Stack: (sp=0x0000000041cac550)
> > >>>> 0x0000000041cac550: 0000000041cac580 00002b4e0f903c5b
> > >>>> 0x0000000041cac560: 0000000041cac590 0000000000000003
> > >>>> 0x0000000041cac570: 00002aac9289cf50 00002aadab2015a8
> > >>>> 0x0000000041cac580: 0000000041cac5c0 00002b4e0f72e388
> > >>>> 0x0000000041cac590: 0000000041cac5c0 00002aac9289cf40
> > >>>> 0x0000000041cac5a0: 0000000000000005 00002b4e0fc86330
> > >>>> 0x0000000041cac5b0: 0000000000000000 00002b4e0fd8c740
> > >>>> 0x0000000041cac5c0: 0000000041cac5f0 00002b4e0f903b7f
> > >>>> 0x0000000041cac5d0: 0000000041cac610 0000000000000003
> > >>>> 0x0000000041cac5e0: 00002aaccb1750f8 00002aaccea41570
> > >>>> 0x0000000041cac5f0: 0000000041cac610 00002b4e0f931548
> > >>>> 0x0000000041cac600: 00002b4e0fc861d8 00002aadd4052ab0
> > >>>> 0x0000000041cac610: 0000000041cac640 00002b4e0f903d1a
> > >>>> 0x0000000041cac620: 0000000041cac650 0000000000000003
> > >>>> 0x0000000041cac630: 000000005bc7d6d0 00002b4e0fd8c740
> > >>>> 0x0000000041cac640: 0000000041cac650 00002b4e0f90411c
> > >>>> 0x0000000041cac650: 0000000041cac680 00002b4e0fa1d16e
> > >>>> 0x0000000041cac660: 0000000000000000 000000005bc7d6d0
> > >>>> 0x0000000041cac670: 0000000000000002 00002b4e0fd8c740
> > >>>> 0x0000000041cac680: 0000000041cac6c0 00002b4e0fa74640
> > >>>> 0x0000000041cac690: 0000000041cac6b0 000000005bc7d6d0
> > >>>> 0x0000000041cac6a0: 0000000000000002 00002b4e0fd8c740
> > >>>> 0x0000000041cac6b0: 0000000000000001 00002b4e0fd8c740
> > >>>> 0x0000000041cac6c0: 0000000041cac700 00002b4e0f9a52da
> > >>>> 0x0000000041cac6d0: 000000000000bfc0 0000000000000000
> > >>>> 0x0000000041cac6e0: 00002b4e0fd8c740 000000005bc7d6d0
> > >>>> 0x0000000041cac6f0: 00002b4e0fd8c740 0000000000000001
> > >>>> 0x0000000041cac700: 0000000041cac750 00002b4e0f6feb80
> > >>>> 0x0000000041cac710: 449dae1d9ae42358 3ff0cccccccccccd
> > >>>> 0x0000000041cac720: 00002aad289aa680 0000000000000001
> > >>>> 0x0000000041cac730: 0000000000000000 0000000041cac780
> > >>>> 0x0000000041cac740: 0000000000000001 000000005bc7d6d0
> > >>>>
> > >>>> Instructions: (pc=0x00002b4e0f69ea2a)
> > >>>> 0x00002b4e0f69ea1a: 89 e5 48 83 f9 05 74 38 48 8b 56 08 48 83 c2 10
> > >>>> 0x00002b4e0f69ea2a: 48 8b b2 a0 00 00 00 ba 01 00 00 00 83 e6 07 48
> > >>>>
> > >>>> Stack: [0x0000000041bad000,0x0000000041cae000],
> > sp=0x0000000041cac550,
> > >>>> free space=1021k
> > >>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
> > >>>>
> > >> C=native
> > >>
> > >>>> code)
> > >>>> V [libjvm.so+0x265a2a]
> > >>>> V [libjvm.so+0x4cac5b]
> > >>>> V [libjvm.so+0x2f5388]
> > >>>> V [libjvm.so+0x4cab7f]
> > >>>> V [libjvm.so+0x4f8548]
> > >>>> V [libjvm.so+0x4cad1a]
> > >>>> V [libjvm.so+0x4cb11c]
> > >>>> V [libjvm.so+0x5e416e]
> > >>>> V [libjvm.so+0x63b640]
> > >>>> V [libjvm.so+0x56c2da]
> > >>>> V [libjvm.so+0x2c5b80]
> > >>>> V [libjvm.so+0x2c8866]
> > >>>> V [libjvm.so+0x2c7f10]
> > >>>> V [libjvm.so+0x2551ba]
> > >>>> V [libjvm.so+0x254a6a]
> > >>>> V [libjvm.so+0x254778]
> > >>>> V [libjvm.so+0x2c579c]
> > >>>> V [libjvm.so+0x23502a]
> > >>>> V [libjvm.so+0x2c5b0e]
> > >>>> V [libjvm.so+0x661a5e]
> > >>>> V [libjvm.so+0x66e48a]
> > >>>> V [libjvm.so+0x66da32]
> > >>>> V [libjvm.so+0x66dcb4]
> > >>>> V [libjvm.so+0x66d7ae]
> > >>>> V [libjvm.so+0x50628a]
> > >>>>
> > >>>> VM_Operation (0x000000004076bd20): GenCollectForAllocation, mode:
> > >>>>
> > >>> safepoint,
> > >>>
> > >>>> requested by thread 0x000000005c42d800
> > >>>>
> > >>>>
> > >>>> --------------- P R O C E S S ---------------
> > >>>>
> > >>>> Java Threads: ( => current thread )
> > >>>> 0x000000005c466400 JavaThread "btpool0-502" [_thread_blocked,
> > >>>>
> > >> id=4508,
> > >>
> > >>>> stack(0x0000000046332000,0x0000000046433000)]
> > >>>> 0x000000005c2a2400 JavaThread "btpool0-501" [_thread_blocked,
> > >>>>
> > >> id=4507,
> > >>
> > >>>> stack(0x00000000428f8000,0x00000000429f9000)]
> > >>>> 0x000000005c0fec00 JavaThread "btpool0-500" [_thread_blocked,
> > >>>>
> > >> id=4506,
> > >>
> > >>>> stack(0x0000000043e0d000,0x0000000043f0e000)]
> > >>>> 0x000000005c2ce400 JavaThread "btpool0-498" [_thread_blocked,
> > >>>>
> > >> id=4504,
> > >>
> > >>>> stack(0x0000000042dfd000,0x0000000042efe000)]
> > >>>> 0x000000005be69000 JavaThread "btpool0-497" [_thread_blocked,
> > >>>>
> > >> id=4503,
> > >>
> > >>>> stack(0x0000000045f2e000,0x000000004602f000)]
> > >>>> 0x000000005c30e000 JavaThread "btpool0-496" [_thread_blocked,
> > >>>>
> > >> id=4251,
> > >>
> > >>>> stack(0x0000000044e1d000,0x0000000044f1e000)]
> > >>>> 0x000000005c0ff800 JavaThread "btpool0-495" [_thread_blocked,
> > >>>>
> > >> id=4250,
> > >>
> > >>>> stack(0x0000000043706000,0x0000000043807000)]
> > >>>> 0x000000005c752c00 JavaThread "btpool0-494" [_thread_blocked,
> > >>>>
> > >> id=4249,
> > >>
> > >>>> stack(0x0000000045d2c000,0x0000000045e2d000)]
> > >>>> 0x000000005bf31400 JavaThread "btpool0-493" [_thread_in_native,
> > >>>>
> > >>> id=4248,
> > >>>
> > >>>> stack(0x0000000042cfc000,0x0000000042dfd000)]
> > >>>> 0x000000005c47e000 JavaThread "btpool0-485" [_thread_blocked,
> > >>>>
> > >> id=3426,
> > >>
> > >>>> stack(0x000000004501f000,0x0000000045120000)]
> > >>>> 0x000000005c35d000 JavaThread "btpool0-484" [_thread_blocked,
> > >>>>
> > >> id=2669,
> > >>
> > >>>> stack(0x000000004086f000,0x0000000040970000)]
> > >>>> 0x000000005c583800 JavaThread "btpool0-468" [_thread_blocked,
> > >>>>
> > >> id=1954,
> > >>
> > >>>> stack(0x0000000043d0c000,0x0000000043e0d000)]
> > >>>> 0x000000005c35ec00 JavaThread "btpool0-448" [_thread_in_native,
> > >>>>
> > >>> id=32527,
> > >>>
> > >>>> stack(0x0000000045423000,0x0000000045524000)]
> > >>>> 0x000000005c2fd000 JavaThread "btpool0-437" [_thread_blocked,
> > >>>>
> > >> id=30735,
> > >>
> > >>>> stack(0x0000000046433000,0x0000000046534000)]
> > >>>> 0x000000005bc77c00 JavaThread "DestroyJavaVM" [_thread_blocked,
> > >>>>
> > >>> id=32230,
> > >>>
> > >>>> stack(0x00000000404b8000,0x00000000405b9000)]
> > >>>> 0x000000005c42d800 JavaThread "pool-1-thread-1" [_thread_blocked,
> > >>>> id=32272, stack(0x000000004066d000,0x000000004076e000)]
> > >>>> 0x000000005c1e9400 JavaThread "Timer-1" [_thread_blocked, id=32271,
> > >>>> stack(0x0000000040da5000,0x0000000040ea6000)]
> > >>>> 0x000000005c0b2c00 JavaThread "btpool0-9 - Acceptor0 SocketConnector
> > >>>>
> > >> @
> > >>
> > >>>> 0.0.0.0:8983" [_thread_in_native, id=32268,
> > >>>> stack(0x0000000043201000,0x0000000043302000)]
> > >>>> 0x000000005c176800 JavaThread "Timer-0" [_thread_blocked, id=32258,
> > >>>> stack(0x00000000427f7000,0x00000000428f8000)]
> > >>>> 0x000000005be81800 JavaThread "Low Memory Detector" daemon
> > >>>> [_thread_blocked, id=32256,
> > >>>>
> > >> stack(0x00000000425f5000,0x00000000426f6000)]
> > >>
> > >>>> 0x000000005be7fc00 JavaThread "CompilerThread1" daemon
> > >>>>
> > >>> [_thread_blocked,
> > >>>
> > >>>> id=32255, stack(0x00000000416f3000,0x00000000417f4000)]
> > >>>> 0x000000005be7c400 JavaThread "CompilerThread0" daemon
> > >>>>
> > >>> [_thread_blocked,
> > >>>
> > >>>> id=32254, stack(0x00000000415f2000,0x00000000416f3000)]
> > >>>> 0x000000005be7ac00 JavaThread "Signal Dispatcher" daemon
> > >>>>
> > >>> [_thread_blocked,
> > >>>
> > >>>> id=32253, stack(0x00000000414f1000,0x00000000415f2000)]
> > >>>> 0x000000005be79400 JavaThread "Surrogate Locker Thread (CMS)" daemon
> > >>>> [_thread_blocked, id=32252,
> > >>>>
> > >> stack(0x000000004025d000,0x000000004035e000)]
> > >>
> > >>>> 0x000000005be4dc00 JavaThread "Finalizer" daemon [_thread_blocked,
> > >>>> id=32251, stack(0x00000000424f4000,0x00000000425f5000)]
> > >>>> 0x000000005be4c800 JavaThread "Reference Handler" daemon
> > >>>>
> > >>> [_thread_blocked,
> > >>>
> > >>>> id=32250, stack(0x00000000423f3000,0x00000000424f4000)]
> > >>>>
> > >>>> Other Threads:
> > >>>> =>0x000000005be47400 VMThread [stack:
> > >>>>
> > >>> 0x0000000041bad000,0x0000000041cae000]
> > >>>
> > >>>> [id=32249]
> > >>>> 0x000000005be83800 WatcherThread [stack:
> > >>>> 0x00000000426f6000,0x00000000427f7000] [id=32257]
> > >>>>
> > >>>> VM state:at safepoint (normal execution)
> > >>>>
> > >>>> VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
> > >>>> [0x000000005bc75420/0x000000005bc75470] Threads_lock - owner thread:
> > >>>> 0x000000005be47400
> > >>>> [0x000000005bc75c90/0x000000005bc75cc0] Heap_lock - owner thread:
> > >>>> 0x000000005c42d800
> > >>>>
> > >>>> Heap
> > >>>> par new generation total 153344K, used 88065K [0x00002aaaae1f0000,
> > >>>> 0x00002aaab8850000, 0x00002aaab8850000)
> > >>>> eden space 136320K, 52% used [0x00002aaaae1f0000,
> > >>>>
> > >> 0x00002aaab2750590,
> > >>
> > >>>> 0x00002aaab6710000)
> > >>>> from space 17024K, 100% used [0x00002aaab77b0000,
> > 0x00002aaab8850000,
> > >>>> 0x00002aaab8850000)
> > >>>> to space 17024K, 99% used [0x00002aaab6710000,
> > 0x00002aaab77afb00,
> > >>>> 0x00002aaab77b0000)
> > >>>> concurrent mark-sweep generation total 12363392K, used 10670634K
> > >>>> [0x00002aaab8850000, 0x00002aadab1f0000, 0x00002aadab1f0000)
> > >>>> concurrent-mark-sweep perm gen total 35344K, used 21182K
> > >>>> [0x00002aadab1f0000, 0x00002aadad474000, 0x00002aadb05f0000)
> > >>>>
> > >>>> Dynamic libraries:
> > >>>> 40000000-40009000 r-xp 00000000 08:05 12505572
> > >>>> /usr/local/jdk1.6.0_06/bin/java
> > >>>> 40108000-4010a000 rwxp 00008000 08:05 12505572
> > >>>> /usr/local/jdk1.6.0_06/bin/java
> > >>>> 4015c000-4015d000 ---p 4015c000 00:00 0
> > >>>> 4015d000-4025d000 rwxp 4015d000 00:00 0
> > >>>> 4025d000-40260000 ---p 4025d000 00:00 0
> > >>>> 40260000-4035e000 rwxp 40260000 00:00 0
> > >>>> 4035e000-40361000 rwxp 4035e000 00:00 0
> > >>>> 40361000-4045f000 rwxp 40361000 00:00 0
> > >>>> 404b8000-404bb000 ---p 404b8000 00:00 0
> > >>>> 404bb000-405b9000 rwxp 404bb000 00:00 0
> > >>>> 4066d000-40670000 ---p 4066d000 00:00 0
> > >>>> 40670000-4076e000 rwxp 40670000 00:00 0
> > >>>> 4076e000-40771000 rwxp 4076e000 00:00 0
> > >>>> 40771000-4086f000 rwxp 40771000 00:00 0
> > >>>> 4086f000-40872000 ---p 4086f000 00:00 0
> > >>>> 40872000-40970000 rwxp 40872000 00:00 0
> > >>>> 40da5000-40da8000 ---p 40da5000 00:00 0
> > >>>> 40da8000-40ea6000 rwxp 40da8000 00:00 0
> > >>>> 40ea6000-40ea9000 rwxp 40ea6000 00:00 0
> > >>>> 40ea9000-40fa7000 rwxp 40ea9000 00:00 0
> > >>>> 40fec000-40fed000 ---p 40fec000 00:00 0
> > >>>> 40fed000-410ed000 rwxp 40fed000 00:00 0
> > >>>> 410ed000-410ee000 ---p 410ed000 00:00 0
> > >>>> 410ee000-411ee000 rwxp 410ee000 00:00 0
> > >>>> 411ee000-411ef000 ---p 411ee000 00:00 0
> > >>>> 411ef000-412ef000 rwxp 411ef000 00:00 0
> > >>>> 412ef000-412f0000 ---p 412ef000 00:00 0
> > >>>> 412f0000-413f0000 rwxp 412f0000 00:00 0
> > >>>> 413f0000-413f1000 ---p 413f0000 00:00 0
> > >>>> 413f1000-414f1000 rwxp 413f1000 00:00 0
> > >>>> 414f1000-414f4000 ---p 414f1000 00:00 0
> > >>>> 414f4000-415f2000 rwxp 414f4000 00:00 0
> > >>>> 415f2000-415f5000 ---p 415f2000 00:00 0
> > >>>> 415f5000-416f3000 rwxp 415f5000 00:00 0
> > >>>> 416f3000-416f6000 ---p 416f3000 00:00 0
> > >>>> 416f6000-417f4000 rwxp 416f6000 00:00 0
> > >>>> 418f5000-418f8000 rwxp 418f5000 00:00 0
> > >>>> 418f8000-419f6000 rwxp 418f8000 00:00 0
> > >>>> 41bad000-41bae000 ---p 41bad000 00:00 0
> > >>>> 41bae000-41cae000 rwxp 41bae000 00:00 0
> > >>>> 41daf000-41db2000 rwxp 41daf000 00:00 0
> > >>>> 41db2000-41eb0000 rwxp 41db2000 00:00 0
> > >>>> 41eee000-41eef000 ---p 41eee000 00:00 0
> > >>>> 41eef000-41fef000 rwxp 41eef000 00:00 0
> > >>>> 41fef000-41ff0000 ---p 41fef000 00:00 0
> > >>>> 41ff0000-420f0000 rwxp 41ff0000 00:00 0
> > >>>> 420f0000-420f1000 ---p 420f0000 00:00 0
> > >>>> 420f1000-421f1000 rwxp 420f1000 00:00 0
> > >>>> 421f1000-421f2000 ---p 421f1000 00:00 0
> > >>>> 421f2000-422f2000 rwxp 421f2000 00:00 0
> > >>>> 422f2000-422f3000 ---p 422f2000 00:00 0
> > >>>> 422f3000-423f3000 rwxp 422f3000 00:00 0
> > >>>> 423f3000-423f6000 ---p 423f3000 00:00 0
> > >>>> 423f6000-424f4000 rwxp 423f6000 00:00 0
> > >>>> 424f4000-424f7000 ---p 424f4000 00:00 0
> > >>>> 424f7000-425f5000 rwxp 424f7000 00:00 0
> > >>>> 425f5000-425f8000 ---p 425f5000 00:00 0
> > >>>> 425f8000-426f6000 rwxp 425f8000 00:00 0
> > >>>> 426f6000-426f7000 ---p 426f6000 00:00 0
> > >>>> 426f7000-427f7000 rwxp 426f7000 00:00 0
> > >>>> 427f7000-427fa000 ---p 427f7000 00:00 0
> > >>>> 427fa000-428f8000 rwxp 427fa000 00:00 0
> > >>>> 428f8000-428fb000 ---p 428f8000 00:00 0
> > >>>> 428fb000-429f9000 rwxp 428fb000 00:00 0
> > >>>> 42afa000-42afd000 rwxp 42afa000 00:00 0
> > >>>> 42afd000-42bfb000 rwxp 42afd000 00:00 0
> > >>>> 42bfb000-42bfe000 rwxp 42bfb000 00:00 0
> > >>>> 42bfe000-42cfc000 rwxp 42bfe000 00:00 0
> > >>>> 42cfc000-42cff000 ---p 42cfc000 00:00 0
> > >>>> 42cff000-42dfd000 rwxp 42cff000 00:00 0
> > >>>> 42dfd000-42e00000 ---p 42dfd000 00:00 0
> > >>>> 42e00000-42efe000 rwxp 42e00000 00:00 0
> > >>>> 42fff000-43002000 rwxp 42fff000 00:00 0
> > >>>> 43002000-43100000 rwxp 43002000 00:00 0
> > >>>> 43100000-43103000 rwxp 43100000 00:00 0
> > >>>> 43103000-43201000 rwxp 43103000 00:00 0
> > >>>> 43201000-43204000 ---p 43201000 00:00 0
> > >>>> 43204000-43302000 rwxp 43204000 00:00 0
> > >>>> 43302000-43305000 rwxp 43302000 00:00 0
> > >>>> 43305000-43403000 rwxp 43305000 00:00 0
> > >>>> 43706000-43709000 ---p 43706000 00:00 0
> > >>>> 43709000-43807000 rwxp 43709000 00:00 0
> > >>>> 43807000-4380a000 rwxp 43807000 00:00 0
> > >>>> 4380a000-43908000 rwxp 4380a000 00:00 0
> > >>>> 43908000-4390b000 rwxp 43908000 00:00 0
> > >>>> 4390b000-43a09000 rwxp 4390b000 00:00 0
> > >>>> 43a09000-43a0c000 rwxp 43a09000 00:00 0
> > >>>> 43a0c000-43b0a000 rwxp 43a0c000 00:00 0
> > >>>> 43b0a000-43b0d000 rwxp 43b0a000 00:00 0
> > >>>> 43b0d000-43c0b000 rwxp 43b0d000 00:00 0
> > >>>> 43c0b000-43c0e000 rwxp 43c0b000 00:00 0
> > >>>> 43c0e000-43d0c000 rwxp 43c0e000 00:00 0
> > >>>> 43d0c000-43d0f000 ---p 43d0c000 00:00 0
> > >>>> 43d0f000-43e0d000 rwxp 43d0f000 00:00 0
> > >>>> 43e0d000-43e10000 ---p 43e0d000 00:00 0
> > >>>> 43e10000-43f0e000 rwxp 43e10000 00:00 0
> > >>>> 43f0e000-43f11000 rwxp 43f0e000 00:00 0
> > >>>> 43f11000-4400f000 rwxp 43f11000 00:00 0
> > >>>> 4400f000-44012000 rwxp 4400f000 00:00 0
> > >>>> 44012000-44110000 rwxp 44012000 00:00 0
> > >>>> 44312000-44315000 rwxp 44312000 00:00 0
> > >>>> 44315000-44413000 rwxp 44315000 00:00 0
> > >>>> 44413000-44416000 rwxp 44413000 00:00 0
> > >>>> 44416000-44514000 rwxp 44416000 00:00 0
> > >>>> 44514000-44517000 rwxp 44514000 00:00 0
> > >>>> 44517000-44615000 rwxp 44517000 00:00 0
> > >>>> 44615000-44618000 rwxp 44615000 00:00 0
> > >>>> 44618000-44716000 rwxp 44618000 00:00 0
> > >>>> 44716000-44719000 rwxp 44716000 00:00 0
> > >>>> 44719000-44817000 rwxp 44719000 00:00 0
> > >>>> 44817000-4481a000 rwxp 44817000 00:00 0
> > >>>> 4481a000-44918000 rwxp 4481a000 00:00 0
> > >>>> 44a19000-44a1c000 rwxp 44a19000 00:00 0
> > >>>> 44a1c000-44b1a000 rwxp 44a1c000 00:00 0
> > >>>> 44e1d000-44e20000 ---p 44e1d000 00:00 0
> > >>>> 44e20000-44f1e000 rwxp 44e20000 00:00 0
> > >>>> 4501f000-45022000 ---p 4501f000 00:00 0
> > >>>> 45022000-45120000 rwxp 45022000 00:00 0
> > >>>> 45120000-45123000 rwxp 45120000 00:00 0
> > >>>> 45123000-45221000 rwxp 45123000 00:00 0
> > >>>> 45322000-45325000 rwxp 45322000 00:00 0
> > >>>> 45325000-45423000 rwxp 45325000 00:00 0
> > >>>> 45423000-45426000 ---p 45423000 00:00 0
> > >>>> 45426000-45524000 rwxp 45426000 00:00 0
> > >>>> 45524000-45527000 rwxp 45524000 00:00 0
> > >>>> 45527000-45625000 rwxp 45527000 00:00 0
> > >>>> 45625000-45628000 rwxp 45625000 00:00 0
> > >>>> 45628000-45726000 rwxp 45628000 00:00 0
> > >>>> 45726000-45729000 rwxp 45726000 00:00 0
> > >>>> 45729000-45827000 rwxp 45729000 00:00 0
> > >>>> 45928000-4592b000 rwxp 45928000 00:00 0
> > >>>> 4592b000-45a29000 rwxp 4592b000 00:00 0
> > >>>> 45a29000-45a2c000 rwxp 45a29000 00:00 0
> > >>>> 45a2c000-45b2a000 rwxp 45a2c000 00:00 0
> > >>>> 45b2a000-45b2d000 rwxp 45b2a000 00:00 0
> > >>>> 45b2d000-45c2b000 rwxp 45b2d000 00:00 0
> > >>>> 45d2c000-45d2f000 ---p 45d2c000 00:00 0
> > >>>> 45d2f000-45e2d000 rwxp 45d2f000 00:00 0
> > >>>> 45f2e000-45f31000 ---p 45f2e000 00:00 0
> > >>>> 45f31000-4602f000 rwxp 45f31000 00:00 0
> > >>>> 46332000-46335000 ---p 46332000 00:00 0
> > >>>> 46335000-46433000 rwxp 46335000 00:00 0
> > >>>> 46433000-46436000 ---p 46433000 00:00 0
> > >>>> 46436000-46534000 rwxp 46436000 00:00 0
> > >>>> 46534000-46537000 rwxp 46534000 00:00 0
> > >>>> 46537000-46635000 rwxp 46537000 00:00 0
> > >>>> 46635000-46638000 rwxp 46635000 00:00 0
> > >>>> 46638000-46736000 rwxp 46638000 00:00 0
> > >>>> 5bc70000-5d61a000 rwxp 5bc70000 00:00 0
> > >>>> [heap]
> > >>>> 3b32800000-3b3281c000 r-xp 00000000 08:05 11948691
> > >>>> /lib64/ld-2.5.so
> > >>>> 3b32a1b000-3b32a1c000 r-xp 0001b000 08:05 11948691
> > >>>> /lib64/ld-2.5.so
> > >>>> 3b32a1c000-3b32a1d000 rwxp 0001c000 08:05 11948691
> > >>>> /lib64/ld-2.5.so
> > >>>> 3b32c00000-3b32d4c000 r-xp 00000000 08:05 11948841
> > >>>> /lib64/libc-2.5.so
> > >>>> 3b32d4c000-3b32f4c000 ---p 0014c000 08:05 11948841
> > >>>> /lib64/libc-2.5.so
> > >>>> 3b32f4c000-3b32f50000 r-xp 0014c000 08:05 11948841
> > >>>> /lib64/libc-2.5.so
> > >>>> 3b32f50000-3b32f51000 rwxp 00150000 08:05 11948841
> > >>>> /lib64/libc-2.5.so
> > >>>> 3b32f51000-3b32f56000 rwxp 3b32f51000 00:00 0
> > >>>> 3b33000000-3b33002000 r-xp 00000000 08:05 11948843
> > >>>> /lib64/libdl-2.5.so
> > >>>> 3b33002000-3b33202000 ---p 00002000 08:05 11948843
> > >>>> /lib64/libdl-2.5.so
> > >>>> 3b33202000-3b33203000 r-xp 00002000 08:05 11948843
> > >>>> /lib64/libdl-2.5.so
> > >>>> 3b33203000-3b33204000 rwxp 00003000 08:05 11948843
> > >>>> /lib64/libdl-2.5.so
> > >>>> 3b33c00000-3b33c16000 r-xp 00000000 08:05 11948927
> > >>>> /lib64/libpthread-2.5.so
> > >>>> 3b33c16000-3b33e15000 ---p 00016000 08:05 11948927
> > >>>> /lib64/libpthread-2.5.so
> > >>>> 3b33e15000-3b33e16000 r-xp 00015000 08:05 11948927
> > >>>> /lib64/libpthread-2.5.so
> > >>>> 3b33e16000-3b33e17000 rwxp 00016000 08:05 11948927
> > >>>> /lib64/libpthread-2.5.so
> > >>>> 3b33e17000-3b33e1b000 rwxp 3b33e17000 00:00 0
> > >>>> 3b34000000-3b34082000 r-xp 00000000 08:05 11948934
> > >>>> /lib64/libm-2.5.so
> > >>>> 3b34082000-3b34281000 ---p 00082000 08:05 11948934
> > >>>> /lib64/libm-2.5.so
> > >>>> 3b34281000-3b34282000 r-xp 00081000 08:05 11948934
> > >>>> /lib64/libm-2.5.so
> > >>>> 3b34282000-3b34283000 rwxp 00082000 08:05 11948934
> > >>>> /lib64/libm-2.5.so
> > >>>> 3b34400000-3b34407000 r-xp 00000000 08:05 11948928
> > >>>> /lib64/librt-2.5.so
> > >>>> 3b34407000-3b34607000 ---p 00007000 08:05 11948928
> > >>>> /lib64/librt-2.5.so
> > >>>> 3b34607000-3b34608000 r-xp 00007000 08:05 11948928
> > >>>> /lib64/librt-2.5.so
> > >>>> 3b34608000-3b34609000 rwxp 00008000 08:05 11948928
> > >>>> /lib64/librt-2.5.so
> > >>>> 3b35400000-3b35415000 r-xp 00000000 08:05 11948806
> > >>>> /lib64/libnsl-2.5.so
> > >>>> 3b35415000-3b35614000 ---p 00015000 08:05 11948806
> > >>>> /lib64/libnsl-2.5.so
> > >>>> 3b35614000-3b35615000 r-xp 00014000 08:05 11948806
> > >>>> /lib64/libnsl-2.5.so
> > >>>> 3b35615000-3b35616000 rwxp 00015000 08:05 11948806
> > >>>> /lib64/libnsl-2.5.so
> > >>>> 3b35616000-3b35618000 rwxp 3b35616000 00:00 0
> > >>>> 2aaaaaaad000-2aaaaaaaf000 r-xs 00003000 08:05 8675780
> > >>>> /opt/solr-1.3/olx/start.jar
> > >>>> 2aaaaaaaf000-2aaaaaab5000 r--s 00045000 08:02 393249
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/commons-httpclient-3.1.jar
> > >>
> > >>>> 2aaaaaab8000-2aaaaaab9000 ---p 2aaaaaab8000 00:00 0
> > >>>> 2aaaaaab9000-2aaaaaaba000 rwxp 2aaaaaab9000 00:00 0
> > >>>> 2aaaaaaba000-2aaaaaac1000 r-xp 00000000 08:05 12506743
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/native_threads/libhpi.so
> > >>>> 2aaaaaac1000-2aaaaabc2000 ---p 00007000 08:05 12506743
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/native_threads/libhpi.so
> > >>>> 2aaaaabc2000-2aaaaabc4000 rwxp 00008000 08:05 12506743
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/native_threads/libhpi.so
> > >>>> 2aaaaabc4000-2aaaaabc5000 rwxp 2aaaaabc4000 00:00 0
> > >>>> 2aaaaabc5000-2aaaaabcd000 rw-s 00000000 08:02 163842
> > >>>> /tmp/hsperfdata_root/32224
> > >>>> 2aaaaabce000-2aaaaabd8000 r-xp 00000000 08:05 11948665
> > >>>> /lib64/libnss_files-2.5.so
> > >>>> 2aaaaabd8000-2aaaaadd7000 ---p 0000a000 08:05 11948665
> > >>>> /lib64/libnss_files-2.5.so
> > >>>> 2aaaaadd7000-2aaaaadd8000 r-xp 00009000 08:05 11948665
> > >>>> /lib64/libnss_files-2.5.so
> > >>>> 2aaaaadd8000-2aaaaadd9000 rwxp 0000a000 08:05 11948665
> > >>>> /lib64/libnss_files-2.5.so
> > >>>> 2aaaaadd9000-2aaaaade6000 r-xp 00000000 08:05 12506738
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libverify.so
> > >>>> 2aaaaade6000-2aaaaaee5000 ---p 0000d000 08:05 12506738
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libverify.so
> > >>>> 2aaaaaee5000-2aaaaaee8000 rwxp 0000c000 08:05 12506738
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libverify.so
> > >>>> 2aaaaaee8000-2aaaaaf10000 r-xp 00000000 08:05 12506719
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libjava.so
> > >>>> 2aaaaaf10000-2aaaab010000 ---p 00028000 08:05 12506719
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libjava.so
> > >>>> 2aaaab010000-2aaaab017000 rwxp 00028000 08:05 12506719
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libjava.so
> > >>>> 2aaaab017000-2aaaab025000 r-xp 00000000 08:05 12506739
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libzip.so
> > >>>> 2aaaab025000-2aaaab127000 ---p 0000e000 08:05 12506739
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libzip.so
> > >>>> 2aaaab127000-2aaaab12a000 rwxp 00010000 08:05 12506739
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libzip.so
> > >>>> 2aaaab12a000-2aaaab68b000 rwxp 2aaaab12a000 00:00 0
> > >>>> 2aaaab68b000-2aaaae12b000 rwxp 2aaaab68b000 00:00 0
> > >>>> 2aaaae12b000-2aaaae141000 rwxp 2aaaae12b000 00:00 0
> > >>>> 2aaaae141000-2aaaae1eb000 rwxp 2aaaae141000 00:00 0
> > >>>> 2aaaae1f0000-2aaab8850000 rwxp 2aaaae1f0000 00:00 0
> > >>>> 2aaab8850000-2aadab1f0000 rwxp 2aaab8850000 00:00 0
> > >>>> 2aadab1f0000-2aadad474000 rwxp 2aadab1f0000 00:00 0
> > >>>> 2aadad474000-2aadb05f0000 rwxp 2aadad474000 00:00 0
> > >>>> 2aadb05f0000-2aadb0644000 rwxp 2aadb05f0000 00:00 0
> > >>>> 2aadb0644000-2aadb1dd8000 rwxp 2aadb0644000 00:00 0
> > >>>> 2aadb1dd8000-2aadb1dea000 rwxp 2aadb1dd8000 00:00 0
> > >>>> 2aadb1dea000-2aadb1e02000 rwxp 2aadb1dea000 00:00 0
> > >>>> 2aadb1e02000-2aadb36a0000 rwxp 2aadb1e02000 00:00 0
> > >>>> 2aadb36a0000-2aadb36b2000 rwxp 2aadb36a0000 00:00 0
> > >>>> 2aadb36b2000-2aadb36cb000 rwxp 2aadb36b2000 00:00 0
> > >>>> 2aadb36cb000-2aadbf7ba000 rwxp 2aadb36cb000 00:00 0
> > >>>> 2aadbf7fa000-2aadc014c000 rwxp 2aadbf7fa000 00:00 0
> > >>>> 2aadc014c000-2aadc02d7000 r-xs 02df0000 08:05 12506783
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/rt.jar
> > >>>> 2aadc02d7000-2aadc0309000 rwxp 2aadc02d7000 00:00 0
> > >>>> 2aadc0309000-2aadc38d6000 r-xp 00000000 08:05 12116537
> > >>>> /usr/lib/locale/locale-archive
> > >>>> 2aadc38d6000-2aadc38d9000 r-xs 0001d000 08:05 8675779
> > >>>> /opt/solr-1.3/olx/lib/jetty-util-6.1.3.jar
> > >>>> 2aadc38d9000-2aadc38df000 r-xs 0006e000 08:05 8675772
> > >>>> /opt/solr-1.3/olx/lib/jetty-6.1.3.jar
> > >>>> 2aadc38df000-2aadc38e2000 r-xs 0001e000 08:05 8675778
> > >>>> /opt/solr-1.3/olx/lib/servlet-api-2.5-6.1.3.jar
> > >>>> 2aadc38e2000-2aadc38f1000 r-xs 000ee000 08:05 8675774
> > >>>> /opt/solr-1.3/olx/lib/jsp-2.1/ant-1.6.5.jar
> > >>>> 2aadc38f1000-2aadc38fa000 r-xs 00093000 08:05 8675777
> > >>>> /opt/solr-1.3/olx/lib/jsp-2.1/jsp-2.1.jar
> > >>>> 2aadc38fa000-2aadc391d000 r-xs 00344000 08:05 8675776
> > >>>> /opt/solr-1.3/olx/lib/jsp-2.1/core-3.1.1.jar
> > >>>> 2aadc391d000-2aadc3928000 r-xs 00073000 08:05 8675775
> > >>>> /opt/solr-1.3/olx/lib/jsp-2.1/jsp-api-2.1.jar
> > >>>> 2aadc3928000-2aadc392f000 r--s 00079000 08:02 393246
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/wstx-asl-3.2.7.jar
> > >>
> > >>>> 2aadc392f000-2aadc3931000 r--s 00004000 08:02 393223
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/commons-csv-1.0-SNAPSHOT-r609327.jar
> > >>
> > >>>> 2aadc3931000-2aadc3933000 r--s 0000c000 08:02 393233
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/lucene-memory-2.4-dev.jar
> > >>
> > >>>> 2aadc3933000-2aadc3935000 r--s 00008000 08:02 393250
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/commons-logging-1.0.4.jar
> > >>
> > >>>> 2aadc3935000-2aadc3938000 r--s 00014000 08:02 393232
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/lucene-highlighter-2.4-dev.jar
> > >>
> > >>>> 2aadc3938000-2aadc393a000 r--s 0001a000 08:02 393235
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/lucene-snowball-2.4-dev.jar
> > >>
> > >>>> 2aadc393a000-2aadc3942000 r--s 00035000 08:02 393229
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/junit-4.3.jar
> > >>
> > >>>> 2aadc3942000-2aadc394a000 r--s 0004b000 08:02 393227
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/commons-math-1.2.jar
> > >>
> > >>>> 2aadc394a000-2aadc395a000 r--s 000d3000 08:02 393248
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/apache-solr-core-1.3.0.jar
> > >>
> > >>>> 2aadc395a000-2aadc3968000 r--s 000b9000 08:02 393231
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/lucene-core-2.4-dev.jar
> > >>
> > >>>> 2aadc3968000-2aadc396a000 r--s 00014000 08:02 393247
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/apache-solr-common-1.3.0.jar
> > >>
> > >>>> 2aadc396a000-2aadc396d000 r--s 0001f000 08:02 393230
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/lucene-analyzers-2.4-dev.jar
> > >>
> > >>>> 2aadc396d000-2aadc396f000 r--s 0000b000 08:02 393224
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/commons-fileupload-1.2.jar
> > >>
> > >>>> 2aadc396f000-2aadc3973000 r--s 0003c000 08:02 393226
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/commons-lang-2.4.jar
> > >>
> > >>>> 2aadc3973000-2aadc398b000 r--s 00162000 08:02 393285
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/apache-solr-dataimporthandler-1.3.0.jar
> > >>
> > >>>> 2aadc398b000-2aadc398e000 r--s 0001d000 08:02 393245
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/stax-utils.jar
> > >>
> > >>>> 2aadc398e000-2aadc3991000 r--s 00012000 08:02 393225
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/commons-io-1.3.1.jar
> > >>
> > >>>> 2aadc3991000-2aadc3993000 r--s 00007000 08:02 393234
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/lucene-queries-2.4-dev.jar
> > >>
> > >>>> 2aadc3993000-2aadc3995000 r--s 00006000 08:02 393228
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/geronimo-stax-api_1.0_spec-1.0.1.jar
> > >>
> > >>>> 2aadc3995000-2aadc3996000 r--s 00004000 08:02 393236
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/lucene-spellchecker-2.4-dev.jar
> > >>
> > >>>> 2aadc3996000-2aadc3998000 r--s 0000a000 08:02 393222
> > >>>>
> > >>>>
> > >>
> >
> /tmp/Jetty_0_0_0_0_8983_solr.war__solr__k1kf17/webapp/WEB-INF/lib/commons-codec-1.3.jar
> > >>
> > >>>> 2aadc3998000-2aadc39ab000 r-xp 00000000 08:05 12506731
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libnet.so
> > >>>> 2aadc39ab000-2aadc3aac000 ---p 00013000 08:05 12506731
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libnet.so
> > >>>> 2aadc3aac000-2aadc3aaf000 rwxp 00014000 08:05 12506731
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libnet.so
> > >>>> 2aadc3aaf000-2aadc3ab6000 r-xp 00000000 08:05 12506732
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libnio.so
> > >>>> 2aadc3ab6000-2aadc3bb5000 ---p 00007000 08:05 12506732
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libnio.so
> > >>>> 2aadc3bb5000-2aadc3bb7000 rwxp 00006000 08:05 12506732
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/libnio.so
> > >>>> 2aadc3bb7000-2aadc3bbb000 r-xs 000ca000 08:05 12538411
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/ext/localedata.jar
> > >>>> 2aadc3bbb000-2aadc3bc2000 r-xs 00110000 08:05 12506779
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/resources.jar
> > >>>> 2aadc3bc2000-2aadc3bc3000 r-xs 00007000 08:05 8675819
> > >>>> /opt/solr-1.3/olx/solr/lib/lucene-queries-2.4-dev.jar
> > >>>> 2aadc3bc3000-2aadc3bc4000 r-xs 00005000 08:05 8675818
> > >>>> /opt/solr-1.3/olx/solr/lib/RelatedItemsRequestHandler.jar
> > >>>> 2aadc3bc4000-2aadc3bc5000 r-xs 00002000 08:05 8675817
> > >>>> /opt/solr-1.3/olx/solr/lib/OlxHTMLWhiteSpaceTokenizerFactory.jar
> > >>>> 2aadc4000000-2aadc5ce6000 rwxp 2aadc4000000 00:00 0
> > >>>> 2aadc5ce6000-2aadc8000000 ---p 2aadc5ce6000 00:00 0
> > >>>> 2aadc8000000-2aadc9cf8000 rwxp 2aadc8000000 00:00 0
> > >>>> 2aadc9cf8000-2aadcc000000 ---p 2aadc9cf8000 00:00 0
> > >>>> 2aadcc000000-2aadcc061000 rwxp 2aadcc000000 00:00 0
> > >>>> 2aadcc061000-2aadd0000000 ---p 2aadcc061000 00:00 0
> > >>>> 2aadd1c00000-2aadd3c00000 rwxp 2aadd1c00000 00:00 0
> > >>>> 2aadd4000000-2aadd7f59000 rwxp 2aadd4000000 00:00 0
> > >>>> 2aadd7f59000-2aadd8000000 ---p 2aadd7f59000 00:00 0
> > >>>> 2aadd8000000-2aadd88b9000 rwxp 2aadd8000000 00:00 0
> > >>>> 2aadd88b9000-2aaddc000000 ---p 2aadd88b9000 00:00 0
> > >>>> 2b4e0f322000-2b4e0f323000 rwxp 2b4e0f322000 00:00 0
> > >>>> 2b4e0f32c000-2b4e0f32d000 rwxp 2b4e0f32c000 00:00 0
> > >>>> 2b4e0f32d000-2b4e0f334000 r-xp 00000000 08:05 12506703
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/jli/libjli.so
> > >>>> 2b4e0f334000-2b4e0f435000 ---p 00007000 08:05 12506703
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/jli/libjli.so
> > >>>> 2b4e0f435000-2b4e0f437000 rwxp 00008000 08:05 12506703
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/jli/libjli.so
> > >>>> 2b4e0f437000-2b4e0f439000 rwxp 2b4e0f437000 00:00 0
> > >>>> 2b4e0f439000-2b4e0fb2e000 r-xp 00000000 08:05 12506747
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/server/libjvm.so
> > >>>> 2b4e0fb2e000-2b4e0fc2d000 ---p 006f5000 08:05 12506747
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/server/libjvm.so
> > >>>> 2b4e0fc2d000-2b4e0fd77000 rwxp 006f4000 08:05 12506747
> > >>>> /usr/local/jdk1.6.0_06/jre/lib/amd64/server/libjvm.so
> > >>>> 2b4e0fd77000-2b4e0fdb4000 rwxp 2b4e0fd77000 00:00 0
> > >>>> 7fffe6a79000-7fffe6aae000 rwxp 7ffffffca000 00:00 0
> > >>>> [stack]
> > >>>> ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0
> > >>>> [vdso]
> > >>>>
> > >>>> VM Arguments:
> > >>>> jvm_args: -Xmx12240m
> > >>>> -Djava.util.logging.config.file=/opt/solr/olx/solr/conf/logging.conf
> > >>>> -XX:+PrintGCApplicationConcurrentTime
> > >>>>
> > >> -XX:+PrintGCApplicationStoppedTime
> > >>
> > >>>> -XX:+UseConcMarkSweepGC
> > >>>> java_command: start.jar
> > >>>> Launcher Type: SUN_STANDARD
> > >>>>
> > >>>> Environment Variables:
> > >>>>
> > >>>>
> > >>
> >
> PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/java/bin:/root/bin
> > >>
> > >>
> >
> LD_LIBRARY_PATH=/usr/local/jdk1.6.0_06/jre/lib/amd64/server:/usr/local/jdk1.6.0_06/jre/lib/amd64:/usr/local/jdk1.6.0_06/jre/../lib/amd64
> > >>
> > >>>> SHELL=/bin/bash
> > >>>>
> > >>>> Signal Handlers:
> > >>>> SIGSEGV: [libjvm.so+0x6616e0], sa_mask[0]=0x7ffbfeff,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> SIGBUS: [libjvm.so+0x6616e0], sa_mask[0]=0x7ffbfeff,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> SIGFPE: [libjvm.so+0x5044d0], sa_mask[0]=0x7ffbfeff,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> SIGPIPE: [libjvm.so+0x5044d0], sa_mask[0]=0x7ffbfeff,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> SIGXFSZ: [libjvm.so+0x5044d0], sa_mask[0]=0x7ffbfeff,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> SIGILL: [libjvm.so+0x5044d0], sa_mask[0]=0x7ffbfeff,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
> > >>>> SIGUSR2: [libjvm.so+0x506740], sa_mask[0]=0x00000000,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> SIGHUP: [libjvm.so+0x5064a0], sa_mask[0]=0x7ffbfeff,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
> > >>>> SIGTERM: [libjvm.so+0x5064a0], sa_mask[0]=0x7ffbfeff,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> SIGQUIT: [libjvm.so+0x5064a0], sa_mask[0]=0x7ffbfeff,
> > >>>>
> > >> sa_flags=0x10000004
> > >>
> > >>>> --------------- S Y S T E M ---------------
> > >>>>
> > >>>> OS:Red Hat Enterprise Linux Server release 5.3 (Tikanga)
> > >>>>
> > >>>> uname:Linux 2.6.18-128.1.14.el5 #1 SMP Mon Jun 1 15:52:58 EDT 2009
> > >>>>
> > >> x86_64
> > >>
> > >>>> libc:glibc 2.5 NPTL 2.5
> > >>>> rlimit: STACK 10240k, CORE 0k, NPROC infinity, NOFILE 4096, AS
> > infinity
> > >>>> load average:1.39 0.60 0.56
> > >>>>
> > >>>> CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 7
> > >>>>
> > >>> stepping
> > >>>
> > >>>> 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
> > >>>>
> > >>>> Memory: 4k page, physical 16432184k(92220k free), swap
> > >>>>
> > >> 2096472k(2065524k
> > >>
> > >>>> free)
> > >>>>
> > >>>> vm_info: Java HotSpot(TM) 64-Bit Server VM (10.0-b22) for linux-amd64
> > >>>>
> > >> JRE
> > >>
> > >>>> (1.6.0_06-b02), built on Mar 25 2008 01:03:02 by "java_re" with gcc
> > >>>>
> > >> 3.2.2
> > >>
> > >>>> (SuSE Linux)
> > >>>>
> > >>>> time: Sat Sep 26 00:20:34 2009
> > >>>> elapsed time: 30852 seconds
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Sat, Sep 26, 2009 at 11:49 AM, Mark Miller
> > >>>>
> > >>> wrote:
> > >>>
> > >>>>
> > >>>>> Jonathan Ariel wrote:
> > >>>>>
> > >>>>>
> > >>>>>> I have around 8M documents.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>> Thats actually not so bad - I take it you are faceting/sorting on
> > >>>>>
> > >> quite
> > >>
> > >>>>> a few unique fields?
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>> I set up my server to use a different collector and it seems like it
> > >>>>>> decreased from 11% to 4%, of course I need to wait a bit more
> > because
> > >>>>>>
> > >>> it
> > >>>
> > >>>>> is
> > >>>>>
> > >>>>>
> > >>>>>> just a 1 hour old log. But it seems like it is much better now.
> > >>>>>> I will tell you on Monday the results :)
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>> Are you still seeing major collections then? (eg the tenured space
> > >>>>>
> > >> hits
> > >>
> > >>>>> its limit) You might be able to get even better.
> > >>>>>
> > >>>>>
> > >>>>>> On Fri, Sep 25, 2009 at 6:07 PM, Mark Miller
> > >
> > >>>>>>
> > >>>>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>
> > >>>>>>> Thats a good point too - if you can reduce your need for such a
> > >>>>>>>
> > >> large
> > >>
> > >>>>>>> heap, by all means, do so.
> > >>>>>>>
> > >>>>>>> However, considering you already need at least 10GB or you get OOM,
> > >>>>>>>
> > >>> you
> > >>>
> > >>>>>>> have a long way to go with that approach. Good luck :)
> > >>>>>>>
> > >>>>>>> How many docs do you have ? I'm guessing its mostly FieldCache type
> > >>>>>>> stuff, and thats the type of thing you can't really side step,
> > >>>>>>>
> > >> unless
> > >>
> > >>>>>>> you give up the functionality thats using it.
> > >>>>>>>
> > >>>>>>> Grant Ingersoll wrote:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On Sep 25, 2009, at 9:30 AM, Jonathan Ariel wrote:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> Hi to all!
> > >>>>>>>>> Lately my solr servers seem to stop responding once in a while.
> > >>>>>>>>>
> > >> I'm
> > >>
> > >>>>>>>>> using
> > >>>>>>>>> solr 1.3.
> > >>>>>>>>> Of course I'm having more traffic on the servers.
> > >>>>>>>>> So I logged the Garbage Collection activity to check if it's
> > >>>>>>>>>
> > >> because
> > >>
> > >>>>> of
> > >>>>>
> > >>>>>
> > >>>>>>>>> that. It seems like 11% of the time the application runs, it is
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>> stopped
> > >>>>>
> > >>>>>
> > >>>>>>>>> because of GC. And some times the GC takes up to 10 seconds!
> > >>>>>>>>> Is is normal? My instances run on a 16GB RAM, Dual Quad Core
> > Intel
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>> Xeon
> > >>>>>
> > >>>>>
> > >>>>>>>>> servers. My index is around 10GB and I'm giving to the instances
> > >>>>>>>>>
> > >>> 10GB
> > >>>
> > >>>>> of
> > >>>>>
> > >>>>>
> > >>>>>>>>> RAM.
> > >>>>>>>>>
> > >>>>>>>>> How can I check which is the GC that it is being used? If I'm
> > >>>>>>>>>
> > >> right
> > >>
> > >>>>> JVM
> > >>>>>
> > >>>>>
> > >>>>>>>>> Ergonomics should use the Throughput GC, but I'm not 100% sure.
> > Do
> > >>>>>>>>> you have
> > >>>>>>>>> any recommendation on this?
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> As I said in Eteve's thread on JVM settings, some extra time spent
> > >>>>>>>>
> > >> on
> > >>
> > >>>>>>>> application design/debugging will save a whole lot of headache in
> > >>>>>>>> Garbage Collection and trying to tune the gazillion different
> > >>>>>>>>
> > >> options
> > >>
> > >>>>>>>> available. Ask yourself: What is on the heap and does it need to
> > >>>>>>>>
> > >> be
> > >>
> > >>>>>>>> there? For instance, do you, if you have them, really need
> > >>>>>>>>
> > >> sortable
> > >>
> > >>>>>>>> ints? If your servers seem to come to a stop, I'm going to bet
> > >>>>>>>>
> > >> you
> > >>
> > >>>>>>>> have major collections going on. Major collections in a
> > production
> > >>>>>>>> system are very bad. They tend to happen right after commits in
> > >>>>>>>> poorly tuned systems, but can also happen in other places if you
> > >>>>>>>>
> > >> let
> > >>
> > >>>>>>>> things build up due to really large heaps and/or things like
> > really
> > >>>>>>>> large cache settings. I would pull up jConsole and have a look at
> > >>>>>>>> what is happening when the pauses occur. Is it a major
> > collection?
> > >>>>>>>> If so, then hook up a heap analyzer or a profiler and see what is
> > >>>>>>>>
> > >> on
> > >>
> > >>>>>>>> the heap around those times. Then have a look at your
> > >>>>>>>>
> > >> schema/config,
> > >>
> > >>>>>>>> etc. and see if there are things that are memory intensive
> > >>>>>>>>
> > >> (sorting,
> > >>
> > >>>>>>>> faceting, excessively large filter caches).
> > >>>>>>>>
> > >>>>>>>> --------------------------
> > >>>>>>>> Grant Ingersoll
> > >>>>>>>> http://www.lucidimagination.com/
> > >>>>>>>>
> > >>>>>>>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
> > >>>>>>>> using Solr/Lucene:
> > >>>>>>>> http://www.lucidimagination.com/search
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>> --
> > >>>>>>> - Mark
> > >>>>>>>
> > >>>>>>> http://www.lucidimagination.com
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>> --
> > >>>>> - Mark
> > >>>>>
> > >>>>> http://www.lucidimagination.com
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >
> > >
> >
> >
> >
> >
