I have the problem in production and local, with default Solr 6.6 JVM arguments, environments are:
openjdk version "1.8.0_141" OpenJDK Runtime Environment (build 1.8.0_141-8u141-b15-1~deb9u1-b15) OpenJDK 64-Bit Server VM (build 25.141-b15, mixed mode) Linux idx1 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u1 (2017-06-18) x86_64 GNU/Linux and openjdk version "1.8.0_131" OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.17.04.3-b11) OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode) Linux midas 4.10.0-32-generic #36-Ubuntu SMP Tue Aug 8 12:10:06 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Regarding the node that shows the problem, can you reproduce it locally? Fire it up, put some data in, confirm low shared space usage, and execute few thousands queries against it? We immediately see a sharp rise in shared memory, MB's per second until it reaches some sort of plateau. -----Original message----- > From:Shawn Heisey <apa...@elyograg.org> > Sent: Wednesday 23rd August 2017 16:37 > To: solr-user@lucene.apache.org > Subject: Re: Solr uses lots of shared memory! > > On 8/23/2017 7:32 AM, Markus Jelsma wrote: > > Why does it slowly increase over time? > > Why does it appear to correlate to index size? > > Is anyone else seeing this on their 6.6 cloud production or local machines? > > More detailed information included here. My 6.6 dev install is NOT > having the problem, but a much older version IS. > > I grabbed this screenshot only moments ago from a production server > which is exhibiting a large SHR value for the Solr process: > > https://www.dropbox.com/s/q79lo2gft9es06u/idxa1-top-big-shr.png?dl=0 > > This is Solr 4.7.2, with a 10 month uptime for the Solr process, running > with these arguments: > > -DSTOP.KEY=REDACTED > -DSTOP.PORT=8078 > -Djetty.port=8981 > -Dsolr.solr.home=/index/solr4 > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.port=8686 > -Dcom.sun.management.jmxremote > -XX:+PrintReferenceGC > -XX:+PrintAdaptiveSizePolicy > -XX:+PrintGCDetails > -XX:+PrintGCDateStamps > -Xloggc:logs/gc.log-verbose:gc > -XX:+AggressiveOpts > -XX:+UseLargePages > -XX:InitiatingHeapOccupancyPercent=75 > -XX:MaxGCPauseMillis=250 > -XX:G1HeapRegionSize=8m > -XX:+ParallelRefProcEnabled > -XX:+PerfDisableSharedMem > -XX:+UseG1GC > -Dlog4j.configuration=file:etc/log4j.properties > -Xmx8192M > -Xms4096M > > The OS is CentOS 6, with the following Java and kernel: > > java version "1.7.0_72" > Java(TM) SE Runtime Environment (build 1.7.0_72-b14) > Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode) > > Linux idxa1 2.6.32-431.11.2.el6.centos.plus.x86_64 #1 SMP Tue Mar 25 > 21:36:54 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > > I also just grabbed a screenshot from my dev server, running Ubuntu 14, > Solr 6.6.0, a LOT more index data, and a more recent Java version. Solr > has an uptime of about one month. This server was installed with the > service installer script, so it uses bin/solr. It doesn't seem to have > the same problem: > > https://www.dropbox.com/s/85h1weuopa643za/bigindy5-top-small-shr.png?dl=0 > > java version "1.8.0_144" > Java(TM) SE Runtime Environment (build 1.8.0_144-b01) > Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode) > > Linux bigindy5 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC > 2016 x86_64 x86_64 x86_64 GNU/Linux > > The arguments for this one are very similar to the production server: > > -DSTOP.KEY=solrrocks > -DSTOP.PORT=7982 > -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.local.only=false > -Dcom.sun.management.jmxremote.port=18982 > -Dcom.sun.management.jmxremote.rmi.port=18982 > -Dcom.sun.management.jmxremote.ssl=false > -Djetty.home=/opt/solr6/server > -Djetty.port=8982 > -Dlog4j.configuration=file:/index/solr6/log4j.properties > -Dsolr.install.dir=/opt/solr6 > -Dsolr.log.dir=/index/solr6/logs > -Dsolr.log.muteconsole > -Dsolr.solr.home=/index/solr6/data > -Duser.timezone=UTC > -XX:+AggressiveOpts > -XX:+ParallelRefProcEnabled > -XX:+PrintGCApplicationStoppedTime > -XX:+PrintGCDateStamps > -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps > -XX:+PrintHeapAtGC > -XX:+PrintTenuringDistribution > -XX:+UseG1GC > -XX:+UseGCLogFileRotation > -XX:+UseLargePages > -XX:G1HeapRegionSize=8m > -XX:GCLogFileSize=20M > -XX:InitiatingHeapOccupancyPercent=75 > -XX:MaxGCPauseMillis=250 > -XX:NumberOfGCLogFiles=9 > -XX:OnOutOfMemoryError=/opt/solr6/bin/oom_solr.sh 8982 /index/solr6/logs > -Xloggc:/index/solr6/logs/solr_gc.log > -Xms28g > -Xmx28g > -Xss256k > -verbose:gc > > Neither system has any huge pages allocated in the OS, so I doubt that > the UseLargePages option is actually doing anything. I've left it there > in case I *do* enable huge pages, so they will automatically get used. > > Thanks, > Shawn > >
