On 8/23/2017 7:32 AM, Markus Jelsma wrote: > Why does it slowly increase over time? > Why does it appear to correlate to index size? > Is anyone else seeing this on their 6.6 cloud production or local machines?
More detailed information included here. My 6.6 dev install is NOT having the problem, but a much older version IS. I grabbed this screenshot only moments ago from a production server which is exhibiting a large SHR value for the Solr process: https://www.dropbox.com/s/q79lo2gft9es06u/idxa1-top-big-shr.png?dl=0 This is Solr 4.7.2, with a 10 month uptime for the Solr process, running with these arguments: -DSTOP.KEY=REDACTED -DSTOP.PORT=8078 -Djetty.port=8981 -Dsolr.solr.home=/index/solr4 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8686 -Dcom.sun.management.jmxremote -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log-verbose:gc -XX:+AggressiveOpts -XX:+UseLargePages -XX:InitiatingHeapOccupancyPercent=75 -XX:MaxGCPauseMillis=250 -XX:G1HeapRegionSize=8m -XX:+ParallelRefProcEnabled -XX:+PerfDisableSharedMem -XX:+UseG1GC -Dlog4j.configuration=file:etc/log4j.properties -Xmx8192M -Xms4096M The OS is CentOS 6, with the following Java and kernel: java version "1.7.0_72" Java(TM) SE Runtime Environment (build 1.7.0_72-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode) Linux idxa1 2.6.32-431.11.2.el6.centos.plus.x86_64 #1 SMP Tue Mar 25 21:36:54 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux I also just grabbed a screenshot from my dev server, running Ubuntu 14, Solr 6.6.0, a LOT more index data, and a more recent Java version. Solr has an uptime of about one month. This server was installed with the service installer script, so it uses bin/solr. It doesn't seem to have the same problem: https://www.dropbox.com/s/85h1weuopa643za/bigindy5-top-small-shr.png?dl=0 java version "1.8.0_144" Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode) Linux bigindy5 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux The arguments for this one are very similar to the production server: -DSTOP.KEY=solrrocks -DSTOP.PORT=7982 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=18982 -Dcom.sun.management.jmxremote.rmi.port=18982 -Dcom.sun.management.jmxremote.ssl=false -Djetty.home=/opt/solr6/server -Djetty.port=8982 -Dlog4j.configuration=file:/index/solr6/log4j.properties -Dsolr.install.dir=/opt/solr6 -Dsolr.log.dir=/index/solr6/logs -Dsolr.log.muteconsole -Dsolr.solr.home=/index/solr6/data -Duser.timezone=UTC -XX:+AggressiveOpts -XX:+ParallelRefProcEnabled -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+UseG1GC -XX:+UseGCLogFileRotation -XX:+UseLargePages -XX:G1HeapRegionSize=8m -XX:GCLogFileSize=20M -XX:InitiatingHeapOccupancyPercent=75 -XX:MaxGCPauseMillis=250 -XX:NumberOfGCLogFiles=9 -XX:OnOutOfMemoryError=/opt/solr6/bin/oom_solr.sh 8982 /index/solr6/logs -Xloggc:/index/solr6/logs/solr_gc.log -Xms28g -Xmx28g -Xss256k -verbose:gc Neither system has any huge pages allocated in the OS, so I doubt that the UseLargePages option is actually doing anything. I've left it there in case I *do* enable huge pages, so they will automatically get used. Thanks, Shawn
