Hi, Is there anyone that would be able to assist with the issue that I am seeing? I am seeing the same slowness with solr 8.1.1 using java11 as I am seeing with java12, over queries that are run from solr4.10.4 with java8 and solr6.5.1 with java8. Queries that return in less than half a second on solr4 are taking up to 20 seconds with same data indexed to solr 8.1.1 I have posted configs, schemas, and various log files in this shared dropbox folder ( https://www.dropbox.com/sh/2x2k5c9db7d4pt9/AADnHwuJc7a9Fh4KmUD15rS0a?dl=0 ). Any additional help/assistance would be greatly appreciated. Thank you, Russ
*Manzama*a MODERN GOVERNANCE company Russell Bahr Lead Infrastructure Engineer USA & CAN Office: +1 (541) 306 3271 USA & CAN Support: +1 (541) 706 9393 UK Office & Support: +44 (0)203 282 1633 AUS Office & Support: +61 (0) 2 8417 2339 543 NW York Drive, Suite 100, Bend, OR 97703 LinkedIn <http://www.linkedin.com/company/manzama> | Twitter <https://twitter.com/ManzamaInc> | Facebook <http://www.facebook.com/manzamainc> | YouTube <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> On Sun, Oct 20, 2019 at 8:54 PM Russell Bahr <r...@manzama.com> wrote: > Hi Shawn, > per your comments from before > On Oct 15, 2019, 2:28 AM, Shawn Heisey wrote: > > Java 12 is not recommended. It is one of the "new feature" releases > > that only gets 6 months of support. We would recommend Java 8 or Java > > 11. These are the versions with long term support. Probably a good > > thing to be using OpenJDK, as the official Oracle Java now requires > > paying for a license. > > I have rebuilt my 30 server sorl 8 cluster using java11 and increased the > java heap -Xms10433m -Xmx10433m and am seeing the same slowness that I > was seeing with java12. > I have not yet tried to build out the solr 8 collection with java8. Would > it be worthwhile to do that or were you able to see anything in the logs? > > Thank you in advance, > Russ > > *Manzama*a MODERN GOVERNANCE company > > Russell Bahr > Lead Infrastructure Engineer > > USA & CAN Office: +1 (541) 306 3271 > USA & CAN Support: +1 (541) 706 9393 > UK Office & Support: +44 (0)203 282 1633 > AUS Office & Support: +61 (0) 2 8417 2339 > > 543 NW York Drive, Suite 100, Bend, OR 97703 > > LinkedIn <http://www.linkedin.com/company/manzama> | Twitter > <https://twitter.com/ManzamaInc> | Facebook > <http://www.facebook.com/manzamainc> | YouTube > <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> > > > On Wed, Oct 16, 2019 at 11:50 AM Russell Bahr <r...@manzama.com> wrote: > >> Hi Shawn, >> >> Just checking to see if you saw my reply and had any feedback. Thank you >> again for your help. It is much appreciated. >> >> Thank you, >> >> Russ >> >> >> >> >> >> *From: *Russell Bahr <r...@manzama.com> >> *Date: *Tuesday, October 15, 2019 at 11:50 AM >> *To: *"solr-user@lucene.apache.org" <solr-user@lucene.apache.org> >> *Subject: *Re: solr 8.1.1 many time slower returning query results than >> solr 4.10.4 or solr 6.5.1 >> >> >> >> Hi Shawn, >> >> I included the wrong file for solr4 and did not realize until you pointed >> out the heap size. The correct file that is setting the Java environment >> is "Solr 4 tomcat setenv" I have uploaded that to the shared folder along >> with the requested screenshots "Solr 4 top screenshot","Solr 6 top >> screenshot","Solr 8 top screenshot". >> >> >> >> I have also uploaded the solr.log, solr_gc.log, and >> solr_slow_requests.log from a 2 hour period of time where I was running the >> email load test against the solr8 implementation in which the queued tasks >> are taking too long to complete. >> >> >> >> solr_gc.log, solr_gc.log.1, solr_gc.log.2, solr.log, solr.log.10, >> solr.log.6, solr.log.7, solr.log.8, solr.log.9, solr_slow_requests.log >> >> >> >> Let me know if there is any other information that I can provide that may >> help to work through this. >> >> >> >> *Manzama *a MODERN GOVERNANCE company >> >> >> >> Russell Bahr >> Lead Infrastructure Engineer >> >> USA & CAN Office: +1 (541) 306 3271 >> USA & CAN Support: +1 (541) 706 9393 >> UK Office & Support: +44 (0)203 282 1633 >> AUS Office & Support: +61 (0) 2 8417 2339 >> >> 543 NW York Drive, Suite 100, Bend, OR 97703 >> >> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter >> <https://twitter.com/ManzamaInc> | Facebook >> <http://www.facebook.com/manzamainc> | YouTube >> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> >> >> >> >> >> >> On Tue, Oct 15, 2019 at 2:28 AM Shawn Heisey <apa...@elyograg.org> wrote: >> >> On 10/14/2019 1:36 PM, Russell Bahr wrote: >> > Backend replacement of solr4 and hopefully Frontend replacement as well. >> > solr-spec 8.1.1 >> > lucene-spec 8.1.1 >> > Runtime Oracle Corporation OpenJDK 64-Bit Server VM 12 12+33 >> > 1 collection 6 shards 5 replicas per shard 17,919,889 current documents >> (35 days worth of documents) - indexing new documents regularly throughout >> the day, deleting aged out documents nightly. >> >> Java 12 is not recommended. It is one of the "new feature" releases >> that only gets 6 months of support. We would recommend Java 8 or Java >> 11. These are the versions with long term support. Probably a good >> thing to be using OpenJDK, as the official Oracle Java now requires >> paying for a license. >> >> Solr 8 ships with settings that enable the G1GC collector instead of >> CMS, because CMS is deprecated and will disappear in a future Java >> version. We have seen problems with this when the system is >> misconfigured as far as heap size. When the system is properly sized, >> G1 tends to do better than CMS, but when the heap is too large or too >> small, has a tendency to amplify garbage collection problems in >> comparison. >> >> Looking at your solr.in.sh files for each version ... the Solr 4 install >> appears to be setting the heap to 512 megabytes. This is definitely not >> enough for millions of documents, and if this is what the heap size is >> actually set to, would almost certainly run into memory errors >> frequently and have absolutely terrible performance. But you are saying >> that it works well, so I don't think the heap is actually set to 512 >> megabytes. Maybe the bin/solr script has been modified directly to set >> the memory size instead of setting it in solr.in.sh where it should be >> set. >> >> Solr 6 has a heap size of just under 27 gigabytes. Solr 8 has a heap >> size of just under 8 gigabytes. With millions of documents, it is >> likely that 8GB of heap is not quite big enough. >> >> For each of your installations (Solr 4, Solr 6, and Solr 8) can you >> provide the screenshot described at this wiki page? >> >> >> https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue >> >> It would also be helpful to see the GC logs from Solr 8. We would need >> at least one GC log, making sure that they cover at least a few hours, >> including the timeframe when the slow indexing and slow queries were >> observed. >> >> Thanks, >> Shawn >> >>
