Unable to run two multicore Solr instances under Tomcat
Hi, I have been running two multicore Solr instances under Tomcat using a nightly build of 4.0 from September 2011. This has been running fine but when I try to update these instances to the release version of 4.0 I'm hitting problems when the second instance starts up. If I have one instance on the release version and one on the nightly build it also works fine. It's running on a Solaris 10 box using Tomcat 6.0.26 and Java 1.6.0_20 I can run up either instance on it's own and it works fine, it's just when starting both together so I'm pretty sure my configs aren't the issue. Snippet from the log is below, please note that I have had to type this out so there may be some typos, hopefully not! Any ideas? Adam 12-Nov-2012 09:58:50 org.apache.solr.core.SolrResourceLoader locateSolrHome INFO: Using JNDI solr.home: /conf_solr/instance2 12-Nov-2012 09:58:50 org.apache.solr.core.SolrResourceLoader INFO: new SolrResourceLoader for deduced Solr Home: '/conf_solr/instance2/' 12-Nov-2012 09:58:52 org.apache.solr.servlet.SolrDispatchFilter init INFO: SolrDispatchFilter.init() 12-Nov-2012 09:58:52 org.apache.solr.core.SolrResourceLoader locateSolrHome INFO: Using JNDI solr.home /conf_solr/instance2 12-Nov-2012 09:58:52 org.apache.solr.core.CoreContainer$Initializer initialize INFO: looking for solr.xml: /conf_solr/instance2/solr.xml 12-Nov-2012 09:58:52 org.apache.solr.core.CoreContainer INFO: New CoreContainer 15471347 12-Nov-2012 09:58:52 org.apache.solr.core.CoreContainer load INFO: Loading CoreContainer using Solr Home: '/conf_solr/instance2/' 12-Nov-2012 09:58:52 org.apache.solr.core.SolrResourceLoader INFO: new SOlrResourceLoader for directory: '/conf_solr/instance2/' 12-Nov-2012 09:58:52 org.apache.solr.servlet.SolrDispatchFilter init SEVERE: Could not start Solr. Check solr/home property and the logs 12-Nov-2012 09:58:52 org.apache.solr.common.SolrException log SEVERE: null:java.lang.ClassCastException: org.apache.xerces.parsers.XIncludeAwareParserConfiguration cannot be cast to org.apache.xerces.xni.parser.XMLParserConfiguration at org.apache.xerces.parsers.DOMParser.(Unknown Source) at org.apache.xerces.parsers.DOMParser.(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at com.sun.org.apache.xalan.internal.xsltc.trax.SAX2DOM.createDocument(SAX2DOM.java:324) at com.sun.org.apache.xalan.internal.xsltc.trax.SAX2DOM.(SAX2DOM.java:84) at com.sun.org.apache.xalan.internal.xsltc.runtime.output.TranslateOutputHandlerFactory.getSerializationHanlder(TransletOutputHandlerFactory.java:187) at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getOutputHandler(TransformerImpl.java:392) at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:298) at org.apache.solr.core.CoreContainer.copyDoc(CoreContainer.java:551) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:381) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:356) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:308) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:107) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3838) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4488) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at org.apache.catalina.core.StandardHost.start(StandardHost.java:785) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardServic
RE: Unable to run two multicore Solr instances under Tomcat
Hi Erick, Thanks for the info, I figured out that it was a jar problem earlier today but I don't think it is an old jar. Both of the instances I ran included the extraction libraries and it appears that the problem is due to the xercesImpl-2.9.1.jar. If I remove the extraction tool jars from one of the instances, or even just the specific jar, then everything works as normal. Fortunately I only need the extraction tools in one of my instances so this work around is good for now. I can't see any old jars that would interfere, I will try and test this at some point on a clean install of 4.0 and see if the same problem occurs. -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Tue 13/11/2012 12:05 To: solr-user@lucene.apache.org Subject: Re: Unable to run two multicore Solr instances under Tomcat At a guess you have leftover jars from your earlier installation in your classpath that are being picked up. I've always found that figuring out how _that_ happened is...er... "interesting"... Best Erick On Mon, Nov 12, 2012 at 7:44 AM, Adam Neal wrote: > Hi, > > I have been running two multicore Solr instances under Tomcat using a > nightly build of 4.0 from September 2011. This has been running fine but > when I try to update these instances to the release version of 4.0 I'm > hitting problems when the second instance starts up. If I have one instance > on the release version and one on the nightly build it also works fine. > > It's running on a Solaris 10 box using Tomcat 6.0.26 and Java 1.6.0_20 > > I can run up either instance on it's own and it works fine, it's just when > starting both together so I'm pretty sure my configs aren't the issue. > > Snippet from the log is below, please note that I have had to type this > out so there may be some typos, hopefully not! > > Any ideas? > > Adam > > > 12-Nov-2012 09:58:50 org.apache.solr.core.SolrResourceLoader locateSolrHome > INFO: Using JNDI solr.home: /conf_solr/instance2 > 12-Nov-2012 09:58:50 org.apache.solr.core.SolrResourceLoader > INFO: new SolrResourceLoader for deduced Solr Home: '/conf_solr/instance2/' > 12-Nov-2012 09:58:52 org.apache.solr.servlet.SolrDispatchFilter init > INFO: SolrDispatchFilter.init() > 12-Nov-2012 09:58:52 org.apache.solr.core.SolrResourceLoader locateSolrHome > INFO: Using JNDI solr.home /conf_solr/instance2 > 12-Nov-2012 09:58:52 org.apache.solr.core.CoreContainer$Initializer > initialize > INFO: looking for solr.xml: /conf_solr/instance2/solr.xml > 12-Nov-2012 09:58:52 org.apache.solr.core.CoreContainer > INFO: New CoreContainer 15471347 > 12-Nov-2012 09:58:52 org.apache.solr.core.CoreContainer load > INFO: Loading CoreContainer using Solr Home: '/conf_solr/instance2/' > 12-Nov-2012 09:58:52 org.apache.solr.core.SolrResourceLoader > INFO: new SOlrResourceLoader for directory: '/conf_solr/instance2/' > 12-Nov-2012 09:58:52 org.apache.solr.servlet.SolrDispatchFilter init > SEVERE: Could not start Solr. Check solr/home property and the logs > 12-Nov-2012 09:58:52 org.apache.solr.common.SolrException log > SEVERE: null:java.lang.ClassCastException: > org.apache.xerces.parsers.XIncludeAwareParserConfiguration cannot be cast > to org.apache.xerces.xni.parser.XMLParserConfiguration > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown > Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at > com.sun.org.apache.xalan.internal.xsltc.trax.SAX2DOM.createDocument(SAX2DOM.java:324) > at > com.sun.org.apache.xalan.internal.xsltc.trax.SAX2DOM.(SAX2DOM.java:84) > at > com.sun.org.apache.xalan.internal.xsltc.runtime.output.TranslateOutputHandlerFactory.getSerializationHanlder(TransletOutputHandlerFactory.java:187) > at > com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getOutputHandler(TransformerImpl.java:392) > at > com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:298) > at > org.apache.solr.core.CoreContainer.copyDoc(CoreContainer.java:551) > at org.apache.solr.core.CoreContainer.load(CoreContainer.java:381) > at org.apache.solr.core.CoreContainer.load(CoreContainer.java:356) > at > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:308) > at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:107) > at > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) >
RE: Unable to run two multicore Solr instances under Tomcat
Just to wrap up this one. Previously all the lib jars were located in the war file on our setup, this was mainly to ease deployment as it's just a single file. Moving the lib directory external to the war seems to have fixed the issue. Thanks for the pointer Erick. -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Tue 13/11/2012 12:05 To: solr-user@lucene.apache.org Subject: Re: Unable to run two multicore Solr instances under Tomcat At a guess you have leftover jars from your earlier installation in your classpath that are being picked up. I've always found that figuring out how _that_ happened is...er... "interesting"... Best Erick On Mon, Nov 12, 2012 at 7:44 AM, Adam Neal wrote: > Hi, > > I have been running two multicore Solr instances under Tomcat using a > nightly build of 4.0 from September 2011. This has been running fine but > when I try to update these instances to the release version of 4.0 I'm > hitting problems when the second instance starts up. If I have one instance > on the release version and one on the nightly build it also works fine. > > It's running on a Solaris 10 box using Tomcat 6.0.26 and Java 1.6.0_20 > > I can run up either instance on it's own and it works fine, it's just when > starting both together so I'm pretty sure my configs aren't the issue. > > Snippet from the log is below, please note that I have had to type this > out so there may be some typos, hopefully not! > > Any ideas? > > Adam > > > 12-Nov-2012 09:58:50 org.apache.solr.core.SolrResourceLoader locateSolrHome > INFO: Using JNDI solr.home: /conf_solr/instance2 > 12-Nov-2012 09:58:50 org.apache.solr.core.SolrResourceLoader > INFO: new SolrResourceLoader for deduced Solr Home: '/conf_solr/instance2/' > 12-Nov-2012 09:58:52 org.apache.solr.servlet.SolrDispatchFilter init > INFO: SolrDispatchFilter.init() > 12-Nov-2012 09:58:52 org.apache.solr.core.SolrResourceLoader locateSolrHome > INFO: Using JNDI solr.home /conf_solr/instance2 > 12-Nov-2012 09:58:52 org.apache.solr.core.CoreContainer$Initializer > initialize > INFO: looking for solr.xml: /conf_solr/instance2/solr.xml > 12-Nov-2012 09:58:52 org.apache.solr.core.CoreContainer > INFO: New CoreContainer 15471347 > 12-Nov-2012 09:58:52 org.apache.solr.core.CoreContainer load > INFO: Loading CoreContainer using Solr Home: '/conf_solr/instance2/' > 12-Nov-2012 09:58:52 org.apache.solr.core.SolrResourceLoader > INFO: new SOlrResourceLoader for directory: '/conf_solr/instance2/' > 12-Nov-2012 09:58:52 org.apache.solr.servlet.SolrDispatchFilter init > SEVERE: Could not start Solr. Check solr/home property and the logs > 12-Nov-2012 09:58:52 org.apache.solr.common.SolrException log > SEVERE: null:java.lang.ClassCastException: > org.apache.xerces.parsers.XIncludeAwareParserConfiguration cannot be cast > to org.apache.xerces.xni.parser.XMLParserConfiguration > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown > Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at > com.sun.org.apache.xalan.internal.xsltc.trax.SAX2DOM.createDocument(SAX2DOM.java:324) > at > com.sun.org.apache.xalan.internal.xsltc.trax.SAX2DOM.(SAX2DOM.java:84) > at > com.sun.org.apache.xalan.internal.xsltc.runtime.output.TranslateOutputHandlerFactory.getSerializationHanlder(TransletOutputHandlerFactory.java:187) > at > com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getOutputHandler(TransformerImpl.java:392) > at > com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:298) > at > org.apache.solr.core.CoreContainer.copyDoc(CoreContainer.java:551) > at org.apache.solr.core.CoreContainer.load(CoreContainer.java:381) > at org.apache.solr.core.CoreContainer.load(CoreContainer.java:356) > at > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:308) > at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:107) > at > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) > at > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) > at > org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115) > at > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3838) > at > org.apache.catalina.core.StandardCon
Slow commits
I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow commits on one of my cores. The core in question is relatively small (56k docs) and the issue only shows when commiting after a number of deletes, commiting after additions is fine. As an example commiting after deleting approximately 10% of the documents takes around 25mins. The same test on the 4.10.2 instance takes around 1 second. I have done some investigation and the problem appears to be caused by having dynamic fields, the core in question has a large number, performing the same operation on this core with the dynamic fields removed sees a big improvement on the performance with the commit taking 11 seconds (still not quite on a par with 4.10.2). At the moment for this core it is quicker in 5.3.1 to delete everything and reindex the data than it is to delete old documents. Are there any changes that may have caused this or anything I should be doing differently in 5.3.1? I intend to download 5.4.1 next week to see if that improves things. Thanks Adam # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
RE: Slow commits
Just some additional information, the problem is mainly when the dynamic fields are stored. Just having them indexed reduces the commit time to around 20 seconds. Unfortunately I need them stored. From: Adam Neal [Extranet] [an...@mass.co.uk] Sent: 19 February 2016 13:51 To: solr-user@lucene.apache.org Subject: Slow commits I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow commits on one of my cores. The core in question is relatively small (56k docs) and the issue only shows when commiting after a number of deletes, commiting after additions is fine. As an example commiting after deleting approximately 10% of the documents takes around 25mins. The same test on the 4.10.2 instance takes around 1 second. I have done some investigation and the problem appears to be caused by having dynamic fields, the core in question has a large number, performing the same operation on this core with the dynamic fields removed sees a big improvement on the performance with the commit taking 11 seconds (still not quite on a par with 4.10.2). At the moment for this core it is quicker in 5.3.1 to delete everything and reindex the data than it is to delete old documents. Are there any changes that may have caused this or anything I should be doing differently in 5.3.1? I intend to download 5.4.1 next week to see if that improves things. Thanks Adam # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
RE: Slow commits
max heap on both instances is the same at 8gig, and only using around 1.5gig at the time of testing. 5.3.1 index size is 90MB 4.10.2 index size is 125MB From: Shawn Heisey [apa...@elyograg.org] Sent: 19 February 2016 15:45 To: solr-user@lucene.apache.org Subject: Re: Slow commits On 2/19/2016 6:51 AM, Adam Neal [Extranet] wrote: > I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow > commits on one of my cores. The core in question is relatively small (56k > docs) and the issue only shows when commiting after a number of deletes, > commiting after additions is fine. As an example commiting after deleting > approximately 10% of the documents takes around 25mins. The same test on the > 4.10.2 instance takes around 1 second. This sounds like one of two possibilities, with a strong possibility that it's both: * You are encountering extreme garbage collection pauses from a heap that's too small. * The deletes are leading to segment merges that are proceeding slowly. What is the max heap set to on the new version? Do you know what it was set to on the old version? How much disk space is used by your 56000 document index? Thanks, Shawn # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
RE: Slow commits
I'm out of the office now so I don't have the numbers to hand but from memory I think there are probably around 800-1000 fields or so. I will confirm on Monday. If i have time over the weekend I will try and recreate the problem at home and see if I can post up a sample. From: Yonik Seeley [ysee...@gmail.com] Sent: 19 February 2016 16:25 To: solr-user@lucene.apache.org Subject: Re: Slow commits On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] wrote: > I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow > commits on one of my cores. The core in question is relatively small (56k > docs) and the issue only shows when commiting after a number of deletes, > commiting after additions is fine. As an example commiting after deleting > approximately 10% of the documents takes around 25mins. The same test on the > 4.10.2 instance takes around 1 second. > > I have done some investigation and the problem appears to be caused by having > dynamic fields, the core in question has a large number, performing the same > operation on this core with the dynamic fields removed sees a big improvement > on the performance with the commit taking 11 seconds (still not quite on a > par with 4.10.2). Dynamic fields is a Solr schema concept, and does not translate to any differences in Lucene. You may be hitting something due to a large number of fields (at the lucene level, each field name is a different field). How many different fields (i.e. fieldnames) do you have across the entire index? -Yonik # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
RE: Slow commits
Well I got the numbers wrong, there are actually around 66000 fields on the index. I have restructured the index and there are now around 1500 fiields. This has resulted in the commit taking 34 seconds which is acceptable for my usage however it is still significantly slower than the 4.10.2 commit on the original 66000 fields which was around 1 second. From: Adam Neal [Extranet] [an...@mass.co.uk] Sent: 19 February 2016 17:43 To: solr-user@lucene.apache.org Subject: RE: Slow commits I'm out of the office now so I don't have the numbers to hand but from memory I think there are probably around 800-1000 fields or so. I will confirm on Monday. If i have time over the weekend I will try and recreate the problem at home and see if I can post up a sample. From: Yonik Seeley [ysee...@gmail.com] Sent: 19 February 2016 16:25 To: solr-user@lucene.apache.org Subject: Re: Slow commits On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] wrote: > I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow > commits on one of my cores. The core in question is relatively small (56k > docs) and the issue only shows when commiting after a number of deletes, > commiting after additions is fine. As an example commiting after deleting > approximately 10% of the documents takes around 25mins. The same test on the > 4.10.2 instance takes around 1 second. > > I have done some investigation and the problem appears to be caused by having > dynamic fields, the core in question has a large number, performing the same > operation on this core with the dynamic fields removed sees a big improvement > on the performance with the commit taking 11 seconds (still not quite on a > par with 4.10.2). Dynamic fields is a Solr schema concept, and does not translate to any differences in Lucene. You may be hitting something due to a large number of fields (at the lucene level, each field name is a different field). How many different fields (i.e. fieldnames) do you have across the entire index? -Yonik # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. # # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
RE: Slow commits
Yup, that's correct. Not talking massive amounts of data really. The commit performance difference between 4.10.2 and 5.3.1 is huge in this case. From: Susheel Kumar [susheel2...@gmail.com] Sent: 22 February 2016 13:31 To: solr-user@lucene.apache.org Subject: Re: Slow commits Sorry, I see now you mentioned 56K docs which is pretty small. On Mon, Feb 22, 2016 at 8:30 AM, Susheel Kumar wrote: > Adam - how many documents you have in your index? > > Thanks, > Susheel > > On Mon, Feb 22, 2016 at 4:37 AM, Adam Neal [Extranet] > wrote: > >> Well I got the numbers wrong, there are actually around 66000 fields on >> the index. I have restructured the index and there are now around 1500 >> fiields. This has resulted in the commit taking 34 seconds which is >> acceptable for my usage however it is still significantly slower than the >> 4.10.2 commit on the original 66000 fields which was around 1 second. >> >> From: Adam Neal [Extranet] [an...@mass.co.uk] >> Sent: 19 February 2016 17:43 >> To: solr-user@lucene.apache.org >> Subject: RE: Slow commits >> >> I'm out of the office now so I don't have the numbers to hand but from >> memory I think there are probably around 800-1000 fields or so. I will >> confirm on Monday. >> >> If i have time over the weekend I will try and recreate the problem at >> home and see if I can post up a sample. >> >> From: Yonik Seeley [ysee...@gmail.com] >> Sent: 19 February 2016 16:25 >> To: solr-user@lucene.apache.org >> Subject: Re: Slow commits >> >> On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] >> wrote: >> > I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with >> slow commits on one of my cores. The core in question is relatively small >> (56k docs) and the issue only shows when commiting after a number of >> deletes, commiting after additions is fine. As an example commiting after >> deleting approximately 10% of the documents takes around 25mins. The same >> test on the 4.10.2 instance takes around 1 second. >> > >> > I have done some investigation and the problem appears to be caused by >> having dynamic fields, the core in question has a large number, performing >> the same operation on this core with the dynamic fields removed sees a big >> improvement on the performance with the commit taking 11 seconds (still not >> quite on a par with 4.10.2). >> >> Dynamic fields is a Solr schema concept, and does not translate to any >> differences in Lucene. >> You may be hitting something due to a large number of fields (at the >> lucene level, each field name is a different field). How many >> different fields (i.e. fieldnames) do you have across the entire >> index? >> >> -Yonik >> >> >> # >> >> This E-mail is the property of Mass Consultants Ltd. It is confidential >> and intended only for the use of the addressee or with its permission. Use >> by anyone else for any purpose is prohibited. If you are not the >> addressee, you should not use, disclose, copy or distribute this e-mail and >> should notify us of receipt immediately by return e-mail to the address >> where the e-mail originated. >> >> This E-mail may not have been sent through a secure system and >> accordingly (i) its contents should not be relied upon by any person >> without independent verification from Mass Consultants Ltd and (ii) it is >> the responsibility of the recipient to ensure that the onward transmission, >> opening or use of this message and any attachments will not adversely >> affect its systems or data. No responsibility is accepted by Mass >> Consultants Ltd in this regard. >> >> Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may >> be monitored by systems or persons other than the addressee, for the >> purposes of ascertaining whether the communication complies with the law >> and Mass Consultants Ltd's policies. >> >> Mass Consultants Ltd is registered in England No. 1705804, Enterprise >> House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. >> Tel: +44 (0) 1480 222600. >> >> >> # >> >> >> # >> >> This E-mail is
RE: Slow commits
Highest count is fairly equal between string and text. They are not indexed but stored and no docvalues used From: Yonik Seeley [ysee...@gmail.com] Sent: 22 February 2016 14:40 To: solr-user@lucene.apache.org Subject: Re: Slow commits What are the types of the fields with the highest count? I assume they are indexed. Are they stored, and do they have docValues? -Yonik On Mon, Feb 22, 2016 at 4:37 AM, Adam Neal [Extranet] wrote: > Well I got the numbers wrong, there are actually around 66000 fields on the > index. I have restructured the index and there are now around 1500 fiields. > This has resulted in the commit taking 34 seconds which is acceptable for my > usage however it is still significantly slower than the 4.10.2 commit on the > original 66000 fields which was around 1 second. > ____ > From: Adam Neal [Extranet] [an...@mass.co.uk] > Sent: 19 February 2016 17:43 > To: solr-user@lucene.apache.org > Subject: RE: Slow commits > > I'm out of the office now so I don't have the numbers to hand but from memory > I think there are probably around 800-1000 fields or so. I will confirm on > Monday. > > If i have time over the weekend I will try and recreate the problem at home > and see if I can post up a sample. > > From: Yonik Seeley [ysee...@gmail.com] > Sent: 19 February 2016 16:25 > To: solr-user@lucene.apache.org > Subject: Re: Slow commits > > On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] > wrote: >> I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow >> commits on one of my cores. The core in question is relatively small (56k >> docs) and the issue only shows when commiting after a number of deletes, >> commiting after additions is fine. As an example commiting after deleting >> approximately 10% of the documents takes around 25mins. The same test on the >> 4.10.2 instance takes around 1 second. >> >> I have done some investigation and the problem appears to be caused by >> having dynamic fields, the core in question has a large number, performing >> the same operation on this core with the dynamic fields removed sees a big >> improvement on the performance with the commit taking 11 seconds (still not >> quite on a par with 4.10.2). > > Dynamic fields is a Solr schema concept, and does not translate to any > differences in Lucene. > You may be hitting something due to a large number of fields (at the > lucene level, each field name is a different field). How many > different fields (i.e. fieldnames) do you have across the entire > index? > > -Yonik > > # > > This E-mail is the property of Mass Consultants Ltd. It is confidential and > intended only for the use of the addressee or with its permission. Use by > anyone else for any purpose is prohibited. If you are not the addressee, you > should not use, disclose, copy or distribute this e-mail and should notify us > of receipt immediately by return e-mail to the address where the e-mail > originated. > > This E-mail may not have been sent through a secure system and accordingly > (i) its contents should not be relied upon by any person without independent > verification from Mass Consultants Ltd and (ii) it is the responsibility of > the recipient to ensure that the onward transmission, opening or use of this > message and any attachments will not adversely affect its systems or data. No > responsibility is accepted by Mass Consultants Ltd in this regard. > > Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be > monitored by systems or persons other than the addressee, for the purposes of > ascertaining whether the communication complies with the law and Mass > Consultants Ltd's policies. > > Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, > Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 > (0) 1480 222600. > > # > > # > > This E-mail is the property of Mass Consultants Ltd. It is confidential and > intended only for the use of the addressee or with its permission. Use by > anyone else for any purpose is prohibited. If you are not the addressee, you > should not use, disclose, copy or distribute this e-mail and should notify us > of receipt immediately by return e-mail to the address where the e-mail > originated. >
RE: Slow commits
I have also tried it with the fields just indexed and not stored, the performance is the same so I doubt it is related to stored field compression. I can file a JIRA, unfortunately I wont be able to provide example files for this system but I will try and reproduce it with some test data and include that. From: Yonik Seeley [ysee...@gmail.com] Sent: 22 February 2016 15:43 To: solr-user@lucene.apache.org Subject: Re: Slow commits On Mon, Feb 22, 2016 at 10:22 AM, Adam Neal [Extranet] wrote: > Highest count is fairly equal between string and text. They are not indexed > but stored and no docvalues used Ah, that's a big clue... I wonder if it's related to stored fields compression. In Solr 5.5. there is a way to tune compression, but the default is already BEST_SPEED. I don't know if an easy way to turn compression off to see if that is the culprit. Whatever the reasons, this seems to be a large performance regression, and it's not clear if it's expected or not. Could you file a JIRA issue for this? -Yonik > > From: Yonik Seeley [ysee...@gmail.com] > Sent: 22 February 2016 14:40 > To: solr-user@lucene.apache.org > Subject: Re: Slow commits > > > What are the types of the fields with the highest count? I assume > they are indexed. Are they stored, and do they have docValues? > > -Yonik > > > On Mon, Feb 22, 2016 at 4:37 AM, Adam Neal [Extranet] > wrote: >> Well I got the numbers wrong, there are actually around 66000 fields on the >> index. I have restructured the index and there are now around 1500 fiields. >> This has resulted in the commit taking 34 seconds which is acceptable for my >> usage however it is still significantly slower than the 4.10.2 commit on the >> original 66000 fields which was around 1 second. >> >> From: Adam Neal [Extranet] [an...@mass.co.uk] >> Sent: 19 February 2016 17:43 >> To: solr-user@lucene.apache.org >> Subject: RE: Slow commits >> >> I'm out of the office now so I don't have the numbers to hand but from >> memory I think there are probably around 800-1000 fields or so. I will >> confirm on Monday. >> >> If i have time over the weekend I will try and recreate the problem at home >> and see if I can post up a sample. >> ____ >> From: Yonik Seeley [ysee...@gmail.com] >> Sent: 19 February 2016 16:25 >> To: solr-user@lucene.apache.org >> Subject: Re: Slow commits >> >> On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] >> wrote: >>> I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow >>> commits on one of my cores. The core in question is relatively small (56k >>> docs) and the issue only shows when commiting after a number of deletes, >>> commiting after additions is fine. As an example commiting after deleting >>> approximately 10% of the documents takes around 25mins. The same test on >>> the 4.10.2 instance takes around 1 second. >>> >>> I have done some investigation and the problem appears to be caused by >>> having dynamic fields, the core in question has a large number, performing >>> the same operation on this core with the dynamic fields removed sees a big >>> improvement on the performance with the commit taking 11 seconds (still not >>> quite on a par with 4.10.2). >> >> Dynamic fields is a Solr schema concept, and does not translate to any >> differences in Lucene. >> You may be hitting something due to a large number of fields (at the >> lucene level, each field name is a different field). How many >> different fields (i.e. fieldnames) do you have across the entire >> index? >> >> -Yonik >> >> # >> >> This E-mail is the property of Mass Consultants Ltd. It is confidential and >> intended only for the use of the addressee or with its permission. Use by >> anyone else for any purpose is prohibited. If you are not the addressee, >> you should not use, disclose, copy or distribute this e-mail and should >> notify us of receipt immediately by return e-mail to the address where the >> e-mail originated. >> >> This E-mail may not have been sent through a secure system and accordingly >> (i) its contents should not be relied upon by any person without independent >> verification from Mass Consultants Ltd and (ii) it is the responsibility of >> the recipient to ensure that the onwa
