Solr cloud: zkHost in solr.xml gets wiped out
Hi, I'm working on setting up a solr cloud test environment, and the target environment I need to put it in has multiple webapps per tomcat instance. With that in mind, I wanted/had to avoid putting any configs in system properties. I tried putting the zkHost in solr.xml, like this: > > > >hostContext="/"/> > Everything works fine when I first start things up, create collections, upload docs, search, etc. Creating the collection, however, modifies the solr.xml file, and doesn't keep the zkHost setting: > > >hostContext="/"> > instanceDir="directory_shard2_replica1/" transient="false" > name="directory_shard2_replica1" collection="directory"/> > instanceDir="directory_shard1_replica1/" transient="false" > name="directory_shard1_replica1" collection="directory"/> > > With that in mind, once I restart tomcat, it no longer knows it's supposed to be talking to zookeeper, so it looks for local configs and blows up. I traced this back to the code in CoreContainer.java, in the method persistFile(), where it seems to contain no code to write out the zkHost when it updates solr.xml. I upped the logging on my solr instance to verify this code is executing, so I'm pretty sure it's the right spot. Is anyone else using zkHost in their solr.xml successfully? I can't see how it would work given this problem. Does this seem like a bug? If so, I can probably file a report and submit a patch. It seems like this problem may become a non-issue in 5.0, based on comments in the code and some of the discussion in JIRA, but I'm not sure how far off that is. Thanks! -Al Wold
Re: Start custom Java component on Solr start?
I've used a servlet context listener before and it works pretty well. You just have a write a small Java class to receive the event when the app is started, then add it to web.xml. I don't think there's much good official documentation, but this blog post outlines it pretty simply: http://www.mkyong.com/servlet/what-is-listener-servletcontextlistener-example/ -Al On Jun 17, 2013, at 1:03 PM, Otis Gospodnetic wrote: > Hi, > > What is the best thing in Solr to hook into that would allow me to > start (and keep running) a custom piece of code when Solr starts? Say > I want to have something that pulls data from an external queue from > within Solr and indexes it into Solr and I want it start and stop > together with the Solr process. Is there any place in Solr where one > could do that? > > Thanks, > Otis > -- > Solr & ElasticSearch Support > http://sematext.com/
Re: Solr cloud: zkHost in solr.xml gets wiped out
Hi Erick, I tried out your changes from the branch_4x branch. It looks good in terms of preserving the zkHost, but I'm running into an exception because it isn't persisting the instanceDir attribute on the element. I've got a few other things I need to take care of, but as soon as I have time I'll dig in and see if I can figure out what's going on, and see what changed to make this not work. Here are details on what the files looked like before/after CREATE call: original solr.xml: here's what was produced with 4.3 branch + a quick mod to preserve zkHost: here's what was produced with branch_4x 4.4-SNAPSHOT: and here's the error from solr.log after restarting after the CREATE: 2013-06-17 21:37:07,083 1874 [pool-2-thread-1] ERROR org.apache.solr.core.CoreContainer - null:java.lang.NullPointerException: Missing required 'instanceDir' at org.apache.solr.core.CoreDescriptor.doInit(CoreDescriptor.java:133) at org.apache.solr.core.CoreDescriptor.(CoreDescriptor.java:87) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:365) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:221) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:190) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:124) at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:103) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4638) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5294) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1099) at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1621) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) On Jun 16, 2013, at 5:38 AM, Erick Erickson wrote: > Al: > > As it happens, I hope sometime today to put up a patch for SOLR-4910 > that should harden up many things in persisting solr.xml, I'll be sure > to include this. It's kind of a pain to create an automated test for > this, so I'll give it a whirl manually. > > As you say, most of this is going away in 5.0, but it needs to work for 4.x. > > And when I get the patch up, if you could give it a "real world" try > it'd be great! > > Thanks, > Erick > > On Fri, Jun 14, 2013 at 6:15 PM, Al Wold wrote: >> Hi, >> I'm working on setting up a solr cloud test environment, and the target >> environment I need to put it in has multiple webapps per tomcat instance. >> With that in mind, I wanted/had to avoid putting any configs in system >> properties. I tried putting the zkHost in solr.xml, like this: >> >>> >>> >>> >>> >> hostContext="/"/> >>> >> >> Everything works fine when I first start things up, create collections, >> upload docs, search, etc. Creating the collection, however, modifies the >> solr.xml file, and doesn't keep the zkHost setting: >> >>> >>> >>> >> hostContext="/"> >>>>> instanceDir="directory_shard2_replica1/" transient="false" >>> name="directory_shard2_replica1" collection="directory"/> >>>>> instanceDir="directory_shard1_replica1/" transient="false" >>> name="directory_shard1_replica1" collection="directory"/> >>> >>> >> >> >> With that in mind, once I restart tomcat, it no longer knows it's suppose
Re: Solr cloud: zkHost in solr.xml gets wiped out
For the CREATE call, I'm doing it manually per the instructions here: http://wiki.apache.org/solr/SolrCloud Here's the exact URL I'm using: http://asu-solr-cloud.elasticbeanstalk.com/admin/collections?action=CREATE&name=directory&numShards=2&replicationFactor=2&maxShardsPerNode=2 I'm testing out your patch now, and I'll let you know how it goes. Thanks for all the help! -Al On Jun 18, 2013, at 6:47 AM, Erick Erickson wrote: > OK, I think I see what's happening. If you do > NOT specify an instanceDir on the create > (and I'm doing this via the core admin > interface, not SolrJ) then the default is > used, but not persisted. If you _do_ > specify the instance dir, it will be persisted. > > I've put up another quick patch (tested > only in my test case, running full suite > now). Can you give it a whirl? You'll have > to apply the patch over top of the current > 4x, een though the patch is for trunk it > applied to 4x cleanly for me and the tests ran. > > Thanks, > Erick > > On Tue, Jun 18, 2013 at 9:02 AM, Erick Erickson > wrote: >> OK, I put up a very preliminary patch attached to the bug >> if you want to try it out that addresses the extra junk being >> put in the tag. Doesn't address the instanceDir issue >> since I haven't reproduced it yet. >> >> Erick >> >> On Tue, Jun 18, 2013 at 8:46 AM, Erick Erickson >> wrote: >>> Whoa! What's this junk? >>> qt="/admin/cores" wt="javabin" version="2 >>> >>> That shouldn't be being preserved, and the instancedir should be! >>> >>> So I'm guessing you're using SolrJ to create the core, but I just >>> reproduced the problem (at least the 'wt="json" ') bit from the >>> browser and even from one of my internal tests when I added >>> extra parameters. >>> >>> That said, instanceDir is being preserved in my test, so I'm not >>> seeing everything you're seeing, could you cut/paste your >>> create code? I'll see if I can set up a test case for SolrJ to catch >>> this too. >>> >>> See SOLR-4935 >>> >>> Thanks for reporting! >>> >>> On Mon, Jun 17, 2013 at 5:39 PM, Al Wold wrote: >>>> Hi Erick, >>>> I tried out your changes from the branch_4x branch. It looks good in terms >>>> of preserving the zkHost, but I'm running into an exception because it >>>> isn't persisting the instanceDir attribute on the element. >>>> >>>> I've got a few other things I need to take care of, but as soon as I have >>>> time I'll dig in and see if I can figure out what's going on, and see what >>>> changed to make this not work. >>>> >>>> Here are details on what the files looked like before/after CREATE call: >>>> >>>> original solr.xml: >>>> >>>> >>>> >>>> >>>> >>> hostContext="/"/> >>>> >>>> >>>> here's what was produced with 4.3 branch + a quick mod to preserve zkHost: >>>> >>>> >>>> >>>> >>> hostContext="/"> >>>>>>> instanceDir="directory_shard1_replica1/" transient="false" >>>> name="directory_shard1_replica1" collection="directory"/> >>>>>>> instanceDir="directory_shard2_replica1/" transient="false" >>>> name="directory_shard2_replica1" collection="directory"/> >>>> >>>> >>>> >>>> here's what was produced with branch_4x 4.4-SNAPSHOT: >>>> >>>> >>>> >>>> >>> distribUpdateSoTimeout="0" distribUpdateConnTimeout="0" hostPort="8080" >>>> hostContext="/"> >>>>>>> collection="directory" qt="/admin/cores" wt="javabin" version="2"/> >>>>>>> collection="directory" qt="/admin/cores" wt="javabin" version="2"/> >>>> >>>> >>>> >>>> and here's the error from solr.log after restarting after the CREATE: >>>> >>>> 2013-06-17 21:37:07,083 1874 [pool-2-thread-1] ERROR >>>> org.apache.solr.core.CoreContainer - null:java
Re: Solr cloud: zkHost in solr.xml gets wiped out
I just finished a test with the patch, and it looks like all is working well. On Jun 18, 2013, at 12:19 PM, Al Wold wrote: > For the CREATE call, I'm doing it manually per the instructions here: > > http://wiki.apache.org/solr/SolrCloud > > Here's the exact URL I'm using: > > http://asu-solr-cloud.elasticbeanstalk.com/admin/collections?action=CREATE&name=directory&numShards=2&replicationFactor=2&maxShardsPerNode=2 > > I'm testing out your patch now, and I'll let you know how it goes. > > Thanks for all the help! > > -Al > > On Jun 18, 2013, at 6:47 AM, Erick Erickson wrote: > >> OK, I think I see what's happening. If you do >> NOT specify an instanceDir on the create >> (and I'm doing this via the core admin >> interface, not SolrJ) then the default is >> used, but not persisted. If you _do_ >> specify the instance dir, it will be persisted. >> >> I've put up another quick patch (tested >> only in my test case, running full suite >> now). Can you give it a whirl? You'll have >> to apply the patch over top of the current >> 4x, een though the patch is for trunk it >> applied to 4x cleanly for me and the tests ran. >> >> Thanks, >> Erick >> >> On Tue, Jun 18, 2013 at 9:02 AM, Erick Erickson >> wrote: >>> OK, I put up a very preliminary patch attached to the bug >>> if you want to try it out that addresses the extra junk being >>> put in the tag. Doesn't address the instanceDir issue >>> since I haven't reproduced it yet. >>> >>> Erick >>> >>> On Tue, Jun 18, 2013 at 8:46 AM, Erick Erickson >>> wrote: >>>> Whoa! What's this junk? >>>> qt="/admin/cores" wt="javabin" version="2 >>>> >>>> That shouldn't be being preserved, and the instancedir should be! >>>> >>>> So I'm guessing you're using SolrJ to create the core, but I just >>>> reproduced the problem (at least the 'wt="json" ') bit from the >>>> browser and even from one of my internal tests when I added >>>> extra parameters. >>>> >>>> That said, instanceDir is being preserved in my test, so I'm not >>>> seeing everything you're seeing, could you cut/paste your >>>> create code? I'll see if I can set up a test case for SolrJ to catch >>>> this too. >>>> >>>> See SOLR-4935 >>>> >>>> Thanks for reporting! >>>> >>>> On Mon, Jun 17, 2013 at 5:39 PM, Al Wold wrote: >>>>> Hi Erick, >>>>> I tried out your changes from the branch_4x branch. It looks good in >>>>> terms of preserving the zkHost, but I'm running into an exception because >>>>> it isn't persisting the instanceDir attribute on the element. >>>>> >>>>> I've got a few other things I need to take care of, but as soon as I have >>>>> time I'll dig in and see if I can figure out what's going on, and see >>>>> what changed to make this not work. >>>>> >>>>> Here are details on what the files looked like before/after CREATE call: >>>>> >>>>> original solr.xml: >>>>> >>>>> >>>>> >>>>> >>>>> >>>> hostContext="/"/> >>>>> >>>>> >>>>> here's what was produced with 4.3 branch + a quick mod to preserve zkHost: >>>>> >>>>> >>>>> >>>>> >>>> hostContext="/"> >>>>> >>>> instanceDir="directory_shard1_replica1/" transient="false" >>>>> name="directory_shard1_replica1" collection="directory"/> >>>>> >>>> instanceDir="directory_shard2_replica1/" transient="false" >>>>> name="directory_shard2_replica1" collection="directory"/> >>>>> >>>>> >>>>> >>>>> here's what was produced with branch_4x 4.4-SNAPSHOT: >>>>> >>>>> >>>>> >>>>> >>>> distribUpdateSoTimeout="0" distribUpdateConnTimeout="0" hostPort="8080" >>>>> hostContext="/"> >>&
