There's a related issue open. https://issues.apache.org/jira/browse/SOLR-712
On Thu, May 14, 2009 at 7:50 AM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > > Bryan, maybe it's time to stick this in JIRA? > http://wiki.apache.org/solr/HowToContribute > > Thanks, > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > ----- Original Message ---- > > From: Bryan Talbot <btal...@aeriagames.com> > > To: solr-user@lucene.apache.org > > Sent: Wednesday, May 13, 2009 10:11:21 PM > > Subject: Re: Replication master+slave > > > > I think the patch I included earlier covers solr core, but it looks like > at > > least some other extensions (DIH) create and use their own XML parser. > So, if > > this functionality is to extend to all XML files, those will need similar > > patches. > > > > Here's one for DIH: > > > > --- src/main/java/org/apache/solr/handler/dataimport/DataImporter.java > > (revision 774137) > > +++ src/main/java/org/apache/solr/handler/dataimport/DataImporter.java > (working > > copy) > > @@ -148,8 +148,10 @@ > > void loadDataConfig(String configFile) { > > > > try { > > - DocumentBuilder builder = DocumentBuilderFactory.newInstance() > > - .newDocumentBuilder(); > > + DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance(); > > + dbf.setNamespaceAware(true); > > + dbf.setXIncludeAware(true); > > + DocumentBuilder builder = dbf.newDocumentBuilder(); > > Document document = builder.parse(new InputSource(new > StringReader( > > configFile))); > > > > > > > > The only down side I can see to this is it doesn't offer very expressive > > conditional inclusion: the file is included if it's present otherwise > fallback > > inclusions can be used. It's also specific to XML files and obviously > won't > > work for other types of configuration files. However, it is simple and > > effective. > > > > > > -Bryan > > > > > > > > > > On May 13, 2009, at May 13, 6:36 PM, Otis Gospodnetic wrote: > > > > > > > > Coincidentally, from > > > http://www.cloudera.com/blog/2009/05/07/what%E2%80%99s-new-in-hadoop-core-020/: > > > > > > "Hadoop configuration files now support XInclude elements for including > > portions of another configuration file (HADOOP-4944). This mechanism > allows you > > to make configuration files more modular and reusable." > > > > > > So "others are doing it, too". > > > > > > Otis > > > -- > > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > > > > > > > > > ----- Original Message ---- > > >> From: Bryan Talbot > > >> To: solr-user@lucene.apache.org > > >> Sent: Wednesday, May 13, 2009 11:26:41 AM > > >> Subject: Re: Replication master+slave > > >> > > >> I see that Nobel's final comment in SOLR-1154 is that config files > need to be > > >> able to include snippets from external files. In my limited testing, > a > > simple > > >> patch to enable XInclude support seems to work. > > >> > > >> > > >> > > >> --- src/java/org/apache/solr/core/Config.java (revision 774137) > > >> +++ src/java/org/apache/solr/core/Config.java (working copy) > > >> @@ -100,8 +100,10 @@ > > >> if (lis == null) { > > >> lis = loader.openConfig(name); > > >> } > > >> - javax.xml.parsers.DocumentBuilder builder = > > >> DocumentBuilderFactory.newInstance().newDocumentBuilder(); > > >> - doc = builder.parse(lis); > > >> + javax.xml.parsers.DocumentBuilderFactory dbf = > > >> DocumentBuilderFactory.newInstance(); > > >> + dbf.setNamespaceAware(true); > > >> + dbf.setXIncludeAware(true); > > >> + doc = dbf.newDocumentBuilder().parse(lis); > > >> > > >> DOMUtil.substituteProperties(doc, loader.getCoreProperties()); > > >> } catch (ParserConfigurationException e) { > > >> > > >> > > >> > > >> This allows a clause like this to include the contents of > replication.xml if > > it > > >> exists. If it's not found an exception will be thrown. > > >> > > >> > > >> > > >> href=" > http://localhost:8983/solr/corename/admin/file/?file=replication.xml"; > > >> xmlns:xi="http://www.w3.org/2001/XInclude";> > > >> > > >> > > >> > > >> If the file is optional and no exception should be thrown if the file > is > > >> missing, simply include a fallback action: in this case the fallback > is empty > > >> and does nothing. > > >> > > >> > > >> > > >> href=" > http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml"; > > >> xmlns:xi="http://www.w3.org/2001/XInclude";> > > >> > > >> > > >> > > >> > > >> -Bryan > > >> > > >> > > >> > > >> > > >> On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote: > > >> > > >>> I was looking at the same problem, and had a discussion with Noble. > You can > > >>> use a hack to achieve what you want, see > > >>> > > >>> https://issues.apache.org/jira/browse/SOLR-1154 > > >>> > > >>> Thanks, > > >>> > > >>> Jianhan > > >>> > > >>> > > >>> On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot wrote: > > >>> > > >>>> So how are people managing solrconfig.xml files which are largely > the same > > >>>> other than differences for replication? > > >>>> > > >>>> I don't think it's a "good thing" to maintain two copies of the same > file > > >>>> and I'd like to avoid that. Maybe enabling the XInclude feature in > > >>>> DocumentBuilders would make it possible to modularize configuration > files > > to > > >>>> make this possible? > > >>>> > > >>>> > > >>>> > > >> > > > http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)<http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware%28boolean%29> > > >>>> > > >>>> > > >>>> -Bryan > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote: > > >>>> > > >>>> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot > > >>>>>> wrote: > > >>>>> > > >>>>> For replication in 1.4, the wiki at > > >>>>>> http://wiki.apache.org/solr/SolrReplication says that a node can > be both > > >>>>>> the master and a slave: > > >>>>>> > > >>>>>> A node can act as both master and slave. In that case both the > master and > > >>>>>> slave configuration lists need to be present inside the > > >>>>>> ReplicationHandler > > >>>>>> requestHandler in the solrconfig.xml. > > >>>>>> > > >>>>>> What does this mean? Does the core then poll itself for updates? > > >>>>>> > > >>>>> > > >>>>> > > >>>>> No. This type of configuration is meant for "repeaters". Suppose > there are > > >>>>> slaves in multiple data-centers (say data center A and B). There is > always > > >>>>> a > > >>>>> single master (say in A). One of the slaves in B is used as a > master for > > >>>>> the > > >>>>> other slaves in B. Therefore, this one slave in B is both a master > as well > > >>>>> as the slave. > > >>>>> > > >>>>> > > >>>>> > > >>>>>> I'd like to have a single set of configuration files that are > shared by > > >>>>>> masters and slaves and avoid duplicating configuration details in > > >>>>>> multiple > > >>>>>> files (one for master and one for slave) to ease management and > failover. > > >>>>>> Is this possible? > > >>>>>> > > >>>>>> > > >>>>> You wouldn't want the master to be a slave. So I guess you'd need > to have > > >>>>> a > > >>>>> separate file. Also, it needs to be a separate file so that the > slave does > > >>>>> not become a master when the solrconfig.xml is replicated. > > >>>>> > > >>>>> > > >>>>> > > >>>>>> When I attempt to setup a multi server master-slave configuration > and > > >>>>>> include both master and slave replication configuration options, I > into > > >>>>>> some > > >>>>>> problems. I'm running a nightly build from May 7. > > >>>>>> > > >>>>>> > > >>>>> Not sure what happened. Is that the url for this solr (meaning same > solr > > >>>>> url > > >>>>> is master and slave of itself)? If yes, that is not a valid > configuration. > > >>>>> > > >>>>> -- > > >>>>> Regards, > > >>>>> Shalin Shekhar Mangar. > > >>>>> > > >>>> > > >>>> > > > > > -- Regards, Shalin Shekhar Mangar.
