https://issues.apache.org/jira/browse/SOLR-1167
-Bryan
On May 13, 2009, at May 13, 7:20 PM, Otis Gospodnetic wrote:
Bryan, maybe it's time to stick this in JIRA?
http://wiki.apache.org/solr/HowToContribute
Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
From: Bryan Talbot <btal...@aeriagames.com>
To: solr-user@lucene.apache.org
Sent: Wednesday, May 13, 2009 10:11:21 PM
Subject: Re: Replication master+slave
I think the patch I included earlier covers solr core, but it looks
like at
least some other extensions (DIH) create and use their own XML
parser. So, if
this functionality is to extend to all XML files, those will need
similar
patches.
Here's one for DIH:
--- src/main/java/org/apache/solr/handler/dataimport/
DataImporter.java
(revision 774137)
+++ src/main/java/org/apache/solr/handler/dataimport/
DataImporter.java (working
copy)
@@ -148,8 +148,10 @@
void loadDataConfig(String configFile) {
try {
- DocumentBuilder builder = DocumentBuilderFactory.newInstance()
- .newDocumentBuilder();
+ DocumentBuilderFactory dbf =
DocumentBuilderFactory.newInstance();
+ dbf.setNamespaceAware(true);
+ dbf.setXIncludeAware(true);
+ DocumentBuilder builder = dbf.newDocumentBuilder();
Document document = builder.parse(new InputSource(new
StringReader(
configFile)));
The only down side I can see to this is it doesn't offer very
expressive
conditional inclusion: the file is included if it's present
otherwise fallback
inclusions can be used. It's also specific to XML files and
obviously won't
work for other types of configuration files. However, it is simple
and
effective.
-Bryan
On May 13, 2009, at May 13, 6:36 PM, Otis Gospodnetic wrote:
Coincidentally, from
http://www.cloudera.com/blog/2009/05/07/what%E2%80%99s-new-in-hadoop-core-020/
:
"Hadoop configuration files now support XInclude elements for
including
portions of another configuration file (HADOOP-4944). This
mechanism allows you
to make configuration files more modular and reusable."
So "others are doing it, too".
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
From: Bryan Talbot
To: solr-user@lucene.apache.org
Sent: Wednesday, May 13, 2009 11:26:41 AM
Subject: Re: Replication master+slave
I see that Nobel's final comment in SOLR-1154 is that config
files need to be
able to include snippets from external files. In my limited
testing, a
simple
patch to enable XInclude support seems to work.
--- src/java/org/apache/solr/core/Config.java (revision 774137)
+++ src/java/org/apache/solr/core/Config.java (working copy)
@@ -100,8 +100,10 @@
if (lis == null) {
lis = loader.openConfig(name);
}
- javax.xml.parsers.DocumentBuilder builder =
DocumentBuilderFactory.newInstance().newDocumentBuilder();
- doc = builder.parse(lis);
+ javax.xml.parsers.DocumentBuilderFactory dbf =
DocumentBuilderFactory.newInstance();
+ dbf.setNamespaceAware(true);
+ dbf.setXIncludeAware(true);
+ doc = dbf.newDocumentBuilder().parse(lis);
DOMUtil.substituteProperties(doc, loader.getCoreProperties());
} catch (ParserConfigurationException e) {
This allows a clause like this to include the contents of
replication.xml if
it
exists. If it's not found an exception will be thrown.
href="http://localhost:8983/solr/corename/admin/file/?file=replication.xml
"
xmlns:xi="http://www.w3.org/2001/XInclude";>
If the file is optional and no exception should be thrown if the
file is
missing, simply include a fallback action: in this case the
fallback is empty
and does nothing.
href="http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml
"
xmlns:xi="http://www.w3.org/2001/XInclude";>
-Bryan
On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote:
I was looking at the same problem, and had a discussion with
Noble. You can
use a hack to achieve what you want, see
https://issues.apache.org/jira/browse/SOLR-1154
Thanks,
Jianhan
On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot wrote:
So how are people managing solrconfig.xml files which are
largely the same
other than differences for replication?
I don't think it's a "good thing" to maintain two copies of the
same file
and I'd like to avoid that. Maybe enabling the XInclude
feature in
DocumentBuilders would make it possible to modularize
configuration files
to
make this possible?
http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)
-Bryan
On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar
wrote:
On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
wrote:
For replication in 1.4, the wiki at
http://wiki.apache.org/solr/SolrReplication says that a node
can be both
the master and a slave:
A node can act as both master and slave. In that case both
the master and
slave configuration lists need to be present inside the
ReplicationHandler
requestHandler in the solrconfig.xml.
What does this mean? Does the core then poll itself for
updates?
No. This type of configuration is meant for "repeaters".
Suppose there are
slaves in multiple data-centers (say data center A and B).
There is always
a
single master (say in A). One of the slaves in B is used as a
master for
the
other slaves in B. Therefore, this one slave in B is both a
master as well
as the slave.
I'd like to have a single set of configuration files that are
shared by
masters and slaves and avoid duplicating configuration
details in
multiple
files (one for master and one for slave) to ease management
and failover.
Is this possible?
You wouldn't want the master to be a slave. So I guess you'd
need to have
a
separate file. Also, it needs to be a separate file so that
the slave does
not become a master when the solrconfig.xml is replicated.
When I attempt to setup a multi server master-slave
configuration and
include both master and slave replication configuration
options, I into
some
problems. I'm running a nightly build from May 7.
Not sure what happened. Is that the url for this solr (meaning
same solr
url
is master and slave of itself)? If yes, that is not a valid
configuration.
--
Regards,
Shalin Shekhar Mangar.
