Thanks Mark ... indeed, some doc updates would help.
Regarding what seems to be a popular question on sharding. It seems that
it would be a Good Thing that the shards for a collection running HDFS
essentially be pointers to the HDFS-replicated index. Is that what your
thinking is?
I've been following your work recently, would be interested in helping
out on this if there's the chance.
Is there a JIRA yet on this issue?
Thanks,
lajos
On 22/01/2014 16:57, Mark Miller wrote:
Right - solr.hdfs.home is the only setting you should use with SolrCloud.
The documentation should probably be improved.
If you set the data dir or ulog location in solrconfig.xml explicitly, it will
be the same for every collection. SolrCloud shares the solrconfig.xml across
SolrCore’s, and this will not work out.
By setting solr.hdfs.home and leaving the relative defaults, all of the
locations are correctly set for each different collection under solr.hdfs.home
without any effort on your part.
- Mark
On Jan 22, 2014, 7:22:22 AM, Lajos <la...@protulae.com> wrote: Uugh. I just
realised I should have take out the data dir and update log
definitions! Now it works fine.
Cheers,
L
On 22/01/2014 11:47, Lajos wrote:
Hi all,
I've been running Solr on HDFS, and that's fine.
But I have a Cloud installation I thought I'd try on HDFS. I uploaded
the configs for the core that runs in standalone mode already on HDFS
(on another cluster). I specify the HdfsDirectoryFactory, HDFS data dir,
solr.hdfs.home, and HDFS update log path:
<dataDir>hdfs://master:9000/solr/test/data</dataDir>
<directoryFactory name="DirectoryFactory"
class="solr.HdfsDirectoryFactory">
<str name="solr.hdfs.home">hdfs://master:9000/solr</str>
</directoryFactory>
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog>
<str name="dir">hdfs://master:9000/solr/test/ulog</str>
</updateLog>
</updateHandler>
Question is: should I create my collection differently than I would a
normal collection?
If I just try that, Solr will initialise the directory in HDFS as if it
were a single core. It will create shard directories on my nodes, but
not actually put anything in there. And then it will complain mightily
about not being able to forward updates to other nodes. (This same
cluster hosts regular collections, and everything is working fine).
Am I missing a step? Do I have to manually create HDFS directories for
each replica?
Thanks,
L
