I'm doing HDFS input and output in my job, with the following:
hadoop jar /mnt/faas-solr.jar \
-D mapreduce.job.map.class=com.massrel.faassolr.SolrMapper \
--update-conflict-resolver com.massrel.faassolr.SolrConflictResolver
\
--morphline-file /mnt/morphline-ignore.conf \
--zk-host $ZKHOST \
--output-dir hdfs://$MASTERIP:9000/output/ \
--collection $COLLECTION \
--go-live \
--verbose \
hdfs://$MASTERIP:9000/input/
Index creation works,
$ hadoop fs -ls -R hdfs://$MASTERIP:9000/output/results/part-00000
drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data
drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index
-rwxr-xr-x 1 hadoop supergroup 61 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0.fdt
-rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0.fdx
-rwxr-xr-x 1 hadoop supergroup 1681 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0.fnm
-rwxr-xr-x 1 hadoop supergroup 396 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0.si
-rwxr-xr-x 1 hadoop supergroup 67 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene41_0.doc
-rwxr-xr-x 1 hadoop supergroup 37 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene41_0.pos
-rwxr-xr-x 1 hadoop supergroup 508 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene41_0.tim
-rwxr-xr-x 1 hadoop supergroup 305 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene41_0.tip
-rwxr-xr-x 1 hadoop supergroup 120 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene45_0.dvd
-rwxr-xr-x 1 hadoop supergroup 351 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene45_0.dvm
-rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/segments_1
-rwxr-xr-x 1 hadoop supergroup 110 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/index/segments_2
drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/tlog
-rw-r--r-- 1 hadoop supergroup 333 2014-04-17 16:00 hdfs://
10.98.33.114:9000/output/results/part-00000/data/tlog/tlog.0000000000000000000
But the go-live step fails, it's trying to use the HDFS path as the remote
index path?
14/04/17 16:00:31 INFO hadoop.GoLive: Live merging of output shards into
Solr cluster...
14/04/17 16:00:31 INFO hadoop.GoLive: Live merge hdfs://
10.98.33.114:9000/output/results/part-00000 into
http://discover8-test-1d.i.massrel.com:8983/solr
14/04/17 16:00:31 ERROR hadoop.GoLive: Error sending live merge command
java.util.concurrent.ExecutionException:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
directory '/mnt/solr_8983/home/hdfs:/
10.98.33.114:9000/output/results/part-00000/data/index' does not exist
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.apache.solr.hadoop.GoLive.goLive(GoLive.java:126)
at
org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:867)
at
org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:609)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.solr.hadoop.MapReduceIndexerTool.main(MapReduceIndexerTool.java:596)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
directory '/mnt/solr_8983/home/hdfs:/
10.98.33.114:9000/output/results/part-00000/data/index' does not exist
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at
org.apache.solr.client.solrj.request.CoreAdminRequest.process(CoreAdminRequest.java:493)
at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:100)
at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:89)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
14/04/17 16:00:31 INFO hadoop.GoLive: Live merging of index shards into
Solr cluster took 2.31269488E8 secs
14/04/17 16:00:31 INFO hadoop.GoLive: Live merging failed
I'm digging into the code now, but wanted to send this out as a sanity
check.
Thanks,
Brett
