Re: More HDFS and Shard Splitting

2014-11-20 Thread Joseph Obernberger
Just confirmed that you do need to create the core directory before doing the SHARDSPLIT (at least with HDFS) - otherwise it fails saying that it cannot find classes - like the cluster classes. Iv'e noticed that the disk usage on HDFS goes up when I do the split - for example, if I split a 100G sh

Re: More HDFS and Shard Splitting

2014-11-17 Thread Joseph Obernberger
Looks like the shard split failed, and only created one additional shard. I didn't allocate enough memory for 3x - since two additional shards needed to be created. I was allocating 20G for each shard, so in order do the split, I needed to give 60G for the direct memory access. I've now switched

Re: More HDFS and Shard Splitting

2014-11-17 Thread Erick Erickson
Tell us more about your HDFS stuff. Specifically, how do you have your HDFSDirectoryFactory specified in solrconfig.xml? Cause you shouldn't have to do things like create the directory ahead of time I don't think. Best, Erick On Mon, Nov 17, 2014 at 12:17 PM, Joseph Obernberger wrote: > Origina