Re: Solrcloud: external fields and frequent commits

2013-11-24 Thread Erick Erickson
Long blog post on commits and the state of updates here: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ hdfs is perfectly fine with Solr, there's even an HdfsDirectoryFactory for your index. It has its own performance characteristics/tuning param

Re: Solrcloud: external fields and frequent commits

2013-11-23 Thread Flavio Pompermaier
Thanks again for such a detailed description. In our use case we're going to save shards data on hdfs so they all have access to a shared location, it would be great to put such a file in one place in that case :) Do you think that using hdfs as storage is bad for performance? Last question: if I s

Re: Solrcloud: external fields and frequent commits

2013-11-22 Thread Erick Erickson
about <1>. Well, at a high level you're right, of course. Having the EFF stuff in a single place seems more elegant. But then ugly details crop up. I.e. "one place" implies that you'd have to fetch them over the network, potentially a very expensive operation every time there was a commit. Is this

Re: Solrcloud: external fields and frequent commits

2013-11-22 Thread Flavio Pompermaier
On Fri, Nov 22, 2013 at 2:21 PM, Erick Erickson wrote: > 1> I'm not quite sure I understand. External File Fields are keyed > by the unique id of the doc. So every shard _must_ have the > eff available for at least the documents in that shard. At first glance > this doesn't look simple. Perhaps a

Re: Solrcloud: external fields and frequent commits

2013-11-22 Thread Erick Erickson
1> I'm not quite sure I understand. External File Fields are keyed by the unique id of the doc. So every shard _must_ have the eff available for at least the documents in that shard. At first glance this doesn't look simple. Perhaps a bit more explanation of what you're using EFF for? 2> Let's be

Solrcloud: external fields and frequent commits

2013-11-22 Thread Flavio Pompermaier
Hi to all, we're migrating from solr 3.x to solr 4.x to use Solrcloud and I have two big doubts: 1) External fields. When I compute such a file do I have to copy it in the data directory of shards..? The external fields boosts the results of the query to a specific collection, for me it doesn't m