Re: Performance/scaling with custom function queries

2014-06-12 Thread Robert Krüger
Thanks for the info. I will look at that. On Wed, Jun 11, 2014 at 3:47 PM, Joel Bernstein wrote: > In Solr 4.9 there is a feature called RankQueries, that allows you to > plugin your own ranking collector. So, if you wanted to write a > ranking/sorting collector that used a thread per segment, yo

Re: Performance/scaling with custom function queries

2014-06-11 Thread Joel Bernstein
In Solr 4.9 there is a feature called RankQueries, that allows you to plugin your own ranking collector. So, if you wanted to write a ranking/sorting collector that used a thread per segment, you could cleanly plug it in. Joel Bernstein Search Engineer at Heliosearch On Wed, Jun 11, 2014 at 9:39

Re: Performance/scaling with custom function queries

2014-06-11 Thread david.w.smi...@gmail.com
On Wed, Jun 11, 2014 at 7:46 AM, Robert Krüger wrote: > Or will I have to set up distributed search to achieve that? Yes — you have to shard it to achieve that. The shards could be on the same node. There were some discussions this year in JIRA about being able to do thread-per-segment but it

Re: Performance/scaling with custom function queries

2014-06-11 Thread Robert Krüger
Would Solr use multithreading to process the records of a function query as described above? In my scenario concurrent searches are not the issue, rather the speed of one query will be the optimization target. Or will I have to set up distributed search to achieve that? Thanks, Robert On Tue, Ju

Re: Performance/scaling with custom function queries

2014-06-10 Thread Robert Krüger
Great, I was hoping for that. In my case I will have to deal with the worst case scenario, i.e. all documents matching the query, because the only criterion is the fingerprint and the result of the distance/similarity function which will have to be executed for every document. However, I am dealing

Re: Performance/scaling with custom function queries

2014-06-08 Thread Joel Bernstein
You only need to have fast access to the fingerprint field so only that field needs to be in memory. You'll want to review how Lucene DocValues and FieldCache work. Sorting is done with a PriorityQueue so only the top N docs are kept in memory. You'll only need to access the fingerprint field valu