Re: Autosuggest on very large index

2013-08-20 Thread Greg Preston
DocValues looks interesting, a non-inverted field. I'll play with it a bit and see how it works. Thanks for the suggestion. I don't know how many total terms we've got, but each "document" is only 2-5 words/terms on average, and there is a TON of overlap between docs. -Greg On Tue, Aug 20,

Re: Autosuggest on very large index

2013-08-20 Thread Jack Krupansky
Sounds like a problem for DocValues - assuming the number of unique values fits reasonably in memory to avoid I/O. How many unique values do you have or contemplate for two your billion documents? Two possibilities: 1. You need a lot more hardware. 2. You need to scale back your ambitions.

Re: Autosuggest on very large index

2013-08-20 Thread Greg Preston
The filter query would be on a different field (clientId) than the field we want to autosuggest on (title). Or are you proposing we index a compound field that would be clientId+titleTokens so we would then prefix the suggester with clientId+userInput ? Interesting idea. -Greg On Tue, Aug 20,

RE: Autosuggest on very large index

2013-08-20 Thread Markus Jelsma
I am not entirely sure but the Suggester's FST uses prefixes so you may be able to prefix the value you otherwise use for the filter query when you build the suggester. -Original message- > From:Greg Preston > Sent: Tuesday 20th August 2013 20:00 > To: solr-user@lucene.apache.org > Sub