Hi Angel, please share the freesuggester defined in the config. I guess you might have mentioned whitespace as separator in the freesuggester definition as :
<str name="separator"> </str> Which is creaing the trouble. On Tue, Jul 25, 2017 at 9:01 AM, Erick Erickson <erickerick...@gmail.com> wrote: > The shingle filter may use space as the separator between shingles that it > generates. The admin/ analysis page is your friend. > > On Jul 24, 2017 2:45 PM, "Angel Todorov" <attodo...@gmail.com> wrote: > > > Hi Rick, > > > > Yep, that's really weird, because I am using the > StandardTokenizerFactory, > > which is supposed to remove whitespace. Also tried the > > WhitespaceTokenizerFactory. I'll have a look at other analyzers or if > > nothing works maybe implement my own. > > > > I am using a Shingle filter right after the StandardTokenizer, not sure > if > > that has anything to do with it. > > > > > > Thanks, > > Angel > > > > > > On Tue, Jul 25, 2017 at 12:09 AM Rick Leir <rl...@leirtech.com> wrote: > > > > > Angel, > > > The 20 byte is an ASCII space character, which is a separator in most > > > contexts. Breaking the buffer at spaces, you can see 6 non-space > tokens. > > > > > > Have a look at your analysis chain and see why you are getting this. > > > Cheers -- Rick > > > > > > On July 24, 2017 4:27:00 PM EDT, Angel Todorov <attodo...@gmail.com> > > > wrote: > > > >Hi guys, > > > > > > > >I am trying to setup the FreeTextSuggester/ Lookup Factory in a > > > >suggester > > > >definition in SOLR. Unfortunately while the index is building, I am > > > >encountering the following errors: > > > > > > > >*"msg":"tokens must not contain separator byte; got token=[30 20 30 20 > > > >32 > > > >20 72 20 61 6c 6c 65 6e 20 72] but gramCount=6, which is greater than > > > >expected max ngram size=5","trace":"java.lang. > IllegalArgumentException: > > > >tokens must not contain separator byte; got token=[30 20 30 20 32 20 > 72 > > > >20 > > > >61 6c 6c 65 6e 20 72] but gramCount=6, which is greater than expected > > > >max > > > >ngram size=5\r\n\tat > > > > > > >org.apache.lucene.search.suggest.analyzing.FreeTextSuggester.build( > > FreeTextSuggester.java:362)\r\n\tat > > > >* > > > > > > > >I've also opened the following issue, because i don't think it's right > > > >not > > > >to handle this exception: > > > > > > > >https://issues.apache.org/jira/browse/SOLR-11139 > > > > > > > >But my question is about the error in general - why is it occurring? I > > > >only > > > >have English text, nothing special. > > > > > > > >Thanks, > > > >Angel > > > > > > -- > > > Sorry for being brief. Alternate email is rickleir at yahoo dot com > > >
