Would "doing the right thing" include firing the results at different fields based on the language detected? Your answer to Jan seems to indicate not, in which case my original comments stand. The main point is that mixing all the *results* of the analysis chains for multiple languages into a single field will likely result in "interesting" behavior. Not to say it won't be satisfactory in your situation, but there are edge cases.
Best Erick On Fri, Jan 20, 2012 at 9:15 AM, Ted Dunning <ted.dunn...@gmail.com> wrote: > I think you misunderstood what I am suggesting. > > I am suggesting an analyzer that detects the language and then "does the > right thing" according to the language it finds. As such, it would > tokenize and stem English according to English rules, German by German > rules and would probably do a sliding bigram window in Japanese and Chinese. > > On Fri, Jan 20, 2012 at 8:54 AM, Erick Erickson > <erickerick...@gmail.com>wrote: > >> bq: Why not have a polyglot analyzer >> >> That could work, but it makes some compromises and assumes that your >> languages are "close enough", I have absolutely no clue how that would >> work for English and Chinese say. >> >> But it also introduces inconsistencies. Take stemming. Even though you >> could easily stem in the correct language, throwing all those stems >> into the same filed can produce interesting results at search time since >> you run the risk of hitting something produced by one of the other >> analysis chains. >>
