Re: International filters/tokenizers doing too much

2011-06-14 Thread Shawn Heisey
On 6/14/2011 5:34 PM, Robert Muir wrote: On Tue, Jun 14, 2011 at 7:07 PM, Shawn Heisey wrote: Because the text in my index comes in many different languages with no ability to know the language ahead of time, I have a need to use ICUTokenizer and/or the CJK filters, but I have a problem with th

Re: International filters/tokenizers doing too much

2011-06-14 Thread Robert Muir
On Tue, Jun 14, 2011 at 7:07 PM, Shawn Heisey wrote: > Because the text in my index comes in many different languages with no > ability to know the language ahead of time, I have a need to use > ICUTokenizer and/or the CJK filters, but I have a problem with them as they > are implemented currently