Re: multiple tokenizers needed

2009-04-10 Thread Erik Hatcher
Or have the indexing client split the data at these delimiters and just use the CJKAnalyzer. Erik On Apr 10, 2009, at 7:30 AM, Grant Ingersoll wrote: The only thing that comes to mind in a short term way is writing two TokenFilter implementations that wrap the second and third token

Re: multiple tokenizers needed

2009-04-10 Thread Grant Ingersoll
The only thing that comes to mind in a short term way is writing two TokenFilter implementations that wrap the second and third tokenizers On Apr 9, 2009, at 11:00 PM, Ashish P wrote: I want to analyze a text based on pattern ";" and separate on whitespace and it is a Japanese text so use