subject:"RE\: customizing standard tokenizer"

RE: customizing standard tokenizer

2012-02-20 Thread Torsten Krah

Thx, will use the custom tokenizer. Its less error prone than the "workarounds" mentioned. smime.p7s Description: S/MIME cryptographic signature

RE: customizing standard tokenizer

2012-02-17 Thread Steven A Rowe

Hi Torsten, The Lucene StandardTokenizer is written in JFlex (http://jflex.de) - you can see the version 3.X specification at: You can m

Re: customizing standard tokenizer

2012-02-17 Thread Em

Hi Torsten, did you have a look at WordDelimiterTokenFilter? Sounds like it fits your needs. Regards, Em Am 17.02.2012 15:14, schrieb Torsten Krah: > Hi, > > is it possible to extend the standard tokenizer or use a custom one > (possible via extending the standard one) to add some "custom" tok