I have some info and examples for the WikipediaTokenizer in my book, but a
tokenizer does not direct tokens to a field. Rather, you would use the
tokenizer in the analyzer for whatever field you wish to store values in.
You could use the same input for multiple fields and then filter the tokens
I suggest you to look at here:
http://www.javadocexamples.com/java_source/org/apache/lucene/wikipedia/analysis/WikipediaTokenizerTest.java.html
2013/10/4 Ken Krugler
> Hi all,
>
> Where's the documentation on the WikipediaTokenizer?
>
> Specifically I'm wondering how pieces from the source XML
