: ex: "A + W Root Beer" : the field uses a keyword tokenizer to keep the string together, then : it will get converted to "aw root beer" by a custom filter ive made, i : now want to split that up into 3 tokens (aw, root, beer), but seems
TokenFilter's can produce more tokens then they consume ... so your custom TOkenFilter can split the single token "aw root beer" into three tokens if you want it to. (this is how WordDelimiterFilter and SYnonymFilter work) -Hoss
