On 7/4/06, Eric Jain <[EMAIL PROTECTED]> wrote:
Just noticed that in the newTok method in the WordDelimiterFilter, the "position increment" of the token doesn't seem to be copied?
Your'e right, it does look possible that position info can be lost. The fix probably isn't as simple as copying the position increment in newTok()... the original increment should only be copied for the first token generated. -Yonik
