kkharbas opened a new pull request, #16577: URL: https://github.com/apache/lucene/pull/16577
## Summary `FixedShingleFilter` could emit shingles with an incorrect position increment when a base token (e.g. a multi-word synonym like `Wi-Fi`) occupied multiple positions but could not itself form a shingle, and a later token stacked at the same position could. In that case the position increment accumulated by the skipped base token was lost, causing downstream consumers to place the emitted shingle at the wrong position. - Introduced `pendingPosInc` to accumulate position increments from base tokens that don't produce a shingle, so the next emitted shingle carries the full, correct increment. - Added an `end()` override to fold any remaining `pendingPosInc` into the final position increment reported by `TokenStream.end()`. - Added a `reset()` override to clear `pendingPosInc` on reuse. - Added two regression tests (`testPositionIncrementAfterSkippedGraphToken` and `testFirstShinglePositionIncrementAfterSkippedGraphToken`) that reproduce the bug and verify the fix. ## Test plan - [ ] `./gradlew :lucene:analysis:common:test --tests "*.TestFixedShingleFilter"` passes - [ ] Existing `FixedShingleFilter` tests continue to pass unchanged 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
