kamthorn opened a new pull request, #16720:
URL: https://github.com/apache/lucene/pull/16720
### Description
Closes #16719
This PR introduces `ThaiRepeatFilter` and `ThaiRepeatFilterFactory` to
`lucene/analysis/common`, providing support for Thai Maiyamok (`ๆ` / `U+0E46`),
which denotes word repetition (reduplication).
#### Motivation:
In Thai, Maiyamok (`ๆ`) is used after a word or phrase to repeat it:
- `"เร็วๆ"` / `"เร็ว ๆ"` -> `"เร็ว เร็ว"` (very fast)
- `"มากๆ"` / `"มาก ๆ"` -> `"มาก มาก"` (very much)
- `"เด็กๆ"` -> `"เด็ก เด็ก"` (children)
When tokenized by segmenters like `ThaiTokenizer` (based on `BreakIterator`)
or `ICUTokenizer`, `ๆ` is emitted as a standalone token (`["เร็ว", "ๆ"]`). This
distorts term frequency (1 instead of 2), breaks phrase queries (e.g. `"เร็ว
เร็ว"` does not match `["เร็ว", "ๆ"]`), and indexes `ๆ` as a meaningless token.
In other tokenization pipelines (e.g. `WhitespaceTokenizer`), `ๆ` may remain
attached at the end of the word (`["เร็วๆ"]`).
#### Features:
- **`ThaiRepeatFilter`**:
- Handles standalone Maiyamok tokens (`["เร็ว", "ๆ"]\ -> `["เร็ว",
"เร็ว"]`).
- Handles multiple consecutive Maiyamok (`["มาก", "ๆ", "ๆ"]\ -> `["มาก",
"มาก", "มาก"]`).
- Handles attached trailing Maiyamok (`["เร็วๆ"]\ -> `["เร็ว", "เร็ว"]`,
`"มากๆๆ"\ -> `["มาก", "มาก", "มาก"]`).
- Gracefully discards dangling Maiyamok at stream start with no preceding
token.
- **`ThaiRepeatFilterFactory`**:
- SPI name: `thaiRepeat`. Registered in
`META-INF/services/org.apache.lucene.analysis.TokenFilterFactory`.
- **`ThaiAnalyzer`**:
- Integrated `ThaiRepeatFilter` into the analysis pipeline.
- **Tests**:
- Added `TestThaiRepeatFilter` covering standalone, attached, multiple,
sentence, dangling, and `ThaiTokenizer` integration.
- Added `TestThaiRepeatFilterFactory`.
- Added test case in `TestThaiAnalyzer`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
