kamthorn opened a new issue, #16719: URL: https://github.com/apache/lucene/issues/16719
### Description In the Thai language, the character Maiyamok (`ๆ` / `U+0E46`) denotes reduplication (repetition) of the preceding word or phrase to convey emphasis, plurality, or frequency: - `"เร็วๆ"` (or `"เร็ว ๆ"`) means `"เร็ว เร็ว"` (very fast) - `"มากๆ"` (or `"มาก ๆ"`) means `"มาก มาก"` (very much) - `"เด็กๆ"` means `"เด็ก เด็ก"` (children) - `"บ่อยๆ"` means `"บ่อย บ่อย"` (frequently) #### Current Problem: When Thai text is tokenized (e.g. using `ThaiTokenizer` with `java.text.BreakIterator`, or `ICUTokenizer`), `ๆ` is classified in Unicode as `Character.MODIFIER_LETTER` and emitted as an isolated token: `"เร็วๆ"` -> `["เร็ว", "ๆ"]` This leads to several search retrieval issues: 1. **Term Frequency distortion**: The repeated word `"เร็ว"` only has a term frequency of 1 instead of 2. 2. **Phrase search failures**: A phrase query for `"เร็ว เร็ว"` will fail to match a document containing `"เร็วๆ"` because position 2 is occupied by `"ๆ"` instead of `"เร็ว"`. 3. **Punctuation noise**: The symbol `"ๆ"` is indexed as a searchable token despite having no independent lexical meaning. 4. **Standalone / attached token inconsistency**: Different tokenizers or input texts either emit `ๆ` as a separate token (`["เร็ว", "ๆ"]`) or attached (`["เร็วๆ"]`). #### Proposed Solution: We propose adding: - `ThaiRepeatFilter`: A `TokenFilter` that duplicates the preceding term upon encountering Maiyamok (`ๆ`), handling both standalone tokens (`["เร็ว", "ๆ"]`) and tokens with trailing Maiyamok (`["เร็วๆ"]`). - `ThaiRepeatFilterFactory`: SPI factory with name `thaiRepeat`. - Integrate `ThaiRepeatFilter` into `ThaiAnalyzer`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
