kamthorn opened a new issue, #16716: URL: https://github.com/apache/lucene/issues/16716
### Description Currently, Apache Lucene provides language-specific normalization filters for many complex scripts and languages (e.g. `ArabicNormalizationFilter`, `HindiNormalizationFilter`, `BengaliNormalizationFilter`, `GermanNormalizationFilter`), but lacks normalization support for the Thai language. Thai script suffers from high orthographic variation, common keyboard typing errors, and Unicode encoding ambiguities: 1. **Double / repeated vowels and tone marks (Deduplication):** Users frequently type repeated combining marks by accident (e.g., `ดีี` with duplicate Sara Ii `\u0E35`, `ไม้้` with duplicate Mai Tho `\u0E49`). 2. **Double Sara E -> Sara Ae composition:** Typing two consecutive Sara E (`เเ` `\u0E40\u0E40`) instead of Sara Ae (`แ` `\u0E41`), which look virtually identical visually but produce completely different tokens. 3. **Decomposed Sara Am recomposition:** Sara Am (`ำ` `\u0E33`) is often typed or decomposed as Nikhahit (`\u0E4D`) + Sara Aa (`\u0E32`) or Nikhahit + Tone mark + Sara Aa (e.g. `ทํางาน` vs `ทำงาน`, `นํ้า` vs `น้ำ`). 4. **Keystroke ordering anomalies (Canonical Reordering):** Typing tone marks or Thanthakhat before upper/lower vowels (e.g. `ก` + `้` + `ิ` instead of standard `ก` + `ิ` + `้`), or typing follow vowels before tone marks (e.g. `น` + `ำ` + `้` instead of `น` + `้` + `ำ`). 5. **Lakkhangyao normalization:** Replacing `\u0E45` with Sara Aa `\u0E32` when not preceded by Ru (`ฤ`) or Lu (`ฦ`). 6. **Zero-width characters:** Removing invisible zero-width spaces (ZWSP `\u200B`, ZWNJ `\u200C`). 7. **Dangling marks removal:** Removing stray non-base combining marks at the beginning of tokens. We propose adding: - `ThaiNormalizer`: In-place character array normalization for Thai text. - `ThaiNormalizationFilter`: A `TokenFilter` applying `ThaiNormalizer`. - `ThaiNormalizationFilterFactory`: SPI factory with name `thaiNormalization`. - Integrate `ThaiNormalizationFilter` into the default `ThaiAnalyzer` pipeline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
