kamthorn opened a new pull request, #16718: URL: https://github.com/apache/lucene/pull/16718
### Description Fixes #5323 Relates to #14730 The Thai stopword list in `lucene/analysis/common/src/resources/org/apache/lucene/analysis/th/stopwords.txt` was originally derived from Sukhum et al. (2010) *"Opinion Detection in Thai Political News Columns Based on Subjectivity Analysis"*. Because it was extracted from political news editorial text, it contains numerous high-frequency domain-specific verbs and common nouns (e.g. `ผล`, `ทาง`, `เปิด`, `เปิดเผย`, `ออก`, `ส่ง`, `ขอ`, `จัด`, `วัน`, `รับ`, `ร่วม`, `ราย`, `พบ`, `ผ่าน`, `มา`, `ลง`, `ตั้ง`, `ด้าน`, `ช่วง`, `ขึ้น`, `เริ่ม`, `เห็น`, `เข้า`, `แบบ`, `แรก`, `ทำ`, `นำ`, `กล่าว`). #### Problems Caused: In Thai, words are written without spaces between them. A word segmentation component like `ThaiTokenizer` (based on `BreakIterator`) segments compound words into their constituent root words: - `"ที่ผ่านมา"` (in the past / recently) -> `[ที่]` `[ผ่าน]` `[มา]` - `"เปิดตัวสินค้า"` (product launch) -> `[เปิด]` `[ตัว]` `[สินค้า]` - `"ส่งออก"` (export) -> `[ส่ง]` `[ออก]` - `"ผลการเรียน"` (academic result) -> `[ผล]` `[การ]` `[เรียน]` - `"วันหยุด"` (holiday) -> `[วัน]` `[หยุด]` - `"รับสมัคร"` (recruitment) -> `[รับ]` `[สมัคร]` - `"แบบฟอร์ม"` (form) -> `[แบบ]` `[ฟอร์ม]` When the stopword filter strips these common verbs/nouns, crucial search constituents or even entire queries vanish completely. For example, in #14730, `"ที่ผ่านมา"` produced zero tokens because `ที่`, `ผ่าน`, and `มา` were all in the stopword list. Similarly, in #5323 (LUCENE-4253), users reported that `ThaiAnalyzer` lost critical terms due to `StopFilter`. #### Changes: 1. Removed 28 content verbs and nouns that cause over-filtering of compound words from `stopwords.txt`. 2. Retained grammatical function words (prepositions, conjunctions, pronouns, particles, auxiliaries). 3. Added canonical spellings for `สำหรับ` and `ทำให้`. 4. Added test cases in `TestThaiAnalyzer` verifying that compound queries such as `"ที่ผ่านมา"`, `"เปิดตัวสินค้า"`, `"ส่งออก"`, `"วันหยุด"`, and `"ผลการเรียน"` preserve their meaningful content tokens. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
