indrajithekanayake opened a new issue, #16724: URL: https://github.com/apache/lucene/issues/16724
### Description Lucene currently has no analyzer for Sinhala (ISO 639-1: si), the primary language of around 16 million speakers in Sri Lanka. Sinhala text today falls back to StandardAnalyzer, which tokenizes but does no normalization, stopword removal or stemming. This was proposed before in #10085 and #10086 in 2019 (seems duplicate issues) but never materialised. I'd like to pick this up and take it through properly. If maintainers agree, those two could be closed in favour of this one. ### Proposed scope Following the structure of existing Indic analyzers (bn, hi, te), under `lucene/analysis/common/src/java/org/apache/lucene/analysis/si`: - `SinhalaAnalyzer`: StandardTokenizer → DecimalDigitFilter → SinhalaNormalizationFilter → StopFilter → SetKeywordMarkerFilter → SinhalaStemFilter - `SinhalaNormalizationFilter` (+ factory): Sinhala-specific normalization. The Sinhala block (U+0D80–U+0DFF) isn't covered by IndicNormalizer. Key issues are handling of ZWJ (U+200D) in conjunct forms like yansaya, rakaransaya and touching letters, and canonicalizing equivalent vowel sign sequences. - `SinhalaStemFilter` (+ factory): rule-based affix stripper (prefixes, inflectional and derivational suffixes, with stem-ending restoration), re-implemented from the approach in the #10085 as a proper TokenFilter with tests. - `stopwords.txt`: derived from frequency analysis of the UCSC 10M-word Sinhala corpus (Weerasinghe, Herath and Welgama, 2009), then manually filtered to function words (particles, postpositions, pronouns, auxiliaries). - Tests extending BaseTokenStreamTestCase, plus random-data tests as with other analyzers ### Evaluation I have access to the UCSC 10M-word Sinhala corpus described in: Ruvan Weerasinghe, Dulip Herath and Viraj Welgama. 2009. Corpus-based Sinhala Lexicon. In Proceedings of the 7th Workshop on Asian Language Resources (ALR7), pages 17–23. https://aclanthology.org/W09-3403/ The corpus itself won't be included in Lucene. Derived resources (the stopword list) will be contributed under the Apache License 2.0 with the corpus owners' permission. Planned evaluation on the corpus: - Normalization: reduction in distinct token forms, with a manual check for false merges - Stemming: vocabulary reduction vs StandardAnalyzer, and Paice under/over-stemming indices on [N] manually grouped word families, compared against the #10085 stemmer ### Questions for maintainers 1. Is a light rule-based stemmer acceptable here, similar to BengaliStemmer, or would you prefer a dictionary/Hunspell-based approach? I'm happy to open a PR once the scope is agreed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
