costin opened a new pull request, #16371:
URL: https://github.com/apache/lucene/pull/16371
Add a token filter that applies Unicode simple case folding to normalize
text for case-insensitive matching, without requiring the ICU analysis plugin.
Performs simple case folding for ~1200 BMP codepoints, which differs from
Character.toLowerCase() such as: Greek final sigma (ς→σ), micro sign to Greek
mu (µ→μ) and others. For ASCII text, folding is identical to lowercasing.
Supplementary codepoints (> U+FFFF) fall back to Character.toLowerCase().
Uses a flat char[65536] lookup table (128 KB) lazily initialized on first
use (once per JVM), built from ~1200 {codepoint, folded} pairs (which get
discarded after initialization) generated from ICU4J via ./gradlew
:lucene:core:generateFoldTable
Inspired by #14389.
### Benchmark
AMD EPYC (c5a.2xlarge), JDK 25, 256 tokens x 8 chars
Baseline is Character.toLowerCase()
| Distribution | Baseline (ops/ms) | FoldTable (ops/ms) | Speedup |
|---|---|---|---|
| english | 0.0739 | 0.0774 | 1.05x |
| german | 0.0699 | 0.0882 | 1.26x |
| turkish | 0.0565 | 0.1044 | 1.85x |
| russian | 0.0137 | 0.1676 | 12.23x |
| greek | 0.0136 | 0.1649 | 12.14x |
| armenian | 0.0138 | 0.1673 | 12.13x |
| japanese | 0.0180 | 0.1653 | 9.16x |
| cherokee | 0.0101 | 0.1676 | 16.62x |
| mixed | 0.0178 | 0.1046 | 5.86x |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
