reugn opened a new pull request, #16754: URL: https://github.com/apache/lucene/pull/16754
Kuromoji and Nori accept a user dictionary of any size and compile it into an FST that lives as long as the analyzer. The FST already computes its own size, but nothing exposed it, so callers could not measure or bound what a user dictionary costs. This makes `UserDictionary` in both modules, and the shared `morph.TokenInfoFST` they build on, implement `Accountable`. `ramBytesUsed()` covers the FST plus the per-entry data held next to it. `morph.TokenInfoFST` also backs the bundled system dictionaries, so `TokenInfoDictionary.getInstance().getFST().ramBytesUsed()` now reports those too: about 2.2 MB for Japanese and 5.7 MB for Korean. The other dictionary classes are out of scope here, since their bundled instances are fixed in size and shared per JVM rather than built from caller input. Additive only, no changes to existing behavior or signatures. The size is computed once at construction, since the state is immutable. Tests pin the reported sizes against independently computed values, including a `RamUsageTester` cross-check, and add the first test for `morph.TokenInfoFST`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
