uschindler commented on PR #15900: URL: https://github.com/apache/lucene/pull/15900#issuecomment-4173406063
We have kind of this token filter already: https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/CodepointCountFilter.java This one drops tokens that are too long or too short. And this one counts codepoints not UTF16 / java characters. If you need to limit storage you have to filter on the TermToBytesRef attribute (number of bytes). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
