Adrien Grand created LUCENE-10203:
-------------------------------------
Summary: Improve reuse of StringTokenStream
Key: LUCENE-10203
URL: https://issues.apache.org/jira/browse/LUCENE-10203
Project: Lucene - Core
Issue Type: Improvement
Reporter: Adrien Grand
This issue is a follow-up to
https://lists.apache.org/thread.html/rdcc6bd085a0e8ac6db22a1ef7dd3228197481b62bec3c6fe4972e50a%40%3Cdev.lucene.apache.org%3E.
StringField has a different mechanism for reusing token streams compared to
TextField: while TextField relies on {{Analyzer#reuseStrategy}} to reuse token
streams across inputs, StringField relies on IndexingChain passing the
previously produced token stream as the `reuse` parameter of
{{IndexableField#tokenStream}}. However one downside of this approach is that
it can only reuse token streams within a single segment. And some nightly
profiles suggest that not reusing across segments still gives room for
attribute initialization to be a hotspot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]