[
https://issues.apache.org/jira/browse/LUCENE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384068#comment-17384068
]
Adrien Grand commented on LUCENE-9959:
--------------------------------------
I actually like the changes made in #180. It does add a new level of
indirection through a new public class, but this is already what we're doing
with e.g. doc values where you need to pull a reader first, then advance the
iterator, and only then you can retrieve a value. It's slightly more annoying
to use with this new indirection, but maybe the benefits are worth the hassle?
When I'm thinking of a machine like Mike's beast with its 128 cores, it doesn't
sound right to keep 128 states *per segment* for stored fields and term vectors
at all times when:
- these states use tens of kB,
- while not entirely free to create, these states aren't super expensive to
create either,
- it's very unlikely that many of these states get used at the same time,
because hopefully more time is spent running queries than fetching stored
fields or term vectors for top hits.
bq. I noticed your original PR https://github.com/apache/lucene/pull/137 that
led to this Jira and also touched on stored fields has not been merged yet, do
you plan to merge it any time soon, or will you have more changes for it?
I had put it on hold to see whether we should explore changing the API like you
did rather than still caching stored fields readers per thread but removing as
much state as possible like my PR does.
If the new API proves controversial, I'd be open to an alternative that would
consist of keeping the previous API and pulling a new TermVectorsReader (resp.
StoredFieldsReader) internally every time that term vectors (resp. stored
fields) are requested instead of the previous approach that consisted of
caching instances in a threadlocal.
> Can we remove threadlocals of stored fields and term vectors
> ------------------------------------------------------------
>
> Key: LUCENE-9959
> URL: https://issues.apache.org/jira/browse/LUCENE-9959
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Adrien Grand
> Priority: Minor
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> [~rmuir] suggested removing these threadlocals at
> https://github.com/apache/lucene/pull/137#issuecomment-840111367.
> These threadlocals are trappy if you manage many segments and threads within
> the same JVM, or worse: non-fixed threadpools. The challenge is to keep the
> API easy to use.
> We could take advantage of 9.0 to change the stored fields API?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
