[
https://issues.apache.org/jira/browse/LUCENE-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298094#comment-17298094
]
Adrien Grand commented on LUCENE-9387:
--------------------------------------
This is breaking enough that it should probably be done in a major, so I made
it a 9.0 blocker to make sure we consider it.
I've recently been doing some tests on an index to compare how much memory it
actually used (by opening the index as many times as possible until the JVM
OOMEs) vs. how much {{Accountable#ramBytesUsed}} reported.
{{Accountable#ramBytesUsed}} reported a memory usage of 48kB while the test
that tries to measure actual memory usage reported 832kB. I'm not especially
surprised, as there are many things that add up and contribute to memory usage
like index inputs, various threadlocals, multiplied by the number of fields /
number of segments / number of threads. I don't think we can realistically make
memory usage accounting much more accurate without entering rabbit holes such
as introducing memory accounting on IndexInput (NIOFS uses more heap memory as
it needs a buffer) or thread locals (the more threads have access to an index
reader, the higher memory usage) and I doubt that returning a number that is
wrong by a factor 20 is actually useful, so I'm leaning towards proceeding with
no longer implementing Accountable.
> Remove RAM accounting from LeafReader
> -------------------------------------
>
> Key: LUCENE-9387
> URL: https://issues.apache.org/jira/browse/LUCENE-9387
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Adrien Grand
> Priority: Blocker
> Fix For: master (9.0)
>
>
> Context for this issue can be found at
> https://lists.apache.org/thread.html/r06b6a63d8689778bbc2736ec7e4e39bf89ae6973c19f2ec6247690fd%40%3Cdev.lucene.apache.org%3E.
> RAM accounting made sense when readers used lots of memory. E.g. when norms
> were on heap, we could return memory usage of the norms array and memory
> estimates would be very close to actual memory usage.
> However nowadays, readers consume very little memory, so RAM accounting has
> become less valuable. Furthermore providing good estimates has become
> incredibly complex as we can no longer focus on a couple main contributors to
> memory usage, but would need to start considering things that we historically
> ignored, such as field infos, segment infos, NIOFS buffers, etc.
> Let's remove RAM accounting from LeafReader?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
