[ https://issues.apache.org/jira/browse/LUCENE-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298094#comment-17298094 ]
Adrien Grand commented on LUCENE-9387: -------------------------------------- This is breaking enough that it should probably be done in a major, so I made it a 9.0 blocker to make sure we consider it. I've recently been doing some tests on an index to compare how much memory it actually used (by opening the index as many times as possible until the JVM OOMEs) vs. how much {{Accountable#ramBytesUsed}} reported. {{Accountable#ramBytesUsed}} reported a memory usage of 48kB while the test that tries to measure actual memory usage reported 832kB. I'm not especially surprised, as there are many things that add up and contribute to memory usage like index inputs, various threadlocals, multiplied by the number of fields / number of segments / number of threads. I don't think we can realistically make memory usage accounting much more accurate without entering rabbit holes such as introducing memory accounting on IndexInput (NIOFS uses more heap memory as it needs a buffer) or thread locals (the more threads have access to an index reader, the higher memory usage) and I doubt that returning a number that is wrong by a factor 20 is actually useful, so I'm leaning towards proceeding with no longer implementing Accountable. > Remove RAM accounting from LeafReader > ------------------------------------- > > Key: LUCENE-9387 > URL: https://issues.apache.org/jira/browse/LUCENE-9387 > Project: Lucene - Core > Issue Type: Improvement > Reporter: Adrien Grand > Priority: Blocker > Fix For: master (9.0) > > > Context for this issue can be found at > https://lists.apache.org/thread.html/r06b6a63d8689778bbc2736ec7e4e39bf89ae6973c19f2ec6247690fd%40%3Cdev.lucene.apache.org%3E. > RAM accounting made sense when readers used lots of memory. E.g. when norms > were on heap, we could return memory usage of the norms array and memory > estimates would be very close to actual memory usage. > However nowadays, readers consume very little memory, so RAM accounting has > become less valuable. Furthermore providing good estimates has become > incredibly complex as we can no longer focus on a couple main contributors to > memory usage, but would need to start considering things that we historically > ignored, such as field infos, segment infos, NIOFS buffers, etc. > Let's remove RAM accounting from LeafReader? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org