navneet1v commented on issue #15054: URL: https://github.com/apache/lucene/issues/15054#issuecomment-3186300687
> If you read/copy the stuff multi-threaded access is totally wrong (unless you have clones of indexinputs that do random access or similar in parallel). +1 on this. On [JDK docs](https://docs.oracle.com/en/java/javase/22/core/memory-segments-and-arenas.html#GUID-01CE34E8-7BCB-4540-92C4-E127C1F62711:~:text=A%20shared%20arena%2C%20which,of%20a%20shared%20arena. ) it says that shared arena has no owner thread if I am reading it correctly. One possible solution I can see here is: * Lucene [groups the segment files](https://github.com/apache/lucene/blob/83df2ff10e2f72aa58fb9480aae970c622458c93/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L144-L162) in a shared arena. What if for the remote store uploads, @gbbafna you can try using a different grouping criteria(there is no direct api for this, but it shouldn't be hard to build and test). This will ensure that arena created for uploads is separate from the readers. Now once all the IndexInputs are closed for uploads the arena would get free. This should also be easy to test. Reason for suggesting above solution is: As per my reading of code I see that Lucene don't close the arena for a segment till all the [references of IndexInputs are closed](https://github.com/apache/lucene/blob/83df2ff10e2f72aa58fb9480aae970c622458c93/lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java#L101-L119). So what this mean is during the upload you are creating IndexInputs(which I would say is a bad practice, we should always do clone of IndexInput) and then these IndexInputs do get closed, but same segments are getting searched are still open, hence arena will never be closed which means the VM space will still be occupied. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
