uschindler commented on issue #15054: URL: https://github.com/apache/lucene/issues/15054#issuecomment-3184410243
Hi, I don't exactly know what your remote directory is doing. I assume it is used for some synchronization of segment files over the wire so it uses mainly copyFrom. But nevertheless: When you open files with READONCE, which is correct for synchronizing index segments accross the network, I'd strong request to really use READONCE. If you read/copy the stuff multi-threaded access is totally wrong (unless you have clones of indexinputs that do random access or similar in parallel). I'd strongly recommend to refactor your own code to use READONCE in a way like it is designed for: "like an InputStream and only from one thread". So I'd recommend to have the code possibly multi-threaded but open, read and copy and close the file in one thread. In addition, for just copying files over the wire, I'd recommend to just use NIOFSDirectory for the remote directory. Why do you want to read files one time and add extra pging costs for data that's only copied from somewhere else? Have you though of using Lucene's Replicator module? The problems with DEFAULT vs READONCE with number of mappings come from the fact that you have for sure some leak on managing open files, as @onestardao said. The fact that `-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1` does not help is also a signal for this. It looks like with a confined arena you would also see the same issue, but due to the problem you an't use it (fails with IllegalStateException). To me this really looks like a leak in closing IndexInputs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
