uschindler commented on issue #13325: URL: https://github.com/apache/lucene/issues/13325#issuecomment-2158241513
Hi, I reopened this to make a further investigation. @dsmiley talked to me at Berlinbuzzwords and he also commented on #13146. He has seen a major slowdown on Apache Solr with Java 21 and using Lucene 9.8 (or like that). The issue there was partly 3x slower queries (as far as I understood) in an environment where many many Solr Cores with man many softcommits (IndexReader repoens for NRT) are happening at the same time. So think of about 1000 indexes on the same JVM and all of them are repopen indexes every second. There is seems to be the problem that closing one IndexInput seems to deoptimize every other MemorySegment although not related / belonging to the same shared Arena. I try to write a JMH benchmark without Lucene code to reproduce the issue and measure how large the issue is (depending on how often you close an IndexInput and how many parallel threads are using other MemorySegments, also those not tied to same Indexinput). My idea is: - Have many threads working on "hot" MemorySegments allocated with different shared Arenas. - Have a few threads closing those Arenas. Theoretically, the benchmark should not slow if the few threads close Arenas which are not bound to and hot workers. What we have seen in dacapo bench is that it seems to affect all MemorySegments negatively, not only those which are affected by the close. of Arena. Once I have the benchmark I will open issue at OpenJDK. In the meantime, we could "preserve" the old Lucene 9.1 (the version without MemorySegments) in the "misc" module of Lucene as LegacyMMapDirectory. This would allow people to use it, but it is strinly discouraged. It won't work anymore with Java 24 (or around that time). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
