[I] Absolutely horrible Lucene performance with JDK 23 (Lucene 9.11.1 and 10.0.0) [lucene]

[via GitHub]

derreisende77 opened a new issue, #13959:
URL: https://github.com/apache/lucene/issues/13959

   ### Description
   
   I am using Lucene in my app for several years happily with JDKs up to 22.
   My use case searches through film data and Lucene can return fairly huge 
result sets to my app - which as of now never was a problem. 
   I upgraded my app to JDK 23.0.1 on my MacBook Air macOS 15.01 16GB RAM.
   
   ```
   openjdk version "23.0.1" 2024-10-15
   OpenJDK Runtime Environment (build 23.0.1+13)
   OpenJDK 64-Bit Server VM (build 23.0.1+13, mixed mode, sharing)
   ```
   
   and started to notice **horrible** Lucene performance.
   With the following code snippet I do query my results:
   
   ```java
   var reader = DirectoryReader.open(list.getLuceneDirectory());
   final var searcher = new IndexSearcher(reader);
   final var docs = searcher.search(finalQuery, list.size());
   final var hit_length = docs.scoreDocs.length;
   
   var storedFields = searcher.storedFields();
   // the for loop takes ages with JDK23...
   for (final var hit : docs.scoreDocs) {
        var docId = hit.doc;
        //storedFields.prefetch(docId); //<-- doesn't change anything
        var d = storedFields.document(docId, INTEREST_SET); //<-- this takes 
ages
        //filmNrSet.add(Integer.parseInt(d.get(LuceneIndexKeys.ID)));
   }
   ```
   
   In the explored use case Lucene always returned the expected 558333 hits out 
of 802k documents.
   99% of the app runs take **5.7 seconds** to get the result. However when I 
am lucky *1% of the app runs* do get the same result back in **756 
milliseconds**.
   If Lucene is delivering fast, it will stay fast, if it is slow it will 
remain slow. I have no idea how this is triggered.
   
   I moved from `NRTCachingDirectory` to `MMapDirectory` but the performance 
remained bad. Tried some other stuff from internet - same result.
   
   I switched back to JDK 22.
   
   ```
   openjdk version "22.0.2" 2024-07-16
   OpenJDK Runtime Environment (build 22.0.2+11)
   OpenJDK 64-Bit Server VM (build 22.0.2+11, mixed mode, sharing)
   ```
   
   The same source which performed just horrible with JDK23 was **consistently 
fast** with JDK 22:
   
   ```
   Search took: 763.8 ms
   ```
   
   I am using the following flags for the JVM:
   ```
   -ea
   -XX:+UseShenandoahGC
   -XX:ShenandoahGCHeuristics=compact
   -XX:+UseStringDeduplication
   -XX:MaxRAMPercentage=50.0
   --enable-native-access=ALL-UNNAMED
   --add-modules jdk.incubator.vector
   ```
   
   I removed `--enable-native-access=ALL-UNNAMED` and `--add-modules 
jdk.incubator.vector` for testing purposes but the performance remained bad 
with JDK23.
   
   I made the same tests with JDK23 on my Windows 11 AMD Ryzen 4900H 16GB RAM 
laptop. There I get the results back in **13.63 seconds** with JDK 23. JDK 22 
does the same consistently in **2.1 seconds**.
   
   Switching between Lucene `9.11.1` and `10.0.0` made no difference, always 
shitty performance with JDK 23 both on macOS and windows. Consistent 
performance with JDK 22.
   
   ### Version and environment details
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org