uschindler opened a new pull request #518: URL: https://github.com/apache/lucene/pull/518
**INFO: This is a followup of #177: It's the same code base, but with API changes from JDK 18 applied** This is just a draft PR for a first insight on memory mapping improvements in JDK 18+. Some background information: Starting with JDK-14, there is a new incubating module "jdk.incubator.foreign" that has a new, not yet stable API for accessing off-heap memory (and later it will also support calling functions using classical MethodHandles that are located in libraries like .so or .dll files). This incubator module has several versions: - first version: https://openjdk.java.net/jeps/370 (slow, very buggy and thread confinement, so making it unuseable with Lucene) - second version: https://openjdk.java.net/jeps/383 (still thread confinement, but now allows transfer of "ownership" to other threads; this is still impossible to use with Lucene. - third version in JDK 16: https://openjdk.java.net/jeps/393 (this version has included "Support for shared segments"). This now allows us to safely use the same external mmaped memory from different threads and also unmap it! This was implemented in the previous pull request #173 - fourth version in JDK 17, included in build 25: https://openjdk.java.net/jeps/412 . This mainly changes the API around the scopes. Instead of having segments explicitely made "shared", we can assign them to some resource scope which control their behaviour. The resourceScope is produced one time for each IndexInput instance (not clones) and owns all segments. When the resourceScope is closed, all segments get invalid - and we throw `AlreadyClosedException`. The big problem is slowness du to heavy use of new instances just to copy memory between segments and java heap. This drives garbage collector crazy. - fifth version in JDK 18, included in build 26: https://openjdk.java.net/jeps/419 (actual version). This mainly cleans up the API. From Lucene's persepctive the `MemorySegment` API now has `System.arraycopy()`-like APIs to copy memory between heap and memory segments. This improves speed. It also handles byte-swapping automatically. This module more or less overcomes several problems: - ByteBuffer API is limited to 32bit (in fact MMapDirectory has to chunk in 1 GiB portions) - There is no official way to unmap ByteBuffers when the file is no longer used. There is a way to use `sun.misc.Unsafe` and forcefully unmap segments, but any IndexInput accessing the file from another thread will crush the JVM with SIGSEGV or SIGBUS. We learned to live with that and we happily apply the unsafe unmapping, but that's the main issue. @uschindler had many discussions with the team at OpenJDK and finally with the third incubator, we have an API that works with Lucene. It was very fruitful discussions (thanks to @mcimadamore !) With the third incubator we are now finally able to do some tests (especially performance). As this is an incubating module, this PR first changes a bit the build system: - disable `-Werror` for `:lucene:core` - add the incubating module to compiler of `:lucene:core` and enable it for all test builds. This is important, as you have to pass `--add-modules jdk.incubator.foreign` also at runtime! The code basically just modifies `MMapDirectory` to use LONG instead of INT for the chunk size parameter. In addition it adds `MemorySegmentIndexInput` that is a copy of our `ByteBufferIndexInput` (still there, but unused), but using MemorySegment instead of ByteBuffer behind the scenes. It works in exactly the same way, just the try/catch blocks for supporting EOFException or moving to another segment were rewritten. It passes all tests and it looks like you can use it to read indexes. The default chunk size is now 16 GiB (but you can raise or lower it as you like; tests are doing this). Of course you can set it to Long.MAX_VALUE, in that case every index file is always mapped to one big memory mapping. My testing with Windows 10 have shown, that this is *not a good idea!!!*. Huge mappings fragment address space over time and as we can only use like 43 or 46 bits (depending on OS), the fragmentation will at some point kill you. So 16 GiB looks like a good compromise: Most files will be smaller than 6 GiB anyways (unless you optimize your index to one huge segment). So for most Lucene installations, the number of segments will equal the number of open files, so Elasticsearch huge user consumers will be very happy. The sysctl max_map_count may not need to be touched anymore. In addition, this implements `readLongs` in a better way than @jpountz did (no caching or arbitrary objects). The new foreign-vector APIs will in future also be written with MemorySegment in its focus. So you can allocate a vector view on a MemorySegment and let the vectorizer fully work outside java heap inside our mmapped files! :-)_ It would be good if you could checkout this branch and try it in production. According to speed tests it should be as fast as MMAPDirectory, partially also faster because less switching between byte-buffers is needed. With recent optimizations also `long`-based absolute access in loops should be faster. But be aware: - You need JDK 11 or JDK 17 to run Gradle (set `JAVA_HOME` to it) - You need JDK 18-ea-b26 (set `RUNTIME_JAVA_HOME` to it) - The lucene-core.jar will be JDK18 class files and requires JDK-18 to execute. - Also you need to add `--add-modules jdk.incubator.foreign` to the command line of your Java program/Solr server/Elasticsearch server It would be good to get some benchmarks, especially by @rmuir or @mikemccand. _Take your time and enjoy the complexity of setting this up!_ ;-) My plan is the following: - report any bugs or slowness, especially with Hotspot optimizations. The last time I talked to Maurizio, he taked about Hotspot not being able to fully optimize for-loops with long instead of int, so it may take some time until the full performance is there. - wait until the final version of project PANAMA-foreign goes into Java's Core Library (`java.base`, no module needed anymore) - add a MR-JAR for lucene-core.jar and compile the MemorySegmentIndexInput and maybe some helper classes with JDK 18/19 (hopefully?). - Add a self-standing JDK-18 compiled module as external JAR. This can be added to classpath or moudle-path and be used by Elasticsearch or Solr. I will work on a Lucene-external project to do this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
