uschindler commented on issue #13194: URL: https://github.com/apache/lucene/issues/13194#issuecomment-2012136360
> > I just noticed, for madvise no file descriptor is needed. We can implement that straight away using the memory segments. > > ++ I think we have all we need enable this for reading - through a "hints" style API in IOContext. > > I can take a look at it, but probably not until next week. I'm sure you already know this, but here's how we've been abstracting FFI in Elasticsearch, https://github.com/elastic/elasticsearch/tree/main/libs/native/src/main/java/org/elasticsearch/nativeaccess . While the use case is not the same here, the necessary abstractions and framework are not unalike. Thanks, this is great. For the use-case here, I think abstractions are not needed, so we can implement it in our MR-JAR: - we only need `defaultLinker()` so no loadLibrary - looking up function addresses will be guarded with tryCatch and when we get a MethodHandle for `madvise()` or `posix_madvise()` we store it in the provider. If we can't find it or native access was denied by JVM (not allowed to module or by command line) we make it a noop MethodHandle or set it null - after when FileChannel returns a memorySegment, we just call madvise depending on IOContext with the MemorySegment as first parameter and the slice (0...length of segment): https://github.com/apache/lucene/blob/d5f8853fda8f488d73aed09d376e1c1c3c9ea85f/lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java#L96 I will hack a draft PR with the basic setup later (only in main branch with Java 21), exposing madvise() and mlock2() ase examples and then we can do tests on Linux (maybe also macos) by passing different options depending on IOContext. P.S.: The `IOContext` is already available to the provider class (I did this in the original API design for exactly that use case; currently the parameter is unused): https://github.com/apache/lucene/blob/d5f8853fda8f488d73aed09d376e1c1c3c9ea85f/lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java#L33 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
