thecoop opened a new issue, #14422: URL: https://github.com/apache/lucene/issues/14422
### Description Currently Lucene tends to use `ReadAdvice.RANDOM` quite liberally, in various hard-coded locations. We have seen that this causes some problems with recent kernel versions, such as #14408. There are also several other aspects about opening files - such as which file access implementation to use, how to map memory, etc, that are dependent on the context of opening such files, and cannot be determined solely by the codec that is used. For example, you may want to access a flat vector file differently if it’s being used for an exhaustive scan vs being used to access specific vectors. To solve both these issues, Directory implementations could have information on the context a particular file is being opened with, to help it decide how and with what options it should open the file. However, there is no such information provided to Directory classes in the IOContext, and there is no space to provide such information in an extensible way. So there needs to be a way to pass additional context-specific information in an IOContext, to help Directory implementations decide on the ReadAdvice to use, and how and with what options it should open a particular file. There are two main ways I can see of doing this: 1. Make IOContext an interface. This allows custom implementations to be created, which can then be checked for specific types inside the custom Directory implementation to then cast and pull out the required information (prototype can be found [here](https://github.com/apache/lucene/commit/739f816b5492a505b1bc964214a6d2de60558d47)) 2. Add a `Map<String, Object>` or `Map<String, String>` payload to IOContext allowing for arbitrary information to be included with an IOContext, that can then be checked by the Directory implementation As part of this, we could introduce some standard patterns, or some standard map keys for existing Lucene directories to replace the existing ReadAdvice logic, to provide more information on how files should be opened. A later piece of work could be to remove ReadAdvice from IOContext, and provide some other way Directory implementations could determine the access pattern that is likely to be used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
