rmuir commented on PR #13337: URL: https://github.com/apache/lucene/pull/13337#issuecomment-2099251913
> * I'm contemplating changing the signature from `void prefetch()` to `void prefetch(long offset, long length)`. The benefit is that this would allow reading from multiple places with a single `IndexInput`. E.g. one could parallelize the I/O of reading N stored documents on a single `StoredFieldsReader` instance instead of creating one reader (with its own `IndexInput` clone) per document. I'll do this change if there are no objections. To me it makes sense to be very specific with the region that is needed. otherwise all this madvising doesn't make a lot of sense... turning off the OS default readahead and supplying our own values that might be gigabytes? better to not call madvise at all :) I see this stuff as an operating system hint to get better performance, not actually reading data. So requiring a user to slice()/clone() just to give a hint seems like the wrong way. That has too much overhead for some directories (e.g. buffers). also, i'm a little concerned about low-level parallelization of e.g. individual stored documents. seems like a lot of overhead! if you need 10,000 documents ranges, at least make a single `mincore()` call first before issuing 10,000 madvise() calls? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
