andygrove commented on PR #7094: URL: https://github.com/apache/datafusion-comet/pull/7094#issuecomment-6255254797
Thanks for checking the DataFusion 55.2 opener and `FileStream` assumptions. I pushed 6be7a15a49 with a change for each inline comment, and the tuning guide now has a section for the config, covering when it helps, what it costs, and that encrypted scans never prefetch. On the evidence you asked for: I'll run TPC-DS at 1TB next week and share the numbers here. I'll also add to #7093 what it would take to default this to on. I couldn't find an upstream DataFusion issue about `ScanState` planning the next file while the reader runs. The module docs link apache/datafusion#5209 (buffer more than one file open) and #18470 (prefetch the next row group) as the nearest open ones. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
