viirya commented on code in PR #7094:
URL: https://github.com/apache/datafusion-comet/pull/7094#discussion_r4327853639
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -200,6 +200,18 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(false)
+ val COMET_PARQUET_PREFETCH_NEXT_FILE_METADATA_ENABLED: ConfigEntry[Boolean] =
+ conf("spark.comet.parquet.prefetchNextFileMetadata.enabled")
+ .category(CATEGORY_PARQUET)
+ .doc(
+ "When enabled, a native Parquet scan that reads several files in one
task loads the " +
+ "next file's footer and page index in the background while it reads
the current " +
+ "file, so opening the next file doesn't wait for a storage round
trip. This helps " +
+ "most when tasks read many small files from high-latency storage
such as S3. Scans " +
+ "of encrypted Parquet files never load ahead.")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
Two TPC-DS SF1000 runs per arm, 538 s and 559 s off against 496 s and 474 s
on, is a gap well beyond the run-to-run spread, so turning this on by default
makes sense to me. Since these numbers are now the case for the default, could
you add them to the PR description with the setup (storage, cluster size, and
roughly how many files a scan task reads)? Could you also say whether any
single query got slower beyond noise? A query that stops early, or one whose
files get pruned before they open, pays one extra footer and page index read,
and a total over all queries could hide that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
