andygrove opened a new pull request, #7132: URL: https://github.com/apache/datafusion-comet/pull/7132
## Which issue does this PR close? No issue. Follow-up to a discussion of #7094, which prefetches the next Parquet file's metadata but does not reach the native Iceberg scan. ## Rationale for this change `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` defaulted to 1, so the native Iceberg scan opened and read its data files one after another and paid each file's open latency in turn. The docs say the default was chosen so that Iceberg's Java tests, which expect file order from queries without an `ORDER BY`, keep passing. That is a test concern, and every user got the slower default for it. Two files in flight hide much of the per-file latency on tables with many small files or on S3, at a modest memory cost. ## What changes are included in this PR? - The default is now 2. It is read through `createWithEnvVarOrDefault`, so the environment variable `COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT` overrides it. - The Iceberg Spark test jobs (core, extensions and runtime) and `dev/local-ci.sh` set that variable to 1, so Iceberg's tests keep reading files in order. Comet's own Maven suites are not pinned, so they run at 2. - The `CometConf` description now states the benefit, the memory cost, and that batches from different files come back in no particular order above 1. - `iceberg.md` (Tuning), `iceberg-spark-tests.md` and the 1.2.0 section of `migration-guide.md` document the new default, the row-order caveat, and how to restore the old behavior with `1`. ## How are these changes tested? `CometIcebergNativeSuite` passes at the new default (132 succeeded, 0 failed, 2 canceled for Spark 4.1 reasons). I did not run Iceberg's own Spark tests locally. The `run-iceberg-tests` label does, and also checks that the variable reaches the test JVMs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
