andygrove opened a new pull request, #4908: URL: https://github.com/apache/datafusion-comet/pull/4908
## Summary Audited all `CometConf` entries in preparation for the 1.0 release and expanded the [Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html) with the user-facing performance and memory knobs that were previously undocumented. New sections in `tuning.md`: - **Batch Size** — `spark.comet.batchSize` (memory vs. throughput tradeoff) and its shuffle-writer cousin `spark.comet.columnar.shuffle.batch.size` - **Limiting Spill Disk Usage** — `spark.comet.maxTempDirectorySize` (100 GB default) - **Parquet Reader Tuning** — parallel I/O thread pool, range coalescing, filter pushdown / late materialization - **Iceberg Scan Tuning** — `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` - **Reducing Row/Columnar Conversion Overhead** — `spark.comet.exec.transitionRevert.{enabled,maxTransitions}` - **Metrics Overhead** — why `spark.comet.metrics.enabled` is off by default - Expanded **Shuffle Compression** with codec / zstd level knobs Also: - Fixed an incorrect claim that Comet overrides Spark LZ4 shuffle compression with ZSTD (the default codec is `lz4`). - Moved `spark.comet.maxTempDirectorySize` from the `exec` category to `tuning` so it renders in the Memory & Tuning table in `configs.md`, and expanded its doc string. ## Test plan - [ ] Docs build renders new sections correctly - [ ] `configs.md` still generates cleanly (moved config appears in Memory & Tuning table) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
