wForget commented on code in PR #4931: URL: https://github.com/apache/datafusion-comet/pull/4931#discussion_r3583902439
########## docs/source/user-guide/latest/tuning.md: ########## @@ -186,6 +186,14 @@ By default, Spark compresses shuffle files using LZ4 compression. Comet override Compression can be disabled by setting `spark.shuffle.compress=false`, which may result in faster shuffle times in certain environments, such as single-node setups with fast NVMe drives, at the expense of increased disk space usage. +### Native scans + +For native scans it is better to check `spark.sql.files.maxPartitionBytes` which is `128M` by default and +set it to `spark.sql.files.maxPartitionBytes: 64M`. The difference comes from how Apache DataFusion and Spark determine the Parquet row group boundaries. Review Comment: Does this mean we should recommend setting `spark.sql.files.maxPartitionBytes` to half of `parquet.block.size`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
