comphead commented on code in PR #4931: URL: https://github.com/apache/datafusion-comet/pull/4931#discussion_r3588429132
########## docs/source/user-guide/latest/tuning.md: ########## @@ -186,6 +186,14 @@ By default, Spark compresses shuffle files using LZ4 compression. Comet override Compression can be disabled by setting `spark.shuffle.compress=false`, which may result in faster shuffle times in certain environments, such as single-node setups with fast NVMe drives, at the expense of increased disk space usage. +### Native scans + +For native scans it is better to check `spark.sql.files.maxPartitionBytes` which is `128M` by default and +set it to `spark.sql.files.maxPartitionBytes: 64M`. The difference comes from how Apache DataFusion and Spark determine the Parquet row group boundaries. Review Comment: Hm, it actually depends on rowgroups, let me try to rephrase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
