NoahKusaba commented on PR #2579: URL: https://github.com/apache/datafusion-ballista/pull/2579#issuecomment-6138296412
Peak RSS growth during the write, zstd, one 8192-row batch per partition: | partitions | `main` | before the fix | after the fix | |---|---|---|---| | 200 | 36 MB | 139 MB | 36 MB | | 1000 | 140 MB | 655 MB | 140 MB | I also wrote the same 37-partition shuffle (40 batches with a string column) on `main` and on this branch, with lz4 and zstd, with and without spilling. The data files, index files and reported stats are byte-identical, and the spill counts match. `sort_shuffle` bench against `main` (laptop, on AC, criterion's 10 samples): | benchmark | `main` | this branch | change | |---|---|---|---| | no spill, 10 batches, 200 partitions | 71.0 ms | 65.1 ms | −8.8% | | no spill, 50 batches, 200 partitions | 159.4 ms | 140.4 ms | −10.6% | | spill, 8 MB limit, 200 partitions | 1.23 s | 1.00 s | −18.6% | | spill, 2 MB limit, 200 partitions | 2.27 s | 1.77 s | −22.3% | No regression. I wouldn't read these as a speedup though: the change does the same work as `main`, and the `main` baseline ran first, so I expect most of the gap is machine noise. I skipped the TPC-H run since the plan and the output bytes don't change, but I'm happy to run it if you'd like. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
