andygrove opened a new issue, #6774: URL: https://github.com/apache/datafusion-comet/issues/6774
### Describe the bug `ParquetWriter::Remote`, the HDFS path in `native/core/src/execution/operators/parquet_writer.rs`, calls `ArrowWriter::flush()` after every input batch so that it can upload the encoded bytes incrementally. `flush()` closes the in-progress row group, so every batch becomes its own row group. With the default batch size of 8192 rows, a file written natively to HDFS gets a row group every 8192 rows. The local path only calls `write()` and lets `ArrowWriter` cut row groups at its default of 1,048,576 rows. Spark's own writer cuts them at `parquet.block.size` (128 MB). Small row groups mean small column chunks, worse compression and dictionary encoding, a bigger footer, and more work for everything that reads the file. ### Steps to reproduce Write the same five 1,000-row batches through `ParquetWriter::Remote` with an in-memory opendal `Operator`, and through `ParquetWriter::LocalFile`. On main at f80042f787 the remote file has 5 row groups and the local file has 1. ### Expected behavior The remote path cuts row groups where the local path does. It could drop the per-batch `flush()`, call `sync()` so that completed row groups reach the cursor, and upload whatever the cursor holds. Since #6247, `memory_size()` charges the in-progress row group to the task's memory pool, so holding it until the row group is full is accounted for. ### Additional context Found during the #5143 review on 2026-08-04. It predates that PR. Related to #5304, which is about honoring `parquet.block.size`. Fixing #5304 alone would not help here, because the remote path closes a row group on every batch whatever the configured size is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
