andygrove commented on issue #5304: URL: https://github.com/apache/datafusion-comet/issues/5304#issuecomment-6217866177
Measured while auditing nested types in the native writers: for nested rows, the row-count cap makes the row group size grow with the fan-out. On `main` at c230bf4e98 (Spark 4.1.3), 100,000 rows of `array<struct<s: string, d: double>>` with 100 elements each came out of the native writer as one row group of 239 MB (268 MB uncompressed). Spark wrote the same rows as two row groups, of 127 MB and 111 MB. The writer holds the whole in-progress row group in a reservation it can't spill (`reservation.try_resize(writer.memory_size())`), so a row group the pool won't grant fails the task. At 1,000 elements per row this one would be about 2.4 GB. arrow-rs 59 has `WriterProperties::set_max_row_group_bytes`, which `parquet.block.size` could map to. One more property the list above doesn't name: parquet-rs truncates column-chunk min/max statistics to 64 bytes (`DEFAULT_STATISTICS_TRUNCATE_LENGTH`), where parquet-mr writes them in full by default (`parquet.statistics.truncate.length`). Row-group pruning on long strings is weaker as a result, though still correct. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
