CTTY commented on PR #1547: URL: https://github.com/apache/iceberg-rust/pull/1547#issuecomment-3130017652
I've manually tested the rolling writer using some small data with the [ParquetWriter::current_written_size fix](https://github.com/apache/iceberg-rust/issues/1554). Now the generated file size is much closer to the configured `target_file_size`. The difference between the configured `target_file_size` and the actual written file size can vary depending on the batch size. Generally speaking, rolling file writer rolls more precisely when the target file size is much larger compared to the size of each batch, which is expected. | target_file_size | batch_rows |batch_size_in_memory (Bytes) | Written File Size (Bytes) | |---------|--------|-------| --- | | 10 MB | 500K | 8194584 | 12968410 | | 1 MB | 50K | 662424 | 1077332 | | 300 KB | 10K |145816 | 312345 | | 30 KB | 1K | 16440 | 29595 | | 30 KB | 500 | 8360| 28800 | | 30 KB | 100 | 1576| 28218 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
