liurenjie1024 commented on PR #1547: URL: https://github.com/apache/iceberg-rust/pull/1547#issuecomment-3131470917
> I've manually tested the rolling writer using some small data with the [ParquetWriter::current_written_size fix](https://github.com/apache/iceberg-rust/issues/1554). Now the generated file size is much closer to the configured `target_file_size`. > > The difference between the configured `target_file_size` and the actual written file size can vary depending on the batch size. Generally speaking, rolling file writer rolls more precisely when the target file size is much larger compared to the size of each batch, which is expected. > > target_file_size batch_rows batch_size_in_memory (Bytes) Written File Size (Bytes) > 10 MB 500K 8194584 12968410 > 1 MB 50K 662424 1077332 > 300 KB 10K 145816 312345 > 30 KB 1K 16440 29595 > 30 KB 500 8360 28800 > 30 KB 100 1576 28218 Thanks @CTTY for the tests. The difference are acceptable to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
