RussellSpitzer commented on code in PR #11258:
URL: https://github.com/apache/iceberg/pull/11258#discussion_r1792476033
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java:
##########
@@ -185,9 +190,21 @@ public List<Long> splitOffsets() {
return null;
}
+ /*
+ * Data size could reduce after written out due to encoding/compression.
+ * Use the ratio totalRowGroupSize / totalBufferSize to estimate the size
after write out.
+ */
+ private long estimateBufferedSize() {
+ if (totalRowGroupSize == 0 || totalRawBufferedSize == 0 ||
currentRawBufferedSize == 0) {
+ return writeStore.getBufferedSize();
+ }
+
+ return currentRawBufferedSize * totalRowGroupSize / totalRawBufferedSize;
+ }
+
private void checkSize() {
if (recordCount >= nextCheckRecordCount) {
- long bufferedSize = writeStore.getBufferedSize();
+ long bufferedSize = estimateBufferedSize();
double avgRecordSize = ((double) bufferedSize) / recordCount;
if (bufferedSize > (targetRowGroupSize - 2 * avgRecordSize)) {
Review Comment:
Do we still need the -2 * avg records size ? Seems like that shouldn't
really matter now that we are more accurately estimating our buffered size?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
