nssalian opened a new issue, #16325:
URL: https://github.com/apache/iceberg/issues/16325
### Apache Iceberg version
1.10.1 (latest release)
### Query engine
None
### Please describe the bug 🐞
`ParquetWriter.checkSize()` does not enforce
`write.parquet.row-group-size-bytes` when using compressing codecs. The method
compares `writeStore.getBufferedSize()` against the target, but after internal
page flushes this metric reports compressed bytes. With GZIP or ZSTD on
compressible data, the compressed total stays below the target even as the
actual uncompressed data grows well past it. The result is a single oversized
row group for the entire file.
Readers with page size limits will reject the resulting files. For example,
Trino enforces a 500 MB uncompressed page limit and will fail reads. Iceberg
and Spark readers can read the files given sufficient heap, but the configured
row group target is silently ignored.
### Steps to reproduce
```java
Schema schema = new Schema(
Types.NestedField.required(1, "id", Types.LongType.get()),
Types.NestedField.optional(2, "data", Types.StringType.get()));
DataWriter<Record> writer = Parquet.writeData(file)
.schema(schema)
.createWriterFunc(GenericParquetWriter::create)
.overwrite()
.withSpec(PartitionSpec.unpartitioned())
.set("write.parquet.row-group-size-bytes", "8388608") // 8 MB target
.set("write.parquet.page-size-bytes", "1048576")
.set("write.parquet.compression-codec", "gzip")
.build();
Random rng = new Random(42);
for (int i = 0; i < 100; i++) {
GenericRecord record = GenericRecord.create(schema);
record.setField("id", (long) i);
StringBuilder sb = new StringBuilder(512 * 1024);
sb.append("{\"id\":").append(i).append(",\"values\":[");
while (sb.length() < 512 * 1024) {
sb.append(rng.nextInt(100000)).append(',');
}
sb.setCharAt(sb.length() - 1, ']');
sb.append('}');
record.setField("data", sb.toString());
writer.write(record);
}
writer.close();
DataFile dataFile = writer.toDataFile();
// Expected: splitOffsets.size() >= 4 (50 MB uncompressed / 8 MB target)
// Actual: splitOffsets.size() == 1
```
### Expected behavior
With ~50 MB of uncompressed data and an 8 MB row group target, the writer
should produce multiple row groups.
### Actual behavior
The writer produces a single row group. `checkSize()` never triggers a flush
because `getBufferedSize()` reports compressed bytes for internally-flushed
pages, and the compressed total never exceeds the target.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
