YuangGao opened a new issue, #4126: URL: https://github.com/apache/iceberg-python/issues/4126
### Apache Iceberg version main (development) ### Please describe the bug 🐞 `write.parquet.page-row-limit` is mapped to pyarrow's `write_batch_size` ([pyarrow.py#L3002-L3006](https://github.com/apache/iceberg-python/blob/7d51e52bfd42dba2806979ee2554507e95fcfa0a/pyiceberg/io/pyarrow.py#L3002-L3006)), but `write_batch_size` is how many values pyarrow writes before checking the page size, not a per-page row limit. So the property has no effect. Two side effects: - The default of 20000 is always passed as `write_batch_size` (pyarrow's own default is 1024), so pages are size-checked far less often and can overshoot `write.parquet.page-size-bytes`. - Java maps this property to parquet-java's `withPageRowCountLimit`, so the same table writes differently from Java and PyIceberg. pyarrow 23.0.0 added `max_rows_per_page` (apache/arrow#48101), which is the actual per-page row limit. To reproduce, 50,000 int64 rows (column chunk size in bytes): ```python import pyarrow as pa import pyarrow.parquet as pq from pyiceberg.catalog.sql import SqlCatalog data = pa.table({"id": pa.array(range(50_000), pa.int64())}) catalog = SqlCatalog("t", uri="sqlite:////tmp/cat.db", warehouse="file:///tmp/wh") catalog.create_namespace("ns") for name, props in [("a", {}), ("b", {"write.parquet.page-row-limit": "100"})]: table = catalog.create_table(f"ns.{name}", schema=data.schema, properties=props) table.append(data) path = next(iter(table.scan().plan_files())).file.file_path.removeprefix("file://") print(name, pq.ParquetFile(path).metadata.row_group(0).column(0).total_compressed_size) ``` | | size | |---|---| | no properties | 145139 | | `write.parquet.page-row-limit=100` | 145139 (unchanged) | | pyarrow `max_rows_per_page=100`, for comparison | 181473 | Proposed fix: stop passing `write_batch_size`, and pass `max_rows_per_page` when pyarrow >= 23.0.0. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
