ForestYang20 opened a new issue, #11014: URL: https://github.com/apache/iceberg/issues/11014
### Apache Iceberg version 1.0.0 ### Query engine Spark ### Please describe the bug 🐞 While compacting data files for a partitioned iceberg table in our Glue catalog using an [AWS Glue v4.0 PySpark script](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html), we've encountered unknown behaviour which has resulted in missing data. ```py sql = f""" CALL spark_catalog.system.rewrite_data_files( table => '{database_name}.{table}', options => map( 'max-concurrent-file-group-rewrites', '50', 'partial-progress.enabled', 'true', 'partial-progress.max-commits', '20', 'target-file-size-bytes', '268435456', 'min-input-files', '10', 'max-file-group-size-bytes', '34359738368', 'rewrite-job-order', 'bytes-desc' ) ) """ ``` The result is having a series of 20 commits, some of which are shown below, where within each, the number of `added-records` is not equal to the number of `deleted-records`. It's not the first time we have run the procedure on the table. | snapshot_id | parent_id | Operation | Summary | |---------------------|---------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 4660292584715833520 | 1491128304405983876 | replace | {added-data-files=6, total-equality-deletes=0, a**dded-records=561738**, deleted-data-files=688, **deleted-records=561742**, total-records=934942885, removed-files-size=42109583, changed-partition-count=6, total-position-deletes=20000, added-files-size=40233613, total-delete-files=32, total-files-size=398800921100, total-data-files=17739} | | 1491128304405983876 | 2725485409088750181 | replace | {added-data-files=6, total-equality-deletes=0, **added-records=424937**, deleted-data-files=685, **deleted-records=424945**, total-records=934942889, removed-files-size=78756903, changed-partition-count=6, total-position-deletes=20000, added-files-size=76231054, total-delete-files=32, total-files-size=398802797070, total-data-files=18421} | | 2725485409088750181 | 3178524085360111885 | replace | {added-data-files=8, total-equality-deletes=0, **added-records=3139079**, deleted-data-files=163, **deleted-records=3139079**, total-records=934942897, removed-files-size=390896283, changed-partition-count=6, total-position-deletes=20000, added-files-size=389858318, total-delete-files=32, total-files-size=398805322919, total-data-files=19100} | Our table has the definition: ``` CREATE TABLE database.table ( position bigint, type string, recordedat timestamp, aggregateid string, aggregateversion bigint, payload string, month bigint, year bigint, day bigint, hour bigint) PARTITIONED BY (month(`recordedat`), `type`) LOCATION 's3://<bucket>/<path>' TBLPROPERTIES ( 'table_type'='iceberg', 'vacuum_max_snapshot_age_seconds'='604800', 'format'='parquet', 'vacuum_max_metadata_files_to_keep'='1000', 'optimize_rewrite_delete_file_threshold'='2', 'optimize_rewrite_data_file_threshold'='5', 'vacuum_min_snapshots_to_keep'='1000' ); ``` ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
