j1wonpark opened a new issue, #4375: URL: https://github.com/apache/amoro/issues/4375
## What happened? Amoro’s Iceberg 1.8.1 writer can commit changes to native V3 tables created by Iceberg 1.9+, but drops the required `next-row-id` field. Subsequent loads with Iceberg 1.9+ fail: ```text IllegalArgumentException: Cannot parse missing long: next-row-id ``` Expected: reject automatic changes to unsupported format versions and report the reason without modifying metadata. ## Affects Versions Confirmed on `master` at `bdd5f5816`, using Iceberg 1.8.1. ## What table formats are you seeing the problem on? Iceberg ## What engines are you seeing the problem on? AMS, Optimizer, Spark ## How to reproduce 1. Create a `format-version=3` table using Spark with Iceberg 1.11.0 in a catalog managed by Amoro. 2. Insert multiple batches and let Amoro commit a self-optimizing compaction. 3. The new metadata loses `next-row-id`, and Spark can no longer load the table. Observed after one full compaction, reducing 12 data files to one. Also reproduced locally with an Iceberg 1.9.2 append followed by a 1.8.1 tag commit. ## Relevant log output ```text IllegalArgumentException: Cannot parse missing long: next-row-id ``` ## Anything else Iceberg 1.8.1 interprets the absent `row-lineage` flag in newer V3 metadata as disabled and omits `next-row-id` when writing. The existing V3 check only disables Parquet row-group merging; row-based rewriting still commits. Other metadata-changing maintenance operations are also affected. Protection should cover execution and commit boundaries, including retries and recovered jobs. This fix would prevent unsafe changes, not add V3 support or repair affected tables. ## Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
