andygrove opened a new pull request, #6242: URL: https://github.com/apache/datafusion-comet/pull/6242
## Which issue does this PR close? No open issue. This follows up on #6214, where the flake was first diagnosed. ## Rationale for this change The Iceberg 1.9 shard-2 job has failed twice in two days on the same test, `TestRewriteDataFilesAction.testParallelPartialProgressWithMaxFailedCommitsLargerThanTotalFileGroup`: in the 2026-09-25 nightly at `formatVersion = 3` (#6214), and on #6155 at `formatVersion = 2`. Both runs failed with "1 rewrite commits failed. This is more than the maximum allowed failures of 0", and both passed on a re-run. This is a known upstream flake, apache/iceberg#12889. The test has three threads commit ten file groups one at a time against a Hadoop table. A commit that keeps losing the race runs out of Iceberg's default four commit retries, and because the test allows no failed commits, that one starved commit fails it. None of this involves Comet code. Upstream fixed the test in apache/iceberg#13208 and apache/iceberg#13598. Both fixes are in 1.10.0 but not 1.9.1, which is why the 1.10 shard-2 job keeps passing. The Iceberg 1.9 job runs only in the nightly and on pull requests labeled `run-iceberg-tests`, so each failure either turns a nightly red or costs a labeled pull request a re-run. ## What changes are included in this PR? `dev/diffs/iceberg/1.9.1.diff` is regenerated from an `apache-iceberg-1.9.1` checkout with the existing diff and the two upstream commits applied: - apache/iceberg#13208 raises `partial-progress.max-failed-commits` from 0 to 1 and renames the test to `testParallelPartialProgressWithMaxCommitsLargerThanTotalGroupCount`. - apache/iceberg#13598 then expects at least 10 snapshots instead of exactly 11, because a tolerated failure leaves one fewer commit. Upstream changed every Spark version's copy of the test, so this PR changes both the `spark/v3.4` and `spark/v3.5` copies in 1.9.1, although CI runs only `v3.5` for 1.9.1. The resulting method is identical to the one in 1.10.0. The diff gains those two file sections and nothing else. Before making the edit, I checked that regenerating the unmodified diff reproduced the committed file exactly. ## How are these changes tested? The regenerated diff applies cleanly to a fresh `apache-iceberg-1.9.1` checkout, reproduces itself when regenerated from there, and leaves the same tree as the edited clone. I applied the upstream commits as three-way merges rather than as patches, because the line the second commit replaces, `shouldHaveSnapshots(table, 11);`, appears twice in the file and an offset patch could land on the wrong test. Iceberg's test sources can't be built locally here, so this PR carries `run-iceberg-tests`, which compiles the patched Iceberg and runs the test in the 1.9 shard-2 job. One green run can't show that a flake is gone. The case for the fix is upstream's, and the 1.10 shard-2 job, which runs the fixed test, has passed in every nightly since 2026-09-16. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
