jackMintao opened a new pull request, #18204:
URL: https://github.com/apache/iceberg/pull/18204
## Why
`compute_partition_stats` writes a partition statistics file and then
registers
it in table metadata via
`table.updatePartitionStatistics().setPartitionStatistics(file).commit()`.
`SetPartitionStatistics.commit()` commits metadata exactly once:
TableMetadata base = ops.current();
TableMetadata newMetadata = internalApply(base);
ops.commit(base, newMetadata);
If another writer commits between the compute step and the commit step, the
commit fails with `CommitFailedException`. The Spark action fails even though
the partition stats file has already been written, leaving an orphaned file
on
disk and no registered statistics. Other metadata updates already retry in
this
situation — `SetStatistics` (see #15148), `SnapshotProducer`,
`SetLocation`-style updates, and `BaseTransaction` — but
`SetPartitionStatistics`
does not.
## What
Retry `SetPartitionStatistics` commits using the standard commit retry
configuration, mirroring the `SetStatistics` implementation added in #15148:
- `commit.retry.num-retries` (default 4)
- `commit.retry.min-wait-ms` / `commit.retry.max-wait-ms` (default 100 ms /
60 s)
- `commit.retry.total-timeout-ms` (default 30 min), exponential backoff
- retry only on `CommitFailedException`; `CommitStateUnknownException` is
propagated unchanged
Each attempt refreshes the base metadata and re-applies the pending updates:
taskOps -> {
TableMetadata base = taskOps.refresh();
TableMetadata updated = internalApply(base);
taskOps.commit(base, updated);
}
Refreshing is required for the retry to make progress: a failed commit leaves
the cached metadata unchanged, so re-using the old base would fail forever.
`internalApply` is side-effect free and safe to apply to the refreshed base,
so
no state needs resetting between attempts. Only the metadata commit is
retried —
the stats file itself is written once, before the commit loop.
## Tests
Two new cases in `TestSetPartitionStatistics`:
- `setPartitionStatisticsRetrySuccess`: injects two `CommitFailedException`s
and
asserts the commit eventually succeeds with the stats file registered.
- `setPartitionStatisticsRetryWithConcurrentModification`: triggers a real
concurrent append during the first commit attempt and asserts the retry
rebases on the refreshed metadata and registers the stats file.
Both tests fail without the production change.
Verified locally with:
./gradlew :iceberg-core:test --tests "*PartitionStat*" \
:iceberg-core:spotlessCheck :iceberg-core:checkstyleMain
:iceberg-core:checkstyleTest
## Notes
- REST catalogs already retry commits server-side (`CatalogHandlers#commit`);
the client-side retry adds a bounded second layer, the same as existing
snapshot-producing operations.
- Behavior for callers that already retried the whole operation is unchanged:
the retry only covers transient commit conflicts, not validation failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
