waterWang opened a new pull request, #17832: URL: https://github.com/apache/iceberg/pull/17832
## Problem When a table is loaded with `snapshot-loading-mode=refs`, `RESTSessionCatalog.loadTable()` wraps the returned `TableMetadata` with a lazy `snapshotsSupplier` so that resolving a snapshot outside the retained-refs window transparently re-fetches the full snapshot history (`snapshots=all`). That lazy fallback is **lost on every subsequent `table.refresh()` (and commit)**, because `RESTTableOperations` neither sends the snapshot mode on refresh nor re-installs the supplier. The refreshed metadata is therefore partial with no fallback, and any code path that resolves a snapshot outside the refs window returns `null` and throws. Spark Structured Streaming and Flink streaming are affected, because both call `table.refresh()` every micro-batch/cycle and then walk snapshot ancestry incrementally. ## Fix Thread the `SnapshotMode` into `RESTTableOperations` (constructor param, passed from the `newTableOps(...)` builders in `RESTSessionCatalog`), send it as the `snapshots` query parameter on `refresh()`, and re-install the lazy `snapshotsSupplier` in `REFS` mode whenever the metadata location changes (`updateCurrentMetadata`). - In the default (`ALL`) mode this is a **no-op** — identical to today. - In `refs` mode it is a correctness improvement: a lagging streaming job triggers one full `snapshots=all` fetch only on the refresh cycle that actually needs history. ## Test Added `testTableSnapshotLoadingRefreshKeepsLazySupplier` in `TestRESTCatalog` which loads a table in `refs` mode, advances the table so the metadata location changes, refreshes, and asserts the lazy supplier is re-installed and can still resolve the full snapshot history. Fixes #17830 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
