gauravsharmadd commented on PR #18008:
URL: https://github.com/apache/iceberg/pull/18008#issuecomment-6133298566
We hit this exact failure mode in production (Spark 4.0 / Iceberg 1.11.0)
and wanted to add corroborating evidence, since this issue and its fix (#18008)
have been open for over a month with no review.
Our setup: a Trigger.AvailableNow streaming job where a fresh driver/JVM is
submitted on a periodic schedule (not a long-lived process), reading from an
Iceberg table with min-snapshots-to-keep tuned fairly low relative to commit
volume. We confirmed via direct inspection of the checkpoint directory that:
- The top-level Spark offsets/<N> log (engine-managed) advanced correctly
across dozens of real batches, with accurate, non-duplicated row counts
throughout.
- The per-source sources/0/offsets/0 file — read once by
InitialOffsetStore.initialOffset() at the very first bootstrap — was never
rewritten across all subsequent successful invocations.
- The query ran successfully for roughly a day (our table's effective
retention window given its commit rate), then failed on the very next
invocation with exactly IllegalStateException: Cannot load current offset at
snapshot <id>, the
snapshot was expired or removed — where <id> was precisely the value
frozen in sources/0/offsets/0 from the original bootstrap, not anything related
to the stream's actual (correctly-advanced) position.
- This recurred independently across multiple instances of the same job
over several weeks, each "fixed" at the time only by a manual checkpoint reset
— which, per this issue, just restarts the same countdown rather than resolving
anything.
We also confirmed by reading the source that
prepareForTriggerAvailableNow()'s call to latestOffset(initialOffset,
ReadLimit.allAvailable()) is what hits validateCurrentSnapshotExists() against
the stale cached value, matching the fix in #18008 exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
