RussellSpitzer commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1171557146
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -212,7 +212,14 @@ private List<FileScanTask> planFiles(StreamingOffset
startOffset, StreamingOffse
currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L,
false);
}
- if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+ Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+ if (snapshot == null) {
+ throw new IllegalStateException(
+ String.format("Failed to find expected snapshot %d",
currentOffset.snapshotId()));
Review Comment:
I think we can probably improve this message. "Cannot load current offset at
snapshot x, the snapshot was expired or removed"?
I'm trying to figure out how we can give the user a bit more information
about what to do in this situation. Our currentOffset should always have a
valid snapshot ID so I think it's probably ok to just say that that the
snapshot was actually expired if we can't find it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
