singhpk234 commented on code in PR #14824:
URL: https://github.com/apache/iceberg/pull/14824#discussion_r2616480828
##########
core/src/main/java/org/apache/iceberg/rest/ScanTaskIterable.java:
##########
@@ -201,21 +224,36 @@ public boolean hasNext() {
return true;
}
- while (true) {
- if (isDone()) {
- return false;
+ while (!shutdown.get()) {
+ // Check for worker failure
+ RuntimeException workerFailure = failure.get();
+ if (workerFailure != null) {
+ throw workerFailure;
}
try {
nextTask = taskQueue.poll(QUEUE_POLL_TIMEOUT_MS,
TimeUnit.MILLISECONDS);
- if (nextTask != null) {
+
+ // Re-check for worker failure after poll - failure could have been
set while polling
+ workerFailure = failure.get();
+ if (workerFailure != null) {
+ throw workerFailure;
+ }
+
+ if (nextTask == DUMMY_TASK) {
+ nextTask = null;
+ shutdown.set(true); // Mark as done so while loop exits on
subsequent calls
Review Comment:
we need this for multiple `hasNext()` call as post the poison pill is
consumed next hasNext will be stuck .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
