singhpk234 commented on code in PR #14824:
URL: https://github.com/apache/iceberg/pull/14824#discussion_r2616479358
##########
core/src/main/java/org/apache/iceberg/rest/ScanTaskIterable.java:
##########
@@ -137,16 +131,50 @@ public void run() {
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
+ failure.compareAndSet(null, new RuntimeException("PlanWorker was
interrupted", e));
} catch (Exception e) {
- throw new RuntimeException("Worker failed processing planTask", e);
+ failure.compareAndSet(null, new RuntimeException("Worker failed
processing planTask", e));
} finally {
- int remaining = activeWorkers.decrementAndGet();
+ handleWorkerExit();
+ }
+ }
+
+ private void handleWorkerExit() {
+ int remainingActiveWorkers = activeWorkers.decrementAndGet();
+ boolean noWorkLeft =
+ remainingActiveWorkers == 0 && planTasks.isEmpty() &&
initialFileScanTasks.isEmpty();
+
+ if (noWorkLeft || shutdown.get()) {
+ signalCompletion();
+ } else if (remainingActiveWorkers == 0 && hasRemainingWork()) {
+ // if there is still work to be done but no active workers, it means
all workers have
+ // failed. no need to respawn workers since the iterator will fail on
next()
+ // and throw back the failure.
+ failure.compareAndSet(
+ null,
+ new IllegalStateException("Workers have exited but there is still
work to be done"));
Review Comment:
This should ideally not happen this would mean that all worker got
terminated but there is work left, failure would have been set ideally but just
adding here for sanity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
