github-actions[bot] commented on code in PR #66913:
URL: https://github.com/apache/doris/pull/66913#discussion_r3853269926
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -552,59 +628,307 @@ private void initPrunedPartitions() throws UserException
{
throw new UserException(ExceptionUtils.getRootCauseMessage(e), e);
}
partitionInit = true;
+ ensureHmsRuntimeGeneration();
}
@Override
public void startSplit(int numBackends) {
+ ensureHmsRuntimeGeneration();
if (prunedPartitions.isEmpty()) {
splitAssignment.finishSchedule();
+ releaseFsViewOnce();
return;
}
- AtomicInteger numFinishedPartitions = new AtomicInteger(0);
+ acquireFsView();
ExecutorService scheduleExecutor =
Env.getCurrentEnv().getExtMetaCacheMgr().getScheduleExecutor();
+ Executor producerExecutor =
Env.getCurrentEnv().getExtMetaCacheMgr().getFileListingExecutor();
long startTime = System.currentTimeMillis();
- CompletableFuture.runAsync(() -> {
- for (HivePartition partition : prunedPartitions) {
- if (batchException.get() != null || splitAssignment.isStop()) {
- break;
- }
- try {
- splittersOnFlight.acquire();
- } catch (InterruptedException e) {
- batchException.set(new UserException(e.getMessage(), e));
- break;
+ BatchFsViewOwner createdOwner = new BatchFsViewOwner(splitAssignment,
fsViewLease);
Review Comment:
[P1] Connect batch cancellation to the split assignment
This owner is registered only with `StatementContext`, but
coordinator/`FileQueryScanNode` cancellation reaches Hudi through
`splitAssignment.stop()`. Because no close handle is installed with
`SplitAssignment.addCloseable()`, that path does not call
`BatchFsViewOwner.close()`: a producer blocked on the semaphore and accepted
filesystem tasks keep running and pinning the exact fs-view until separate
statement teardown happens. Register an idempotent owner handle before producer
submission, and test `assignment.stop()` with an already-started blocked task,
proving interruption is requested immediately while lease release still waits
for actual terminality.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntry.java:
##########
@@ -521,7 +521,7 @@ public void close() {
return;
}
invalidateAll();
- pendingRemovalNotifications.clear();
+ drainRemovalNotificationsOnClose();
Review Comment:
[P1] Linearize removal-token ownership with close
This one-shot drain can miss a concurrent notification in both directions. A
cleanup worker can poll a token, then see `closed` and return without invoking
it; alternatively an `onRemoval` callback that already passed its open check
can enqueue after this drain, while `scheduleRemovalCleanup` refuses closed
entries. Hudi/Iceberg then never run `evict()`/`retire()`, pinning the fs-view
or table/FileIO/catalog generation indefinitely. Make a polled token
non-droppable and make post-close publication invoke inline (or otherwise join
the final drain), with barrier tests for both interleavings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
