github-actions[bot] commented on code in PR #68432:
URL: https://github.com/apache/doris/pull/68432#discussion_r4080722924
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/SourceOffsetProvider.java:
##########
@@ -206,6 +206,11 @@ default int pendingSplitCount() {
return 0;
}
+ /** Snapshot phase lasts until splitting and consumption of produced
splits both finish. */
+ default boolean isSnapshotPhase() {
+ return !noMoreSplits() || pendingSplitCount() > 0;
Review Comment:
[P2] Keep the per-task snapshot phase check O(1)
`StreamingMultiTblTask.before()` now calls this for every task. For
`JdbcSourceOffsetProvider`, `noMoreSplits()` can call
`computeCdcRemainingTables()` under `splitsLock`, scanning all finished and
pending splits whenever `currentSplittingTable` is null (table boundaries and
final backlog). A many-table snapshot, especially one with one split per table,
therefore re-walks a growing split history once per task, adding O(N^2 / P) FE
work while holding the same lock needed by split publication and offset
commits. The previous backend decision inspected the selected `JdbcOffset` and
did not add this scan. Please maintain or cache an O(1) JDBC phase/completion
state before querying it on every task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
