OIiveirra commented on code in PR #68791:
URL: https://github.com/apache/doris/pull/68791#discussion_r4255191372
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java:
##########
@@ -280,12 +310,23 @@ public Multimap<Backend, Split>
computeScanRangeAssignment(List<Split> splits) t
}
case RANDOM: {
randomCandidates.reset();
- candidateNodes =
selectNodes(Config.split_assigner_min_random_candidate_num, randomCandidates);
+ candidateNodes = consistentHashSpreadNum == 0 ?
backends
+ :
selectNodes(Config.split_assigner_min_random_candidate_num, randomCandidates);
break;
}
case CONSISTENT_HASHING: {
- candidateNodes = consistentHash.getNode(split,
-
Config.split_assigner_min_consistent_hash_candidate_num);
+ if (consistentHashSpreadNum == 0) {
+ candidateNodes = backends;
+ } else {
+ if (isSpreadEnabled()) {
+ int candidateCount =
Math.min(consistentHashSpreadNum, backends.size());
+ candidateNodes = candidateCount >=
backends.size() ? backends
+ : consistentHash.getNode(split,
candidateCount);
Review Comment:
Fixed in 267ca885472. MaxComputeScanRange now includes the read-session ID
together with the synthetic split-type path in getSplitIdentity(); offsets and
lengths remain unchanged. Added coverage for equal coordinates from different
sessions in both row-offset and byte-size modes, plus a ten-BE/spread=2 policy
test verifying that distinct identities escape a single candidate pair while
every assignment stays within its own hash candidates. This corrects hash
placement for synthetic work, including spread=1; it does not promise stable
placement across newly created ODPS sessions. All 70 targeted FE/connector
tests and the standard FE build with Checkstyle passed locally.
##########
fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRange.java:
##########
@@ -46,6 +46,14 @@ default Optional<String> getPath() {
return Optional.empty();
}
+ /**
+ * Returns a stable identity used to assign connector ranges to
external-scan backends. Connectors whose JNI
+ * ranges do not have a path should override this with their serialized
split identity.
+ */
+ default String getSplitIdentity() {
+ return getPath().orElse("");
Review Comment:
Fixed in 267ca885472. IcebergScanRange uses its immutable serializedSplit as
the identity for JNI metadata tasks and retains the existing path identity for
native data and native position-delete ranges. No SPI signature, planner, wire
parameters or BE reader changes were needed. Added
same-dummy-path/different-task and repeated-task identity tests, native-path
preservation checks, and a ten-BE/spread=2 policy test. All 70 targeted
FE/connector tests and the standard FE build with Checkstyle passed locally.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java:
##########
@@ -239,6 +261,7 @@ public Multimap<Backend, Split>
computeScanRangeAssignment(List<Split> splits) t
for (List<Backend> backendList : backendMap.values()) {
backends.addAll(backendList);
}
+ rebuildAssignedWeightCounts();
Review Comment:
Fixed in 267ca885472. Removed the unconditional per-batch histogram rebuild
and its helper. Production updates continue to use the existing
setAssignedWeight/addAssignedWeight methods, so incremental counts are retained
across streamed batches without changing the assignment lock or scheduling
rules. The test that injected weights now invokes the existing
weight-maintenance method; new singleton-batch coverage checks histogram
consistency for spread=0/1/2. All 70 targeted FE/connector tests and the
standard FE build with Checkstyle passed locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
