This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c83e45dd222 [fix](split)Fixed the bug that batch mode split could not
query data in multiple be scenarios. (#46218)
c83e45dd222 is described below
commit c83e45dd2222ad89d315d339388c57ce4587554e
Author: daidai <changyu...@selectdb.com>
AuthorDate: Tue Dec 31 21:59:29 2024 +0800
[fix](split)Fixed the bug that batch mode split could not query data in
multiple be scenarios. (#46218)
### What problem does this PR solve?
Problem Summary:
In multiple be scenarios, batch mode split sometimes could not query
data.
The reason is that the estimated `numApproximateSplits()` may be
relatively small, and the value after dividing the current number of be
may be 0. As a result, the split will not be distributed to be, and the
query result will be empty.
We need to take the max of the value after division and 1.
---
.../src/main/java/org/apache/doris/datasource/FileQueryScanNode.java | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java
index 16b19ac77b2..69115e969b0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java
@@ -340,7 +340,10 @@ public abstract class FileQueryScanNode extends
FileScanNode {
totalFileSize = fileSplit.getLength() * selectedSplitNum;
long maxWaitTime =
ConnectContext.get().getSessionVariable().getFetchSplitsMaxWaitTime();
// Not accurate, only used to estimate concurrency.
- int numSplitsPerBE = numApproximateSplits() /
backendPolicy.numBackends();
+ // Here, we must take the max of 1, because
+ // in the case of multiple BEs, `numApproximateSplits() /
backendPolicy.numBackends()` may be 0,
+ // and finally numSplitsPerBE is 0, resulting in no data being
queried.
+ int numSplitsPerBE = Math.max(numApproximateSplits() /
backendPolicy.numBackends(), 1);
for (Backend backend : backendPolicy.getBackends()) {
SplitSource splitSource = new SplitSource(backend,
splitAssignment, maxWaitTime);
splitSources.add(splitSource);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
