[GitHub] [doris] adonis0147 opened a new pull request, #15866: [regression-test](MTMV) Make the case test_create_mtmv more robust
adonis0147 opened a new pull request, #15866: URL: https://github.com/apache/doris/pull/15866 # Proposed changes 1. Use the state of MTMV task as the loop condition. 2. Check the data in materialized view. ## Problem summary 1. The case used a retry strategy as the loop condition, it may not be stable while the host machine is busy. 2. The case didn't check the final data in materialized view. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15864: [fix](sort spill) fix bug of failed to create spilled file
github-actions[bot] commented on PR #15864: URL: https://github.com/apache/doris/pull/15864#issuecomment-1379943253 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15669: [feature](struct-type) adjust the vexpr for struct type
github-actions[bot] commented on code in PR #15669:
URL: https://github.com/apache/doris/pull/15669#discussion_r1067808419
##
be/src/vec/columns/column_struct.cpp:
##
@@ -23,10 +23,10 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int ILLEGAL_COLUMN;
-extern const int NOT_IMPLEMENTED;
-extern const int CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE;
-extern const int LOGICAL_ERROR;
+const int ILLEGAL_COLUMN = 44;
Review Comment:
warning: unused variable 'ILLEGAL_COLUMN'
[clang-diagnostic-unused-const-variable]
```cpp
const int ILLEGAL_COLUMN = 44;
^
```
##
be/src/vec/columns/column_struct.cpp:
##
@@ -23,10 +23,10 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int ILLEGAL_COLUMN;
-extern const int NOT_IMPLEMENTED;
-extern const int CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE;
-extern const int LOGICAL_ERROR;
+const int ILLEGAL_COLUMN = 44;
+const int NOT_IMPLEMENTED = 48;
+const int LOGICAL_ERROR = 49;
+const int CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE = 120;
Review Comment:
warning: unused variable 'CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE'
[clang-diagnostic-unused-const-variable]
```cpp
const int CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE = 120;
^
```
##
be/src/vec/columns/column_struct.cpp:
##
@@ -23,10 +23,10 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int ILLEGAL_COLUMN;
-extern const int NOT_IMPLEMENTED;
-extern const int CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE;
-extern const int LOGICAL_ERROR;
+const int ILLEGAL_COLUMN = 44;
+const int NOT_IMPLEMENTED = 48;
+const int LOGICAL_ERROR = 49;
Review Comment:
warning: unused variable 'LOGICAL_ERROR'
[clang-diagnostic-unused-const-variable]
```cpp
const int LOGICAL_ERROR = 49;
^
```
##
be/src/vec/columns/column_struct.cpp:
##
@@ -23,10 +23,10 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int ILLEGAL_COLUMN;
-extern const int NOT_IMPLEMENTED;
-extern const int CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE;
-extern const int LOGICAL_ERROR;
+const int ILLEGAL_COLUMN = 44;
+const int NOT_IMPLEMENTED = 48;
Review Comment:
warning: unused variable 'NOT_IMPLEMENTED'
[clang-diagnostic-unused-const-variable]
```cpp
const int NOT_IMPLEMENTED = 48;
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly opened a new pull request, #15867: [fix](nereids)upgrade signature datatype bug
englefly opened a new pull request, #15867: URL: https://github.com/apache/doris/pull/15867 # Proposed changes ComputeSignatureHelper.upgradeDateOrDateTimeToV2() we upgrate return date type, but forget to upgrade arguments datatype. The same problem in upgradeDecimalV2ToV3() Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15605: [improvement](projection) Improve projection performance
github-actions[bot] commented on PR #15605: URL: https://github.com/apache/doris/pull/15605#issuecomment-1379954131 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15780: [feature](multi-catalog) support clickhouse jdbc catalog
github-actions[bot] commented on PR #15780: URL: https://github.com/apache/doris/pull/15780#issuecomment-1379955922 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15669: [feature](struct-type) adjust the vexpr for struct type
github-actions[bot] commented on code in PR #15669:
URL: https://github.com/apache/doris/pull/15669#discussion_r1067818064
##
be/src/vec/data_types/data_type_struct.cpp:
##
@@ -23,14 +23,14 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int BAD_ARGUMENTS;
-extern const int DUPLICATE_COLUMN;
-extern const int EMPTY_DATA_PASSED;
-extern const int NOT_FOUND_COLUMN_IN_BLOCK;
-extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
-extern const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH;
-extern const int ILLEGAL_INDEX;
-extern const int LOGICAL_ERROR;
+const int NOT_FOUND_COLUMN_IN_BLOCK = 10;
+const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH = 13;
+const int DUPLICATE_COLUMN = 15;
Review Comment:
warning: unused variable 'DUPLICATE_COLUMN'
[clang-diagnostic-unused-const-variable]
```cpp
const int DUPLICATE_COLUMN = 15;
^
```
##
be/src/vec/data_types/data_type_struct.cpp:
##
@@ -23,14 +23,14 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int BAD_ARGUMENTS;
-extern const int DUPLICATE_COLUMN;
-extern const int EMPTY_DATA_PASSED;
-extern const int NOT_FOUND_COLUMN_IN_BLOCK;
-extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
-extern const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH;
-extern const int ILLEGAL_INDEX;
-extern const int LOGICAL_ERROR;
+const int NOT_FOUND_COLUMN_IN_BLOCK = 10;
+const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH = 13;
Review Comment:
warning: unused variable 'SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH'
[clang-diagnostic-unused-const-variable]
```cpp
const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH = 13;
^
```
##
be/src/vec/data_types/data_type_struct.cpp:
##
@@ -23,14 +23,14 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int BAD_ARGUMENTS;
-extern const int DUPLICATE_COLUMN;
-extern const int EMPTY_DATA_PASSED;
-extern const int NOT_FOUND_COLUMN_IN_BLOCK;
-extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
-extern const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH;
-extern const int ILLEGAL_INDEX;
-extern const int LOGICAL_ERROR;
+const int NOT_FOUND_COLUMN_IN_BLOCK = 10;
+const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH = 13;
+const int DUPLICATE_COLUMN = 15;
+const int BAD_ARGUMENTS = 36;
+const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH = 42;
+const int LOGICAL_ERROR = 49;
Review Comment:
warning: unused variable 'LOGICAL_ERROR'
[clang-diagnostic-unused-const-variable]
```cpp
const int LOGICAL_ERROR = 49;
^
```
##
be/src/vec/data_types/data_type_struct.cpp:
##
@@ -23,14 +23,14 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int BAD_ARGUMENTS;
-extern const int DUPLICATE_COLUMN;
-extern const int EMPTY_DATA_PASSED;
-extern const int NOT_FOUND_COLUMN_IN_BLOCK;
-extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
-extern const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH;
-extern const int ILLEGAL_INDEX;
-extern const int LOGICAL_ERROR;
+const int NOT_FOUND_COLUMN_IN_BLOCK = 10;
+const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH = 13;
+const int DUPLICATE_COLUMN = 15;
+const int BAD_ARGUMENTS = 36;
+const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH = 42;
+const int LOGICAL_ERROR = 49;
+const int EMPTY_DATA_PASSED = 92;
Review Comment:
warning: unused variable 'EMPTY_DATA_PASSED'
[clang-diagnostic-unused-const-variable]
```cpp
const int EMPTY_DATA_PASSED = 92;
^
```
##
be/src/vec/data_types/data_type_struct.cpp:
##
@@ -23,14 +23,14 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int BAD_ARGUMENTS;
-extern const int DUPLICATE_COLUMN;
-extern const int EMPTY_DATA_PASSED;
-extern const int NOT_FOUND_COLUMN_IN_BLOCK;
-extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
-extern const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH;
-extern const int ILLEGAL_INDEX;
-extern const int LOGICAL_ERROR;
+const int NOT_FOUND_COLUMN_IN_BLOCK = 10;
+const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH = 13;
+const int DUPLICATE_COLUMN = 15;
+const int BAD_ARGUMENTS = 36;
+const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH = 42;
+const int LOGICAL_ERROR = 49;
+const int EMPTY_DATA_PASSED = 92;
+const int ILLEGAL_INDEX = 127;
Review Comment:
warning: unused variable 'ILLEGAL_INDEX'
[clang-diagnostic-unused-const-variable]
```cpp
const int ILLEGAL_INDEX = 127;
^
```
##
be/src/vec/data_types/data_type_struct.cpp:
##
@@ -23,14 +23,14 @@
namespace doris::vectorized {
namespace ErrorCodes {
-extern const int BAD_ARGUMENTS;
-extern const int DUPLICATE_COLUMN;
-extern const int EMPTY_DATA_PASSED;
-extern const int NOT_FOUND_COLUMN_IN_BLOCK;
-extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
-extern const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH;
-extern const int ILLEGAL_INDEX;
-extern const int LOGICAL_ERROR;
+const int NOT_FOUND_COLUMN_IN_BLOCK = 10;
[GitHub] [doris] github-actions[bot] commented on pull request #15841: [Bug](join) fix bug null aware left anti join not correct result
github-actions[bot] commented on PR #15841: URL: https://github.com/apache/doris/pull/15841#issuecomment-1379967101 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] spaces-X opened a new issue, #15868: [Feature] support QuantileState in vectorized engine
spaces-X opened a new issue, #15868: URL: https://github.com/apache/doris/issues/15868 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description From #8234 pre-aggregation for quantile type is supported in non-vectorized engine. ### Use case Already added in #8234 ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15605: [improvement](projection) Improve projection performance
github-actions[bot] commented on PR #15605: URL: https://github.com/apache/doris/pull/15605#issuecomment-1379974411 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15623: [fix](nereids) binding group by key on agg.output if output is slot
github-actions[bot] commented on PR #15623: URL: https://github.com/apache/doris/pull/15623#issuecomment-1379974541 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15623: [fix](nereids) binding group by key on agg.output if output is slot
github-actions[bot] commented on PR #15623: URL: https://github.com/apache/doris/pull/15623#issuecomment-1379974498 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #15623: [fix](nereids) binding group by key on agg.output if output is slot
morrySnow merged PR #15623: URL: https://github.com/apache/doris/pull/15623 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15605: [improvement](projection) Improve projection performance
github-actions[bot] commented on PR #15605: URL: https://github.com/apache/doris/pull/15605#issuecomment-1379977953 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](nereids) binding group by key on agg.output if output is slot (#15623)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d23646793c [fix](nereids) binding group by key on agg.output if output
is slot (#15623)
d23646793c is described below
commit d23646793c71ea44e8e9f6b5f28acd9b5b51e237
Author: minghong
AuthorDate: Thu Jan 12 16:34:56 2023 +0800
[fix](nereids) binding group by key on agg.output if output is slot (#15623)
case 1
`select count(1) from t1 join t2 on t1.a = t2.a group by a`
`group by a` is ambiguous
case 2
`select t1.a from t1 join t2 on t1.a = t2.a group by a`
`group by a` is bound on t1.a
---
.../nereids/rules/analysis/BindSlotReference.java | 40 +++-
.../rules/analysis/BindSlotReferenceTest.java | 55 ++
2 files changed, 93 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
index bc0c9325ae..7fbcb9fde0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
@@ -74,6 +74,7 @@ import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
import org.apache.commons.lang.StringUtils;
import java.util.ArrayList;
@@ -227,8 +228,20 @@ public class BindSlotReference implements
AnalysisRuleFactory {
group by key cannot bind with agg func
plan:
agg(group_by v, output sum(k) as v)
-
throw AnalysisException
+
+CASE 4
+ sql:
+`select count(1) from t1 join t2 group by a`
+ we cannot bind `group by a`, because it is ambiguous
(t1.a and t2.a)
+
+CASE 5
+ following case 4, if t1.a is in agg.output, we can bind
`group by a` to t1.a
+ sql
+select t1.a
+from t1 join t2 on t1.a = t2.a
+group by a
+ group_by_key is bound on t1.a
*/
duplicatedSlotNames.stream().forEach(dup ->
childOutputsToExpr.remove(dup));
Map aliasNameToExpr = output.stream()
@@ -261,8 +274,31 @@ public class BindSlotReference implements
AnalysisRuleFactory {
}
return groupBy;
}).collect(Collectors.toList());
+/*
+according to case 4 and case 5, we construct boundSlots
+*/
+Set outputSlotNames = Sets.newHashSet();
+Set outputSlots = output.stream()
+.filter(SlotReference.class::isInstance)
+.peek(slot -> outputSlotNames.add(slot.getName()))
+.map(NamedExpression::toSlot).collect(
+Collectors.toSet());
+//suppose group by key is a.
+// if both t1.a and t2.a are in agg.child.output, and t1.a
in agg.output,
+// bind group_by_key a with t1.a
+// ` .filter(slot ->
!outputSlotNames.contains(slot.getName()))`
+// is used to avoid add t2.a into boundSlots
+Set boundSlots = agg.child().getOutputSet().stream()
+.filter(slot ->
!outputSlotNames.contains(slot.getName()))
+.collect(Collectors.toSet());
+
+boundSlots.addAll(outputSlots);
+SlotBinder binder = new
SlotBinder(toScope(Lists.newArrayList(boundSlots)), ctx.cascadesContext);
+
+List groupBy = replacedGroupBy.stream()
+.map(expression -> binder.bind(expression))
+.collect(Collectors.toList());
-List groupBy = bind(replacedGroupBy,
agg.children(), agg, ctx.cascadesContext);
List unboundGroupBys = Lists.newArrayList();
boolean hasUnbound = groupBy.stream().anyMatch(
expression -> {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/BindSlotReferenceTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/BindSlotReferenceTest.java
index e63618cdd7..abc18a70c7 100644
---
a/fe/fe-core/src/test/j
[GitHub] [doris] Henry2SS opened a new issue, #15869: [Enhancement] make max running txn per db more flexible
Henry2SS opened a new issue, #15869: URL: https://github.com/apache/doris/issues/15869 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Now, we have a fe config `max_running_txn_num_per_db`. It limits every database's transaction number. But in JD, we have a problem. Some users have broker load with large amount of data. Their task will affect other users load task. We need a limit that can be flexibly changed. ### Solution Add transaction quota of every database. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg commented on a diff in pull request #15859: [Bug](sort)fix merge sorter might div zero when block bytes less than block rows
jacktengg commented on code in PR #15859:
URL: https://github.com/apache/doris/pull/15859#discussion_r1067837923
##
regression-test/suites/demo_p0/merge_sort.groovy:
##
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// when rows is bigger than block bytes, it might cause diving zero error
+suite("merge_sort_rows_bigger_than_bytes") {
+
+sql """ DROP TABLE IF EXISTS B """
+sql """
+CREATE TABLE IF NOT EXISTS B
+(
+b_id int
+)
+DISTRIBUTED BY HASH(b_id) BUCKETS 1
+PROPERTIES("replication_num" = "1");
+"""
+
+sql " INSERT INTO B values (1);"
+
+qt_sql """
+SELECT subq_0.`c1` AS c1
+FROM
+(SELECT version() AS c0,
+ ref_0.`id` AS c1
+FROM test_streamload_action1 AS ref_0
+LEFT JOIN B AS ref_9 ON (ref_0.`id` = ref_9.`b_id`)
+WHERE ref_9.`b_id` IS NULL) AS subq_0
+WHERE subq_0.`c0` IS NOT NULL
+ORDER BY subq_0.`c0` DESC
Review Comment:
need add another other by to ensure test result stability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Henry2SS commented on pull request #15830: [enhancement](load) change transaction limit from db level to user level
Henry2SS commented on PR #15830: URL: https://github.com/apache/doris/pull/15830#issuecomment-1379980886 > You'd better explain why we change it, e.g. requests from users or resolving a problem. Thx for your advice. Done. Add a related issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nothing-go-reade opened a new issue, #15870: update about file
nothing-go-reade opened a new issue, #15870: URL: https://github.com/apache/doris/issues/15870 1 cp thirdparty/*.sh /var/local/thirdparty 2 cp thirdparty/patches /var/local/thirdparty -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15858: [fix](Nereids): mergeGroup() consider GroupExpr in same/different Group
hello-stephen commented on PR #15858: URL: https://github.com/apache/doris/pull/15858#issuecomment-1379982156 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.66 seconds load time: 495 seconds storage size: 17122884559 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112083859_clickbench_pr_78436.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg commented on a diff in pull request #15859: [Bug](sort)fix merge sorter might div zero when block bytes less than block rows
jacktengg commented on code in PR #15859:
URL: https://github.com/apache/doris/pull/15859#discussion_r1067837923
##
regression-test/suites/demo_p0/merge_sort.groovy:
##
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// when rows is bigger than block bytes, it might cause diving zero error
+suite("merge_sort_rows_bigger_than_bytes") {
+
+sql """ DROP TABLE IF EXISTS B """
+sql """
+CREATE TABLE IF NOT EXISTS B
+(
+b_id int
+)
+DISTRIBUTED BY HASH(b_id) BUCKETS 1
+PROPERTIES("replication_num" = "1");
+"""
+
+sql " INSERT INTO B values (1);"
+
+qt_sql """
+SELECT subq_0.`c1` AS c1
+FROM
+(SELECT version() AS c0,
+ ref_0.`id` AS c1
+FROM test_streamload_action1 AS ref_0
+LEFT JOIN B AS ref_9 ON (ref_0.`id` = ref_9.`b_id`)
+WHERE ref_9.`b_id` IS NULL) AS subq_0
+WHERE subq_0.`c0` IS NOT NULL
+ORDER BY subq_0.`c0` DESC
Review Comment:
need add another order by to ensure test result stability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #15833: [fix](Nereids): add unusedFlag to avoid job run used GroupExprssion
morrySnow commented on code in PR #15833:
URL: https://github.com/apache/doris/pull/15833#discussion_r1067844668
##
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java:
##
@@ -465,6 +468,8 @@ public Group mergeGroup(Group source, Group destination) {
source.moveOwnership(destination);
groups.remove(source.getGroupId());
}
+
+needMergeGroups.forEach(this::mergeGroup);
Review Comment:
consider a scenario:
if we have two entry in the map: B -> A and C -> B.
when we do merge group. we first merge B to A, and then B is removed. after
that, we want to merge C to B, but B is not exist anymore.
So, maybe we need to use **disjoint set** do find a common parent, and then
merge all other group to it in one **disjoint set**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ByteYue commented on a diff in pull request #15859: [Bug](sort)fix merge sorter might div zero when block bytes less than block rows
ByteYue commented on code in PR #15859:
URL: https://github.com/apache/doris/pull/15859#discussion_r1067846403
##
regression-test/suites/demo_p0/merge_sort.groovy:
##
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// when rows is bigger than block bytes, it might cause diving zero error
+suite("merge_sort_rows_bigger_than_bytes") {
+
+sql """ DROP TABLE IF EXISTS B """
+sql """
+CREATE TABLE IF NOT EXISTS B
+(
+b_id int
+)
+DISTRIBUTED BY HASH(b_id) BUCKETS 1
+PROPERTIES("replication_num" = "1");
+"""
+
+sql " INSERT INTO B values (1);"
+
+qt_sql """
+SELECT subq_0.`c1` AS c1
+FROM
+(SELECT version() AS c0,
+ ref_0.`id` AS c1
+FROM test_streamload_action1 AS ref_0
+LEFT JOIN B AS ref_9 ON (ref_0.`id` = ref_9.`b_id`)
+WHERE ref_9.`b_id` IS NULL) AS subq_0
+WHERE subq_0.`c0` IS NOT NULL
+ORDER BY subq_0.`c0` DESC
Review Comment:
> need add another order by to ensure test result stability.
OK, I'll refresh the regression case after I've rebuilt the be binary and
run the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15854: [fix](Nereids): throw exception when aliasedQuery has no alias
github-actions[bot] commented on PR #15854: URL: https://github.com/apache/doris/pull/15854#issuecomment-1379990541 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15854: [fix](Nereids): throw exception when aliasedQuery has no alias
github-actions[bot] commented on PR #15854: URL: https://github.com/apache/doris/pull/15854#issuecomment-1379990481 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-spark-connector] caoliang-web opened a new pull request, #64: Return specific error information after optimizing the exception
caoliang-web opened a new pull request, #64: URL: https://github.com/apache/doris-spark-connector/pull/64 ## Problem Summary: Return specific error information after optimizing the exception ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko commented on a diff in pull request #15633: [enhancement](compaction) Optimize calculating level size of input rowset in SizeBasedCumulativeCompactionPolicy
platoneko commented on code in PR #15633:
URL: https://github.com/apache/doris/pull/15633#discussion_r1067854490
##
be/src/olap/cumulative_compaction_policy.cpp:
##
@@ -335,13 +325,12 @@ int
SizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
return transient_size;
}
-int SizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
-for (auto& i : _levels) {
-if (size >= i) {
-return i;
-}
-}
-return 0;
+int64_t SizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
+if (size < 1024) return 0;
+int64_t max_level = (int64_t)1
Review Comment:
Calculate the first 2 power number <= the given size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15807: [Update](deps) add clucene deps for doris inverted index
github-actions[bot] commented on PR #15807: URL: https://github.com/apache/doris/pull/15807#issuecomment-1380003973 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on a diff in pull request #15830: [enhancement](load) change transaction limit from db level to user level
caiconghui commented on code in PR #15830:
URL: https://github.com/apache/doris/pull/15830#discussion_r1067866074
##
docs/en/docs/admin-manual/config/fe-config.md:
##
@@ -1005,7 +1005,7 @@ MasterOnly：true
Maximum percentage of data that can be filtered (due to reasons such as data
is irregularly) , The default value is 0.
- `max_running_txn_num_per_db`
+ `defalut_db_running_txn_num`
Review Comment:
```suggestion
`defalut_db_max_running_txn_num`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15859: [Bug](sort)fix merge sorter might div zero when block bytes less than block rows
github-actions[bot] commented on PR #15859: URL: https://github.com/apache/doris/pull/15859#issuecomment-1380017280 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #15871: [refactor](rpc fn) decouple vectorized remote function from row-based one
Gabriel39 opened a new pull request, #15871: URL: https://github.com/apache/doris/pull/15871 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15780: [feature](multi-catalog) support clickhouse jdbc catalog
github-actions[bot] commented on PR #15780: URL: https://github.com/apache/doris/pull/15780#issuecomment-1380021068 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15871: [refactor](rpc fn) decouple vectorized remote function from row-based one
github-actions[bot] commented on PR #15871: URL: https://github.com/apache/doris/pull/15871#issuecomment-1380021887 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yuanyuan8983 opened a new pull request, #15872: [style](FE)change PaloFe to DorisFe
yuanyuan8983 opened a new pull request, #15872: URL: https://github.com/apache/doris/pull/15872 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg commented on pull request #15859: [Bug](sort)fix merge sorter might div zero when block bytes less than block rows
jacktengg commented on PR #15859: URL: https://github.com/apache/doris/pull/15859#issuecomment-1380025087 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15859: [Bug](sort)fix merge sorter might div zero when block bytes less than block rows
github-actions[bot] commented on PR #15859: URL: https://github.com/apache/doris/pull/15859#issuecomment-1380026632 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qidaye commented on a diff in pull request #15821: [Feature-WIP](inverted index) inverted index writer's implementation
qidaye commented on code in PR #15821:
URL: https://github.com/apache/doris/pull/15821#discussion_r1067885554
##
be/src/olap/olap_common.h:
##
@@ -198,6 +198,22 @@ enum ReaderType {
READER_CHECKSUM = 4,
};
+constexpr bool field_is_slice_type(const FieldType& field_type) {
+return field_type == OLAP_FIELD_TYPE_VARCHAR || field_type ==
OLAP_FIELD_TYPE_CHAR ||
+ field_type == OLAP_FIELD_TYPE_STRING;
+}
+
+constexpr bool field_is_numeric_type(const FieldType& field_type) {
+return field_type == OLAP_FIELD_TYPE_INT || field_type ==
OLAP_FIELD_TYPE_UNSIGNED_INT ||
+ field_type == OLAP_FIELD_TYPE_BIGINT || field_type ==
OLAP_FIELD_TYPE_SMALLINT ||
+ field_type == OLAP_FIELD_TYPE_UNSIGNED_TINYINT ||
+ field_type == OLAP_FIELD_TYPE_UNSIGNED_SMALLINT ||
+ field_type == OLAP_FIELD_TYPE_TINYINT || field_type ==
OLAP_FIELD_TYPE_DOUBLE ||
+ field_type == OLAP_FIELD_TYPE_FLOAT || field_type ==
OLAP_FIELD_TYPE_DATE ||
+ field_type == OLAP_FIELD_TYPE_DATETIME || field_type ==
OLAP_FIELD_TYPE_LARGEINT ||
Review Comment:
Remeber the new data type, like DECIMALV3 and DATETIMEV2 and so on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qidaye commented on a diff in pull request #15821: [Feature-WIP](inverted index) inverted index writer's implementation
qidaye commented on code in PR #15821:
URL: https://github.com/apache/doris/pull/15821#discussion_r1067885554
##
be/src/olap/olap_common.h:
##
@@ -198,6 +198,22 @@ enum ReaderType {
READER_CHECKSUM = 4,
};
+constexpr bool field_is_slice_type(const FieldType& field_type) {
+return field_type == OLAP_FIELD_TYPE_VARCHAR || field_type ==
OLAP_FIELD_TYPE_CHAR ||
+ field_type == OLAP_FIELD_TYPE_STRING;
+}
+
+constexpr bool field_is_numeric_type(const FieldType& field_type) {
+return field_type == OLAP_FIELD_TYPE_INT || field_type ==
OLAP_FIELD_TYPE_UNSIGNED_INT ||
+ field_type == OLAP_FIELD_TYPE_BIGINT || field_type ==
OLAP_FIELD_TYPE_SMALLINT ||
+ field_type == OLAP_FIELD_TYPE_UNSIGNED_TINYINT ||
+ field_type == OLAP_FIELD_TYPE_UNSIGNED_SMALLINT ||
+ field_type == OLAP_FIELD_TYPE_TINYINT || field_type ==
OLAP_FIELD_TYPE_DOUBLE ||
+ field_type == OLAP_FIELD_TYPE_FLOAT || field_type ==
OLAP_FIELD_TYPE_DATE ||
+ field_type == OLAP_FIELD_TYPE_DATETIME || field_type ==
OLAP_FIELD_TYPE_LARGEINT ||
Review Comment:
Remeber the new data types, like DECIMALV3 and DATETIMEV2 and so on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #15832: [Feature](remote) Cooldown cold data to object storage only one replica
dataroaring commented on code in PR #15832:
URL: https://github.com/apache/doris/pull/15832#discussion_r1067666463
##
be/src/common/config.h:
##
@@ -815,6 +815,7 @@ CONF_Int32(concurrency_per_dir, "2");
CONF_mInt64(cooldown_lag_time_sec, "10800"); // 3h
CONF_mInt64(max_sub_cache_file_size, "104857600"); // 100MB
CONF_mInt64(file_cache_alive_time_sec, "604800"); // 1 week
+CONF_Bool(cooldown_single_remote_file, "false"); // create single remote
file when cooldown.
Review Comment:
We should not add the config. If it is used during development, we can
rename it to internal_cooldown_single_remote_replica.
##
be/src/io/fs/file_reader_options.h:
##
@@ -39,6 +39,7 @@ class CachePathPolicy {
// path: the path of file which will be cached
// return value: the cache path of the given file.
virtual std::string get_cache_path(const std::string& path) const { return
""; }
+std::string cache_path;
Review Comment:
Please refer to 29492f0d6c455f22aef0ae0b19f436fcd18b7d52.
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -46,9 +46,8 @@ using io::FileCacheManager;
Status Segment::open(io::FileSystemSPtr fs, const std::string& path, uint32_t
segment_id,
RowsetId rowset_id, TabletSchemaSPtr tablet_schema,
+ const io::FileReaderOptions& reader_options,
std::shared_ptr* output) {
-io::FileReaderOptions
reader_options(io::cache_type_from_string(config::file_cache_type),
- io::SegmentCachePathPolicy());
Review Comment:
Please refer to 29492f0d6c455f22aef0ae0b19f436fcd18b7d52.
##
be/src/olap/rowset/beta_rowset_writer.cpp:
##
@@ -385,8 +385,10 @@ Status BetaRowsetWriter::_load_noncompacted_segments(
auto seg_path =
BetaRowset::segment_file_path(_context.rowset_dir,
_context.rowset_id, seg_id);
std::shared_ptr segment;
+io::FileReaderOptions
reader_options(io::cache_type_from_string(config::file_cache_type),
+ io::SegmentCachePathPolicy());
auto s = segment_v2::Segment::open(fs, seg_path, seg_id, rowset_id(),
Review Comment:
Please refer to 29492f0d6c455f22aef0ae0b19f436fcd18b7d52.
##
be/src/olap/push_handler.cpp:
##
@@ -203,6 +203,8 @@ Status PushHandler::_convert_v2(TabletSharedPtr cur_tablet,
RowsetSharedPtr* cur
context.rowset_state = PREPARED;
context.segments_overlap = OVERLAP_UNKNOWN;
context.tablet_schema = tablet_schema;
+context.oldest_write_timestamp = UnixSeconds();
+context.newest_write_timestamp = UnixSeconds();
Review Comment:
timestamps are set in BetaRowsetWriter::build, we can remove either?
##
be/src/olap/tablet_meta.h:
##
@@ -113,7 +113,7 @@ class TabletMeta {
// Init `RowsetMeta._fs` if rowset is local.
void init_rs_metas_fs(const io::FileSystemSPtr& fs);
-void to_meta_pb(TabletMetaPB* tablet_meta_pb);
+void to_meta_pb(bool only_remote, TabletMetaPB* tablet_meta_pb);
Review Comment:
only_include_remote_rowset is a much more specific name.
##
be/src/olap/tablet.cpp:
##
@@ -1683,11 +1690,23 @@ Status Tablet::cooldown() {
LOG(WARNING) << "Failed to own cumu_compaction_lock. tablet=" <<
tablet_id();
return Status::Error();
}
+
+if (!config::cooldown_single_remote_file ||
+_tablet_meta->cooldown_replica_id() == _tablet_meta->replica_id()) {
+RETURN_IF_ERROR(_cooldown_upload_data());
+} else {
+RETURN_IF_ERROR(_cooldown_use_remote_data());
+}
+return Status::OK();
+}
+
+Status Tablet::_cooldown_upload_data() {
Review Comment:
cooldown_data is a better name?
##
be/src/olap/tablet.cpp:
##
@@ -1729,18 +1757,118 @@ Status Tablet::cooldown() {
has_shutdown = tablet_state() == TABLET_SHUTDOWN;
if (!has_shutdown) {
modify_rowsets(to_add, to_delete);
-_self_owned_remote_rowsets.insert(to_add.front());
+if (!config::cooldown_single_remote_file) {
+_self_owned_remote_rowsets.insert(to_add.front());
+}
save_meta();
}
}
-if (has_shutdown) {
+if (has_shutdown && !config::cooldown_single_remote_file) {
record_unused_remote_rowset(new_rowset_id, dest_fs->resource_id(),
to_add.front()->num_segments());
return Status::Aborted("tablet {} has shutdown", tablet_id());
}
return Status::OK();
}
+Status Tablet::_read_remote_tablet_meta(FileSystemSPtr fs, TabletMetaPB*
tablet_meta_pb) {
+std::string remote_meta_path =
+BetaRowset::remote_tablet_meta_path(tablet_id(),
_tablet_meta->cooldown_replica_id());
+bool exist = false;
+RETURN_IF_ERROR(fs->exists(remote_meta_path, &exist));
+if
[GitHub] [doris] github-actions[bot] commented on pull request #15817: [fix](planner) move join reorder to the single node planner
github-actions[bot] commented on PR #15817: URL: https://github.com/apache/doris/pull/15817#issuecomment-1380064685 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15817: [fix](planner) move join reorder to the single node planner
github-actions[bot] commented on PR #15817: URL: https://github.com/apache/doris/pull/15817#issuecomment-1380064803 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Henry2SS commented on a diff in pull request #15830: [enhancement](load) change transaction limit from db level to user level
Henry2SS commented on code in PR #15830: URL: https://github.com/apache/doris/pull/15830#discussion_r1067911159 ## docs/en/docs/admin-manual/config/fe-config.md: ## @@ -1005,7 +1005,7 @@ MasterOnly：true Maximum percentage of data that can be filtered (due to reasons such as data is irregularly) , The default value is 0. - `max_running_txn_num_per_db` + `defalut_db_running_txn_num` Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko commented on a diff in pull request #15832: [Feature](remote) Cooldown cold data to object storage only one replica
platoneko commented on code in PR #15832: URL: https://github.com/apache/doris/pull/15832#discussion_r1067913105 ## fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java: ## @@ -1486,11 +1486,11 @@ private void createJob(long dbId, OlapTable olapTable, Map
[GitHub] [doris] hello-stephen commented on pull request #15859: [Bug](sort)fix merge sorter might div zero when block bytes less than block rows
hello-stephen commented on PR #15859: URL: https://github.com/apache/doris/pull/15859#issuecomment-1380077818 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.29 seconds load time: 560 seconds storage size: 17123097128 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112095621_clickbench_pr_78803.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15766: [pick](compaction) pick vertical&ordered_data compaction to 1.2
github-actions[bot] commented on PR #15766: URL: https://github.com/apache/doris/pull/15766#issuecomment-1380078939 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15766: [pick](compaction) pick vertical&ordered_data compaction to 1.2
github-actions[bot] commented on PR #15766: URL: https://github.com/apache/doris/pull/15766#issuecomment-1380079006 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15836: [fix](multi catalog)Support iceberg schema evolution.
github-actions[bot] commented on PR #15836: URL: https://github.com/apache/doris/pull/15836#issuecomment-1380079776 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15860: [Bug](datev2) Fix bugs for datev2
hello-stephen commented on PR #15860: URL: https://github.com/apache/doris/pull/15860#issuecomment-1380089376 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.79 seconds load time: 494 seconds storage size: 17122600336 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112100511_clickbench_pr_78450.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #15872: [style](FE)change PaloFe to DorisFe
Yukang-Lian commented on PR #15872: URL: https://github.com/apache/doris/pull/15872#issuecomment-1380091314 https://github.com/apache/doris/blob/d23646793c71ea44e8e9f6b5f28acd9b5b51e237/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java#L124 maybe this line also needs to be modified. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko opened a new pull request, #15873: [feature](cooldown) Decouple storage policy and resource (BE)
platoneko opened a new pull request, #15873: URL: https://github.com/apache/doris/pull/15873 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin opened a new pull request, #15874: [fix](brokerload) fix be core dump casued by broker load
luozenglin opened a new pull request, #15874: URL: https://github.com/apache/doris/pull/15874 # Proposed changes Issue Number: close 15846 ## Problem summary The BrokerFileReader is refactored in #15622, but the _client is not initialized ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15829: [fix](olap) dictionary cannot be sorted after inserting some null values
github-actions[bot] commented on PR #15829: URL: https://github.com/apache/doris/pull/15829#issuecomment-1380098991 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15829: [fix](olap) dictionary cannot be sorted after inserting some null values
github-actions[bot] commented on PR #15829: URL: https://github.com/apache/doris/pull/15829#issuecomment-1380099075 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15874: [fix](brokerload) fix be core dump casued by broker load
github-actions[bot] commented on PR #15874: URL: https://github.com/apache/doris/pull/15874#issuecomment-1380099710 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15873: [feature](cooldown) Decouple storage policy and resource (BE)
github-actions[bot] commented on code in PR #15873:
URL: https://github.com/apache/doris/pull/15873#discussion_r1067942363
##
be/src/io/fs/s3_file_system.h:
##
@@ -70,13 +71,13 @@ class S3FileSystem final : public RemoteFileSystem {
};
// Guarded by external lock.
-void set_ak(std::string ak) { _s3_conf.ak = std::move(ak); }
-
-// Guarded by external lock.
-void set_sk(std::string sk) { _s3_conf.sk = std::move(sk); }
+void set_conf(S3Conf s3_conf) { _s3_conf = std::move(s3_conf); }
std::string get_key(const Path& path) const;
+private:
+S3FileSystem(S3Conf&& s3_conf, std::string&& id);
+
private:
Review Comment:
warning: redundant access specifier has the same accessibility as the
previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
**be/src/io/fs/s3_file_system.h:77:** previously declared here
```cpp
private:
^
```
##
be/test/olap/remote_rowset_gc_test.cpp:
##
@@ -52,9 +53,14 @@ class RemoteRowsetGcTest : public testing::Test {
s3_conf.region = config::test_s3_region;
Review Comment:
warning: no member named 'test_s3_region' in namespace 'doris::config'
[clang-diagnostic-error]
```cpp
s3_conf.region = config::test_s3_region;
^
```
##
be/test/olap/remote_rowset_gc_test.cpp:
##
@@ -52,9 +53,14 @@
s3_conf.region = config::test_s3_region;
s3_conf.bucket = config::test_s3_bucket;
Review Comment:
warning: no member named 'test_s3_bucket' in namespace 'doris::config'
[clang-diagnostic-error]
```cpp
s3_conf.bucket = config::test_s3_bucket;
^
```
##
be/test/olap/tablet_cooldown_test.cpp:
##
@@ -50,10 +52,15 @@
s3_conf.endpoint = config::test_s3_endpoint;
s3_conf.region = config::test_s3_region;
Review Comment:
warning: no member named 'test_s3_region' in namespace 'doris::config'
[clang-diagnostic-error]
```cpp
s3_conf.region = config::test_s3_region;
^
```
##
be/test/olap/tablet_test.cpp:
##
@@ -313,7 +312,8 @@
TabletSharedPtr _tablet(new Tablet(_tablet_meta, nullptr));
_tablet->init();
-_tablet->set_storage_policy("test_policy_name");
+constexpr int64_t storage_policy_id = 1;
+_tablet->set_storage_policy_id(storage_policy_id);
_tablet->_rs_version_map[ptr1->version()] = rowset1;
_tablet->_rs_version_map[ptr2->version()] = rowset2;
Review Comment:
warning: '_rs_version_map' is a private member of 'doris::Tablet'
[clang-diagnostic-error]
```cpp
_tablet->_rs_version_map[ptr2->version()] = rowset2;
^
```
**be/src/olap/tablet.h:410:** declared private here
```cpp
std::unordered_map
_rs_version_map;
^
```
##
be/test/olap/tablet_cooldown_test.cpp:
##
@@ -50,10 +52,15 @@ class TabletCooldownTest : public testing::Test {
s3_conf.endpoint = config::test_s3_endpoint;
Review Comment:
warning: no member named 'test_s3_endpoint' in namespace 'doris::config'
[clang-diagnostic-error]
```cpp
s3_conf.endpoint = config::test_s3_endpoint;
^
```
##
be/test/olap/tablet_cooldown_test.cpp:
##
@@ -50,10 +52,15 @@
s3_conf.endpoint = config::test_s3_endpoint;
s3_conf.region = config::test_s3_region;
s3_conf.bucket = config::test_s3_bucket;
Review Comment:
warning: no member named 'test_s3_bucket' in namespace 'doris::config'
[clang-diagnostic-error]
```cpp
s3_conf.bucket = config::test_s3_bucket;
^
```
##
be/test/olap/tablet_cooldown_test.cpp:
##
@@ -50,10 +52,15 @@
s3_conf.endpoint = config::test_s3_endpoint;
s3_conf.region = config::test_s3_region;
s3_conf.bucket = config::test_s3_bucket;
-s3_conf.prefix = "tablet_cooldown_test";
-auto s3_fs = std::make_shared(std::move(s3_conf),
kResourceId);
+s3_conf.prefix = config::test_s3_prefix + "/tablet_cooldown_test";
Review Comment:
warning: no member named 'test_s3_prefix' in namespace 'doris::config'
[clang-diagnostic-error]
```cpp
s3_conf.prefix = config::test_s3_prefix + "/tablet_cooldown_test";
^
```
##
be/src/olap/tablet_meta.cpp:
##
@@ -865,7 +865,7 @@ bool operator==(const TabletMeta& a, const TabletMeta& b) {
}
if (a._in_restore_mode != b._in_restore_mode) return false;
if (a._preferred_rowset_type != b._preferred_rowset_type) return false;
-if (a._storage_policy != b._storage_policy) return false;
+if (a._storage_policy_id !=
[GitHub] [doris] dutyu opened a new pull request, #15875: [Feature](profile) set sql and defaultDb fields in show-load-profile.
dutyu opened a new pull request, #15875:
URL: https://github.com/apache/doris/pull/15875
# Proposed changes
When execute `show load profile '/'`, the value of `SQL` and `DefaultDb`
columns are all 'N/A', but we can fill these fields，the result of this pr is as
follows:
Execute `show load profile '/'\G`:
```sql
MySQL [test_d]> show load profile '/'\G
*** 1. row ***
QueryId: 652326
User: N/A
DefaultDb: default_cluster:test_d
SQL: LOAD LABEL
`default_cluster:test_d`.`dds_broker_load8589513645460` (APPEND DATA INFILE
('hdfs://360jinronglycc/user/hive/warehouse/fin_ads.db/ads_zbf_xry_flow_v4_faqi_after/*')
INTO TABLE ads_zbf_xry_flow_v4_faqi_after FORMAT AS 'ORC' (dt, risk_layer,
channel_id, flow_type, credit_cust, credit_amt, trial_try_cust, trial_cust,
irr_rate, trial_cp_cust, coupon_cust, crawl_cust, psw_cust, pcs_cust,
face_cust, draw_cust) SET (`dt` = `dt`, `risk_layer` = `risk_layer`,
`channel_id` = `channel_id`, `flow_type` = `flow_type`, `credit_cust` =
`credit_cust`, `credit_amt` = `credit_amt`, `trial_try_cust` =
`trial_try_cust`, `trial_cust` = `trial_cust`, `irr_rate` = `irr_rate`,
`trial_cp_cust` = `trial_cp_cust`, `coupon_cust` = `coupon_cust`, `crawl_cust`
= `crawl_cust`, `psw_cust` = `psw_cust`, `pcs_cust` = `pcs_cust`, `face_cust` =
`face_cust`, `draw_cust` = `draw_cust`)) WITH BROKER broker_jingfen
("dfs.client.failover.proxy.provider" = "org.apache.hadoop.hdfs.server.nameno
de.ha.ConfiguredFailoverProxyProvider", "hadoop.security.authentication" =
"kerberos", "kerberos_principal" = "b...@lycc.com", "kerberos_keytab" =
"/etc/bolt_safe_lycc.keytab", "dfs.namenode.rpc-address.360jinronglycc.nn1" =
"oaos245.finloan.lycc.qihoo.net:8020", "_DORIS_STORAGE_TYPE_" = "BROKER",
"dfs.namenode.rpc-address.360jinronglycc.nn2" =
"oaos246.finloan.lycc.qihoo.net:8020", "dfs.nameservices" = "360jinronglycc",
"dfs.ha.namenodes.360jinronglycc" = "nn1, nn2") PROPERTIES
("max_filter_ratio" = "0", "timeout" = "3")
QueryType: Load
StartTime: 2023-01-12 18:33:34
EndTime: 2023-01-12 18:33:46
TotalTime: 11s613ms
QueryState: N/A
1 row in set (0.01 sec)
```
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [x] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [x] No Need
3. Has document been added or modified:
- [ ] Yes
- [x] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [x] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [x] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zbtzbtzbt opened a new pull request, #15876: [fix] remove echo database passwd
zbtzbtzbt opened a new pull request, #15876: URL: https://github.com/apache/doris/pull/15876 # Proposed changes echo password is dangerous, we should delete it.  ## Problem summary 1. remove echo passwd 2. add some necessary information (e.g. add timer for data load) 3. remove some redundant code ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15876: [fix] remove echo database passwd
github-actions[bot] commented on PR #15876:
URL: https://github.com/apache/doris/pull/15876#issuecomment-1380132618
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3901438284";) output.
shellcheck errors
```
'shellcheck ' found no issues.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- tools/ssb-tools/bin/create-ssb-tables.sh.orig
+++ tools/ssb-tools/bin/create-ssb-tables.sh
--- tools/ssb-tools/bin/load-ssb-data.sh.orig
+++ tools/ssb-tools/bin/load-ssb-data.sh
@@ -281,4 +281,4 @@
end_time=$(date +%s)
echo "End time: $(date)"
-echo "Finish load ssb data, Time taken: $((end_time-start_time)) seconds"
+echo "Finish load ssb data, Time taken: $((end_time - start_time)) seconds"
--- tools/tpch-tools/bin/create-tpch-tables.sh.orig
+++ tools/tpch-tools/bin/create-tpch-tables.sh
--- tools/tpch-tools/bin/load-tpch-data.sh.orig
+++ tools/tpch-tools/bin/load-tpch-data.sh
@@ -236,4 +236,4 @@
end_time=$(date +%s)
echo "End time: $(date)"
-echo "Finish load tpch data, Time taken: $((end_time-start_time)) seconds"
+echo "Finish load tpch data, Time taken: $((end_time - start_time)) seconds"
--- tools/tpch-tools/bin/run-tpch-queries.sh.orig
+++ tools/tpch-tools/bin/run-tpch-queries.sh
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15760: [Chore](build) enable -Wextra and remove some -Wno
hello-stephen commented on PR #15760: URL: https://github.com/apache/doris/pull/15760#issuecomment-1380145442 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.64 seconds load time: 496 seconds storage size: 17123165845 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112105109_clickbench_pr_78547.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #15874: [fix](brokerload) fix be core dump casued by broker load
dataroaring commented on PR #15874: URL: https://github.com/apache/doris/pull/15874#issuecomment-1380151176 We should add regression test for broker load. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15874: [fix](brokerload) fix be core dump casued by broker load
github-actions[bot] commented on PR #15874: URL: https://github.com/apache/doris/pull/15874#issuecomment-1380151781 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15874: [fix](brokerload) fix be core dump casued by broker load
github-actions[bot] commented on PR #15874: URL: https://github.com/apache/doris/pull/15874#issuecomment-1380151721 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15876: [fix](terminal) remove echo database passwd
github-actions[bot] commented on PR #15876:
URL: https://github.com/apache/doris/pull/15876#issuecomment-1380156336
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3901582592";) output.
shellcheck errors
```
'shellcheck ' found no issues.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- tools/ssb-tools/bin/create-ssb-tables.sh.orig
+++ tools/ssb-tools/bin/create-ssb-tables.sh
--- tools/ssb-tools/bin/load-ssb-data.sh.orig
+++ tools/ssb-tools/bin/load-ssb-data.sh
@@ -281,4 +281,4 @@
end_time=$(date +%s)
echo "End time: $(date)"
-echo "Finish load ssb data, Time taken: $((end_time-start_time)) seconds"
+echo "Finish load ssb data, Time taken: $((end_time - start_time)) seconds"
--- tools/tpch-tools/bin/create-tpch-tables.sh.orig
+++ tools/tpch-tools/bin/create-tpch-tables.sh
--- tools/tpch-tools/bin/load-tpch-data.sh.orig
+++ tools/tpch-tools/bin/load-tpch-data.sh
@@ -236,4 +236,4 @@
end_time=$(date +%s)
echo "End time: $(date)"
-echo "Finish load tpch data, Time taken: $((end_time-start_time)) seconds"
+echo "Finish load tpch data, Time taken: $((end_time - start_time)) seconds"
--- tools/tpch-tools/bin/run-tpch-queries.sh.orig
+++ tools/tpch-tools/bin/run-tpch-queries.sh
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15605: [improvement](projection) Improve projection performance
github-actions[bot] commented on code in PR #15605:
URL: https://github.com/apache/doris/pull/15605#discussion_r1067986937
##
be/src/vec/exec/join/vjoin_node_base.cpp:
##
@@ -104,51 +104,92 @@ void VJoinNodeBase::_construct_mutable_join_block() {
_join_block.columns() - 1,
remove_nullable(_join_block.get_by_position(_join_block.columns() - 1).column));
}
+_join_block_column_num = _join_block.columns();
}
Status VJoinNodeBase::_build_output_block(Block* origin_block, Block*
output_block) {
auto is_mem_reuse = output_block->mem_reuse();
-MutableBlock mutable_block =
-is_mem_reuse
-? MutableBlock(output_block)
-:
MutableBlock(VectorizedUtils::create_empty_columnswithtypename(row_desc()));
auto rows = origin_block->rows();
-// TODO: After FE plan support same nullable of output expr and origin
block and mutable column
-// we should replace `insert_column_datas` by `insert_range_from`
-
-auto insert_column_datas = [](auto& to, const auto& from, size_t rows) {
-if (to->is_nullable() && !from.is_nullable()) {
-auto& null_column = reinterpret_cast(*to);
-null_column.get_nested_column().insert_range_from(from, 0, rows);
-null_column.get_null_map_column().get_data().resize_fill(rows, 0);
-} else {
-to->insert_range_from(from, 0, rows);
-}
-};
+auto origin_col_num = origin_block->columns();
if (rows != 0) {
-auto& mutable_columns = mutable_block.mutable_columns();
+SCOPED_TIMER(_projection_timer);
+if (!is_mem_reuse) {
+ColumnsWithTypeAndName empty_columns_with_type =
+VectorizedUtils::create_empty_block(row_desc());
+MutableBlock mutable_block = MutableBlock(empty_columns_with_type);
+// If not mem reuse, it means output block is empty, so that we
create empty column for it
+// Then the empty column maybe swap to origin_block, origin_block
will reuse the column to
+// avoid call malloc and page fault
+output_block->swap(mutable_block.to_block());
+}
+Columns columns;
+// origin block id --> output block id
+// This mapping is used to help reuse the columns from output block to
reduce the page fault
+std::map col_mapping;
+Columns resusable_columns;
+size_t output_size = row_desc().num_materialized_slots();
if (_output_expr_ctxs.empty()) {
-DCHECK(mutable_columns.size() ==
row_desc().num_materialized_slots());
-for (int i = 0; i < mutable_columns.size(); ++i) {
-insert_column_datas(mutable_columns[i],
*origin_block->get_by_position(i).column,
-rows);
+for (int i = 0; i < output_size; ++i) {
+columns.emplace_back(origin_block->get_by_position(i).column);
+col_mapping[i] = i;
}
} else {
-DCHECK(mutable_columns.size() ==
row_desc().num_materialized_slots());
-SCOPED_TIMER(_projection_timer);
-for (int i = 0; i < mutable_columns.size(); ++i) {
-auto result_column_id = -1;
+for (int i = 0; i < output_size; ++i) {
+int result_column_id = -1;
RETURN_IF_ERROR(_output_expr_ctxs[i]->execute(origin_block,
&result_column_id));
-auto column_ptr =
origin_block->get_by_position(result_column_id)
-
.column->convert_to_full_column_if_const();
-insert_column_datas(mutable_columns[i], *column_ptr, rows);
+auto& column_ptr =
origin_block->get_by_position(result_column_id).column;
+LOG(INFO) << "1";
+if (!column_ptr->is_column_const()) {
Review Comment:
warning: no member named 'is_column_const' in 'doris::vectorized::IColumn';
did you mean 'is_column_array'? [clang-diagnostic-error]
```suggestion
if (!column_ptr->is_column_array()) {
```
**be/src/vec/columns/column.h:590:** 'is_column_array' declared here
```cpp
virtual bool is_column_array() const { return false; }
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #15217: [fix](memory) Fix too many repeat cause OOM
xinyiZzz commented on PR #15217: URL: https://github.com/apache/doris/pull/15217#issuecomment-1380174035 > This pr sets the repeat count to the smaller value of count and `repeat_max_num`, which may cause the result to be inconsistent with the user's expectations and there is no error prompt. Why not throw an exception when count is greater than `repeat_max_num`? throw an exception seems to be a better solution -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zbtzbtzbt commented on pull request #15876: [fix](terminal) remove echo database passwd
zbtzbtzbt commented on PR #15876: URL: https://github.com/apache/doris/pull/15876#issuecomment-1380183534 use shfmt to format. e.g. `shfmt -w tools/ssb-tools/bin/load-ssb-data.sh` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15780: [feature](multi-catalog) support clickhouse jdbc catalog
github-actions[bot] commented on PR #15780: URL: https://github.com/apache/doris/pull/15780#issuecomment-1380187343 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15832: [Feature](remote) Cooldown cold data to object storage only one replica
github-actions[bot] commented on code in PR #15832:
URL: https://github.com/apache/doris/pull/15832#discussion_r1068015594
##
be/src/io/fs/file_reader_options.h:
##
@@ -38,23 +38,25 @@
public:
// path: the path of file which will be cached
// return value: the cache path of the given file.
-virtual std::string get_cache_path(const std::string& path) const { return
""; }
+virtual std::string get_cache_path() const { return ""; }
+virtual void set_cache_path(const std::string& cache_path) {}
};
class NoCachePathPolicy : public CachePathPolicy {
public:
NoCachePathPolicy() = default;
-std::string get_cache_path(const std::string& path) const override {
return path; }
+std::string get_cache_path() const override { return ""; }
};
class SegmentCachePathPolicy : public CachePathPolicy {
public:
SegmentCachePathPolicy() = default;
-std::string get_cache_path(const std::string& path) const override {
-// the segment file path is
{rowset_dir}/{schema_hash}/{rowset_id}_{seg_num}.dat
-// cache path is: {rowset_dir}/{schema_hash}/{rowset_id}_{seg_num}/
-return path.substr(0, path.size() - 4) + "/";
-}
+
+void set_cache_path(const std::string& cache_path) { _cache_path =
cache_path; }
Review Comment:
warning: annotate this function with 'override' or (rarely) 'final'
[modernize-use-override]
```suggestion
void set_cache_path(const std::string& cache_path) override {
_cache_path = cache_path; }
```
##
be/src/olap/tablet.cpp:
##
@@ -1717,27 +1735,136 @@ Status Tablet::cooldown() {
new_rowset_meta->set_resource_id(dest_fs->resource_id());
new_rowset_meta->set_fs(dest_fs);
new_rowset_meta->set_creation_time(time(nullptr));
+
+// write remote tablet meta
+TabletMetaPB remote_tablet_meta_pb;
+_tablet_meta->to_meta_pb(true, &remote_tablet_meta_pb);
+new_rowset_meta->to_rowset_pb(remote_tablet_meta_pb.add_rs_metas());
+// upload rowset_meta to remote fs.
+RETURN_IF_ERROR(_write_remote_tablet_meta(dest_fs, remote_tablet_meta_pb));
+
RowsetSharedPtr new_rowset;
RowsetFactory::create_rowset(_schema, _tablet_path, new_rowset_meta,
&new_rowset);
std::vector to_add {std::move(new_rowset)};
std::vector to_delete {std::move(old_rowset)};
-bool has_shutdown = false;
{
std::unique_lock meta_wlock(_meta_lock);
-has_shutdown = tablet_state() == TABLET_SHUTDOWN;
-if (!has_shutdown) {
+if (tablet_state() != TABLET_SHUTDOWN) {
modify_rowsets(to_add, to_delete);
-_self_owned_remote_rowsets.insert(to_add.front());
save_meta();
}
}
-if (has_shutdown) {
-record_unused_remote_rowset(new_rowset_id, dest_fs->resource_id(),
-to_add.front()->num_segments());
-return Status::Aborted("tablet {} has shutdown", tablet_id());
+return Status::OK();
+}
+
+Status Tablet::_read_remote_tablet_meta(FileSystemSPtr fs, TabletMetaPB*
tablet_meta_pb) {
+std::string remote_meta_path =
+BetaRowset::remote_tablet_meta_path(tablet_id(),
_tablet_meta->cooldown_replica_id());
+bool exist = false;
+RETURN_IF_ERROR(fs->exists(remote_meta_path, &exist));
+if (exist) {
+IOContext io_ctx;
+io::FileReaderSPtr tablet_meta_reader;
+RETURN_IF_ERROR(fs->open_file(remote_meta_path, &tablet_meta_reader,
&io_ctx));
+if (tablet_meta_reader == nullptr) {
+return Status::InternalError("tablet_meta_reader is null");
+}
+auto file_size = tablet_meta_reader->size();
+size_t bytes_read;
+uint8_t* buf = new uint8_t[file_size];
+Slice slice(buf, file_size);
+IOContext io_ctx;
+Status st = tablet_meta_reader->read_at(0, slice, io_ctx, &bytes_read);
+if (!st.ok()) {
+tablet_meta_reader->close();
Review Comment:
warning: redefinition of 'io_ctx' [clang-diagnostic-error]
```cpp
Context io_ctx;
^
```
**be/src/olap/tablet.cpp:1769:** previous definition is here
```cpp
s(remote_meta_path, &exist));
^
```
##
be/src/io/fs/file_reader_options.h:
##
@@ -38,23 +38,25 @@ class CachePathPolicy {
public:
// path: the path of file which will be cached
// return value: the cache path of the given file.
-virtual std::string get_cache_path(const std::string& path) const { return
""; }
+virtual std::string get_cache_path() const { return ""; }
+virtual void set_cache_path(const std::string& cache_path) {}
};
class NoCachePathPolicy : public CachePathPolicy {
public:
NoCachePathPolicy() = default;
-std::string get_cache_path(const std::string& path) const override {
return path; }
+std::string get_cache_path() const override { ret
[GitHub] [doris] hello-stephen commented on pull request #15862: [feature](multi-catalog) support oracle jdbc catalog
hello-stephen commented on PR #15862: URL: https://github.com/apache/doris/pull/15862#issuecomment-1380197082 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.27 seconds load time: 488 seconds storage size: 17123040156 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112113516_clickbench_pr_78613.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15832: [Feature](remote) Cooldown cold data to object storage only one replica
github-actions[bot] commented on code in PR #15832:
URL: https://github.com/apache/doris/pull/15832#discussion_r1068039602
##
be/src/olap/tablet.cpp:
##
@@ -1717,27 +1735,135 @@ Status Tablet::cooldown() {
new_rowset_meta->set_resource_id(dest_fs->resource_id());
new_rowset_meta->set_fs(dest_fs);
new_rowset_meta->set_creation_time(time(nullptr));
+
+// write remote tablet meta
+TabletMetaPB remote_tablet_meta_pb;
+_tablet_meta->to_meta_pb(true, &remote_tablet_meta_pb);
+new_rowset_meta->to_rowset_pb(remote_tablet_meta_pb.add_rs_metas());
+// upload rowset_meta to remote fs.
+RETURN_IF_ERROR(_write_remote_tablet_meta(dest_fs, remote_tablet_meta_pb));
+
RowsetSharedPtr new_rowset;
RowsetFactory::create_rowset(_schema, _tablet_path, new_rowset_meta,
&new_rowset);
std::vector to_add {std::move(new_rowset)};
std::vector to_delete {std::move(old_rowset)};
-bool has_shutdown = false;
{
std::unique_lock meta_wlock(_meta_lock);
-has_shutdown = tablet_state() == TABLET_SHUTDOWN;
-if (!has_shutdown) {
+if (tablet_state() != TABLET_SHUTDOWN) {
modify_rowsets(to_add, to_delete);
-_self_owned_remote_rowsets.insert(to_add.front());
save_meta();
}
}
-if (has_shutdown) {
-record_unused_remote_rowset(new_rowset_id, dest_fs->resource_id(),
-to_add.front()->num_segments());
-return Status::Aborted("tablet {} has shutdown", tablet_id());
+return Status::OK();
+}
+
+Status Tablet::_read_remote_tablet_meta(FileSystemSPtr fs, TabletMetaPB*
tablet_meta_pb) {
+std::string remote_meta_path =
+BetaRowset::remote_tablet_meta_path(tablet_id(),
_tablet_meta->cooldown_replica_id());
+bool exist = false;
+RETURN_IF_ERROR(fs->exists(remote_meta_path, &exist));
+if (exist) {
+io::IOContext io_ctx;
+io::FileReaderSPtr tablet_meta_reader;
+RETURN_IF_ERROR(fs->open_file(remote_meta_path, &tablet_meta_reader,
&io_ctx));
Review Comment:
warning: no type named 'IOContext' in namespace 'doris::io'; did you mean
simply 'IOContext'? [clang-diagnostic-error]
```suggestion
mote_meta_path, &exist));IOContext
```
**be/src/olap/iterators.h:36:** 'IOContext' declared here
```cpp
struct IOContext {
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15718: [fix](schema scanner)change schema_scanner::get_next_row to get_next_block
github-actions[bot] commented on PR #15718: URL: https://github.com/apache/doris/pull/15718#issuecomment-1380226343 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15843: [vectorzied](jdbc) fix jdbc executor for get result by batch and memo…
github-actions[bot] commented on PR #15843: URL: https://github.com/apache/doris/pull/15843#issuecomment-1380233020 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 commented on pull request #15748: [Enhancement](grouping) Add a switch for users to force using alias name in group by and having clause
xy720 commented on PR #15748: URL: https://github.com/apache/doris/pull/15748#issuecomment-1380233483 Update docs and add regression test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15832: [Feature](remote) Cooldown cold data to object storage only one replica
github-actions[bot] commented on code in PR #15832:
URL: https://github.com/apache/doris/pull/15832#discussion_r1068050963
##
be/test/io/cache/remote_file_cache_test.cpp:
##
@@ -141,7 +141,11 @@ class RemoteFileCacheTest : public ::testing::Test {
EXPECT_NE("", writer.min_encoded_key().to_string());
EXPECT_NE("", writer.max_encoded_key().to_string());
-st = segment_v2::Segment::open(fs, path, 0, {}, query_schema, res);
+io::SegmentCachePathPolicy cache_policy;
+cache_policy.set_cache_path(segment_cache_path(0));
Review Comment:
warning: use of undeclared identifier 'segment_cache_path'
[clang-diagnostic-error]
```cpp
cache_policy.set_cache_path(segment_cache_path(0));
^
```
##
be/test/tools/benchmark_tool.cpp:
##
@@ -363,7 +363,11 @@ class SegmentBenchmark : public BaseBenchmark {
writer.finalize(&file_size, &index_size);
file_writer->close();
-Segment::open(fs, path, "", seg_id, {}, &_tablet_schema, res);
+io::SegmentCachePathPolicy cache_policy;
+cache_policy.set_cache_path(segment_cache_path(seg_id));
Review Comment:
warning: use of undeclared identifier 'segment_cache_path'
[clang-diagnostic-error]
```cpp
cache_policy.set_cache_path(segment_cache_path(seg_id));
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15864: [fix](sort spill) fix bug of failed to create spilled file
hello-stephen commented on PR #15864: URL: https://github.com/apache/doris/pull/15864#issuecomment-1380242924 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.88 seconds load time: 492 seconds storage size: 17121768362 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112121644_clickbench_pr_78600.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] tempestLXC commented on issue #15694: [Bug] register txn replica failed
tempestLXC commented on issue #15694: URL: https://github.com/apache/doris/issues/15694#issuecomment-1380244050 我也遇到同样的问题了 但是是在 binlog load 时出现的 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15781: [typo](docs)Update ARRAY.md
github-actions[bot] commented on PR #15781: URL: https://github.com/apache/doris/pull/15781#issuecomment-1380258364 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15821: [Feature-WIP](inverted index) inverted index writer's implementation
github-actions[bot] commented on PR #15821: URL: https://github.com/apache/doris/pull/15821#issuecomment-1380260666 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15867: [fix](nereids)upgrade signature datatype bug
hello-stephen commented on PR #15867: URL: https://github.com/apache/doris/pull/15867#issuecomment-1380264553 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.55 seconds load time: 510 seconds storage size: 17122419051 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112123543_clickbench_pr_78912.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #15823: [Feature-WIP](inverted index) inverted index reader's implementation, and add mysql_fulltext regression case to test fulltext query
Tanya-W commented on code in PR #15823:
URL: https://github.com/apache/doris/pull/15823#discussion_r1068074792
##
be/src/vec/exec/scan/vscan_node.cpp:
##
@@ -966,6 +966,24 @@ Status VScanNode::_normalize_compound_predicate(
},
active_range);
+_compound_value_ranges.emplace_back(active_range);
+}
+} else if (TExprNodeType::MATCH_PRED == child_expr->node_type()) {
+SlotDescriptor* slot = nullptr;
+ColumnValueRangeType* range_on_slot = nullptr;
+if (_is_predicate_acting_on_slot(child_expr,
in_predicate_checker, &slot,
Review Comment:
match_checker in function `_normalize_match_predicate`
##
be/src/olap/reader.cpp:
##
@@ -452,6 +452,8 @@ void TabletReader::_init_conditions_param(const
ReaderParams& read_params) {
ColumnPredicate* predicate =
parse_to_predicate(_tablet_schema, tmp_cond,
_predicate_mem_pool.get());
if (predicate != nullptr) {
+auto predicate_params = predicate->predicate_params();
+predicate_params->value = condition.condition_values[0];
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #15823: [Feature-WIP](inverted index) inverted index reader's implementation, and add mysql_fulltext regression case to test fulltext query
Tanya-W commented on code in PR #15823:
URL: https://github.com/apache/doris/pull/15823#discussion_r1068073938
##
be/src/vec/exec/scan/vscan_node.h:
##
@@ -334,6 +334,11 @@ class VScanNode : public ExecNode {
SlotDescriptor* slot,
ColumnValueRange& range,
PushDownType* pdt);
+template
+Status _normalize_match_in_compound_predicate(vectorized::VExpr* expr,
VExprContext* expr_ctx,
Review Comment:
`_normalize_match_in_compound_predicate` means match_predicate in compound
node tree, `in` is a preposition, not means in_predicate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #15823: [Feature-WIP](inverted index) inverted index reader's implementation, and add mysql_fulltext regression case to test fulltext query
Tanya-W commented on code in PR #15823:
URL: https://github.com/apache/doris/pull/15823#discussion_r1068074792
##
be/src/vec/exec/scan/vscan_node.cpp:
##
@@ -966,6 +966,24 @@ Status VScanNode::_normalize_compound_predicate(
},
active_range);
+_compound_value_ranges.emplace_back(active_range);
+}
+} else if (TExprNodeType::MATCH_PRED == child_expr->node_type()) {
+SlotDescriptor* slot = nullptr;
+ColumnValueRangeType* range_on_slot = nullptr;
+if (_is_predicate_acting_on_slot(child_expr,
in_predicate_checker, &slot,
Review Comment:
match_checker in `_normalize_match_predicate`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15794: [bugfix](datetimev2) fix coredump when load datatime data to doris from parquet
github-actions[bot] commented on PR #15794: URL: https://github.com/apache/doris/pull/15794#issuecomment-1380291177 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15830: [enhancement](load) change transaction limit from db level to user level
hello-stephen commented on PR #15830: URL: https://github.com/apache/doris/pull/15830#issuecomment-1380299854 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.88 seconds load time: 489 seconds storage size: 17122161492 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112125817_clickbench_pr_78919.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #15823: [Feature-WIP](inverted index) inverted index reader's implementation, and add mysql_fulltext regression case to test fulltext query
Tanya-W commented on code in PR #15823:
URL: https://github.com/apache/doris/pull/15823#discussion_r1068092190
##
be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:
##
@@ -17,30 +17,288 @@
#include "olap/rowset/segment_v2/inverted_index_reader.h"
-#include "common/status.h"
+#include
+#include
+
+#include "common/config.h"
+#include "gutil/strings/strip.h"
+#include "io/fs/file_system.h"
+#include "olap/key_coder.h"
+#include "olap/rowset/segment_v2/inverted_index_compound_directory.h"
+#include "olap/rowset/segment_v2/inverted_index_compound_reader.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet_schema.h"
+#include "olap/utils.h"
+#include "runtime/string_value.h"
+#include "util/time.h"
namespace doris {
namespace segment_v2 {
+bool InvertedIndexReader::_is_match_query(InvertedIndexQueryType query_type) {
+return (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_ALL_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_PHRASE_QUERY);
+}
+
bool InvertedIndexReader::indexExists(io::Path& index_file_path) {
bool exists = false;
RETURN_IF_ERROR(_fs->exists(index_file_path, &exists));
return exists;
}
+std::vector FullTextIndexReader::get_analyse_result(
+const std::wstring& field_name, const std::wstring& value,
+InvertedIndexQueryType query_type, InvertedIndexParserType
analyser_type) {
+std::vector analyse_result;
+std::shared_ptr analyzer;
+if (analyser_type == InvertedIndexParserType::PARSER_STANDARD) {
Review Comment:
here reader default analyzer is SimpleAnalyzer, writer default analyzer is
also SimpleAnalyzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #15823: [Feature-WIP](inverted index) inverted index reader's implementation, and add mysql_fulltext regression case to test fulltext query
Tanya-W commented on code in PR #15823:
URL: https://github.com/apache/doris/pull/15823#discussion_r1068093312
##
be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:
##
@@ -17,30 +17,288 @@
#include "olap/rowset/segment_v2/inverted_index_reader.h"
-#include "common/status.h"
+#include
+#include
+
+#include "common/config.h"
+#include "gutil/strings/strip.h"
+#include "io/fs/file_system.h"
+#include "olap/key_coder.h"
+#include "olap/rowset/segment_v2/inverted_index_compound_directory.h"
+#include "olap/rowset/segment_v2/inverted_index_compound_reader.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet_schema.h"
+#include "olap/utils.h"
+#include "runtime/string_value.h"
+#include "util/time.h"
namespace doris {
namespace segment_v2 {
+bool InvertedIndexReader::_is_match_query(InvertedIndexQueryType query_type) {
+return (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_ALL_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_PHRASE_QUERY);
+}
+
bool InvertedIndexReader::indexExists(io::Path& index_file_path) {
bool exists = false;
RETURN_IF_ERROR(_fs->exists(index_file_path, &exists));
return exists;
}
+std::vector FullTextIndexReader::get_analyse_result(
+const std::wstring& field_name, const std::wstring& value,
+InvertedIndexQueryType query_type, InvertedIndexParserType
analyser_type) {
+std::vector analyse_result;
+std::shared_ptr analyzer;
+if (analyser_type == InvertedIndexParserType::PARSER_STANDARD) {
+analyzer =
std::make_shared();
+} else {
+// default
+analyzer = std::make_shared>();
+}
+
+std::unique_ptr reader(
+new lucene::util::StringReader(value.c_str()));
+std::unique_ptr token_stream(
+analyzer->tokenStream(field_name.c_str(), reader.get()));
+
+lucene::analysis::Token token;
+
+while (token_stream->next(&token)) {
+std::string tk =
+lucene::util::Misc::toString(token.termBuffer(),
token.termLength());
+analyse_result.emplace_back(tk);
+}
+
+if (token_stream != nullptr) {
+token_stream->close();
+}
+
+if (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_ALL_QUERY) {
+std::set unrepeated_result(analyse_result.begin(),
analyse_result.end());
+analyse_result.assign(unrepeated_result.begin(),
unrepeated_result.end());
+}
+
+return analyse_result;
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #15823: [Feature-WIP](inverted index) inverted index reader's implementation, and add mysql_fulltext regression case to test fulltext query
Tanya-W commented on code in PR #15823:
URL: https://github.com/apache/doris/pull/15823#discussion_r1068095484
##
be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:
##
@@ -17,30 +17,288 @@
#include "olap/rowset/segment_v2/inverted_index_reader.h"
-#include "common/status.h"
+#include
+#include
+
+#include "common/config.h"
+#include "gutil/strings/strip.h"
+#include "io/fs/file_system.h"
+#include "olap/key_coder.h"
+#include "olap/rowset/segment_v2/inverted_index_compound_directory.h"
+#include "olap/rowset/segment_v2/inverted_index_compound_reader.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet_schema.h"
+#include "olap/utils.h"
+#include "runtime/string_value.h"
+#include "util/time.h"
namespace doris {
namespace segment_v2 {
+bool InvertedIndexReader::_is_match_query(InvertedIndexQueryType query_type) {
+return (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_ALL_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_PHRASE_QUERY);
+}
+
bool InvertedIndexReader::indexExists(io::Path& index_file_path) {
bool exists = false;
RETURN_IF_ERROR(_fs->exists(index_file_path, &exists));
return exists;
}
+std::vector FullTextIndexReader::get_analyse_result(
+const std::wstring& field_name, const std::wstring& value,
+InvertedIndexQueryType query_type, InvertedIndexParserType
analyser_type) {
+std::vector analyse_result;
+std::shared_ptr analyzer;
+if (analyser_type == InvertedIndexParserType::PARSER_STANDARD) {
+analyzer =
std::make_shared();
+} else {
+// default
+analyzer = std::make_shared>();
+}
+
+std::unique_ptr reader(
+new lucene::util::StringReader(value.c_str()));
+std::unique_ptr token_stream(
+analyzer->tokenStream(field_name.c_str(), reader.get()));
+
+lucene::analysis::Token token;
+
+while (token_stream->next(&token)) {
+std::string tk =
+lucene::util::Misc::toString(token.termBuffer(),
token.termLength());
+analyse_result.emplace_back(tk);
+}
+
+if (token_stream != nullptr) {
+token_stream->close();
+}
+
+if (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_ALL_QUERY) {
+std::set unrepeated_result(analyse_result.begin(),
analyse_result.end());
+analyse_result.assign(unrepeated_result.begin(),
unrepeated_result.end());
+}
+
+return analyse_result;
+}
+
+Status FullTextIndexReader::new_iterator(const TabletIndex* index_meta,
+ InvertedIndexIterator** iterator) {
+*iterator = new InvertedIndexIterator(index_meta, this);
+return Status::OK();
+}
+
+Status FullTextIndexReader::query(const std::string& column_name, const void*
query_value,
+ InvertedIndexQueryType query_type,
+ InvertedIndexParserType analyser_type,
+ roaring::Roaring* bit_map) {
+std::string search_str = reinterpret_cast(query_value)->to_string();
+LOG(INFO) << column_name
Review Comment:
changed to debug log
##
be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:
##
@@ -17,30 +17,288 @@
#include "olap/rowset/segment_v2/inverted_index_reader.h"
-#include "common/status.h"
+#include
+#include
+
+#include "common/config.h"
+#include "gutil/strings/strip.h"
+#include "io/fs/file_system.h"
+#include "olap/key_coder.h"
+#include "olap/rowset/segment_v2/inverted_index_compound_directory.h"
+#include "olap/rowset/segment_v2/inverted_index_compound_reader.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet_schema.h"
+#include "olap/utils.h"
+#include "runtime/string_value.h"
+#include "util/time.h"
namespace doris {
namespace segment_v2 {
+bool InvertedIndexReader::_is_match_query(InvertedIndexQueryType query_type) {
+return (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_ALL_QUERY ||
+query_type == InvertedIndexQueryType::MATCH_PHRASE_QUERY);
+}
+
bool InvertedIndexReader::indexExists(io::Path& index_file_path) {
bool exists = false;
RETURN_IF_ERROR(_fs->exists(index_file_path, &exists));
return exists;
}
+std::vector FullTextIndexReader::get_analyse_result(
+const std::wstring& field_name, const std::wstring& value,
+InvertedIndexQueryType query_type, InvertedIndexParserType
analyser_type) {
+std::vector analyse_result;
+std::shared_ptr analyzer;
+if (analyser_type == InvertedIndexParserType::PARSER_STANDARD) {
+analyzer =
std::make_shared();
+} else {
+// default
+analyzer = std::make_shared>();
+}
+
+std::unique_ptr reader(
+ne
[GitHub] [doris] github-actions[bot] commented on pull request #14531: [improvement](meta) make database,table,column names to support unicode (replace PR #13467 with this)
github-actions[bot] commented on PR #14531: URL: https://github.com/apache/doris/pull/14531#issuecomment-1380315382 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #15823: [Feature-WIP](inverted index) inverted index reader's implementation, and add mysql_fulltext regression case to test fulltext query
Tanya-W commented on code in PR #15823:
URL: https://github.com/apache/doris/pull/15823#discussion_r1068100535
##
be/src/olap/rowset/segment_v2/segment_iterator.cpp:
##
@@ -580,6 +581,10 @@ Status
SegmentIterator::_apply_index_except_leafnode_of_andnode() {
}
if (!res.ok()) {
+if (res.code() == ErrorCode::INVERTED_INDEX_FILE_NOT_FOUND &&
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] tempestLXC opened a new issue, #15877: [Bug] 通过canal订阅binlog , 同步数据不生效
tempestLXC opened a new issue, #15877: URL: https://github.com/apache/doris/issues/15877 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.2.0 ### What's Wrong? mysql 表: create table child.source_test ( id int auto_increment comment '111' primary key, name int not null comment '111' ); doris表: CREATE TABLE `test_child_ods`.`target_test` ( `id` int(11), `name` int(11) ) ENGINE=OLAP UNIQUE KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS 8; SYNC JOB: CREATE SYNC `test_child_ods`.`test_child_job` ( FROM `child`.`source_test` INTO `target_test` (id,name) ) FROM BINLOG ( "type"="canal", "canal.server.ip"="xx.xx.xx.xx", "canal.server.port"="1", "canal.destination"="test_child_job", "canal.username"="", "canal.password"="", "canal.debug"="true" ); mysql侧新增一行数据, doris fe日志中打印了订阅表的 binlog日志. 但是doris对应的表没有更新成功. fe 日志: 2023-01-12 20:58:56,584 INFO (Thread-68|113) [ReportHandler.taskReport():376] finished to handle task report from backend 11002, diff task num: 0. cost: 0 ms 2023-01-12 20:58:56,584 INFO (thrift-server-pool-1|151) [ReportHandler.handleReport():179] receive report from be 11002. type: TASK, current queue size: 1 2023-01-12 20:58:59,576 INFO (Thread-68|113) [ReportHandler.taskReport():376] finished to handle task report from backend 11003, diff task num: 0. cost: 0 ms 2023-01-12 20:58:59,576 INFO (thrift-server-pool-5|219) [ReportHandler.handleReport():179] receive report from be 11003. type: TASK, current queue size: 1 2023-01-12 20:59:01,575 INFO (tablet checker|30) [TabletChecker.checkTablets():331] finished to check tablets. unhealth/total/added/in_sched/not_ready: 0/457/0/0/0, cost: 0 ms 2023-01-12 20:59:02,267 INFO (Thread-139|902) [CanalUtils.printSummary():72] --- Batch Summary --> | Batch Id: [33038] ,count : [2] , Mem size : [106] , Time : 2023-01-12 20:59:02 | Start : [mysql-bin.001015:49702262:1673528342000(2023-01-12 20:59:02)] | End : [mysql-bin.001015:49702491:1673528342000(2023-01-12 20:59:02)] -- 2023-01-12 20:59:03,564 INFO (Thread-68|113) [ReportHandler.taskReport():376] finished to handle task report from backend 11001, diff task num: 0. cost: 0 ms 2023-01-12 20:59:03,564 INFO (thrift-server-pool-12|874) [ReportHandler.handleReport():179] receive report from be 11001. type: TASK, current queue size: 1 2023-01-12 20:59:05,438 INFO (thrift-server-pool-7|228) [DatabaseTransactionMgr.commitTransaction():664] transaction:[TransactionState. transaction id: 199223, label: 978ec977c304cc9-811adb1c2311872e, db id: 14994, table id list: 15867, callback id: 15909, coordinator: FE: 172.19.107.56, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1673528335222, commit time: 1673528345437, finish time: -1, reason: attactment: RLTaskTxnCommitAttachment [filteredRows=0, loadedRows=399, unselectedRows=0, receivedBytes=150595, taskExecutionTimeMs=10215, taskId=TUniqueId(hi:682555478914387145, lo:-9143755179528452306), jobId=15909, progress=KafkaProgress [partitionIdToOffset=0_55065579]]] successfully committed 2023-01-12 20:59:05,439 INFO (PUBLISH_VERSION|20) [PublishVersionDaemon.publishVersion():134] send publish tasks for transaction: 199223 2023-01-12 20:59:05,450 INFO (PUBLISH_VERSION|20) [DatabaseTransactionMgr.finishTransaction():962] finish transaction TransactionState. transaction id: 199223, label: 978ec977c304cc9-811adb1c2311872e, db id: 14994, table id list: 15867, callback id: 15909, coordinator: FE: 172.19.107.56, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1673528335222, commit time: 1673528345437, finish time: 1673528345449, reason: attactment: RLTaskTxnCommitAttachment [filteredRows=0, loadedRows=399, unselectedRows=0, receivedBytes=150595, taskExecutionTimeMs=10215, taskId=TUniqueId(hi:682555478914387145, lo:-9143755179528452306), jobId=15909, progress=KafkaProgress [partitionIdToOffset=0_55065579]] successfully 2023-01-12 20:59:06,330 INFO (Thread-139|902) [CanalUtils.printSummary():72] --- Batch Summary --> | Batch Id: [33039] ,count : [3] , Mem size : [146] , Time : 2023-01-12 20:59:06 | Start : [mysql-bin.001015:49719002:1673528345000(2023-01-12 20:59:05)] | End : [mysql-bin.001015:49719173:1673528345000(2023-01-12 20:59:05)] -- 2023-01-12 20:59:06,330 INFO (Thread-139|902) [CanalUtils.printRow():123] > binlog[mysql-bin.001015:49719131] , name[child,source_test] , eventType : INSERT , executeTime : 167
[GitHub] [doris] yixiutt commented on a diff in pull request #15749: [improvement](reader) use union merge when rowset are noneoverlapping
yixiutt commented on code in PR #15749:
URL: https://github.com/apache/doris/pull/15749#discussion_r1068116639
##
be/src/vec/olap/vcollect_iterator.cpp:
##
@@ -129,6 +132,19 @@ Status
VCollectIterator::build_heap(std::vector& rs_reade
new Level1Iterator(_children, _reader, _merge,
_is_reverse, _skip_same));
}
} else {
+bool have_multiple_child = false;
+bool is_first_child = true;
+for (auto iter = _children.begin(); iter != _children.end();) {
+auto s = (*iter)->init_for_union(is_first_child,
have_multiple_child);
+if (!s.ok()) {
+delete (*iter);
+iter = _children.erase(iter);
+} else {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15749: [improvement](reader) use union merge when rowset are noneoverlapping
github-actions[bot] commented on PR #15749: URL: https://github.com/apache/doris/pull/15749#issuecomment-1380358681 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15801: [Enhancement](export) cancel all running coordinators when execute ca…
hello-stephen commented on PR #15801: URL: https://github.com/apache/doris/pull/15801#issuecomment-1380371548 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.68 seconds load time: 490 seconds storage size: 17123207626 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112133944_clickbench_pr_78771.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 opened a new pull request, #15878: [fix](planner) Keep type of null literal expr when register conjuncts
Kikyou1997 opened a new pull request, #15878: URL: https://github.com/apache/doris/pull/15878 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15871: [refactor](rpc fn) decouple vectorized remote function from row-based one
hello-stephen commented on PR #15871: URL: https://github.com/apache/doris/pull/15871#issuecomment-1380386649 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.12 seconds load time: 493 seconds storage size: 17122714651 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112135149_clickbench_pr_78786.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju opened a new pull request, #15879: [fix](struct-type) struct create failed fix
cambyzju opened a new pull request, #15879: URL: https://github.com/apache/doris/pull/15879 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #15876: [fix](terminal) remove echo database passwd
adonis0147 commented on code in PR #15876:
URL: https://github.com/apache/doris/pull/15876#discussion_r1068146932
##
tools/ssb-tools/bin/gen-ssb-data.sh:
##
@@ -126,7 +126,7 @@ date
cd -
# move data to $SSB_DATA_DIR
-mv "${SSB_DBGEN_DIR}"/*.tbl* "${SSB_DATA_DIR}/"
+mv "${SSB_DBGEN_DIR}"*.tbl* "${SSB_DATA_DIR}/"
Review Comment:
`"${SSB_DBGEN_DIR}"*.tbl*` looks weird. I think the original one is better.
Maybe we should remove the `/` from the assignment of `SSB_DBGEN_DIR` in
[#L32](https://github.com/apache/doris/blob/f8f4cba64f8e75910b0a447c1813a2203e5c99b4/tools/ssb-tools/bin/gen-ssb-data.sh#L32).
##
tools/ssb-tools/bin/gen-ssb-data.sh:
##
@@ -126,7 +126,7 @@ date
cd -
# move data to $SSB_DATA_DIR
-mv "${SSB_DBGEN_DIR}"/*.tbl* "${SSB_DATA_DIR}/"
+mv "${SSB_DBGEN_DIR}"*.tbl* "${SSB_DATA_DIR}/"
# check data
-du -sh "${SSB_DATA_DIR}"/*.tbl*
+du -sh "${SSB_DATA_DIR}"*.tbl*
Review Comment:
Ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15879: [fix](struct-type) struct create failed fix
github-actions[bot] commented on PR #15879: URL: https://github.com/apache/doris/pull/15879#issuecomment-1380393881 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15879: [fix](struct-type) struct create failed fix
github-actions[bot] commented on PR #15879: URL: https://github.com/apache/doris/pull/15879#issuecomment-1380394160 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #15807: [Update](deps) add clucene deps for doris inverted index
morningman merged PR #15807: URL: https://github.com/apache/doris/pull/15807 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
