[GitHub] [doris] morningman opened a new pull request, #14016: [improvement](profile) support ordinary user to get query profile via http api
morningman opened a new pull request, #14016:
URL: https://github.com/apache/doris/pull/14016
# Proposed changes
Issue Number: close #xxx
## Problem summary
In previous implementation, the following http api can only be used by admin
or root user:
```
GET /rest/v2/manager/query/query_info
GET /rest/v2/manager/query/trace/{trace_id}
GET /rest/v2/manager/query/sql/{query_id}
GET /rest/v2/manager/query/profile/text/{query_id}
GET /rest/v2/manager/query/profile/graph/{query_id}
GET /rest/v2/manager/query/profile/json/{query_id}
GET /rest/v2/manager/query/profile/fragments/{query_id}
```
This pr change the behavior, that ordinary users can get profile of their
own queries.
And if no auth, `bad request` will be returned.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz opened a new pull request, #14015: [feature-wip](multi-catalog) fix page index
wsjz opened a new pull request, #14015: URL: https://github.com/apache/doris/pull/14015 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate after push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r1015076884
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicates.java:
##
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * infer additional predicates for `LogicalFilter` and `LogicalJoin`.
+ */
+public class InferPredicates extends DefaultPlanRewriter {
+PredicatePropagation propagation = new PredicatePropagation();
+PullUpPredicates pollUpPredicates = new PullUpPredicates();
+
+/**
+ * The logic is as follows:
+ * 1. poll up bottom predicate then infer additional predicates
+ * for example:
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id =
t2.id
+ * 1. poll up bottom predicate
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id
= t2.id and t.id = 1
+ * 2. infer
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id
= t2.id and t.id = 1 and t2.id = 1
+ * finally transformed sql:
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id
= t2.id and t2.id = 1
+ * 2. put these predicates into `otherJoinConjuncts` , these predicates
are processed in the next
+ * round of predicate push-down
Review Comment:
Good comment :D, move before the class definition.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/rewrite/VisitorRewriteJob.java:
##
@@ -0,0 +1,56 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.jobs.rewrite;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.jobs.Job;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.jobs.JobType;
+import org.apache.doris.nereids.memo.Group;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+
+import java.util.Objects;
+
+/**
+ * Use visitor to rewrite the plan.
+ */
+public class VisitorRewriteJob extends Job {
+private final Group group;
+
+private final DefaultPlanRewriter planRewriter;
+
+/**
+ * Constructor.
+ */
+public VisitorRewriteJob(CascadesContext cascadesContext,
DefaultPlanRewriter rewriter, boolean once) {
+super(JobType.VISITOR_REWRITE, cascadesContext.getCurrentJobContext(),
once);
+this.group =
Objects.requireNonNull(cascadesContext.getMemo().getRoot(), "group cannot be
null");
+this.planRewriter = Objects.requireNonNull(rewriter, "planRewriter
cannot be null");
+}
+
+@Override
+public void execute() {
+GroupExpression logicalExpressio
[doris] annotated tag 1.1.4-rc01 updated (07344af06a -> 4422fe1579)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to annotated tag 1.1.4-rc01 in repository https://gitbox.apache.org/repos/asf/doris.git *** WARNING: tag 1.1.4-rc01 was modified! *** from 07344af06a (commit) to 4422fe1579 (tag) tagging 07344af06a6e572cf734cff69d3afce267111bbe (commit) replaces 1.1.1-rc03 by yiguolei on Mon Nov 7 16:08:10 2022 +0800 - Log - 1.1.4 rc01 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue opened a new pull request, #14017: [docs](odbc) fix docs for sqlserver odbc table
nextdreamblue opened a new pull request, #14017: URL: https://github.com/apache/doris/pull/14017 Signed-off-by: nextdreamblue # Proposed changes Issue Number: close #xxx ## Problem summary fix docs for sqlserver odbc table ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate after push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r1015094281
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PredicatePropagation.java:
##
@@ -0,0 +1,102 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import
org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * derive additional predicates.
+ * for example:
+ * a = b and a = 1 => b = 1
+ */
+public class PredicatePropagation {
+
+/**
+ * infer additional predicates.
+ */
+public Set infer(Set predicates) {
+Set inferred = Sets.newHashSet();
+for (Expression predicate : predicates) {
+if (canEquivalentInfer(predicate)) {
+List newInferred = predicates.stream()
+.filter(p -> !p.equals(predicate))
+.map(p -> doInfer(predicate, p))
+.collect(Collectors.toList());
+inferred.addAll(newInferred);
+}
+}
+inferred.removeAll(predicates);
+return inferred;
+}
+
+/**
+ * Use the left or right child of `leftSlotEqualToRightSlot` to replace
the left or right child of `expression`
+ * Now only support infer `ComparisonPredicate`.
+ * TODO: We should determine whether `expression` satisfies the condition
for replacement
+ * eg: Satisfy `expression` is non-deterministic
+ */
+private Expression doInfer(Expression leftSlotEqualToRightSlot, Expression
expression) {
+return expression.accept(new DefaultExpressionRewriter() {
+
+@Override
+public Expression visit(Expression expr, Void context) {
+return expr;
+}
+
+@Override
+public Expression visitComparisonPredicate(ComparisonPredicate cp,
Void context) {
+if (!cp.left().isConstant() && !cp.right().isConstant()) {
+return cp;
+}
+return super.visit(cp, context);
+}
+
+@Override
+public Expression visitSlotReference(SlotReference slotReference,
Void context) {
+if (slotReference.equals(leftSlotEqualToRightSlot.child(0))) {
+return leftSlotEqualToRightSlot.child(1);
+} else if
(slotReference.equals(leftSlotEqualToRightSlot.child(1))) {
+return leftSlotEqualToRightSlot.child(0);
+} else {
+return slotReference;
+}
+}
Review Comment:
Declare the shape in the visitComparisonPredicate will more clearer.
```java
@Override
public Expression visitComparisonPredicate(ComparisonPredicate cp, Void
context) {
if (cp.left().isSlot() && cp.right().isConstant()) {
return swapSlot(cp);
} else if (cp.left().isConstant() && cp.right().isSlot()) {
return swapSlot(cp);
}
return super.visit(cp, context);
}
private Expression swapSlot(Expression expr) {
return expr.rewriteUp(e -> {
if (e.equals(leftSlotEqualToRightSlot.child(0)) {
return leftSlotEqualToRightSlot.child(1);
} else if (e.equals(leftSlotEqualToRightSlot.child(1)) {
return leftSlotEqualToRightSlot.child(0);
} else {
return e;
}
});
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us
[doris] annotated tag 1.1.4-rc01 updated (8890a58dc8 -> fe5a3e126e)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to annotated tag 1.1.4-rc01 in repository https://gitbox.apache.org/repos/asf/doris.git *** WARNING: tag 1.1.4-rc01 was modified! *** from 8890a58dc8 (commit) to fe5a3e126e (tag) tagging 8890a58dc812379569d3b188687fd98c1b5b6b0c (commit) replaces 1.1.1-rc03 by yiguolei on Mon Nov 7 16:11:52 2022 +0800 - Log - 1.1.4 rc01 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #14014: [enhancement](load) shrink reserved buffer for page builder (#14012)
zhannngchen commented on code in PR #14014:
URL: https://github.com/apache/doris/pull/14014#discussion_r1015120327
##
be/src/olap/rowset/segment_v2/column_writer.cpp:
##
@@ -274,6 +274,7 @@ Status ScalarColumnWriter::init() {
// create page builder
PageBuilderOptions opts;
opts.data_page_size = _opts.data_page_size;
+opts.dict_page_size = _opts.data_page_size; // init smaller dict page,
grow if need more
Review Comment:
We can leave `dict_page_size` as 1M, and not to change the `data_page_size`
when we initialize the `_dict_builder`. So you can reserve the faststring with
`min(data_page_size, dict_page_size)` and use `dict_page_size` to check if the
page is full.
##
be/src/olap/rowset/segment_v2/binary_plain_page.h:
##
@@ -158,6 +166,7 @@ class BinaryPlainPageBuilder : public PageBuilder {
uint32_t _last_value_size = 0;
faststring _first_value;
faststring _last_value;
+bool _is_dict_page;
Review Comment:
move it to `PageBuilderOptions` is better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] elisondba2022 commented on issue #5796: flinksql doris insert exception:stream load error: too many filtered rows
elisondba2022 commented on issue #5796: URL: https://github.com/apache/doris/issues/5796#issuecomment-1305249013 Caused by: org.apache.flink.util.SerializedThrowable: stream load error: too many filtered rows, see more in http://192.168.168.215:8040/api/_load_error_log?file=__shard_25/error_log_insert_stmt_874b6649b376fa9b-d6cb775095dfb4b1_874b6649b376fa9b_d6cb775095dfb4b1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lit2430 opened a new issue, #14020: [Bug] errCode = 2, detailMessage = Unknown column 'E' in 'table list'
lit2430 opened a new issue, #14020: URL: https://github.com/apache/doris/issues/14020 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 0.15.3 ### What's Wrong? When writing data, the imported data field is Scientific notation. When the field type in doris is decimal, tell me error "detailMessage = Unknown column 'E' in 'table list'"; https://user-images.githubusercontent.com/91309604/200253883-f7f220a7-7969-4049-a328-ca33326fe0ff.png";> ### What You Expected? this error Should be tell me "Type conversion exception" ?instead of "errCode = 2, detailMessage = Unknown column 'E' in 'table list'" ### How to Reproduce? [issure.md](https://github.com/apache/doris/files/9949453/issure.md) ### Anything Else? no ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #14014: [enhancement](load) shrink reserved buffer for page builder (#14012)
freemandealer commented on code in PR #14014:
URL: https://github.com/apache/doris/pull/14014#discussion_r1015127547
##
be/src/olap/rowset/segment_v2/binary_plain_page.h:
##
@@ -158,6 +166,7 @@ class BinaryPlainPageBuilder : public PageBuilder {
uint32_t _last_value_size = 0;
faststring _first_value;
faststring _last_value;
+bool _is_dict_page;
Review Comment:
indeed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #13430: [feature](inverted index)WIP inverted index api: SQL syntax and metadata
xiaokang commented on code in PR #13430:
URL: https://github.com/apache/doris/pull/13430#discussion_r1015131165
##
fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java:
##
@@ -126,29 +153,37 @@ public boolean isSetIfNotExists() {
public enum IndexType {
BITMAP,
+INVERTED,
+}
+public boolean isInvertedIndex() {
+return (this.indexType == IndexType.INVERTED);
}
public void checkColumn(Column column, KeysType keysType) throws
AnalysisException {
-if (indexType == IndexType.BITMAP) {
+if (indexType == IndexType.BITMAP || indexType == IndexType.INVERTED) {
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #13430: [feature](inverted index)WIP inverted index api: SQL syntax and metadata
xiaokang commented on code in PR #13430:
URL: https://github.com/apache/doris/pull/13430#discussion_r1015132859
##
fe/fe-core/src/main/java/org/apache/doris/catalog/TableIndexes.java:
##
@@ -56,6 +58,26 @@ public TableIndexes(List indexes, Map
properties) {
this.properties = properties;
}
+private synchronized int incAndGetMaxIndexUniqueId() {
Review Comment:
change to Env.getCurrentEnv().getNextId()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #13430: [feature](inverted index)WIP inverted index api: SQL syntax and metadata
xiaokang commented on code in PR #13430:
URL: https://github.com/apache/doris/pull/13430#discussion_r1015134047
##
fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java:
##
@@ -48,12 +52,18 @@ public IndexDef(String indexName, boolean ifNotExists,
List columns, Ind
} else {
this.comment = comment;
}
+if (properties == null) {
+this.properties = new HashMap<>();
+} else {
+this.properties = properties;
+}
}
public void analyze() throws AnalysisException {
-if (indexType == IndexDef.IndexType.BITMAP) {
+if (indexType == IndexDef.IndexType.BITMAP
+|| indexType == IndexDef.IndexType.INVERTED) {
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 closed pull request #13771: [new] To complete the mysql、hive、pgsql external table to doris by shell
Toms1999 closed pull request #13771: [new] To complete the mysql、hive、pgsql external table to doris by shell URL: https://github.com/apache/doris/pull/13771 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 opened a new pull request, #14023: [extension]new e_table_to_doris by shell
Toms1999 opened a new pull request, #14023: URL: https://github.com/apache/doris/pull/14023 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #13681: [enhancement](Nereids) support otherJoinConjuncts in cascades join reorder
morrySnow commented on code in PR #13681:
URL: https://github.com/apache/doris/pull/13681#discussion_r1015133792
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java:
##
@@ -111,14 +117,26 @@ public Rule build() {
}
}
// replace hashJoinConjuncts
-newBottomHashJoinConjuncts =
JoinUtils.replaceHashConjuncts(
+newBottomHashJoinConjuncts =
JoinUtils.replaceJoinConjuncts(
newBottomHashJoinConjuncts, outputToInput);
-newTopHashJoinConjuncts = JoinUtils.replaceHashConjuncts(
+newTopHashJoinConjuncts = JoinUtils.replaceJoinConjuncts(
newTopHashJoinConjuncts, inputToOutput);
-// Add all slots used by hashOnCondition when projects not
empty.
-// TODO: Does nonHashOnCondition also need to be
considered.
-Map> abOnUsedSlots =
newTopHashJoinConjuncts.stream()
+// replace otherJoinConjuncts
+newBottomOtherJoinConjuncts =
JoinUtils.replaceJoinConjuncts(
+newBottomOtherJoinConjuncts, outputToInput);
+newTopOtherJoinConjuncts = JoinUtils.replaceJoinConjuncts(
+newTopOtherJoinConjuncts, inputToOutput);
+
+if (newBottomHashJoinConjuncts == null ||
newTopHashJoinConjuncts == null
+|| newBottomOtherJoinConjuncts == null ||
newTopOtherJoinConjuncts == null) {
Review Comment:
why prohibit other join conjuncts is null
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscom.java:
##
@@ -52,34 +52,38 @@ public class InnerJoinLAsscom extends
OneExplorationRuleFactory {
public Rule build() {
return innerLogicalJoin(innerLogicalJoin(), group())
.when(topJoin -> checkReorder(topJoin, topJoin.left()))
-// TODO: handle otherJoinCondition
-.when(topJoin -> topJoin.getOtherJoinConjuncts().isEmpty())
-.when(topJoin ->
topJoin.left().getOtherJoinConjuncts().isEmpty())
.then(topJoin -> {
LogicalJoin bottomJoin =
topJoin.left();
GroupPlan a = bottomJoin.left();
GroupPlan b = bottomJoin.right();
GroupPlan c = topJoin.right();
// split HashJoinConjuncts.
-Map> splitOn =
splitHashConjuncts(topJoin.getHashJoinConjuncts(),
-bottomJoin);
-List newTopHashConjuncts = splitOn.get(true);
-List newBottomHashConjuncts =
splitOn.get(false);
+Map> splitHashConjunts =
splitConjuncts(topJoin.getHashJoinConjuncts(),
+bottomJoin, bottomJoin.getHashJoinConjuncts());
+List newTopHashConjuncts =
splitHashConjunts.get(true);
+List newBottomHashConjuncts =
splitHashConjunts.get(false);
+Preconditions.checkState(!newTopHashConjuncts.isEmpty(),
+"LAsscom newTopHashJoinConjuncts join can't
empty");
if (newBottomHashConjuncts.size() == 0) {
return null;
}
-// TODO: split otherCondition.
+// split OtherJoinConjuncts.
+Map> splitOtherConjunts =
splitConjuncts(topJoin.getOtherJoinConjuncts(),
+bottomJoin, bottomJoin.getOtherJoinConjuncts());
+List newTopOtherConjuncts =
splitOtherConjunts.get(true);
+List newBottomOtherConjuncts =
splitOtherConjunts.get(false);
Review Comment:
chould these two list be **null**?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscomProject.java:
##
@@ -117,14 +131,25 @@ public Rule build() {
}
}
// replace hashJoinConjuncts
-newBottomHashJoinConjuncts =
JoinUtils.replaceHashConjuncts(
+newBottomHashJoinConjuncts =
JoinUtils.replaceJoinConjuncts(
newBottomHashJoinConjuncts, outputToInput);
-newTopHashJoinConjuncts = JoinUtils.replaceHashConjuncts(
+newTopHashJoinConjuncts = JoinUtils.replaceJoinConjuncts(
newTopHashJoinConjuncts, inputToOutput);
+// replace otherJoinConjuncts
+newBottomOtherJoinConjuncts =
JoinUtils.replaceJoinConjuncts(
+newBottomOtherJoinConjuncts, outputToInput);
+newTopOtherJoinConjunc
[GitHub] [doris] Toms1999 closed pull request #14023: [extension]new e_table_to_doris by shell
Toms1999 closed pull request #14023: [extension]new e_table_to_doris by shell URL: https://github.com/apache/doris/pull/14023 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 opened a new pull request, #14025: [extension] e_table_to_doris by shell
Toms1999 opened a new pull request, #14025: URL: https://github.com/apache/doris/pull/14025 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #14010: [typo](docs)fix config doc
yiguolei merged PR #14010: URL: https://github.com/apache/doris/pull/14010 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](docs)fix config doc #14010
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 0031304015 [typo](docs)fix config doc #14010 0031304015 is described below commit 00313040159c0b659380b57f90bc27372c58c43c Author: zy-kkk AuthorDate: Mon Nov 7 17:00:16 2022 +0800 [typo](docs)fix config doc #14010 --- docs/en/docs/admin-manual/config/be-config.md| 4 ++-- docs/zh-CN/docs/admin-manual/config/be-config.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/en/docs/admin-manual/config/be-config.md b/docs/en/docs/admin-manual/config/be-config.md index d4236e6870..43b126a765 100644 --- a/docs/en/docs/admin-manual/config/be-config.md +++ b/docs/en/docs/admin-manual/config/be-config.md @@ -844,9 +844,9 @@ The number of sliced tablets, plan the layout of the tablet, and avoid too many * Description: Control gc of tcmalloc, in performance mode doirs releases memory of tcmalloc cache when usgae >= 90% * mem_limit, otherwise, doris releases memory of tcmalloc cache when usage >= 50% * mem_limit; * Default value: performance -### `memory_limitation_per_thread_for_schema_change` +### `memory_limitation_per_thread_for_schema_change_bytes` -Default: 2 （G） +Default: 2147483648 Maximum memory allowed for a single schema change task diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md b/docs/zh-CN/docs/admin-manual/config/be-config.md index 440a5bf7c8..b3d187e0d4 100644 --- a/docs/zh-CN/docs/admin-manual/config/be-config.md +++ b/docs/zh-CN/docs/admin-manual/config/be-config.md @@ -845,9 +845,9 @@ txn 管理器中每个 txn_partition_map 的最大 txns 数，这是一种自我 * 描述：控制tcmalloc的回收。如果配置为performance，内存使用超过mem_limit的90%时，doris会释放tcmalloc cache中的内存，如果配置为compact，内存使用超过mem_limit的50%时，doris会释放tcmalloc cache中的内存。 * 默认值：performance -### `memory_limitation_per_thread_for_schema_change` +### `memory_limitation_per_thread_for_schema_change_bytes` -默认值：2 （GB） +默认值：2147483648 单个schema change任务允许占用的最大内存 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13745: [enhancement](Nereids) remove unnecessary decimal cast
github-actions[bot] commented on PR #13745: URL: https://github.com/apache/doris/pull/13745#issuecomment-1305288663 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13745: [enhancement](Nereids) remove unnecessary decimal cast
github-actions[bot] commented on PR #13745: URL: https://github.com/apache/doris/pull/13745#issuecomment-1305288612 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow closed pull request #13976: [feature](Nereids) support statement having aggregate function in order by list
morrySnow closed pull request #13976: [feature](Nereids) support statement having aggregate function in order by list URL: https://github.com/apache/doris/pull/13976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (0031304015 -> 22b4c6af20)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 0031304015 [typo](docs)fix config doc #14010
add 22b4c6af20 [feature](Nereids) support statement having aggregate
function in order by list (#13976)
No new revisions were added by this update.
Summary of changes:
.../doris/nereids/jobs/batch/AnalyzeRulesJob.java | 11 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 19 +-
.../org/apache/doris/nereids/rules/RuleType.java | 9 +-
.../doris/nereids/rules/analysis/BindFunction.java | 19 +-
.../nereids/rules/analysis/BindSlotReference.java | 33 ++-
...eHaving.java => ResolveAggregateFunctions.java} | 87 ++--
.../datasets/clickbench/AnalyzeClickBenchTest.java | 237 +
.../ClickBenchTestBase.java} | 6 +-
.../datasets/clickbench/ClickBenchUtils.java | 181
...est.java => ResolveAggregateFunctionsTest.java} | 236
10 files changed, 753 insertions(+), 85 deletions(-)
rename
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/{ResolveHaving.java
=> ResolveAggregateFunctions.java} (63%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/{ssb/SSBTestBase.java
=> clickbench/ClickBenchTestBase.java} (86%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/ClickBenchUtils.java
rename
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/{HavingClauseTest.java
=> ResolveAggregateFunctionsTest.java} (60%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #13681: [enhancement](Nereids) support otherJoinConjuncts in cascades join reorder
jackwener commented on code in PR #13681:
URL: https://github.com/apache/doris/pull/13681#discussion_r1015163530
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscomProject.java:
##
@@ -89,20 +87,36 @@ public Rule build() {
Set bExprIdSet =
InnerJoinLAsscomProject.getExprIdSetForB(bottomJoin.right(),
newRightProjects);
-/* ** split HashConjuncts ** */
-Map> splitOn =
InnerJoinLAsscomProject.splitHashConjunctsWithAlias(
-topJoin.getHashJoinConjuncts(), bottomJoin,
bExprIdSet);
-List newTopHashJoinConjuncts =
splitOn.get(true);
+/* ** split Conjuncts ** */
+Map> splitHashJoinConjuncts
+= InnerJoinLAsscomProject.splitConjunctsWithAlias(
+topJoin.getHashJoinConjuncts(), bottomJoin,
bottomJoin.getHashJoinConjuncts(), bExprIdSet);
+List newTopHashJoinConjuncts =
splitHashJoinConjuncts.get(true);
+
Preconditions.checkState(!newTopHashJoinConjuncts.isEmpty(),
+"LAsscom newTopHashJoinConjuncts join can't
empty");
+// When newTopHashJoinConjuncts.size() !=
bottomJoin.getHashJoinConjuncts().size()
+// It means that topHashJoinConjuncts contain A, B, C, we
should LAsscom.
if (topJoin.getJoinType() != bottomJoin.getJoinType()
&& newTopHashJoinConjuncts.size() !=
bottomJoin.getHashJoinConjuncts().size()) {
return null;
}
-List newBottomHashJoinConjuncts =
splitOn.get(false);
+List newBottomHashJoinConjuncts =
splitHashJoinConjuncts.get(false);
if (newBottomHashJoinConjuncts.size() == 0) {
return null;
}
-/* ** replace HashConjuncts by projects **
*/
+Map> splitOtherJoinConjuncts
+= InnerJoinLAsscomProject.splitConjunctsWithAlias(
Review Comment:
This function is not universal (it is obvious in the parameters).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] catpineapple commented on pull request #13772: [feature](planner) add multi partition
catpineapple commented on PR #13772: URL: https://github.com/apache/doris/pull/13772#issuecomment-1305299819 > And please add some regression tests for this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] catpineapple closed pull request #13772: [feature](planner) add multi partition
catpineapple closed pull request #13772: [feature](planner) add multi partition URL: https://github.com/apache/doris/pull/13772 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 closed pull request #14025: [extension] e_table_to_doris by shell
Toms1999 closed pull request #14025: [extension] e_table_to_doris by shell URL: https://github.com/apache/doris/pull/14025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhouaoe closed pull request #14013: [Enhancement] Doris broker support aliyun-oss
zhouaoe closed pull request #14013: [Enhancement] Doris broker support aliyun-oss URL: https://github.com/apache/doris/pull/14013 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhouaoe opened a new pull request, #14026: [Enhancement] Doris broker support aliyun-oss #13665
zhouaoe opened a new pull request, #14026:
URL: https://github.com/apache/doris/pull/14026
# Proposed changes
Issue Number: close #13665
## Problem summary
Describe your changes.
1 Upgrade fs_broker module hadoop2.8.3->hadoop2.9.1
2 Broker support oss://
3 Version of jar file hadoop-huaweicloud used by broker is set to 2.8.3
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [x] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [x] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [x] No
- [ ] No Need
4. Does it need to update dependencies:
- [x] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [x] No
## Further comments
**1. Test case :doris broker升级hadoop2.9.1后访问obs**
验证hadoop2.9.1兼容华为obs的访问
LOAD LABEL demo.load_oss_label_hw (
DATA INFILE(
"obs://weinan-test1.obs.cn-east-3.myhuaweicloud.com/data2.csv"
) INTO TABLE example_tbl_hw COLUMNS TERMINATED BY ","
) WITH BROKER "broker_za" (
"fs.obs.access.key" = "x",
"fs.obs.secret.key" = "xxx",
"fs.obs.endpoint" = "https://obs.cn-east-3.myhuaweicloud.com";
)
测试结果符合预期，兼容obs的访问
执行结果：
后台记录：
结果检查：
**2. Test cast :Doris broker支持OSS协议测试**
broker中hadoop升级2.9.1后支持oss://的导入导出
1 Load with Broker
用例：先导入6条数据，再通过OSS导入7条数据（在原始数据基础上增加了一条）
原始数据：
1,2017-10-01,北京,20,0,2017-10-01 06:00:00,20,10,10
1,2017-10-01,北京,20,0,2017-10-01 07:00:00,15,2,2
10001,2017-10-01,北京,30,1,2017-10-01 17:05:45,2,22,22
10002,2017-10-02,上海,20,1,2017-10-02 12:59:12,200,5,5
10003,2017-10-02,广州,32,0,2017-10-02 11:20:00,30,11,11
10004,2017-10-01,深圳,35,0,2017-10-01 10:00:15,100,3,3
10004,2017-10-03,深圳,35,0,2017-10-03 10:20:22,11,6,6
OSS上的数据data.csv
1,2017-10-01,北京,20,0,2017-10-01 06:00:00,20,10,10
1,2017-10-01,北京,20,0,2017-10-01 07:00:00,15,2,2
10001,2017-10-01,北京,30,1,2017-10-01 17:05:45,2,22,22
10002,2017-10-02,上海,20,1,2017-10-02 12:59:12,200,5,5
10003,2017-10-02,广州,32,0,2017-10-02 11:20:00,30,11,11
10004,2017-10-01,深圳,35,0,2017-10-01 10:00:15,100,3,3
10004,2017-10-03,深圳,35,0,2017-10-03 10:20:22,11,6,6
10005,2017-10-03,深圳,35,0,2017-10-03 10:20:22,11,6,6
导入语句
LOAD LABEL demo.load_oss_label_1
(
DATA INFILE("oss://otsosstest/doris/data.csv")
INTO TABLE example_tbl
COLUMNS TERMINATED BY ","
)
WITH BROKER "broker_za"
(
"fs.oss.endpoint" = "https://x";,
"fs.oss.accessKeyId" = "x",
"fs.oss.accessKeySecret"="x"
)
执行结果：符合预期
1.前端执行：
2.Load任务
3.数据检查
2 Export with Borker
用例：将刚才的7条数据导入到OSS上
EXPORT TABLE demo.example_tbl
TO "oss://otsosstest/doris/export_broker/01/"
PROPERTIES
(
"label" = "export_from_doris_18",
"column_separator"=",",
"timeout" = "3600"
)
WITH BROKER "broker_za"
(
"fs.oss.endpoint" = "https://oss-cn-hangzhou.aliyuncs.com";,
"fs.oss.accessKeyId" = "x",
"fs.oss.accessKeySecret"="x"
);
测试结果符合预期
执行结果
后台执行结果
oss上导出的文件：
oss上导出的文件内容：
3 Outfile export with Broker
select * from demo.example_tbl
into outfile "oss://streamoss-1/doris/newbroker/01"
FORMAT AS CSV
PROPERTIES
(
"broker.name" = "broker_za",
"broker.fs.oss.endpoint" = "https://oss-cn-hangzhou.aliyuncs.com";,
"broker.fs.oss.accessKeyId" = "xx",
"broker.fs.oss.accessKeySecret"="xxx",
"column_separator" = ",",
"line_delimiter" =
[GitHub] [doris] pengxiangyu commented on a diff in pull request #13865: [feature](remote)Only query can use local cache when reading remote files.
pengxiangyu commented on code in PR #13865:
URL: https://github.com/apache/doris/pull/13865#discussion_r1015187159
##
be/src/olap/rowset/segment_v2/segment.h:
##
@@ -74,7 +74,8 @@ class Segment : public std::enable_shared_from_this {
uint32_t num_rows() const { return _footer.num_rows(); }
-Status new_column_iterator(const TabletColumn& tablet_column,
ColumnIterator** iter);
+Status new_column_iterator(const TabletColumn& tablet_column, const
IOContext& io_ctx,
Review Comment:
init is called just in new_column_iterator()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #14014: [enhancement](load) shrink reserved buffer for page builder (#14012)
freemandealer commented on code in PR #14014:
URL: https://github.com/apache/doris/pull/14014#discussion_r1015194633
##
be/src/olap/rowset/segment_v2/column_writer.cpp:
##
@@ -274,6 +274,7 @@ Status ScalarColumnWriter::init() {
// create page builder
PageBuilderOptions opts;
opts.data_page_size = _opts.data_page_size;
+opts.dict_page_size = _opts.data_page_size; // init smaller dict page,
grow if need more
Review Comment:
Thanks for your suggestions. The code is modified accordingly and tested
again using the config above. Everything goes okay now I hope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 opened a new pull request, #14027: [extension] new e_mysql_to_doris by shell
Toms1999 opened a new pull request, #14027: URL: https://github.com/apache/doris/pull/14027 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 closed pull request #14027: [extension] new e_mysql_to_doris by shell
Toms1999 closed pull request #14027: [extension] new e_mysql_to_doris by shell URL: https://github.com/apache/doris/pull/14027 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu opened a new issue, #14028: [Enhancement] decommission should safety drop be immediately when BE's tablets all in recycled state
dutyu opened a new issue, #14028: URL: https://github.com/apache/doris/issues/14028 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description When use decommission cmd to drop BE, some tablets on BE may be recycled, but BE's meta data holds these tablets. Because BE will be droped only when BE's meta data does not containes any tablet, so we always need to wait until trash is expired. So we can check BE's tablets, if all tablets of this BE is in recycled status, we can safety drop the BE immediately. ### Solution _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu closed issue #13579: [Enhancement] decommission should safety drop be immediately when BE's tablets all in recycled state
dutyu closed issue #13579: [Enhancement] decommission should safety drop be immediately when BE's tablets all in recycled state URL: https://github.com/apache/doris/issues/13579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #13879: [feature](nereids) Support row policy
924060929 commented on code in PR #13879:
URL: https://github.com/apache/doris/pull/13879#discussion_r1015227471
##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java:
##
@@ -43,33 +43,41 @@
public class UnboundRelation extends LogicalLeaf implements Relation, Unbound {
private final List nameParts;
+private boolean policyChecked;
+
public UnboundRelation(List nameParts) {
-this(nameParts, Optional.empty(), Optional.empty());
+this(nameParts, false);
+}
+
+public UnboundRelation(List nameParts, boolean policyChecked) {
+this(nameParts, policyChecked, Optional.empty(), Optional.empty());
}
Review Comment:
The policyChecked hacking the UnboundRelation, I can imagine that there will
be lots of field need added.
How about generate a CheckPolicy in the LogicalPlanBuilder?
And then you can write a rule to generate a new filter to the
logicalRelation, like this:
```java
logicalCheckPolicy(logicalRelation()).thenApply(ctx ->
LogicalCheckPolicy checkPolicy = ctx.root;
LogicalRelation relation = checkPolicy.child();
Optional filter = checkPolicy.getFilter(relation);
if (filter.isEmpty()) {
return relation; // no any policy, so we remove the
LogicalCheckPolicy Plan
}
// has policy, we should remove LogicalCheckPolicy and add a filter to
relation
Expression filter = new NereidsParser().parseExpression(filter.get());
return new LogicalFilter(filter, relation);
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #13879: [feature](nereids) Support row policy
924060929 commented on code in PR #13879:
URL: https://github.com/apache/doris/pull/13879#discussion_r1015227471
##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java:
##
@@ -43,33 +43,41 @@
public class UnboundRelation extends LogicalLeaf implements Relation, Unbound {
private final List nameParts;
+private boolean policyChecked;
+
public UnboundRelation(List nameParts) {
-this(nameParts, Optional.empty(), Optional.empty());
+this(nameParts, false);
+}
+
+public UnboundRelation(List nameParts, boolean policyChecked) {
+this(nameParts, policyChecked, Optional.empty(), Optional.empty());
}
Review Comment:
The policyChecked hacking the UnboundRelation, I can imagine that there will
be lots of field need added.
How about generate a LogicalCheckPolicy in the LogicalPlanBuilder?
And then you can write a rule to generate a new filter to the
logicalRelation, like this:
```java
logicalCheckPolicy(logicalRelation()).thenApply(ctx ->
LogicalCheckPolicy checkPolicy = ctx.root;
LogicalRelation relation = checkPolicy.child();
Optional filter = checkPolicy.getFilter(relation);
if (filter.isEmpty()) {
return relation; // no any policy, so we remove the
LogicalCheckPolicy Plan
}
// has policy, we should remove LogicalCheckPolicy and add a filter to
relation
Expression filter = new NereidsParser().parseExpression(filter.get());
return new LogicalFilter(filter, relation);
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #14029: [refactor](new-scan) remove old vectorized scan node
morningman opened a new pull request, #14029: URL: https://github.com/apache/doris/pull/14029 # Proposed changes Issue Number: close #xxx ## Problem summary Remove following vectorized scan node - volap_scan_node - vjdbc_scan_node - vodbc_scan_node - ves_scan_node Use new scan node in `vec/exec/scan` instead. And remove BE config `enable_new_scan_node` ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #13879: [feature](nereids) Support row policy
924060929 commented on code in PR #13879:
URL: https://github.com/apache/doris/pull/13879#discussion_r1015227471
##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java:
##
@@ -43,33 +43,41 @@
public class UnboundRelation extends LogicalLeaf implements Relation, Unbound {
private final List nameParts;
+private boolean policyChecked;
+
public UnboundRelation(List nameParts) {
-this(nameParts, Optional.empty(), Optional.empty());
+this(nameParts, false);
+}
+
+public UnboundRelation(List nameParts, boolean policyChecked) {
+this(nameParts, policyChecked, Optional.empty(), Optional.empty());
}
Review Comment:
The policyChecked hacking the UnboundRelation, I can imagine that there will
be lots of field need added.
How about generate a LogicalCheckPolicy in the LogicalPlanBuilder?
And then you can write a rule to generate a new filter to the
logicalRelation, like this:
```java
logicalCheckPolicy(logicalRelation()).thenApply(ctx ->
LogicalCheckPolicy checkPolicy = ctx.root;
LogicalRelation relation = checkPolicy.child();
Optional filter = checkPolicy.getFilter(relation);
if (filter.isEmpty()) {
return relation; // no any policy, so we remove the
LogicalCheckPolicy Plan
}
// has policy, we should remove LogicalCheckPolicy and add a filter to
relation
Expression filter = new NereidsParser().parseExpression(filter.get());
return new LogicalFilter(filter, relation);
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #12011: [fix](auth) remove USAGE_PRIV in CatalogPrivEntry when loading meta image from old version
morningman commented on PR #12011: URL: https://github.com/apache/doris/pull/12011#issuecomment-1305381534 Fixed in #12016 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #13879: [feature](nereids) Support row policy
924060929 commented on code in PR #13879:
URL: https://github.com/apache/doris/pull/13879#discussion_r1015246202
##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java:
##
@@ -43,33 +43,41 @@
public class UnboundRelation extends LogicalLeaf implements Relation, Unbound {
private final List nameParts;
+private boolean policyChecked;
+
public UnboundRelation(List nameParts) {
-this(nameParts, Optional.empty(), Optional.empty());
+this(nameParts, false);
+}
+
+public UnboundRelation(List nameParts, boolean policyChecked) {
+this(nameParts, policyChecked, Optional.empty(), Optional.empty());
}
Review Comment:
Use logicalRelation can let you not need to care whether the table is exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14015: [feature-wip](multi-catalog) fix page index
hello-stephen commented on PR #14015: URL: https://github.com/apache/doris/pull/14015#issuecomment-1305398428 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.58 seconds load time: 430 seconds storage size: 17154644854 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107102546_clickbench_pr_41059.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #14030: [feature](Nereids) bind alias in group by list
morrySnow opened a new pull request, #14030: URL: https://github.com/apache/doris/pull/14030 # Proposed changes support query having alias in group by list, such as: ```sql SELECT c1 AS a, SUM(c2) FROM t GROUP BY a; ``` ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14014: [enhancement](load) shrink reserved buffer for page builder (#14012)
hello-stephen commented on PR #14014: URL: https://github.com/apache/doris/pull/14014#issuecomment-1305401172 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.98 seconds load time: 462 seconds storage size: 17180543366 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107102822_clickbench_pr_41162.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 opened a new pull request, #14031: [extension] new e_table_to_doris by shell
Toms1999 opened a new pull request, #14031: URL: https://github.com/apache/doris/pull/14031 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 commented on pull request #14031: [extension] new e_table_to_doris by shell
Toms1999 commented on PR #14031: URL: https://github.com/apache/doris/pull/14031#issuecomment-1305403280 upupup -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #14014: [enhancement](load) shrink reserved buffer for page builder (#14012)
zhannngchen commented on code in PR #14014:
URL: https://github.com/apache/doris/pull/14014#discussion_r1015267691
##
be/src/olap/rowset/segment_v2/binary_plain_page.h:
##
@@ -53,8 +53,14 @@ class BinaryPlainPageBuilder : public PageBuilder {
}
bool is_page_full() override {
-// data_page_size is 0, do not limit the page size
-return _options.data_page_size != 0 && _size_estimate >
_options.data_page_size;
+bool ret;
+if (_options.is_dict_page) {
+// data_page_size is 0, do not limit the page size
+ret = _options.data_page_size != 0 && _size_estimate >
_options.dict_page_size;
Review Comment:
should be `_options.dict_page_size != 0`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14016: [improvement](profile) support ordinary user to get query profile via http api
hello-stephen commented on PR #14016: URL: https://github.com/apache/doris/pull/14016#issuecomment-1305419133 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.14 seconds load time: 447 seconds storage size: 17180157957 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107104440_clickbench_pr_41060.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #14007: [feat](Nereids) add graph simplifier
jackwener merged PR #14007: URL: https://github.com/apache/doris/pull/14007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener closed issue #13903: [Feature] Add graph simplifier
jackwener closed issue #13903: [Feature] Add graph simplifier URL: https://github.com/apache/doris/issues/13903 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (22b4c6af20 -> f2978fb6ff)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 22b4c6af20 [feature](Nereids) support statement having aggregate function in order by list (#13976) add f2978fb6ff [feat](Nereids) add graph simplifier (#14007) No new revisions were added by this update. Summary of changes: .../rules/joinreorder/HyperGraphJoinReorder.java | 1 - .../HyperGraphJoinReorderGroupPlan.java| 1 - .../nereids/rules/joinreorder/hypergraph/Edge.java | 69 +++- .../joinreorder/hypergraph/GraphSimplifier.java| 383 + .../rules/joinreorder/hypergraph/HyperGraph.java | 84 - .../nereids/rules/joinreorder/hypergraph/Node.java | 20 ++ .../rules/joinreorder/hypergraph/Receiver.java | 6 +- .../HyperGraphJoinReorderGroupPlanTest.java| 1 + .../joinreorder/HyperGraphJoinReorderTest.java | 1 + .../hypergraph/GraphSimplifierTest.java| 71 .../joinreorder/hypergraph/HyperGraphTest.java | 59 ++-- .../doris/nereids/util/HyperGraphBuilder.java | 101 ++ .../org/apache/doris/nereids/util/PlanChecker.java | 8 + 13 files changed, 746 insertions(+), 59 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifier.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifierTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/util/HyperGraphBuilder.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate after push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r1015106069
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicates.java:
##
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * infer additional predicates for `LogicalFilter` and `LogicalJoin`.
+ */
+public class InferPredicates extends DefaultPlanRewriter {
+PredicatePropagation propagation = new PredicatePropagation();
+PullUpPredicates pollUpPredicates = new PullUpPredicates();
+
+/**
+ * The logic is as follows:
+ * 1. poll up bottom predicate then infer additional predicates
+ * for example:
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id =
t2.id
+ * 1. poll up bottom predicate
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id
= t2.id and t.id = 1
+ * 2. infer
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id
= t2.id and t.id = 1 and t2.id = 1
+ * finally transformed sql:
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id
= t2.id and t2.id = 1
+ * 2. put these predicates into `otherJoinConjuncts` , these predicates
are processed in the next
+ * round of predicate push-down
+ */
+@Override
+public Plan visitLogicalJoin(LogicalJoin
join, JobContext context) {
+join = (LogicalJoin) super.visit(join,
context);
+Plan left = join.left();
+Plan right = join.right();
+Set expressions = getAllExpressions(left, right,
join.getOnClauseCondition());
+List otherJoinConjuncts =
Lists.newArrayList(join.getOtherJoinConjuncts());
+switch (join.getJoinType()) {
+case INNER_JOIN:
+case CROSS_JOIN:
+case LEFT_SEMI_JOIN:
+case RIGHT_SEMI_JOIN:
+otherJoinConjuncts.addAll(inferNewPredicate(left,
expressions));
+otherJoinConjuncts.addAll(inferNewPredicate(right,
expressions));
+break;
+case LEFT_OUTER_JOIN:
+case LEFT_ANTI_JOIN:
+otherJoinConjuncts.addAll(inferNewPredicate(right,
expressions));
+break;
+case RIGHT_OUTER_JOIN:
+case RIGHT_ANTI_JOIN:
+otherJoinConjuncts.addAll(inferNewPredicate(left,
expressions));
+break;
+default:
+return join;
+}
+return join.withOtherJoinConjuncts(otherJoinConjuncts);
+}
+
+/**
+ * reference `inferOn`
+ */
+@Override
+public Plan visitLogicalFilter(LogicalFilter filter,
JobContext context) {
+filter = (LogicalFilter) super.visit(filter, context);
+Set filterPredicates = filter.accept(pollUpPredicates,
null);
+Set filterChildPredicates =
filter.child(0).accept(pollUpPredicates, null);
Review Comment:
This code seem like every plan child has many repeated traverse the all
child plan tree by the pollUpPredicates.
You should add a cache for every plan in PollUpPredicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-ma
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate after push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r1015106069
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicates.java:
##
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * infer additional predicates for `LogicalFilter` and `LogicalJoin`.
+ */
+public class InferPredicates extends DefaultPlanRewriter {
+PredicatePropagation propagation = new PredicatePropagation();
+PullUpPredicates pollUpPredicates = new PullUpPredicates();
+
+/**
+ * The logic is as follows:
+ * 1. poll up bottom predicate then infer additional predicates
+ * for example:
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id =
t2.id
+ * 1. poll up bottom predicate
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id
= t2.id and t.id = 1
+ * 2. infer
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id
= t2.id and t.id = 1 and t2.id = 1
+ * finally transformed sql:
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id
= t2.id and t2.id = 1
+ * 2. put these predicates into `otherJoinConjuncts` , these predicates
are processed in the next
+ * round of predicate push-down
+ */
+@Override
+public Plan visitLogicalJoin(LogicalJoin
join, JobContext context) {
+join = (LogicalJoin) super.visit(join,
context);
+Plan left = join.left();
+Plan right = join.right();
+Set expressions = getAllExpressions(left, right,
join.getOnClauseCondition());
+List otherJoinConjuncts =
Lists.newArrayList(join.getOtherJoinConjuncts());
+switch (join.getJoinType()) {
+case INNER_JOIN:
+case CROSS_JOIN:
+case LEFT_SEMI_JOIN:
+case RIGHT_SEMI_JOIN:
+otherJoinConjuncts.addAll(inferNewPredicate(left,
expressions));
+otherJoinConjuncts.addAll(inferNewPredicate(right,
expressions));
+break;
+case LEFT_OUTER_JOIN:
+case LEFT_ANTI_JOIN:
+otherJoinConjuncts.addAll(inferNewPredicate(right,
expressions));
+break;
+case RIGHT_OUTER_JOIN:
+case RIGHT_ANTI_JOIN:
+otherJoinConjuncts.addAll(inferNewPredicate(left,
expressions));
+break;
+default:
+return join;
+}
+return join.withOtherJoinConjuncts(otherJoinConjuncts);
+}
+
+/**
+ * reference `inferOn`
+ */
+@Override
+public Plan visitLogicalFilter(LogicalFilter filter,
JobContext context) {
+filter = (LogicalFilter) super.visit(filter, context);
+Set filterPredicates = filter.accept(pollUpPredicates,
null);
+Set filterChildPredicates =
filter.child(0).accept(pollUpPredicates, null);
Review Comment:
This code seem like every plan child has many repeated traverse the all
child plan tree by the pollUpPredicates.
You should add a cache for every plan in PollUpPredicates for speed up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsu
[GitHub] [doris] yuanyuan8983 closed pull request #13933: [test](delete) Add more deletion test cases
yuanyuan8983 closed pull request #13933: [test](delete) Add more deletion test cases URL: https://github.com/apache/doris/pull/13933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #13879: [feature](nereids) Support row policy
924060929 commented on code in PR #13879:
URL: https://github.com/apache/doris/pull/13879#discussion_r1015227471
##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java:
##
@@ -43,33 +43,41 @@
public class UnboundRelation extends LogicalLeaf implements Relation, Unbound {
private final List nameParts;
+private boolean policyChecked;
+
public UnboundRelation(List nameParts) {
-this(nameParts, Optional.empty(), Optional.empty());
+this(nameParts, false);
+}
+
+public UnboundRelation(List nameParts, boolean policyChecked) {
+this(nameParts, policyChecked, Optional.empty(), Optional.empty());
}
Review Comment:
The policyChecked hacking the UnboundRelation, I can imagine that there will
be lots of fields need added.
How about generate a LogicalCheckPolicy in the LogicalPlanBuilder?
And then you can write a rule to generate a new filter to the
logicalRelation, like this:
```java
logicalCheckPolicy(logicalRelation()).thenApply(ctx ->
LogicalCheckPolicy checkPolicy = ctx.root;
LogicalRelation relation = checkPolicy.child();
Optional filter = checkPolicy.getFilter(relation);
if (filter.isEmpty()) {
return relation; // no any policy, so we remove the
LogicalCheckPolicy Plan
}
// has policy, we should remove LogicalCheckPolicy and add a filter to
relation
Expression filter = new NereidsParser().parseExpression(filter.get());
return new LogicalFilter(filter, relation);
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen opened a new pull request, #14032: [debug] check num_segments != segment_key_bounds.size()
zhannngchen opened a new pull request, #14032: URL: https://github.com/apache/doris/pull/14032 # Proposed changes Issue Number: close #xxx ## Problem summary just run pipeline for debug, don't merge ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly opened a new pull request, #14033: [fix](nereids) let right anti join generates runtime-filter
englefly opened a new pull request, #14033: URL: https://github.com/apache/doris/pull/14033 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang opened a new issue, #14034: [Bug] like '' (empty string) get wrong result with all rows
xiaokang opened a new issue, #14034:
URL: https://github.com/apache/doris/issues/14034
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
master
### What's Wrong?
like '' (empty string) match all rows mistakenly
### What You Expected?
only match '' (empty string) rows
### How to Reproduce?
-- create table
create table test_like(id int, str string) DUPLICATE KEY(id)
DISTRIBUTED BY RANDOM BUCKETS 1 PROPERTIES ("replication_allocation" =
"tag.location.default: 1");
-- insert data
insert into test_like values(1, 'i like doris'), (2, ''), (3, ' ');
-- query, will get wrong result with all rows
select * from test_like where str like '';
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang opened a new pull request, #14035: fix like '' (empty string) get wrong result with all rows
xiaokang opened a new pull request, #14035: URL: https://github.com/apache/doris/pull/14035 # Proposed changes Issue Number: close #14034 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #13745: [enhancement](Nereids) remove unnecessary decimal cast
morrySnow merged PR #13745: URL: https://github.com/apache/doris/pull/13745 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (f2978fb6ff -> 4ea1b39cb2)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f2978fb6ff [feat](Nereids) add graph simplifier (#14007) add 4ea1b39cb2 [enhancement](Nereids) remove unnecessary decimal cast (#13745) No new revisions were added by this update. Summary of changes: .../expression/rewrite/rules/SimplifyCastRule.java | 20 +++- .../nereids/trees/expressions/literal/Literal.java | 2 +- .../expression/rewrite/ExpressionRewriteTest.java| 18 ++ 3 files changed, 38 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TaoZex commented on pull request #14001: [typo](doc) fix get-starting doc
TaoZex commented on PR #14001: URL: https://github.com/apache/doris/pull/14001#issuecomment-1305471652 @zy-kkk PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13973: [Enhancement](function) add to_bitmap() function with int type
github-actions[bot] commented on PR #13973: URL: https://github.com/apache/doris/pull/13973#issuecomment-1305476601 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13973: [Enhancement](function) add to_bitmap() function with int type
github-actions[bot] commented on PR #13973: URL: https://github.com/apache/doris/pull/13973#issuecomment-1305476646 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yuanyuan8983 opened a new pull request, #14036: [test] delete-where-in-test
yuanyuan8983 opened a new pull request, #14036: URL: https://github.com/apache/doris/pull/14036 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly opened a new pull request, #14037: [opt](nereids) q21 anti and semi join reorder
englefly opened a new pull request, #14037: URL: https://github.com/apache/doris/pull/14037 # Proposed changes estimation of anti and semi join need re-work. we just let tpch q21 pass. Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 opened a new pull request, #14038: [fix](test) fix regression-test result is unstable to failed
zhangstar333 opened a new pull request, #14038: URL: https://github.com/apache/doris/pull/14038 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Pace2Car opened a new pull request, #14039: [Enhancement] forEach() replace stream().forEach()
Pace2Car opened a new pull request, #14039: URL: https://github.com/apache/doris/pull/14039 # Proposed changes Issue Number: close #13889 ## Problem summary Replace stream().forEach() by forEach() ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13967: [fix](load) fix a bug that reduce memory work on hard limit might be …
github-actions[bot] commented on PR #13967: URL: https://github.com/apache/doris/pull/13967#issuecomment-1305510930 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13967: [fix](load) fix a bug that reduce memory work on hard limit might be …
github-actions[bot] commented on PR #13967: URL: https://github.com/apache/doris/pull/13967#issuecomment-1305510980 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14014: [enhancement](load) shrink reserved buffer for page builder (#14012)
github-actions[bot] commented on PR #14014: URL: https://github.com/apache/doris/pull/14014#issuecomment-1305511357 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin commented on a diff in pull request #13985: [enhancement](profile) add instanceNum, tableIds to profile.
luozenglin commented on code in PR #13985:
URL: https://github.com/apache/doris/pull/13985#discussion_r1015346073
##
fe/fe-core/src/main/java/org/apache/doris/common/util/ProfileManager.java:
##
@@ -67,6 +67,11 @@ public class ProfileManager {
public static final String DEFAULT_DB = "Default Db";
public static final String SQL_STATEMENT = "Sql Statement";
public static final String IS_CACHED = "Is Cached";
+
+public static final String TOTAL_INSTANCES_NUM = "Total Instances Num";
+
+public static final String INSTANCES_NUM_PER_FRAGMENT = "Instances Num Per
Fragment";
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo closed issue #13773: [Bug] Using groupId as join key may cause wrong result
wangbo closed issue #13773: [Bug] Using groupId as join key may cause wrong result URL: https://github.com/apache/doris/issues/13773 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly closed pull request #13839: Estimate q21
englefly closed pull request #13839: Estimate q21 URL: https://github.com/apache/doris/pull/13839 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly opened a new pull request, #14040: [feature](nereids) let user define right deep tree penalty by session variable
englefly opened a new pull request, #14040: URL: https://github.com/apache/doris/pull/14040 # Proposed changes it is hard for us to find a proper factor for all queries. default is 0.7 Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on issue #13999: 以小时自动创建分区，分区创建的时间越来越迟，最后可能第一个按小时的分区不会创建导致数据丢失
nextdreamblue commented on issue #13999: URL: https://github.com/apache/doris/issues/13999#issuecomment-1305532864 目前动态分区的后台调度周期是由 fe的 dynamic_partition_check_interval_seconds 参数来控制的，每sleep这么多秒就会进入下一次判断，而每次执行添加新partition，删除过期partition的耗时，就导致下次任务的调度会有顺延才会再执行 其实延伸一下，这个地方会存在一个调度不及时的问题。当前dynamic_partition_check_interval_seconds参数默值是600秒，10分钟才调度一次动态分区的检查和添加（如果需要），如果一个fe调度任务在每小时的09、19、29、39、49、59分去执行，这就存在一种情况：当到了需要添加新分区的整小时时，比如按hour的0分时，按天的0点0分时，新分区的添加是会延迟9分钟才执行的，这就有可能导致这段没有及时添加新分区的时间里，新数据的插入因为没有对应的分区，而会丢失的，但是这种插入是不会报错的。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue opened a new pull request, #14041: [Enhancement](Dynamic) set dynamic partition scheduler daemon sleep t…
nextdreamblue opened a new pull request, #14041: URL: https://github.com/apache/doris/pull/14041 …ime automaticly Signed-off-by: nextdreamblue # Proposed changes Issue Number: close #13999 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14002: [Bug](udf) Make UDF's type always nullable
github-actions[bot] commented on PR #14002: URL: https://github.com/apache/doris/pull/14002#issuecomment-1305543042 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14002: [Bug](udf) Make UDF's type always nullable
github-actions[bot] commented on PR #14002: URL: https://github.com/apache/doris/pull/14002#issuecomment-1305542986 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #14042: [feature](nereids) binding slot in order by that not show in project
morrySnow opened a new pull request, #14042: URL: https://github.com/apache/doris/pull/14042 # Proposed changes 1. binding slot in order by that not show in project, such as: ```sql SELECT c1 FROM t WHERE c2 > 0 ORDER BY c3 ``` 2. not check unbound when bind slot reference. Instead, do it in analysis check. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 6. Does it need to update dependencies: - [ ] Yes - [ ] No 7. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #14002: [Bug](udf) Make UDF's type always nullable
Gabriel39 merged PR #14002: URL: https://github.com/apache/doris/pull/14002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug](udf) Make UDF's type always nullable (#14002)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1c2532b9dc [Bug](udf) Make UDF's type always nullable (#14002)
1c2532b9dc is described below
commit 1c2532b9dcb7f270a48aa348ee844222b54b1270
Author: Gabriel
AuthorDate: Mon Nov 7 20:51:31 2022 +0800
[Bug](udf) Make UDF's type always nullable (#14002)
---
.../org/apache/doris/catalog/ScalarFunction.java | 1 +
.../data/javaudf_p0/test_javaudf_null.out | 26
.../main/java/org/apache/doris/udf/NullTest.java | 26
.../suites/javaudf_p0/test_javaudf_null.groovy | 70 ++
4 files changed, 123 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/ScalarFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/ScalarFunction.java
index a04cc5aaa3..371c7328f2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/ScalarFunction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/ScalarFunction.java
@@ -339,6 +339,7 @@ public class ScalarFunction extends Function {
fn.prepareFnSymbol = prepareFnSymbol;
fn.closeFnSymbol = closeFnSymbol;
fn.setLocation(location);
+fn.nullableMode = NullableMode.ALWAYS_NULLABLE;
return fn;
}
diff --git a/regression-test/data/javaudf_p0/test_javaudf_null.out
b/regression-test/data/javaudf_p0/test_javaudf_null.out
new file mode 100644
index 00..dc59ed24af
--- /dev/null
+++ b/regression-test/data/javaudf_p0/test_javaudf_null.out
@@ -0,0 +1,26 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select_default --
+1
+2
+3
+4
+5
+6
+7
+8
+9
+
+-- !select --
+\N
+
+-- !select --
+\N
+\N
+\N
+\N
+\N
+\N
+\N
+\N
+\N
+
diff --git
a/regression-test/java-udf-src/src/main/java/org/apache/doris/udf/NullTest.java
b/regression-test/java-udf-src/src/main/java/org/apache/doris/udf/NullTest.java
new file mode 100644
index 00..4675fca99b
--- /dev/null
+++
b/regression-test/java-udf-src/src/main/java/org/apache/doris/udf/NullTest.java
@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.udf;
+
+import org.apache.hadoop.hive.ql.exec.UDF;
+
+public class NullTest extends UDF {
+public Integer evaluate(Integer i) {
+return null;
+}
+}
diff --git a/regression-test/suites/javaudf_p0/test_javaudf_null.groovy
b/regression-test/suites/javaudf_p0/test_javaudf_null.groovy
new file mode 100644
index 00..e2699f1b4b
--- /dev/null
+++ b/regression-test/suites/javaudf_p0/test_javaudf_null.groovy
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.Paths
+
+suite("test_javaudf_null") {
+def tableName = "test_javaudf_null"
+def jarPath =
"""${context.file.parent}/jars/java-udf-case-jar-with-dependencies.jar"""
+
+log.info("Jar path: ${jarPath}".toString())
+try {
+sql """ DROP TABLE IF EXISTS ${tableName} """
+sql """
+CREATE TABLE IF NOT EXISTS ${tableName} (
+`user_id` INT NOT NULL COMMENT ""
+)
+DISTRIBUTED BY HASH(user_i
[GitHub] [doris] github-actions[bot] commented on pull request #14030: [feature](Nereids) bind alias in group by list
github-actions[bot] commented on PR #14030: URL: https://github.com/apache/doris/pull/14030#issuecomment-1305591399 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14030: [feature](Nereids) bind alias in group by list
github-actions[bot] commented on PR #14030: URL: https://github.com/apache/doris/pull/14030#issuecomment-1305591458 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] htyoung opened a new issue, #14043: [Bug] grouping sets + case when or coalesce get wrong result
htyoung opened a new issue, #14043: URL: https://github.com/apache/doris/issues/14043 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version version：1.1.3-rc02 ### What's Wrong? Test SQL: select coalesce(col1,"all") as `col1` ,count(*) as `cnt` from ( select null as col1 union all select "a" as col1 ) t group by grouping sets ( (col1) ,() ) ; ### What You Expected? The expected result is: col1 cnt all 2 all 1 a 1 but the actual result is : col1 cnt NULL 2 all 1 a 1 so, I test the similar sql with hive, mysql(8.0+ and 5.7+) also get the expected result. https://user-images.githubusercontent.com/9734567/200318848-4cd3f10e-fbba-47ce-92bf-925b5c4c2732.png";> https://user-images.githubusercontent.com/9734567/200319156-df0e4dec-ce61-48f7-a48f-9cfeed6c9fa5.png";>  ### How to Reproduce? just run the test sql directly ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 commented on pull request #14031: [extension] new e_table_to_doris by shell
Toms1999 commented on PR #14031: URL: https://github.com/apache/doris/pull/14031#issuecomment-1305606254 upupup -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname opened a new pull request, #14044: [Enhancement](Nereids)optimize groupExpressionMatch
sohardforaname opened a new pull request, #14044: URL: https://github.com/apache/doris/pull/14044 # Proposed changes Issue Number: close #xxx ## Problem summary optimize groupExpressionMatch by reducing unnecessary iterator creation. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14001: [typo](doc) fix get-starting doc
github-actions[bot] commented on PR #14001: URL: https://github.com/apache/doris/pull/14001#issuecomment-1305616954 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] JNSimba merged pull request #14001: [typo](doc) fix get-starting doc
JNSimba merged PR #14001: URL: https://github.com/apache/doris/pull/14001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](doc) fix get_start doc (#14001)
This is an automated email from the ASF dual-hosted git repository. diwu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 241801ca17 [typo](doc) fix get_start doc (#14001) 241801ca17 is described below commit 241801ca17029b3b4a30dabae2a5bfede416111e Author: TaoZex <45089228+tao...@users.noreply.github.com> AuthorDate: Mon Nov 7 21:28:45 2022 +0800 [typo](doc) fix get_start doc (#14001) --- docs/zh-CN/docs/get-starting/get-starting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh-CN/docs/get-starting/get-starting.md b/docs/zh-CN/docs/get-starting/get-starting.md index 645d50d7ae..faa9cb72d9 100644 --- a/docs/zh-CN/docs/get-starting/get-starting.md +++ b/docs/zh-CN/docs/get-starting/get-starting.md @@ -179,7 +179,7 @@ Doris FE 的停止可以通过下面的命令完成 cd apache-doris-x.x.x/be ``` -修改 FE 配置文件 `conf/be.conf` ，这里我们主要修改两个参数：`priority_networks'` 及 `storage_root` ，如果你需要更多优化配置，请参考 [BE 参数配置](../admin-manual/config/be-config)说明，进行调整。 +修改 BE 配置文件 `conf/be.conf` ，这里我们主要修改两个参数：`priority_networks'` 及 `storage_root` ，如果你需要更多优化配置，请参考 [BE 参数配置](../admin-manual/config/be-config)说明，进行调整。 1. 添加 priority_networks 参数 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee opened a new pull request, #14045: [thirdpart](lib) Add lock free queue of concurrentqueue
HappenLee opened a new pull request, #14045: URL: https://github.com/apache/doris/pull/14045 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #14042: [feature](nereids) binding slot in order by that not show in project
924060929 commented on code in PR #14042:
URL: https://github.com/apache/doris/pull/14042#discussion_r1015424230
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ResolveSortProject.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableList.Builder;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * resolve sort to add slot reference in sort into it's child - project if
necessary. such as:
+ * SELECT c1 FROM t WHERE c2 > 0 ORDER BY c3
+ */
+public class ResolveSortProject extends OneAnalysisRuleFactory {
+@Override
+public Rule build() {
+return logicalSort(logicalProject()).when(sort ->
sort.getExpressions().stream()
Review Comment:
```suggestion
return logicalSort(logicalProject()).when(Plan::canBind).when(sort
-> sort.getExpressions().stream()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14045: [thirdpart](lib) Add lock free queue of concurrentqueue
github-actions[bot] commented on PR #14045:
URL: https://github.com/apache/doris/pull/14045#issuecomment-1305623543
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3410771540";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In thirdparty/build-thirdparty.sh line 1512:
cp *.h "${TP_INSTALL_DIR}/include/"
^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes
won't become options.
For more information:
https://www.shellcheck.net/wiki/SC2035 -- Use ./*glob* or -- *glob* so
name...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' found no issues.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14045: [thirdpart](lib) Add lock free queue of concurrentqueue
github-actions[bot] commented on PR #14045: URL: https://github.com/apache/doris/pull/14045#issuecomment-1305624338 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14045: [thirdpart](lib) Add lock free queue of concurrentqueue
github-actions[bot] commented on PR #14045: URL: https://github.com/apache/doris/pull/14045#issuecomment-1305624386 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #14042: [feature](nereids) binding slot in order by that not show in project
924060929 commented on code in PR #14042:
URL: https://github.com/apache/doris/pull/14042#discussion_r1015430108
##
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/AnalyzeRulesJob.java:
##
@@ -55,7 +56,8 @@ public AnalyzeRulesJob(CascadesContext cascadesContext,
Optional scope) {
new ProjectToGlobalAggregate()
)),
topDownBatch(ImmutableList.of(
-new ResolveAggregateFunctions()
+new ResolveAggregateFunctions(),
+new ResolveSortProject()
Review Comment:
Rename to BindXxx?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #14046: [fix](priv) fix meta replay bug when upgrading from 1.1.x to 1.2.x
morningman opened a new pull request, #14046: URL: https://github.com/apache/doris/pull/14046 # Proposed changes Issue Number: close #xxx ## Problem summary Follow up #12016 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14026: [Enhancement] Doris broker support aliyun-oss
hello-stephen commented on PR #14026: URL: https://github.com/apache/doris/pull/14026#issuecomment-1305650196 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.82 seconds load time: 436 seconds storage size: 17154741679 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107135430_clickbench_pr_41128.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
