[doris-website] branch master updated: backup s3 example
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 9d5ee640f1d backup s3 example 9d5ee640f1d is described below commit 9d5ee640f1dad3bd893283b486d36a5fa3c7a64c Author: jiafeng.zhang AuthorDate: Wed Jul 27 15:02:27 2022 +0800 backup s3 example backup s3 example --- .../Backup-and-Restore/CREATE-REPOSITORY.md | 19 ++- .../Backup-and-Restore/CREATE-REPOSITORY.md | 16 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/CREATE-REPOSITORY.md b/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/CREATE-REPOSITORY.md index dab7e695a2c..f2e75390721 100644 --- a/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/CREATE-REPOSITORY.md +++ b/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/CREATE-REPOSITORY.md @@ -122,8 +122,25 @@ PROPERTIES ### Keywords ``` -CREATE, REPOSITORY +6. Create a repository named minio_repo to link minio storage directly through the s3 protocol. + ``` +CREATE REPOSITORY `minio_repo` +WITH S3 +ON LOCATION "s3://minio_repo" +PROPERTIES +( +"AWS_ENDPOINT" = "http://minio.com";, +"AWS_ACCESS_KEY" = "MINIO_USER", +"AWS_SECRET_KEY"="MINIO_PASSWORD", +"AWS_REGION" = "REGION", +"use_path_style" = "true" +); +``` + +### Keywords + +CREATE, REPOSITORY ### Best Practice diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/CREATE-REPOSITORY.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/CREATE-REPOSITORY.md index a78618a2aef..c43e6c73303 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/CREATE-REPOSITORY.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/CREATE-REPOSITORY.md @@ -120,6 +120,22 @@ PROPERTIES ); ``` +6. 创建名为 minio_repo 的仓库，直接通过 s3 协议链接 minio. + +```sql +CREATE REPOSITORY `minio_repo` +WITH S3 +ON LOCATION "s3://minio_repo" +PROPERTIES +( +"AWS_ENDPOINT" = "http://minio.com";, +"AWS_ACCESS_KEY" = "MINIO_USER", +"AWS_SECRET_KEY"="MINIO_PASSWORD", +"AWS_REGION" = "REGION", +"use_path_style" = "true" +); +``` + ### Keywords CREATE, REPOSITORY - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11246: [WIP](optional) using hash set to distinct single value
morrySnow commented on code in PR #11246:
URL: https://github.com/apache/doris/pull/11246#discussion_r930692117
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AggregateInfo.java:
##
@@ -283,6 +285,12 @@ public static boolean
estimateIfContainsMultiDistinct(List dis
hasMultiDistinct = true;
}
}
+
+// for vectorized execution, we force it to using hash set to execution
+if (!hasMultiDistinct && distinctAggExprs.size() == 1 &&
distinctAggExprs.get(0).getFnParams().isDistinct()
+&& VectorizedUtil.isVectorized() &&
ConnectContext.get().getSessionVariable().enableSingleDistinctColumnOpt()) {
+hasMultiDistinct = true;
Review Comment:
nit: if we use 2 stage aggregate for one distinct aggregate, maybe we should
change function name and variable name to get a better description of what's
actually being done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on issue #10541: [Bug] DCHECK fail on aggregate
BiteThet commented on issue #10541: URL: https://github.com/apache/doris/issues/10541#issuecomment-1196341799 fixed on #10650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt closed issue #10541: [Bug] DCHECK fail on aggregate
BiteThet closed issue #10541: [Bug] DCHECK fail on aggregate URL: https://github.com/apache/doris/issues/10541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on issue #11224: [Bug] fe-core: Compilation failure
BiteThet commented on issue #11224: URL: https://github.com/apache/doris/issues/11224#issuecomment-1196342869 fixed on #11215 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt closed issue #11224: [Bug] fe-core: Compilation failure
BiteThet closed issue #11224: [Bug] fe-core: Compilation failure URL: https://github.com/apache/doris/issues/11224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 merged pull request #11211: (Git)[Nereids] ignore ANTLR4 generated files and dir.
924060929 merged PR #11211: URL: https://github.com/apache/doris/pull/11211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: ignore antlr generated (#11211)
This is an automated email from the ASF dual-hosted git repository. huajianlan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 2db642d9d3 ignore antlr generated (#11211) 2db642d9d3 is described below commit 2db642d9d3b7f05299029d678dec90b493e8312b Author: Shuo Wang AuthorDate: Wed Jul 27 15:07:56 2022 +0800 ignore antlr generated (#11211) This PR proposes to ignore tracking the following file and dir auto-generated by ANTLR4: fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.tokens fe/fe-core/src/main/antlr4/org/apache/doris/nereids/gen/ --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index e743211fac..ff50e5b3da 100644 --- a/.gitignore +++ b/.gitignore @@ -69,6 +69,8 @@ samples/doris-demo/remote-udf-cpp-demo/function_server_demo fe/mocked/ fe/fe-core/gen/ fe/fe-core/src/main/resources/static/ +fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.tokens +fe/fe-core/src/main/antlr4/org/apache/doris/nereids/gen/ # BE be/build*/ - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt closed pull request #9313: [Feature] [Backend] support build be with c++20
BiteThet closed pull request #9313: [Feature] [Backend] support build be with c++20 URL: https://github.com/apache/doris/pull/9313 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 merged pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
924060929 merged PR #11162: URL: https://github.com/apache/doris/pull/11162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature] (Nereids) add rule to push down predicate through aggregate (#11162)
This is an automated email from the ASF dual-hosted git repository.
huajianlan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new daf2e27202 [feature] (Nereids) add rule to push down predicate through
aggregate (#11162)
daf2e27202 is described below
commit daf2e2720250416c2908380794c51d8803b7a629
Author: minghong
AuthorDate: Wed Jul 27 15:16:15 2022 +0800
[feature] (Nereids) add rule to push down predicate through aggregate
(#11162)
add rule to push predicates down to aggregation node
add PushDownPredicatesThroughAggregation.java
add ut for PushDownPredicatesThroughAggregation
For example:
```
Logical plan tree:
any_node
|
filter (a>0 and b>0)
|
group by(a, c)
|
scan
```
transformed to:
```
project
|
upper filter (b>0)
|
group by(a, c)
|
bottom filter (a>0)
|
scan
```
Note:
'a>0' could be push down, because 'a' is in group by keys;
but 'b>0' could not push down, because 'b' is not in group by keys.
---
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../logical/PushPredicateThroughAggregation.java | 109 +++
.../PushDownPredicateThroughAggregationTest.java | 206 +
3 files changed, 316 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index 2485fad8a4..fa6df6d5f7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -43,6 +43,7 @@ public enum RuleType {
COLUMN_PRUNE_PROJECTION(RuleTypeClass.REWRITE),
// predicate push down rules
PUSH_DOWN_PREDICATE_THROUGH_JOIN(RuleTypeClass.REWRITE),
+PUSH_DOWN_PREDICATE_THROUGH_AGGREGATION(RuleTypeClass.REWRITE),
// column prune rules,
COLUMN_PRUNE_AGGREGATION_CHILD(RuleTypeClass.REWRITE),
COLUMN_PRUNE_FILTER_CHILD(RuleTypeClass.REWRITE),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java
new file mode 100644
index 00..bc4155bdec
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ *filter (a>0 and b>0)
+ * |
+ *group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ *
[GitHub] [doris] BiteTheDDDDt closed pull request #9235: [Enhancement] [Vectorized] replace std::move with assume_mutable
BiteThet closed pull request #9235: [Enhancement] [Vectorized] replace std::move with assume_mutable URL: https://github.com/apache/doris/pull/9235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11210: [enhancement](Nereids)enable explain query for nereids planner
wangshuo128 commented on code in PR #11210:
URL: https://github.com/apache/doris/pull/11210#discussion_r930703991
##
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java:
##
@@ -46,14 +49,25 @@ public class NereidsParser {
* a single packet.
* https://dev.mysql.com/doc/internals/en/com-set-option.html
*/
-public List parseSQL(String originStr) throws Exception {
-List logicalPlanList = parseMultiple(originStr);
-List statementBaseList = new ArrayList<>();
-for (LogicalPlan logicalPlan : logicalPlanList) {
+public List parseSQL(String originStr) {
+List logicalPlans = parseMultiple(originStr);
+List statementBases = new ArrayList<>();
+for (LogicalPlan logicalPlan : logicalPlans) {
+// TODO: this is a trick to support explain. Since we do not
support any other command in a short time.
+// It is acceptable. In the future, we need to refactor this.
+ExplainOptions explainOptions = null;
+if (logicalPlan instanceof ExplainCommand) {
Review Comment:
I think it's clear like this:
```java
if (logicalPlan instanceof ExplainCommand) {
ExplainCommand explainCommand = (ExplainCommand) logicalPlan;
LogicalPlan innerPlan = explainCommand.getLogicalPlan();
LogicalPlanAdapter logicalPlanAdapter = new
LogicalPlanAdapter(innerPlan);
logicalPlanAdapter.setIsExplain(new ExplainOptions(
explainCommand.getLevel() == ExplainLevel.VERBOSE,
explainCommand.getLevel() == ExplainLevel.GRAPH));
statementBases.add(logicalPlanAdapter);
} else {
statementBases.add(new LogicalPlanAdapter(logicalPlan));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11209: [feature] (Nereids) support limit clause
morrySnow commented on code in PR #11209:
URL: https://github.com/apache/doris/pull/11209#discussion_r930696935
##
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##
@@ -119,6 +119,12 @@ sortItem
: expression ordering = (ASC | DESC)?
;
+limitClause
+: (LIMIT INTEGER_VALUE)?
+| (LIMIT INTEGER_VALUE OFFSET INTEGER_VALUE)?
+| (LIMIT INTEGER_VALUE COMMA INTEGER_VALUE)?
Review Comment:
(LIMIT limit=INTEGER_VALUE ((OFFSET | COMMA) offset=INTEGER_VALUE)?)?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalLimit.java:
##
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.IntegerLiteral;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Logical limit plan
+ * eg: select * from table limit 10
+ * limit: 10
+ *
+ * eg: select * from table order by a limit 100, 10
+ * limit: 10
+ * offset 100
+ */
+public class LogicalLimit extends
LogicalUnary {
+private long limit = Long.MAX_VALUE;
+private long offset = 0L;
+
+public LogicalLimit(long limit, long offset, CHILD_TYPE child) {
+this(limit, offset, Optional.empty(), Optional.empty(), child);
+}
+
+public LogicalLimit(long limit, long offset, Optional
groupExpression,
+Optional logicalProperties,
CHILD_TYPE child) {
+super(PlanType.LOGICAL_LIMIT, groupExpression, logicalProperties,
child);
+this.limit = limit;
+this.offset = offset;
+}
+
+public long getLimit() {
+return limit;
+}
+
+public long getOffset() {
+return offset;
+}
+
+@Override
+public List computeOutput(Plan input) {
+return input.getOutput();
+}
+
+@Override
+public String toString() {
+return "LogicalLimit (" + offset + "," + limit + ")";
Review Comment:
```suggestion
return "LogicalLimit (offset=" + offset + ", limit=" + limit + ")";
```
##
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##
@@ -532,16 +537,52 @@ public List
visitNamedExpressionSeq(NamedExpressionSeqContext namedC
* @return List of OrderKey
*/
@Override
-public List visitQueryOrganization(QueryOrganizationContext ctx)
{
Review Comment:
nit: we could just add sort node and limit node to plan in
withQueryOrganization directly and remove this function and related data
structure like LimitAndOffset..
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLimit.java:
##
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.n
[GitHub] [doris] github-actions[bot] commented on pull request #11145: [enhancement](workflow) Optimize workflows
github-actions[bot] commented on PR #11145: URL: https://github.com/apache/doris/pull/11145#issuecomment-1196358980 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11145: [enhancement](workflow) Optimize workflows
github-actions[bot] commented on PR #11145: URL: https://github.com/apache/doris/pull/11145#issuecomment-1196358941 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
Kikyou1997 commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r930709610
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Slot.java:
##
@@ -30,4 +32,8 @@ public Slot toSlot() {
public Slot withNullable(boolean newNullable) {
throw new RuntimeException("Do not implement");
}
+
+public Slot withQualifier(List qualifiers) {
+throw new RuntimeException("Do not implement");
Review Comment:
"Not implemented" may be a better exception message here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
sohardforaname commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r930711483
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Slot.java:
##
@@ -30,4 +32,8 @@ public Slot toSlot() {
public Slot withNullable(boolean newNullable) {
throw new RuntimeException("Do not implement");
}
+
+public Slot withQualifier(List qualifiers) {
+throw new RuntimeException("Do not implement");
Review Comment:
ok，I will change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #11209: [feature] (Nereids) support limit clause
morrySnow commented on PR #11209: URL: https://github.com/apache/doris/pull/11209#issuecomment-1196361946 we should remove `limit` and `offset` in SortNode and `limit` in AbstractPlan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
Kikyou1997 commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r930718759
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalSubQueryAlias.java:
##
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * The node of logical plan for sub query and alias
+ *
+ * @param param
+ */
+public class LogicalSubQueryAlias extends
LogicalUnary {
+private final String alias;
+
+public LogicalSubQueryAlias(String tableAlias, CHILD_TYPE child) {
+this(tableAlias, Optional.empty(), Optional.empty(), child);
+}
+
+public LogicalSubQueryAlias(String tableAlias, Optional
groupExpression,
+Optional logicalProperties, CHILD_TYPE child) {
+super(PlanType.LOGICAL_SUBQUERY_ALIAS, groupExpression,
logicalProperties, child);
+this.alias = tableAlias;
+
+}
+
+public List computeOutput(Plan input) {
+return input.getOutput().stream()
+.map(slot -> slot.withQualifier(ImmutableList.of(alias)))
+.collect(Collectors.toList());
+}
+
+public String getAlias() {
+return alias;
+}
+
+@Override
+public String toString() {
+return "LogicalSubQueryAlias (" + alias.toString() + ")";
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
+}
+if (o == null || getClass() != o.getClass()) {
+return false;
+}
+LogicalSubQueryAlias that = (LogicalSubQueryAlias) o;
+return alias.equals(that.alias);
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(alias);
+}
+
+@Override
+public Plan withChildren(List children) {
+Preconditions.checkArgument(children.size() == 1);
+return new LogicalSubQueryAlias<>(alias, children.get(0));
+}
+
+@Override
+public R accept(PlanVisitor visitor, C context) {
+return visitor.visitSubQueryAlias((LogicalSubQueryAlias) this,
context);
+}
+
+@Override
+public List getExpressions() {
+return Collections.emptyList();
+}
+
+@Override
+public Plan withGroupExpression(Optional groupExpression)
{
+return new LogicalSubQueryAlias<>(alias, groupExpression,
Optional.of(logicalProperties), child());
+}
Review Comment:
Will `LogicalSubQueryAlias` still exist in the Memo? If not, I think it'd
be better to not overwrite this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
Kikyou1997 commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r930718759
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalSubQueryAlias.java:
##
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * The node of logical plan for sub query and alias
+ *
+ * @param param
+ */
+public class LogicalSubQueryAlias extends
LogicalUnary {
+private final String alias;
+
+public LogicalSubQueryAlias(String tableAlias, CHILD_TYPE child) {
+this(tableAlias, Optional.empty(), Optional.empty(), child);
+}
+
+public LogicalSubQueryAlias(String tableAlias, Optional
groupExpression,
+Optional logicalProperties, CHILD_TYPE child) {
+super(PlanType.LOGICAL_SUBQUERY_ALIAS, groupExpression,
logicalProperties, child);
+this.alias = tableAlias;
+
+}
+
+public List computeOutput(Plan input) {
+return input.getOutput().stream()
+.map(slot -> slot.withQualifier(ImmutableList.of(alias)))
+.collect(Collectors.toList());
+}
+
+public String getAlias() {
+return alias;
+}
+
+@Override
+public String toString() {
+return "LogicalSubQueryAlias (" + alias.toString() + ")";
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
+}
+if (o == null || getClass() != o.getClass()) {
+return false;
+}
+LogicalSubQueryAlias that = (LogicalSubQueryAlias) o;
+return alias.equals(that.alias);
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(alias);
+}
+
+@Override
+public Plan withChildren(List children) {
+Preconditions.checkArgument(children.size() == 1);
+return new LogicalSubQueryAlias<>(alias, children.get(0));
+}
+
+@Override
+public R accept(PlanVisitor visitor, C context) {
+return visitor.visitSubQueryAlias((LogicalSubQueryAlias) this,
context);
+}
+
+@Override
+public List getExpressions() {
+return Collections.emptyList();
+}
+
+@Override
+public Plan withGroupExpression(Optional groupExpression)
{
+return new LogicalSubQueryAlias<>(alias, groupExpression,
Optional.of(logicalProperties), child());
+}
Review Comment:
Will `LogicalSubQueryAlias` still exist in the Memo? If not, I think it'd
be better to throw a exception here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] solitary159 opened a new issue, #11247: BE.alive is false when i begain sql
solitary159 opened a new issue, #11247: URL: https://github.com/apache/doris/issues/11247 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.0 ### What's Wrong? nvaicat 链接doris数据库，执行sql之后3个be节点中的一个会掉 ### What You Expected? 查看这个是bug还是我的配置或sql或者其他问题 ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
sohardforaname commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r930723297
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalSubQueryAlias.java:
##
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * The node of logical plan for sub query and alias
+ *
+ * @param param
+ */
+public class LogicalSubQueryAlias extends
LogicalUnary {
+private final String alias;
+
+public LogicalSubQueryAlias(String tableAlias, CHILD_TYPE child) {
+this(tableAlias, Optional.empty(), Optional.empty(), child);
+}
+
+public LogicalSubQueryAlias(String tableAlias, Optional
groupExpression,
+Optional logicalProperties, CHILD_TYPE child) {
+super(PlanType.LOGICAL_SUBQUERY_ALIAS, groupExpression,
logicalProperties, child);
+this.alias = tableAlias;
+
+}
+
+public List computeOutput(Plan input) {
+return input.getOutput().stream()
+.map(slot -> slot.withQualifier(ImmutableList.of(alias)))
+.collect(Collectors.toList());
+}
+
+public String getAlias() {
+return alias;
+}
+
+@Override
+public String toString() {
+return "LogicalSubQueryAlias (" + alias.toString() + ")";
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
+}
+if (o == null || getClass() != o.getClass()) {
+return false;
+}
+LogicalSubQueryAlias that = (LogicalSubQueryAlias) o;
+return alias.equals(that.alias);
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(alias);
+}
+
+@Override
+public Plan withChildren(List children) {
+Preconditions.checkArgument(children.size() == 1);
+return new LogicalSubQueryAlias<>(alias, children.get(0));
+}
+
+@Override
+public R accept(PlanVisitor visitor, C context) {
+return visitor.visitSubQueryAlias((LogicalSubQueryAlias) this,
context);
+}
+
+@Override
+public List getExpressions() {
+return Collections.emptyList();
+}
+
+@Override
+public Plan withGroupExpression(Optional groupExpression)
{
+return new LogicalSubQueryAlias<>(alias, groupExpression,
Optional.of(logicalProperties), child());
+}
Review Comment:
Will not, but it's not need to throw a exception as their is always a node
below the node so the child() will never be null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] solitary159 commented on issue #11247: BE.alive is false when i begain sql
solitary159 commented on issue #11247: URL: https://github.com/apache/doris/issues/11247#issuecomment-1196374980 W0727 14:40:58.582839 8567 internal_service.cpp:446] apply filter meet error: Invalid argument: fragment-id: 10c8dc9bf6d94460-a3fb832330c749a9 W0727 14:40:58.592717 8557 internal_service.cpp:446] apply filter meet error: Invalid argument: fragment-id: 10c8dc9bf6d94460-a3fb832330c74db4 F0727 14:57:59.556994 9758 vslot_ref.cpp:63] Check failed: _column_id >= 0 (-1 vs. 0) , SlotRef(slot_id=419 type=VARCHAR codegen=false) column id: -1, name: pname, is nulable: 1, @ 0x55aa0d75bbf9 doris::vectorized::VSlotRef::execute() @ 0x55aa0d755a24 doris::vectorized::VInPredicate::execute() @ 0x55aa0d761719 doris::vectorized::VCaseExpr::execute() @ 0x55aa0d17b01b doris::vectorized::VExprContext::get_output_block_after_execute_exprs() @ 0x55aa0e2e16a9 doris::vectorized::VMysqlResultWriter::append_block() @ 0x55aa0e2a1cbc doris::vectorized::VResultSink::send() @ 0x55aa0cb89115 doris::PlanFragmentExecutor::open_vectorized_internal() @ 0x55aa0cb8a82f doris::PlanFragmentExecutor::open() @ 0x55aa0cb041c5 doris::FragmentExecState::execute() @ 0x55aa0cb08e7a doris::FragmentMgr::_exec_actual() @ 0x55aa0cb1241c std::_Function_handler<>::_M_invoke() @ 0x55aa0ccb5ca9 doris::ThreadPool::dispatch_thread() @ 0x55aa0ccaf29f doris::Thread::supervise_thread() @ 0x7f4a69fb7609 start_thread @ 0x7f4a6a0f1133 clone @ (nil) (unknown) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: config
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new d8c8faa8661 config d8c8faa8661 is described below commit d8c8faa86619e776675137b326f45cdd9fe8b688 Author: jiafeng.zhang AuthorDate: Wed Jul 27 15:42:48 2022 +0800 config --- docs/admin-manual/config/be-config.md | 49 +- docs/admin-manual/config/fe-config.md | 10 - .../current/admin-manual/config/be-config.md | 49 +- .../current/admin-manual/config/fe-config.md | 10 - 4 files changed, 74 insertions(+), 44 deletions(-) diff --git a/docs/admin-manual/config/be-config.md b/docs/admin-manual/config/be-config.md index f16194f7b3b..0e1aec9be4b 100644 --- a/docs/admin-manual/config/be-config.md +++ b/docs/admin-manual/config/be-config.md @@ -672,11 +672,17 @@ Default: 10737418240 BloomFilter/Min/Max and other statistical information cache capacity +### `kafka_api_version_request` + +Default: true + +If the dependent Kafka version is lower than 0.10.0.0, this value should be set to false. + ### `kafka_broker_version_fallback` Default: 0.10.0 -If the dependent Kafka version is lower than the Kafka client version that routine load depends on, the value set by the fallback version kafka_broker_version_fallback will be used, and the valid values are: 0.9.0, 0.8.2, 0.8.1, 0.8.0. +If the dependent Kafka version is lower than 0.10.0.0, the value set by the fallback version kafka_broker_version_fallback will be used if the value of kafka_api_version_request is set to false, and the valid values are: 0.9.0.x, 0.8.x.y. ### `load_data_reserve_hours` @@ -1437,36 +1443,18 @@ The size of the buffer before flashing * Default: 3 -### `track_new_delete` +### `enable_tcmalloc_hook` * Type: bool * Description: Whether Hook TCmalloc new/delete, currently consume/release tls mem tracker in Hook. * Default: true -### `mem_tracker_level` - -* Type: int16 -* Description: The level at which MemTracker is displayed on the Web page equal or lower than this level will be displayed on the Web page - ``` -OVERVIEW = 0 -TASK = 1 -INSTANCE = 2 -VERBOSE = 3 - ``` -* Default: 0 - ### `mem_tracker_consume_min_size_bytes` * Type: int32 * Description: The minimum length of TCMalloc Hook when consume/release MemTracker. Consume size smaller than this value will continue to accumulate to avoid frequent calls to consume/release of MemTracker. Decreasing this value will increase the frequency of consume/release. Increasing this value will cause MemTracker statistics to be inaccurate. Theoretically, the statistical value of a MemTracker differs from the true value = ( mem_tracker_consume_min_size_bytes * the number of BE thr [...] * Default: 1048576 -### `memory_leak_detection` - -* Type: bool -* Description: Whether to start memory leak detection, when MemTracker is a negative value, it is considered that a memory leak has occurred, but the actual MemTracker records inaccurately will also cause a negative value, so this feature is in the experimental stage. -* Default: false - ### `max_segment_num_per_rowset` * Type: int32 @@ -1531,3 +1519,24 @@ Translated with www.DeepL.com/Translator (free version) * Type: int32 * Description: The maximum amount of data read by each OlapScanner. * Default: 1024 + +### `enable_quick_compaction` +* Type: bool +* Description: enable quick compaction,It is mainly used in the scenario of frequent import of small amount of data. The problem of -235 can be effectively avoided by merging the imported versions in time through the mechanism of rapid compaction. The definition of small amount of data is currently defined according to the number of rows +* Default: false + +### `quick_compaction_max_rows` +* Type: int32 +* Description: When the number of imported rows is less than this value, it is considered that this import is an import of small amount of data, which will be selected during quick compaction +* Default: 1000 + +### `quick_compaction_batch_size` +* Type: int32 +* Description: trigger time, when import times reach quick_compaction_batch_size will trigger immediately +* Default: 10 + +### `quick_compaction_min_rowsets` +* Type: int32 +* Description: at least the number of versions to be compaction, and the number of rowsets with a small amount of data in the selection. If it is greater than this value, the real compaction will be carried out +* Default: 10 + diff --git a/docs/admin-manual/config/fe-config.md b/docs/admin-manual/config/fe-config.md index f37e64564ee..13c6e8374f5 100644 --- a/docs/admin-manual/config/fe-config.md +++ b/docs/admin-manual/config/fe-config.md @@ -1458,6 +1458,12 @@ MasterOnly：true Default broker load timeout +### broker_timeout_ms + +De
[GitHub] [doris] morrySnow commented on a diff in pull request #11179: [feature](nereids) Add stats derive framework for new optimizer
morrySnow commented on code in PR #11179:
URL: https://github.com/apache/doris/pull/11179#discussion_r930710682
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/SlotReference.java:
##
@@ -45,17 +45,16 @@ public SlotReference(String name, DataType dataType,
boolean nullable, List qualifier) {
this.exprId = exprId;
-this.name = name;
this.dataType = dataType;
this.qualifier = qualifier;
this.nullable = nullable;
+this.name = name;
Review Comment:
move back?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/PlanVisitor.java:
##
@@ -59,8 +59,8 @@ public R visitLogicalRelation(LogicalRelation relation, C
context) {
}
-public R visitLogicalAggregate(LogicalAggregate relation, C context)
{
-return visit(relation, context);
+public R visitLogicalAggregate(LogicalAggregate agg, C context) {
+return visit(agg, context);
Review Comment:
```suggestion
public R visitLogicalAggregate(LogicalAggregate aggregate, C
context) {
return visit(aggregate, context);
```
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/SlotReference.java:
##
@@ -45,17 +45,16 @@ public SlotReference(String name, DataType dataType,
boolean nullable, Listhttp://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.stats;
+
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import
org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionVisitor;
+import org.apache.doris.statistics.ColumnStats;
+
+import java.util.Map;
+
+
+/**
+ * Calculate selectivity of the filter.
+ */
+public class FilterSelectivityCalculator extends
DefaultExpressionVisitor {
+
+private final Map slotRefToStats;
+
+public FilterSelectivityCalculator(Map slotRefToStats) {
+this.slotRefToStats = slotRefToStats;
+}
+
+/**
+ * Do estimate for the given expr.
+ */
+public double estimate(Expression expression) {
+// For a comparison predicate, only when it's left side is a slot and
right side is a literal, we would
+// consider it as a valid predicate.
+if (expression instanceof ComparisonPredicate
+&& !(expression.child(0) instanceof SlotReference
+&& expression.child(1) instanceof Literal)) {
+return 1.0;
+}
+return expression.accept(this, null);
+}
+
+@Override
+public Double visitCompoundPredicate(CompoundPredicate compoundPredicate,
Void context) {
+Expression leftExpr = compoundPredicate.child(0);
+Expression rightExpr = compoundPredicate.child(1);
+double leftSel = 1;
+double rightSel = 1;
+leftSel = estimate(leftExpr);
+rightSel = estimate(rightExpr);
+return compoundPredicate instanceof Or ? leftSel + rightSel - leftSel
* rightSel : leftSel * rightSel;
+}
+
+// TODO: If right value greater than the max value or less than min value
in column stats, return 0.0 .
+@Override
+public Double visitEqualTo(EqualTo equalTo, Void context) {
+SlotReference left = (SlotReference) equalTo.left();
+ColumnStats columnStats = slotRefToStats.get(left);
+if (columnStats == null) {
+return Expression.DEFAULT_SELECTIVITY;
Review Comment:
i think it is better put DEFAULT_SELECTIVITY in this class
##
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java:
##
@@ -57,7 +59,7 @@ public GroupExpression(Plan plan) {
* @param plan {@link Plan} to reference
* @param children children groups in memo
*/
-public GroupExpression(Plan plan, List children) {
+public GroupExpression(Plan plan, @NotNull List children) {
Review Comment:
why add this annotation?
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStats.java:
##
@@ -76,6 +7
[GitHub] [doris] englefly opened a new pull request, #11248: [feature] add rewrite rule to merge consecutive filter nodes
englefly opened a new pull request, #11248: URL: https://github.com/apache/doris/pull/11248 # Proposed changes this rule aims to merge consecutive filters. For example: logical plan tree: project | filter(a>0) | filter(b>0) | scan transformed to: project | filter(a>0 and b>0) | scan Issue Number: close #xxx ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
Kikyou1997 commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r930728800
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalSubQueryAlias.java:
##
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * The node of logical plan for sub query and alias
+ *
+ * @param param
+ */
+public class LogicalSubQueryAlias extends
LogicalUnary {
+private final String alias;
+
+public LogicalSubQueryAlias(String tableAlias, CHILD_TYPE child) {
+this(tableAlias, Optional.empty(), Optional.empty(), child);
+}
+
+public LogicalSubQueryAlias(String tableAlias, Optional
groupExpression,
+Optional logicalProperties, CHILD_TYPE child) {
+super(PlanType.LOGICAL_SUBQUERY_ALIAS, groupExpression,
logicalProperties, child);
+this.alias = tableAlias;
+
+}
+
+public List computeOutput(Plan input) {
+return input.getOutput().stream()
+.map(slot -> slot.withQualifier(ImmutableList.of(alias)))
+.collect(Collectors.toList());
+}
+
+public String getAlias() {
+return alias;
+}
+
+@Override
+public String toString() {
+return "LogicalSubQueryAlias (" + alias.toString() + ")";
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
+}
+if (o == null || getClass() != o.getClass()) {
+return false;
+}
+LogicalSubQueryAlias that = (LogicalSubQueryAlias) o;
+return alias.equals(that.alias);
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(alias);
+}
+
+@Override
+public Plan withChildren(List children) {
+Preconditions.checkArgument(children.size() == 1);
+return new LogicalSubQueryAlias<>(alias, children.get(0));
+}
+
+@Override
+public R accept(PlanVisitor visitor, C context) {
+return visitor.visitSubQueryAlias((LogicalSubQueryAlias) this,
context);
+}
+
+@Override
+public List getExpressions() {
+return Collections.emptyList();
+}
+
+@Override
+public Plan withGroupExpression(Optional groupExpression)
{
+return new LogicalSubQueryAlias<>(alias, groupExpression,
Optional.of(logicalProperties), child());
+}
Review Comment:
Ok, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Echo18322276280 opened a new issue, #11249: https://doris.incubator.apache.org/docs/get-starting/ Help docs describes the error
Echo18322276280 opened a new issue, #11249: URL: https://github.com/apache/doris/issues/11249 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version docs version 1.1 ### What's Wrong? https://doris.incubator.apache.org/docs/sql-manual/sql-reference/Cluster-Management-Statements/ALTER-SYSTEM-DROP-FOLLOWER； https://doris.incubator.apache.org/docs/sql-manual/sql-reference/Cluster-Management-Statements/ALTER-SYSTEM-DROP-OBSERVER； There is an error in the description of the linked page，And the error doesn't stop here，Please check the documentation in this directory about Cluster management！ ### What You Expected? Check for corrections ASAP ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly closed pull request #11248: [feature] add rewrite rule to merge consecutive filter nodes
englefly closed pull request #11248: [feature] add rewrite rule to merge consecutive filter nodes URL: https://github.com/apache/doris/pull/11248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new issue, #11250: [Bug] be core，doris::vectorized::VMergeIteratorContext::copy_row
liaoxin01 opened a new issue, #11250:
URL: https://github.com/apache/doris/issues/11250
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
master
### What's Wrong?
Aborted at 1658899655 (unix time) try "date -d @1658899655" if you are using
GNU date *** *** Current BE git commitID: 352a3819b *** *** SIGABRT unkown
detail explain (@0xace2b) received by PID 708139 (TID 0x7f6f90da0700) from PID
708139; stack trace: *** 0# doris::signal::(anonymous
namespace)::FailureSignalHandler(int, siginfo_t*, void*) at
/home/zcp/repo_center/doris_master/be/src/common/signal_handler.h:420 1#
0x7F6F98B800C0 in /lib/x86_64-linux-gnu/libc.so.6 2# raise in
/lib/x86_64-linux-gnu/libc.so.6 3# abort in /lib/x86_64-linux-gnu/libc.so.6 4#
0x55A2C3543C49 in /mnt/ssd01/doris-master/VEC_ASAN/be/lib/doris_be 5#
0x55A2C353925D at src/[logging.cc:1650](http://logging.cc:1650/) 6#
google::LogMessage::SendToLog() at
src/[logging.cc:1607](http://logging.cc:1607/) 7# google::LogMessage::Flush()
at src/[logging.cc:1477](http://logging.cc:1477/) 8#
google::LogMessageFatal::~LogMessageFatal() at
src/[logging.cc:2227](http://logging.cc:2227/) 9# doris::vectorized::Colu
mnNullable const& assert_cast(doris::vectorized::IColumn const&) at
/home/zcp/repo_center/doris_master/be/src/vec/common/assert_cast.h:50 10#
doris::vectorized::ColumnNullable::insert_from(doris::vectorized::IColumn
const&, unsigned long) at
/home/zcp/repo_center/doris_master/be/src/vec/columns/column_nullable.cpp:202
11#
doris::vectorized::VMergeIteratorContext::copy_row(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_master/be/src/vec/olap/vgeneric_iterators.cpp:187
12# doris::vectorized::VMergeIterator::next_batch(doris::vectorized::Block*) at
/home/zcp/repo_center/doris_master/be/src/vec/olap/vgeneric_iterators.cpp:356
13# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at
/home/zcp/repo_center/doris_master/be/src/olap/rowset/beta_rowset_reader.cpp:191
14# doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row()
at /home/zcp/repo_center/doris_master/be/src/vec/
olap/vcollect_iterator.cpp:207 15#
doris::vectorized::VCollectIterator::Level0Iterator::init() at
/home/zcp/repo_center/doris_master/be/src/vec/olap/vcollect_iterator.cpp:192
16#
doris::vectorized::VCollectIterator::build_heap(std::vector,
std::allocator > >&) at
/home/zcp/repo_center/doris_master/be/src/vec/olap/vcollect_iterator.cpp:67 17#
doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams
const&, std::vector,
std::allocator > >) at
/home/zcp/repo_center/doris_master/be/src/vec/olap/block_reader.cpp:63 18#
doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&)
at /home/zcp/repo_center/doris_master/be/src/vec/olap/block_reader.cpp:135 19#
doris::vectorized::VOlapScanner::open() at
/home/zcp/repo_center/doris_master/be/src/vec/exec/volap_scanner.cpp:130 20#
doris::vectorized::VOla
pScanNode::scanner_thread(doris::vectorized::VOlapScanner) at
/home/zcp/repo_center/doris_master/be/src/vec/exec/volap_scan_node.cpp:399 21#
doris::vectorized::VOlapScanNode::_start_scanner_thread_task(doris::RuntimeState*,
int)::{lambda()#1}::operator()() const at
/home/zcp/repo_center/doris_master/be/src/vec/exec/volap_scan_node.cpp:1887 22#
void std::invoke_impl(std::_invoke_other,
doris::vectorized::VOlapScanNode::_start_scanner_thread_task(doris::RuntimeState*,
int)::{lambda()#1}&) at
/var/local/ldb_toolchain/include/c++/11/bits/invoke.h:61
### What You Expected?
fix it
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #11251: [Improvement][chore] add const to all operator==
BiteThet opened a new pull request, #11251: URL: https://github.com/apache/doris/pull/11251 # Proposed changes part of work with #9312 1. add const to all `operator==` 2. remove template argument to `construtor/desctructor`, and some unsed code. 3. remove some volatile usage [P1152R4](https://wg21.link/P1152R4) ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [X] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 4. Does it affect the original behavior: - [ ] Yes - [X] No - [ ] I don't know 5. Has unit tests been added: - [ ] Yes - [ ] No - [X] No Need 6. Has document been added or modified: - [ ] Yes - [ ] No - [X] No Need 7. Does it need to update dependencies: - [ ] Yes - [X] No 8. Are there any changes that cannot be rolled back: - [ ] Yes - [X] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 commented on issue #11250: [Bug] be core，doris::vectorized::VMergeIteratorContext::copy_row
liaoxin01 commented on issue #11250: URL: https://github.com/apache/doris/issues/11250#issuecomment-1196387907  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen opened a new pull request, #11252: [feature-wip](unique-key-merge-on-write) update counter, DSIP-018
zhannngchen opened a new pull request, #11252: URL: https://github.com/apache/doris/pull/11252 # Proposed changes Issue Number: close #xxx ## Problem Summary: The follow up patch for #11136 ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [x] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 5. Does it need to update dependencies: - [ ] Yes - [x] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #11233: [Doc] Fix readme error link
adonis0147 commented on PR #11233: URL: https://github.com/apache/doris/pull/11233#issuecomment-1196393704 Duplicated. See #11091 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #11253: [fix](compile)compile failed after merge #11117 #10479 and #11162
morrySnow opened a new pull request, #11253: URL: https://github.com/apache/doris/pull/11253 # Proposed changes Issue Number: close #xxx ## Problem Summary: #7 and #10479 changed interface. #11162 use old interface. All of them passed all checked. But after merged all of them, compile failed. ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [x] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 5. Does it need to update dependencies: - [ ] Yes - [x] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on pull request #11203: [enhancement](nereids) Don't normalize expressions when pushing down predicates
englefly commented on PR #11203: URL: https://github.com/apache/doris/pull/11203#issuecomment-1196395415 Could you explain why we should not normalize expressions? And what your suggestion about when should we normalize them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #11254: [feature](Nereids): add MultiJoin.
jackwener opened a new pull request, #11254: URL: https://github.com/apache/doris/pull/11254 # Proposed changes Issue Number: close #xxx ## Problem Summary: Add MultiJoin. In addtion, when `(joinInputs.size() >= 3 && !conjuncts.isEmpty())`, conjunct still can contains onPredicate. ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11017: [enhancement][Storage] refactor create predicate
yiguolei merged PR #11017: URL: https://github.com/apache/doris/pull/11017 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement][Storage] refactor create predicate (#11017)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 461a31b1f6 [enhancement][Storage] refactor create predicate (#11017)
461a31b1f6 is described below
commit 461a31b1f6fe43d7ac2dc8b797f1a96d935689ca
Author: Pxl <952130...@qq.com>
AuthorDate: Wed Jul 27 16:12:23 2022 +0800
[enhancement][Storage] refactor create predicate (#11017)
---
be/src/olap/column_predicate.h | 11 +
be/src/olap/predicate_creator.h| 260
be/src/olap/reader.cpp | 437 ++---
be/src/olap/reader.h | 21 +-
be/src/vec/runtime/vdatetime_value.h | 2 +-
.../java/org/apache/doris/udf/UdfExecutorTest.java | 4 +-
6 files changed, 311 insertions(+), 424 deletions(-)
diff --git a/be/src/olap/column_predicate.h b/be/src/olap/column_predicate.h
index b2480fe672..2edf19d6da 100644
--- a/be/src/olap/column_predicate.h
+++ b/be/src/olap/column_predicate.h
@@ -51,7 +51,18 @@ struct PredicateTypeTraits {
return (type == PredicateType::LT || type == PredicateType::LE ||
type == PredicateType::GT || type == PredicateType::GE);
}
+
static constexpr bool is_bloom_filter(PredicateType type) { return type ==
PredicateType::BF; }
+
+static constexpr bool is_list(PredicateType type) {
+return (type == PredicateType::IN_LIST || type ==
PredicateType::NOT_IN_LIST);
+}
+
+static constexpr bool is_comparison(PredicateType type) {
+return (type == PredicateType::EQ || type == PredicateType::NE ||
+type == PredicateType::LT || type == PredicateType::LE ||
+type == PredicateType::GT || type == PredicateType::GE);
+}
};
class ColumnPredicate {
diff --git a/be/src/olap/predicate_creator.h b/be/src/olap/predicate_creator.h
new file mode 100644
index 00..1605d34337
--- /dev/null
+++ b/be/src/olap/predicate_creator.h
@@ -0,0 +1,260 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+
+#include "olap/column_predicate.h"
+#include "olap/comparison_predicate.h"
+#include "olap/in_list_predicate.h"
+#include "olap/olap_cond.h"
+#include "olap/tablet_schema.h"
+#include "util/date_func.h"
+
+namespace doris {
+
+template
+class PredicateCreator {
+public:
+virtual ColumnPredicate* create(const TabletColumn& column, int index,
+const ConditionType& conditions, bool
opposite,
+MemPool* pool) = 0;
+virtual ~PredicateCreator() = default;
+};
+
+template
+class IntegerPredicateCreator : public PredicateCreator {
+public:
+ColumnPredicate* create(const TabletColumn& column, int index, const
ConditionType& conditions,
+bool opposite, MemPool* pool) override {
+if constexpr (PredicateTypeTraits::is_list(PT)) {
+phmap::flat_hash_set values;
+for (const auto& condition : conditions) {
+values.insert(convert(condition));
+}
+return new InListPredicateBase(index,
std::move(values), opposite);
+} else {
+static_assert(PredicateTypeTraits::is_comparison(PT));
+return new ComparisonPredicateBase(index,
convert(conditions), opposite);
+}
+}
+
+private:
+CppType convert(const std::string& condition) {
+CppType value = 0;
+std::from_chars(condition.data(), condition.data() + condition.size(),
value);
+return value;
+}
+};
+
+template
+class DecimalPredicateCreator : public PredicateCreator {
+public:
+ColumnPredicate* create(const TabletColumn& column, int index, const
ConditionType& conditions,
+bool opposite, MemPool* pool) override {
+if constexpr (PredicateTypeTraits::is_list(PT)) {
+phmap::flat_hash_set values;
+for (const auto& condition : conditions) {
+values.insert(convert(column, condition));
+
[GitHub] [doris] stalary commented on pull request #11233: [Doc] Fix readme error link
stalary commented on PR #11233: URL: https://github.com/apache/doris/pull/11233#issuecomment-1196405015 > Duplicated. See #11091 Thanks for reminding me, I will close this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary closed pull request #11233: [Doc] Fix readme error link
stalary closed pull request #11233: [Doc] Fix readme error link URL: https://github.com/apache/doris/pull/11233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg opened a new issue, #11255: [Bug] dev-1.1.1 insert data from odbc table to olap table cause be coredump
jacktengg opened a new issue, #11255: URL: https://github.com/apache/doris/issues/11255 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version dev-1.1.1 ### What's Wrong? insert data from odbc table to olap table cause be coredump: ``` I0727 15:56:14.075747 64554 odbc_connector.cpp:115] connect success:Driver=MySQL;Server=127.0.0.1;Port=3306;DataBase=test;Uid=root; I0727 15:56:14.076445 64554 odbc_connector.cpp:138] execute success:SELECT `id`, `reason` FROM `tb1` column count:2 I0727 15:56:14.077713 64780 tablets_channel.cpp:60] open tablets channel: (id=7e694eaec9bb40fa-b526b5111a0b5dc4,index_id=11005), tablets num: 10, timeout(s): 300 [New Thread 0x7fff17fe5700 (LWP 65811)] F0727 15:56:14.078720 64554 text_converter.hpp:282] Check failed: false bad slot type: STRING *** Check failure stack trace: *** @ 0x5bd65dfd google::LogMessage::Fail() @ 0x5bd68339 google::LogMessage::SendToLog() @ 0x5bd65966 google::LogMessage::Flush() @ 0x5bd689a9 google::LogMessageFatal::~LogMessageFatal() @ 0x5a406c76 doris::TextConverter::write_column() @ 0x5a404f56 doris::vectorized::VOdbcScanNode::get_next() @ 0x597eb4be doris::PlanFragmentExecutor::get_vectorized_internal() @ 0x597eae95 doris::PlanFragmentExecutor::open_vectorized_internal() @ 0x597ea957 doris::PlanFragmentExecutor::open() @ 0x597033c2 doris::FragmentExecState::execute() @ 0x597062e0 doris::FragmentMgr::_exec_actual() @ 0x59729117 std::__invoke_impl<>() @ 0x597281c7 _ZSt10__invoke_rIvRMN5doris11FragmentMgrEFvSt10shared_ptrINS0_17FragmentExecStateEESt8functionIFvPNS0_20PlanFragmentExecutorJRPS1_RS4_RS9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESI_E4typeEOSJ_DpOSK_ @ 0x59726c3b _ZNSt12_Bind_resultIvFMN5doris11FragmentMgrEFvSt10shared_ptrINS0_17FragmentExecStateEESt8functionIFvPNS0_20PlanFragmentExecutorPS1_S4_S9_EE6__callIvJEJLm0ELm1ELm2T_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE @ 0x59724c57 std::_Bind_result<>::operator()<>() @ 0x59721ff6 std::__invoke_impl<>() @ 0x5971e160 _ZSt10__invoke_rIvRSt12_Bind_resultIvFMN5doris11FragmentMgrEFvSt10shared_ptrINS1_17FragmentExecStateEESt8functionIFvPNS1_20PlanFragmentExecutorPS2_S5_SA_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESI_E4typeEOSJ_DpOSK_ @ 0x5971a436 std::_Function_handler<>::_M_invoke() @ 0x596c7700 std::function<>::operator()() @ 0x59994766 doris::FunctionRunnable::run() @ 0x599925e5 doris::ThreadPool::dispatch_thread() @ 0x599a04be std::__invoke_impl<>() @ 0x5999ff2d std::__invoke<>() @ 0x5999f761 _ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0T_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE @ 0x5999ec17 std::_Bind<>::operator()<>() @ 0x5999d96a std::__invoke_impl<>() @ 0x5999c532 _ZSt10__invoke_rIvRSt5_BindIFMN5doris10ThreadPoolEFvvEPS2_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESA_E4typeEOSB_DpOSC_ @ 0x5999a6e7 std::_Function_handler<>::_M_invoke() @ 0x596c7700 std::function<>::operator()() @ 0x5998660d doris::Thread::supervise_thread() @ 0x770d8ea5 start_thread @ 0x773ebb0d __clone --Type for more, q to quit, c to continue without paging--c Thread 222 "FragmentMgrThre" received signal SIGABRT, Aborted. ``` ### What You Expected? insert OK ### How to Reproduce? Create table in mysql server and insert data: ``` create table tb1 (`id` int(11) NOT NULL COMMENT "主键id", `reason` text null comment "reason"); insert into tb1 values(1, "aaa"); insert into tb1 values(2, "bbb); ``` Creat external and olap tables in doris: ``` CREATE EXTERNAL RESOURCE `mysql_resource` PROPERTIES ( "host" = "127.0.0.1", "port" = "3306", "user" = "root", "password" = "123", "database" = "test", "driver" = "MySQL", "odbc_type" = "mysql", "type" = "odbc_catalog"); DROP TABLE if exists `ex_tb`; CREATE EXTERNAL TABLE if not exists `ex_tb` ( `id` int(11) NOT NULL COMMENT "主键id", `reason` string NULL COMMENT "理由" ) ENGINE=ODBC COMMENT "ODBC 外部表" PROPERTIES ( "odbc_catalog_resource" = "mysql_resource", "database" = "test", "table" = "tb1" ); DROP TABLE if exists `in_tb`; CREATE TABLE if not exists `in_tb` ( `id` int(11) NOT NULL COMMENT "主键id", `reason` string REPLACE_IF_NOT_NULL
[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #11213: [feature-wip](array-type) support the array type in reverse function
carlvinhust2012 commented on code in PR #11213: URL: https://github.com/apache/doris/pull/11213#discussion_r930759895 ## be/src/vec/functions/function_string.cpp: ## @@ -605,7 +606,7 @@ using FunctionStringLocate = using FunctionStringFindInSet = FunctionBinaryToType; -using FunctionReverse = FunctionStringToString; +using FunctionReverse = FunctionReverseCommon; Review Comment: since the 'reverse()' is not the variadic arguments function, it just can register one function with one name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: stream load partition param
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 676a63c0061 stream load partition param 676a63c0061 is described below commit 676a63c0061489b60cdccf2db466d8a0da0ede2e Author: jiafeng.zhang AuthorDate: Wed Jul 27 16:19:42 2022 +0800 stream load partition param stream load partition param --- docs/data-operate/import/import-way/stream-load-manual.md | 4 ++-- .../current/data-operate/import/import-way/stream-load-manual.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/data-operate/import/import-way/stream-load-manual.md b/docs/data-operate/import/import-way/stream-load-manual.md index 17c0c4276d4..0b65f87aa46 100644 --- a/docs/data-operate/import/import-way/stream-load-manual.md +++ b/docs/data-operate/import/import-way/stream-load-manual.md @@ -141,9 +141,9 @@ The number of rows in the original file = `dpp.abnorm.ALL + dpp.norm.ALL` Import the filter conditions specified by the task. Stream load supports filtering of where statements specified for raw data. The filtered data will not be imported or participated in the calculation of filter ratio, but will be counted as `num_rows_unselected`. -+ partition ++ partitions -Partition information for tables to be imported will not be imported if the data to be imported does not belong to the specified Partition. These data will be included in `dpp.abnorm.ALL`. +Partitions information for tables to be imported will not be imported if the data to be imported does not belong to the specified Partition. These data will be included in `dpp.abnorm.ALL`. + columns diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/stream-load-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/stream-load-manual.md index 35d8048a0c7..12299fba999 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/stream-load-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/stream-load-manual.md @@ -142,7 +142,7 @@ Stream Load 由于使用的是 HTTP 协议，所以所有导入任务有关的 导入任务指定的过滤条件。Stream load 支持对原始数据指定 where 语句进行过滤。被过滤的数据将不会被导入，也不会参与 filter ratio 的计算，但会被计入`num_rows_unselected`。 -- partition +- Partitions 待导入表的 Partition 信息，如果待导入数据不属于指定的 Partition 则不会被导入。这些数据将计入 `dpp.abnorm.ALL` - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11081: [improvement]Remove the website build framework from the docs directory
github-actions[bot] commented on PR #11081: URL: https://github.com/apache/doris/pull/11081#issuecomment-1196416966 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11081: [improvement]Remove the website build framework from the docs directory
github-actions[bot] commented on PR #11081: URL: https://github.com/apache/doris/pull/11081#issuecomment-1196417013 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #11195: [feature-wip](unique-key-merge-on-write) update delete bitmap while p…
dataroaring merged PR #11195: URL: https://github.com/apache/doris/pull/11195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](unique-key-merge-on-write) update delete bitmap while publish version (#11195)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 01e108cb7b [feature-wip](unique-key-merge-on-write) update delete
bitmap while publish version (#11195)
01e108cb7b is described below
commit 01e108cb7b0f5085a63a535f8cf571eb293d6d2b
Author: yixiutt <102007456+yixi...@users.noreply.github.com>
AuthorDate: Wed Jul 27 16:26:42 2022 +0800
[feature-wip](unique-key-merge-on-write) update delete bitmap while publish
version (#11195)
1.make version publish work in version order
2.update delete bitmap while publish version, load current version rowset
primary key and search in pre rowsets
3.speed up publish version task by parallel tablet publish task
Co-authored-by: yixiutt
---
be/src/agent/task_worker_pool.cpp | 10 ++
be/src/common/config.h | 2 +
be/src/common/status.h | 3 +-
be/src/olap/olap_server.cpp| 6 +
be/src/olap/rowset/segment_v2/segment.cpp | 6 +-
be/src/olap/rowset/segment_v2/segment.h| 5 +-
be/src/olap/storage_engine.h | 6 +
be/src/olap/tablet.h | 5 +
be/src/olap/task/engine_publish_version_task.cpp | 146 -
be/src/olap/task/engine_publish_version_task.h | 34 +
be/src/olap/txn_manager.cpp| 102 ++
be/src/olap/txn_manager.h | 5 +
.../olap/engine_storage_migration_task_test.cpp| 4 +-
be/test/olap/tablet_clone_test.cpp | 4 +-
be/test/olap/tablet_cooldown_test.cpp | 4 +-
15 files changed, 297 insertions(+), 45 deletions(-)
diff --git a/be/src/agent/task_worker_pool.cpp
b/be/src/agent/task_worker_pool.cpp
index 1380331e7a..40681ce17d 100644
--- a/be/src/agent/task_worker_pool.cpp
+++ b/be/src/agent/task_worker_pool.cpp
@@ -711,6 +711,13 @@ void
TaskWorkerPool::_publish_version_worker_thread_callback() {
res = _env->storage_engine()->execute_task(&engine_task);
if (res.ok()) {
break;
+} else if (res.precise_code() ==
OLAP_ERR_PUBLISH_VERSION_NOT_CONTINUOUS) {
+// version not continuous, put to queue and wait pre version
publish
+// task execute
+std::unique_lock
worker_thread_lock(_worker_thread_lock);
+_tasks.push_back(agent_task_req);
+_worker_thread_condition_variable.notify_one();
+break;
} else {
LOG(WARNING) << "publish version error, retry.
[transaction_id="
<< publish_version_req.transaction_id
@@ -719,6 +726,9 @@ void
TaskWorkerPool::_publish_version_worker_thread_callback() {
std::this_thread::sleep_for(std::chrono::seconds(1));
}
}
+if (res.precise_code() == OLAP_ERR_PUBLISH_VERSION_NOT_CONTINUOUS) {
+continue;
+}
TFinishTaskRequest finish_task_request;
if (!res) {
diff --git a/be/src/common/config.h b/be/src/common/config.h
index ee8698bcb5..9891df2f18 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -81,6 +81,8 @@ CONF_Int32(push_worker_count_normal_priority, "3");
CONF_Int32(push_worker_count_high_priority, "3");
// the count of thread to publish version
CONF_Int32(publish_version_worker_count, "8");
+// the count of tablet thread to publish version
+CONF_Int32(tablet_publish_txn_max_thread, "32");
// the count of thread to clear transaction task
CONF_Int32(clear_transaction_task_worker_count, "1");
// the count of thread to delete
diff --git a/be/src/common/status.h b/be/src/common/status.h
index 830d417a7c..f441ccd759 100644
--- a/be/src/common/status.h
+++ b/be/src/common/status.h
@@ -231,7 +231,8 @@ namespace doris {
M(OLAP_ERR_ROWSET_READ_FAILED, -3111, "", true) \
M(OLAP_ERR_ROWSET_INVALID_STATE_TRANSITION, -3112, "", true) \
M(OLAP_ERR_STRING_OVERFLOW_IN_VEC_ENGINE, -3113, "", true) \
-M(OLAP_ERR_ROWSET_ADD_MIGRATION_V2, -3114, "", true)
+M(OLAP_ERR_ROWSET_ADD_MIGRATION_V2, -3114, "", true) \
+M(OLAP_ERR_PUBLISH_VERSION_NOT_CONTINUOUS, -3115, "", false)
enum ErrorCode {
#define M(NAME, ERRORCODE, DESC, STACKTRACEENABLED) NAME = ERRORCODE,
diff --git a/be/src/olap/olap_server.cpp b/be/src/olap/olap_server.cpp
index fbb1811705..28e9241985 100644
--- a/be/src/olap/olap_server.cpp
+++ b/be/src/olap/olap_server.cpp
@@ -137,6 +137,12 @@ Status StorageEngine::start_bg_threads() {
&_cooldown_tasks_producer_thread));
LOG(INFO) << "cooldown tasks producer thread started";
+// add tablet publish versio
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
Kikyou1997 commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r930770660
##
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##
@@ -104,13 +105,18 @@ public PhysicalPlan plan(LogicalPlan plan,
PhysicalProperties outputProperties,
// cascades style optimize phase.
.setJobContext(outputProperties);
+finalizeAnalyze();
rewrite();
optimize();
// Get plan directly. Just for SSB.
return getRoot().extractPlan();
}
+private void finalizeAnalyze() {
+new FinalizeAnalyzeJob(plannerContext).execute();
Review Comment:
Do we really need a specific job to do this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #10565: [feature-wip](multi-catalog) Support es datasource
morningman commented on code in PR #10565: URL: https://github.com/apache/doris/pull/10565#discussion_r930768575 ## fe/fe-core/pom.xml: ## @@ -624,6 +624,7 @@ under the License. org.apache.iceberg iceberg-core +provided Review Comment: remove this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11166: [doc][fix]Modify docs/community to keep the same as the website site directory structure
github-actions[bot] commented on PR #11166: URL: https://github.com/apache/doris/pull/11166#issuecomment-1196419860 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11166: [doc][fix]Modify docs/community to keep the same as the website site directory structure
github-actions[bot] commented on PR #11166: URL: https://github.com/apache/doris/pull/11166#issuecomment-1196419905 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
sohardforaname commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r930773125
##
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##
@@ -104,13 +105,18 @@ public PhysicalPlan plan(LogicalPlan plan,
PhysicalProperties outputProperties,
// cascades style optimize phase.
.setJobContext(outputProperties);
+finalizeAnalyze();
rewrite();
optimize();
// Get plan directly. Just for SSB.
return getRoot().extractPlan();
}
+private void finalizeAnalyze() {
+new FinalizeAnalyzeJob(plannerContext).execute();
Review Comment:
yes, because we should eliminate all the logical-subquery-alias nodes, or
the rewrite phase will core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #11149: [feature-wip](unique-key-merge-on-write) row id conversion for compaction
dataroaring merged PR #11149: URL: https://github.com/apache/doris/pull/11149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (01e108cb7b -> d4fb27125a)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 01e108cb7b [feature-wip](unique-key-merge-on-write) update delete bitmap while publish version (#11195) add d4fb27125a [feature-wip](unique-key-merge-on-write) row id conversion for compaction (#11149) No new revisions were added by this update. Summary of changes: be/src/olap/compaction.cpp | 4 + be/src/olap/compaction.h | 1 + be/src/olap/iterators.h| 6 ++ be/src/olap/merger.cpp | 26 + be/src/olap/merger.h | 2 + be/src/olap/reader.cpp | 1 + be/src/olap/reader.h | 3 + be/src/olap/rowid_conversion.h | 118 + be/src/olap/rowset/beta_rowset_reader.cpp | 9 ++ be/src/olap/rowset/beta_rowset_reader.h| 6 ++ be/src/olap/rowset/beta_rowset_writer.cpp | 1 + be/src/olap/rowset/beta_rowset_writer.h| 8 ++ be/src/olap/rowset/rowset_reader.h | 8 ++ be/src/olap/rowset/rowset_reader_context.h | 1 + be/src/olap/rowset/rowset_writer.h | 4 + be/src/olap/rowset/segment_v2/segment_iterator.cpp | 39 +-- be/src/olap/rowset/segment_v2/segment_iterator.h | 11 ++ be/src/vec/olap/block_reader.cpp | 13 +++ be/src/vec/olap/block_reader.h | 4 + be/src/vec/olap/vcollect_iterator.cpp | 29 + be/src/vec/olap/vcollect_iterator.h| 24 + be/src/vec/olap/vgeneric_iterators.cpp | 43 be/test/CMakeLists.txt | 1 + be/test/olap/rowid_conversion_test.cpp | 105 ++ be/test/olap/rowset/beta_rowset_test.cpp | 21 25 files changed, 482 insertions(+), 6 deletions(-) create mode 100644 be/src/olap/rowid_conversion.h create mode 100644 be/test/olap/rowid_conversion_test.cpp - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] spaces-X commented on a diff in pull request #11204: [fix](be): fix stack overflow in unhex function
spaces-X commented on code in PR #11204:
URL: https://github.com/apache/doris/pull/11204#discussion_r930779540
##
be/src/udf/udf.h:
##
@@ -255,7 +256,8 @@ class FunctionContext {
// Create a test FunctionContext object. The caller is responsible for
calling delete
// on it. This context has additional debugging validation enabled.
-static FunctionContext* create_test_context();
+// And the default value of mem_pool is nullprt.
Review Comment:
```suggestion
// And the default value of mem_pool is nullptr.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg opened a new pull request, #11256: [bugfix] handle string type in TextConverter::write_column
jacktengg opened a new pull request, #11256: URL: https://github.com/apache/doris/pull/11256 # Proposed changes Issue Number: close #11255 ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [x ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [x ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ x] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ x] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [x ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] spaces-X commented on pull request #11204: [fix](be): fix stack overflow in unhex function
spaces-X commented on PR #11204: URL: https://github.com/apache/doris/pull/11204#issuecomment-1196431095 @dataroaring @yiguolei Fix an ut error and it has been passed in my local env. Please review again thx. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11116: Optimize regexp and like using hyperscan
yiguolei merged PR #6: URL: https://github.com/apache/doris/pull/6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: Optimize regexp and like using hyperscan (#11116)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4ea2c04676 Optimize regexp and like using hyperscan (#6)
4ea2c04676 is described below
commit 4ea2c04676e104a9e25576cf7d4e357577707abf
Author: Kang
AuthorDate: Wed Jul 27 16:43:58 2022 +0800
Optimize regexp and like using hyperscan (#6)
* use hyperscan instead of re2 for regexp and like function
---
be/CMakeLists.txt | 4 ++
be/src/vec/functions/like.cpp | 137 ++
be/src/vec/functions/like.h | 47 ---
3 files changed, 125 insertions(+), 63 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index c4ad9093e6..7f11c4cb9c 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -144,6 +144,9 @@ set_target_properties(backtrace PROPERTIES
IMPORTED_LOCATION ${THIRDPARTY_DIR}/l
add_library(re2 STATIC IMPORTED)
set_target_properties(re2 PROPERTIES IMPORTED_LOCATION
${THIRDPARTY_DIR}/lib/libre2.a)
+add_library(hyperscan STATIC IMPORTED)
+set_target_properties(hyperscan PROPERTIES IMPORTED_LOCATION
${THIRDPARTY_DIR}/lib64/libhs.a)
+
add_library(odbc STATIC IMPORTED)
set_target_properties(odbc PROPERTIES IMPORTED_LOCATION
${THIRDPARTY_DIR}/lib/libodbc.a)
@@ -638,6 +641,7 @@ set(COMMON_THIRDPARTY
thriftnb
glog
re2
+hyperscan
pprof
lz4
libevent
diff --git a/be/src/vec/functions/like.cpp b/be/src/vec/functions/like.cpp
index 522fb0b621..8f84c0f21b 100644
--- a/be/src/vec/functions/like.cpp
+++ b/be/src/vec/functions/like.cpp
@@ -82,6 +82,61 @@ Status
FunctionLikeBase::constant_substring_fn(LikeSearchState* state, const Str
return Status::OK();
}
+Status FunctionLikeBase::constant_regex_fn(LikeSearchState* state, const
StringValue& val,
+ const StringValue& pattern,
unsigned char* result) {
+auto ret = hs_scan(state->hs_database.get(), val.ptr, val.len, 0,
state->hs_scratch.get(),
+ state->hs_match_handler, (void*)result);
+if (ret != HS_SUCCESS && ret != HS_SCAN_TERMINATED) {
+return Status::RuntimeError(fmt::format("hyperscan error: {}", ret));
+}
+
+return Status::OK();
+}
+
+Status FunctionLikeBase::regexp_fn(LikeSearchState* state, const StringValue&
val,
+ const StringValue& pattern, unsigned char*
result) {
+std::string re_pattern(pattern.ptr, pattern.len);
+
+hs_database_t* database = nullptr;
+hs_scratch_t* scratch = nullptr;
+RETURN_IF_ERROR(hs_prepare(nullptr, re_pattern.c_str(), &database,
&scratch));
+
+auto ret =
+hs_scan(database, val.ptr, val.len, 0, scratch,
state->hs_match_handler, (void*)result);
+if (ret != HS_SUCCESS && ret != HS_SCAN_TERMINATED) {
+return Status::RuntimeError(fmt::format("hyperscan error: {}", ret));
+}
+
+hs_free_scratch(scratch);
+hs_free_database(database);
+
+return Status::OK();
+}
+
+// hyperscan compile expression to database and allocate scratch space
+Status FunctionLikeBase::hs_prepare(FunctionContext* context, const char*
expression,
+hs_database_t** database, hs_scratch_t**
scratch) {
+hs_compile_error_t* compile_err;
+if (hs_compile(expression, HS_FLAG_DOTALL, HS_MODE_BLOCK, NULL, database,
&compile_err) !=
+HS_SUCCESS) {
+hs_free_compile_error(compile_err);
+*database = nullptr;
+if (context) context->set_error("hs_compile regex pattern error");
+return Status::RuntimeError("hs_compile regex pattern error");
+}
+hs_free_compile_error(compile_err);
+
+if (hs_alloc_scratch(*database, scratch) != HS_SUCCESS) {
+hs_free_database(*database);
+*database = nullptr;
+*scratch = nullptr;
+if (context) context->set_error("hs_alloc_scratch allocate scratch
space error");
+return Status::RuntimeError("hs_alloc_scratch allocate scratch space
error");
+}
+
+return Status::OK();
+}
+
Status FunctionLikeBase::execute_impl(FunctionContext* context, Block& block,
const ColumnNumbers& arguments, size_t
result,
size_t /*input_rows_count*/) {
@@ -183,28 +238,20 @@ Status FunctionLikeBase::vector_vector(const
ColumnString::Chars& values,
Status FunctionLike::like_fn(LikeSearchState* state, const StringValue& val,
const StringValue& pattern, unsigned char*
result) {
std::string re_pattern;
-RE2::Options opts;
-opts.set_never_nl(false);
-opts.set_dot_nl(true);
convert_like_pattern(state, std::string(pattern.ptr, pattern.len),
&re_pattern);
-re2::RE2 re(re_pattern, opts);
-if (re.ok()) {
-*result = RE2::Fu
[GitHub] [doris] yiguolei merged pull request #11256: [bugfix] handle string type in TextConverter::write_column
yiguolei merged PR #11256: URL: https://github.com/apache/doris/pull/11256 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [bugfix] handle string type in TextConverter::write_column (#11256)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new fae5b6cb59 [bugfix] handle string type in TextConverter::write_column
(#11256)
fae5b6cb59 is described below
commit fae5b6cb59da529d8b7465381ae68b18b8dcd245
Author: TengJianPing <18241664+jackte...@users.noreply.github.com>
AuthorDate: Wed Jul 27 16:46:44 2022 +0800
[bugfix] handle string type in TextConverter::write_column (#11256)
---
be/src/exec/text_converter.hpp | 1 +
1 file changed, 1 insertion(+)
diff --git a/be/src/exec/text_converter.hpp b/be/src/exec/text_converter.hpp
index 1eacaadfb9..6cf4fa94a2 100644
--- a/be/src/exec/text_converter.hpp
+++ b/be/src/exec/text_converter.hpp
@@ -189,6 +189,7 @@ inline bool TextConverter::write_column(const
SlotDescriptor* slot_desc,
reinterpret_cast(col_ptr)->get_data().emplace_back(HyperLogLog(Slice(data,
len)));
break;
}
+case TYPE_STRING:
case TYPE_VARCHAR:
case TYPE_CHAR: {
if (need_escape) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](auth) Forbid grant USAGE_PRIV to database.table (#11234)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new be2ac6aa59 [fix](auth) Forbid grant USAGE_PRIV to database.table
(#11234)
be2ac6aa59 is described below
commit be2ac6aa59b3906caa70a8af7c4bf82e5571d99e
Author: Mingyu Chen
AuthorDate: Wed Jul 27 16:51:15 2022 +0800
[fix](auth) Forbid grant USAGE_PRIV to database.table (#11234)
---
.../org/apache/doris/analysis/CreateRoleStmt.java | 2 +-
.../java/org/apache/doris/analysis/GrantStmt.java | 49 --
.../java/org/apache/doris/analysis/RevokeStmt.java | 4 +-
.../org/apache/doris/mysql/privilege/PaloAuth.java | 48 +
.../org/apache/doris/mysql/privilege/AuthTest.java | 20 -
5 files changed, 70 insertions(+), 53 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
index c218ff3197..ec35fcb7b8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
@@ -56,7 +56,7 @@ public class CreateRoleStmt extends DdlStmt {
// check if current user has GRANT priv on GLOBAL level.
if
(!Env.getCurrentEnv().getAuth().checkGlobalPriv(ConnectContext.get(),
PrivPredicate.GRANT)) {
-
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"CREATE USER");
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"CREATE ROLE");
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
index afe5387b35..3dcdec07cb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
@@ -121,9 +121,9 @@ public class GrantStmt extends DdlStmt {
}
if (tblPattern != null) {
-checkPrivileges(analyzer, privileges, role, tblPattern);
+checkTablePrivileges(privileges, role, tblPattern);
} else {
-checkPrivileges(analyzer, privileges, role, resourcePattern);
+checkResourcePrivileges(privileges, role, resourcePattern);
}
}
@@ -136,86 +136,91 @@ public class GrantStmt extends DdlStmt {
* 5.1 User should has GLOBAL level GRANT_PRIV
* 5.2 or user has DATABASE/TABLE level GRANT_PRIV if grant/revoke to/from
certain database or table.
* 5.3 or user should has 'resource' GRANT_PRIV if grant/revoke to/from
certain 'resource'
+ * 6. Can not grant USAGE_PRIV to database or table
*
- * @param analyzer
* @param privileges
* @param role
* @param tblPattern
* @throws AnalysisException
*/
-public static void checkPrivileges(Analyzer analyzer, List
privileges,
- String role, TablePattern tblPattern)
throws AnalysisException {
+public static void checkTablePrivileges(List privileges,
String role, TablePattern tblPattern)
+throws AnalysisException {
// Rule 1
if (tblPattern.getPrivLevel() != PrivLevel.GLOBAL &&
(privileges.contains(PaloPrivilege.ADMIN_PRIV)
|| privileges.contains(PaloPrivilege.NODE_PRIV))) {
-throw new AnalysisException("ADMIN_PRIV and NODE_PRIV can only be
granted on *.*.*");
+throw new AnalysisException("ADMIN_PRIV and NODE_PRIV can only be
granted/revoke on/from *.*.*");
}
// Rule 2
if (privileges.contains(PaloPrivilege.NODE_PRIV) &&
!Env.getCurrentEnv().getAuth()
.checkGlobalPriv(ConnectContext.get(),
PrivPredicate.OPERATOR)) {
-throw new AnalysisException("Only the user with NODE_PRIV can
grant NODE_PRIV to other user");
+throw new AnalysisException("Only user with NODE_PRIV can
grant/revoke NODE_PRIV to other user");
}
if (role != null) {
// Rule 3 and 4
if
(!Env.getCurrentEnv().getAuth().checkGlobalPriv(ConnectContext.get(),
PrivPredicate.GRANT)) {
-
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"GRANT");
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"GRANT/ROVOKE");
}
} else {
// Rule 5.1 and 5.2
if (tblPattern.getPrivLevel() == PrivLevel.GLOBAL) {
if
(!Env.getCurrentEnv().getAuth().checkGlobalPriv(ConnectContext.get(),
PrivPredicate.GRANT)) {
-
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"GRANT");
+
[GitHub] [doris] yiguolei closed issue #11235: [Bug] FE start failed at replaying drop user edit log
yiguolei closed issue #11235: [Bug] FE start failed at replaying drop user edit log URL: https://github.com/apache/doris/issues/11235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11234: [fix](auth) Forbid grant USAGE_PRIV to database.table
yiguolei merged PR #11234: URL: https://github.com/apache/doris/pull/11234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11253: [fix](compile)compile failed after merge #11117 #10479 and #11162
github-actions[bot] commented on PR #11253: URL: https://github.com/apache/doris/pull/11253#issuecomment-1196447566 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11253: [fix](compile)compile failed after merge #11117 #10479 and #11162
github-actions[bot] commented on PR #11253: URL: https://github.com/apache/doris/pull/11253#issuecomment-1196447616 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on pull request #11213: [feature-wip](array-type) support the array type in reverse function
carlvinhust2012 commented on PR #11213: URL: https://github.com/apache/doris/pull/11213#issuecomment-1196456640 > please also update reverse docs. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #11257: [improvement]Use phmap::flat_hash_set in AggregateFunctionUniq
mrhhsg opened a new pull request, #11257: URL: https://github.com/apache/doris/pull/11257 # Proposed changes Issue Number: close #xxx ## Problem Summary: Test on clikcbench with SQL: ```sql SELECT COUNT(DISTINCT SearchPhrase), count(distinct userid) FROM hits; ``` Executing time reduced to 7.60 sec from 9.73 sec ## Checklist(Required) 1. Type of your changes: - [x] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11179: [feature](nereids) Add stats derive framework for new optimizer
Kikyou1997 commented on code in PR #11179:
URL: https://github.com/apache/doris/pull/11179#discussion_r930816157
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStats.java:
##
@@ -76,6 +76,22 @@ public class ColumnStats {
private LiteralExpr minValue;
private LiteralExpr maxValue;
+public ColumnStats(ColumnStats other) {
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #11251: [Improvement][chore] add const to all operator==
BiteThet commented on PR #11251: URL: https://github.com/apache/doris/pull/11251#issuecomment-1196468933 > P0 fail because fe build fail, I will rebase after #11253 merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Im-Manshushu opened a new issue, #11258: [Feature] JSON data is dynamically written to the Doris table
Im-Manshushu opened a new issue, #11258: URL: https://github.com/apache/doris/issues/11258 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description The flink connector automatically writes the data into the corresponding table by parsing the database name and table name in the JSON data ### Use case Collect data in real time, consume the data in Kafka through Flink, there are multiple table data in a topic, JSON data format, and the data includes database name, table name, field name and data value ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11075: [Doc] update flink connector faq
jackwener commented on code in PR #11075: URL: https://github.com/apache/doris/pull/11075#discussion_r930819530 ## docs/zh-CN/docs/ecosystem/flink-doris-connector.md: ## @@ -467,3 +467,8 @@ WITH ( 'sink.properties.columns' = 'dt,page,user_id,user_id=to_bitmap(user_id)' ) ``` + +2. **errCode = 2, detailMessage = Label [label_0_1] has already been used, relate to txn [19650]** Review Comment: We can add it to https://doris.apache.org/zh-CN/docs/faq/data-faq -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 commented on a diff in pull request #11114: [Doc]Add Introduction to Apache Doris
xy720 commented on code in PR #4: URL: https://github.com/apache/doris/pull/4#discussion_r930817636 ## docs/zh-CN/docs/summary/basic-summary.md: ## @@ -24,6 +24,76 @@ specific language governing permissions and limitations under the License. --> -# Doris 基本概念 +# Doris 介绍 -(TODO) \ No newline at end of file +Apache Doris 是一个基于 MPP 架构的高性能、实时的分析型数据库，以极速易用的特点被人们所熟知，仅需亚秒级响应时间即可返回海量数据下的查询结果，不仅可以支持高并发的点查询场景，也能支持高吞吐的复杂分析场景。基于此，Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、数据湖联邦查询加速等使用场景，用户可以在此之上构建用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用。 + +Apache Doris 最早是诞生于百度广告报表业务的 Palo 项目，2017 年正式对外开源，2018 年 7 月由百度捐赠给 Apache 基金会进行孵化，之后在 Apache 导师的指导下由孵化器项目管理委员会成员进行孵化和运营。目前 Apache Doris 社区已经聚集了来自不同行业近百家企业的 300 余位贡献者，并且每月活跃贡献者人数也接近 100 位。 2022 年 6 月，Apache Doris 成功从 Apache 孵化器毕业，正式成为 Apache 顶级项目（Top-Level Project，TLP） + +Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群体，截止目前， Apache Doris 已经在全球超过 500 家企业的生产环境中得到应用，在中国市值或估值排行前 50 的互联网公司中，有超过 80% 长期使用 Apache Doris，包括百度、美团、小米、京东、字节跳动、腾讯、网易、快手、微博、贝壳等。同时在一些传统行业如金融、能源、制造、电信等领域也有着丰富的应用。 + +# 使用场景 + +如下图所示，数据源经过各种数据集成和加工处理后，通常会入库到实时数仓 Doris 和离线湖仓（Hive, Iceberg, Hudi 中），Apache Doris 被广泛应用在以下场景中。 + + +- 报表分析 + +- 实时看板 （Dashboards） +- 面向企业内部分析师和管理者的报表 +- 面向用户或者客户的高并发报表分析（Customer Facing Analytics）。比如面向网站主的站点分析、面向广告主的广告报表，并发通常要求成千上万的 QPS ，查询延时要求毫秒级响应。著名的电商公司京东在广告报表中使用 Apache Doris ，每天写入 100 亿行数据，查询并发 QPS 上万，99 分位的查询延时 150ms。 + +- 即席查询（Ad-hoc Query）：面向分析师的自助分析，查询模式不固定，要求较高的吞吐。小米公司基于 Doris 构建了增长分析平台（Growing Analytics，GA），利用用户行为数据对业务进行增长分析，平均查询延时 10s，95 分位的查询延时 30s 以内，每天的 SQL 查询量为数万条。 + +- 统一数仓构建 ：一个平台满足统一的数据仓库建设需求，简化繁琐的大数据软件栈。海底捞基于 Doris 构建的统一数仓，替换了原来由 Spark、Hive、Kudu、Hbase、Phoenix 组成的旧架构，架构大大简化。 + +- 数据湖联邦查询：通过外表的方式联邦分析位于 Hive、Iceberg、Hudi 中的数据，在避免数据拷贝的前提下，查询性能大幅提升。 + +# 技术概述 + +Doris**整体架构**如下图所示，Doris 架构非常简单，只有两类进程 + +- **Frontend（FE）**，主要负责用户请求的接入、查询解析规划、元数据的管理、节点管理相关工作。 + +- **Backend（BE）**，主要负责数据存储、查询计划的执行。 + +这两类进程都是可以横向扩展的，单集群可以支持到数百台机器，数十 PB 的存储容量。并且这两类进程通过一致性协议来保证服务的高可用和数据的高可靠。这种高度集成的架构设计极大的降低了一款分布式系统的运维成本。 + + + +在**使用接口**方面，Doris 采用 MySQ L 协议，高度兼容 MySQL 语法，支持标准 SQL，用户可以通过各类客户端工具来访问 Doris，并支持与 BI 工具的无缝对接。 Review Comment: ```suggestion 在**使用接口**方面，Doris 采用 MySQL 协议，高度兼容 MySQL 语法，支持标准 SQL，用户可以通过各类客户端工具来访问 Doris，并支持与 BI 工具的无缝对接。 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #11259: [fix](auth) Forbid grant USAGE_PRIV to database.table (#11234)
yiguolei opened a new pull request, #11259: URL: https://github.com/apache/doris/pull/11259 # Proposed changes Issue Number: close #xxx ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener closed pull request #10228: [doc]Update the zh-CN and en docs downloads link
jackwener closed pull request #10228: [doc]Update the zh-CN and en docs downloads link URL: https://github.com/apache/doris/pull/10228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #11253: [fix](compile)compile failed after merge #11117 #10479 and #11162
BiteThet commented on PR #11253: URL: https://github.com/apache/doris/pull/11253#issuecomment-1196484043 I will merge this pr as fast as possible because master build fail now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] kpfly commented on issue #11247: BE.alive is false when i begain sql
kpfly commented on issue #11247: URL: https://github.com/apache/doris/issues/11247#issuecomment-1196484376 > W0727 14:40:58.582839 8567 internal_service.cpp:446] apply filter meet error: Invalid argument: fragment-id: 10c8dc9bf6d94460-a3fb832330c749a9 W0727 14:40:58.592717 8557 internal_service.cpp:446] apply filter meet error: Invalid argument: fragment-id: 10c8dc9bf6d94460-a3fb832330c74db4 F0727 14:57:59.556994 9758 vslot_ref.cpp:63] Check failed: _column_id >= 0 (-1 vs. 0) , SlotRef(slot_id=419 type=VARCHAR codegen=false) column id: -1, name: pname, is nulable: 1, @ 0x55aa0d75bbf9 doris::vectorized::VSlotRef::execute() @ 0x55aa0d755a24 doris::vectorized::VInPredicate::execute() @ 0x55aa0d761719 doris::vectorized::VCaseExpr::execute() @ 0x55aa0d17b01b doris::vectorized::VExprContext::get_output_block_after_execute_exprs() @ 0x55aa0e2e16a9 doris::vectorized::VMysqlResultWriter::append_block() @ 0x55aa0e2a1cbc doris::vectorized::VResultSink::send() @ 0x55aa0cb89115 doris::PlanFragmentExecutor::open_vectorized_internal() @ 0x55aa0cb8a82f doris::PlanFragmentExecutor::open() @ 0x55aa0cb04 1c5 doris::FragmentExecState::execute() @ 0x55aa0cb08e7a doris::FragmentMgr::_exec_actual() @ 0x55aa0cb1241c std::_Function_handler<>::_M_invoke() @ 0x55aa0ccb5ca9 doris::ThreadPool::dispatch_thread() @ 0x55aa0ccaf29f doris::Thread::supervise_thread() @ 0x7f4a69fb7609 start_thread @ 0x7f4a6a0f1133 clone @ (nil) (unknown) Could you share the sql? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](compile)fix compile failed after merge #11117 #10479 and #11162 (#11253)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 42d76b54dc [fix](compile)fix compile failed after merge #7 #10479
and #11162 (#11253)
42d76b54dc is described below
commit 42d76b54dc2912308d74f2cd5ae430fc48768cb7
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Jul 27 17:25:06 2022 +0800
[fix](compile)fix compile failed after merge #7 #10479 and #11162
(#11253)
fix the compile failed after merge #7 #10479 and #11162
---
.../rules/rewrite/logical/PushPredicateThroughAggregation.java| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java
index bc4155bdec..d2e2912437 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java
@@ -70,14 +70,14 @@ public class PushPredicateThroughAggregation extends
OneRewriteRuleFactory {
return logicalFilter(logicalAggregate()).then(filter -> {
LogicalAggregate aggregate = filter.child();
Set groupBySlots = new HashSet<>();
-for (Expression groupByExpression :
aggregate.getGroupByExpressionList()) {
+for (Expression groupByExpression :
aggregate.getGroupByExpressions()) {
if (groupByExpression instanceof Slot) {
groupBySlots.add((Slot) groupByExpression);
}
}
List pushDownPredicates = Lists.newArrayList();
List filterPredicates = Lists.newArrayList();
-
ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+
ExpressionUtils.extractConjunctive(filter.getPredicates()).forEach(conjunct -> {
Set conjunctSlots = SlotExtractor.extractSlot(conjunct);
if (groupBySlots.containsAll(conjunctSlots)) {
pushDownPredicates.add(conjunct);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt merged pull request #11253: [fix](compile)compile failed after merge #11117 #10479 and #11162
BiteThet merged PR #11253: URL: https://github.com/apache/doris/pull/11253 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11209: [feature] (Nereids) support limit clause
englefly commented on code in PR #11209: URL: https://github.com/apache/doris/pull/11209#discussion_r930831997 ## fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4: ## @@ -119,6 +119,12 @@ sortItem : expression ordering = (ASC | DESC)? ; +limitClause +: (LIMIT INTEGER_VALUE)? +| (LIMIT INTEGER_VALUE OFFSET INTEGER_VALUE)? +| (LIMIT INTEGER_VALUE COMMA INTEGER_VALUE)? Review Comment: `select * from t order by a limit 100, 3` is equivalent to `select * from t order by a limit 3 offset 100` we should follow our `cup` definition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11251: [Improvement][chore] add const to all operator==
github-actions[bot] commented on PR #11251: URL: https://github.com/apache/doris/pull/11251#issuecomment-1196495439 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11251: [Improvement][chore] add const to all operator==
github-actions[bot] commented on PR #11251: URL: https://github.com/apache/doris/pull/11251#issuecomment-1196495482 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11259: [fix](auth) Forbid grant USAGE_PRIV to database.table (#11234)
yiguolei merged PR #11259: URL: https://github.com/apache/doris/pull/11259 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [fix](auth) Forbid grant USAGE_PRIV to database.table (#11234) (#11259)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new 555f5412c3 [fix](auth) Forbid grant USAGE_PRIV to database.table
(#11234) (#11259)
555f5412c3 is described below
commit 555f5412c3d7237a1930bff92d701ccb11644568
Author: yiguolei <676222...@qq.com>
AuthorDate: Wed Jul 27 17:36:53 2022 +0800
[fix](auth) Forbid grant USAGE_PRIV to database.table (#11234) (#11259)
Co-authored-by: Mingyu Chen
---
.../java/org/apache/doris/analysis/GrantStmt.java | 25 ++-
.../java/org/apache/doris/analysis/RevokeStmt.java | 4 +-
.../org/apache/doris/mysql/privilege/PaloAuth.java | 49 +-
.../org/apache/doris/mysql/privilege/AuthTest.java | 20 -
4 files changed, 57 insertions(+), 41 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
index 0c18564962..76849a6b3d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
@@ -120,9 +120,9 @@ public class GrantStmt extends DdlStmt {
}
if (tblPattern != null) {
-checkPrivileges(analyzer, privileges, role, tblPattern);
+checkTablePrivileges(privileges, role, tblPattern);
} else {
-checkPrivileges(analyzer, privileges, role, resourcePattern);
+checkResourcePrivileges(privileges, role, resourcePattern);
}
}
@@ -135,15 +135,15 @@ public class GrantStmt extends DdlStmt {
* 5.1 User should has GLOBAL level GRANT_PRIV
* 5.2 or user has DATABASE/TABLE level GRANT_PRIV if grant/revoke to/from
certain database or table.
* 5.3 or user should has 'resource' GRANT_PRIV if grant/revoke to/from
certain 'resource'
+ * 6. Can not grant USAGE_PRIV to database or table
*
- * @param analyzer
* @param privileges
* @param role
* @param tblPattern
* @throws AnalysisException
*/
-public static void checkPrivileges(Analyzer analyzer, List
privileges,
- String role, TablePattern tblPattern)
throws AnalysisException {
+public static void checkTablePrivileges(List privileges,
String role, TablePattern tblPattern)
+throws AnalysisException {
// Rule 1
if (tblPattern.getPrivLevel() != PrivLevel.GLOBAL &&
(privileges.contains(PaloPrivilege.ADMIN_PRIV)
|| privileges.contains(PaloPrivilege.NODE_PRIV))) {
@@ -153,7 +153,7 @@ public class GrantStmt extends DdlStmt {
// Rule 2
if (privileges.contains(PaloPrivilege.NODE_PRIV) &&
!Catalog.getCurrentCatalog().getAuth()
.checkGlobalPriv(ConnectContext.get(),
PrivPredicate.OPERATOR)) {
-throw new AnalysisException("Only the user with NODE_PRIV can
grant NODE_PRIV to other user");
+throw new AnalysisException("Only user with NODE_PRIV can
grant/revoke NODE_PRIV to other user");
}
if (role != null) {
@@ -178,18 +178,23 @@ public class GrantStmt extends DdlStmt {
}
}
}
+
+// Rule 6
+if (privileges.contains(PaloPrivilege.USAGE_PRIV)) {
+throw new AnalysisException("Can not grant/revoke USAGE_PRIV
to/from database or table");
+}
}
-public static void checkPrivileges(Analyzer analyzer, List
privileges,
- String role, ResourcePattern
resourcePattern) throws AnalysisException {
+public static void checkResourcePrivileges(List privileges,
String role,
+ResourcePattern resourcePattern) throws AnalysisException {
// Rule 1
if (privileges.contains(PaloPrivilege.NODE_PRIV)) {
-throw new AnalysisException("Can not grant NODE_PRIV to any other
users or roles");
+throw new AnalysisException("Can not grant/revoke NODE_PRIV
to/from any other users or roles");
}
// Rule 2
if (resourcePattern.getPrivLevel() != PrivLevel.GLOBAL &&
privileges.contains(PaloPrivilege.ADMIN_PRIV)) {
-throw new AnalysisException("ADMIN_PRIV privilege can only be
granted on resource *");
+throw new AnalysisException("ADMIN_PRIV privilege can only be
granted/revoked on/from resource *");
}
if (role != null) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/RevokeStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/RevokeStmt.java
index 9d2ce60c7c..837ff2001b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/RevokeStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/RevokeStmt.java
@@ -111,9 +111,9 @@ public class Re
[GitHub] [doris] github-actions[bot] commented on pull request #11254: [feature](Nereids): add MultiJoin.
github-actions[bot] commented on PR #11254: URL: https://github.com/apache/doris/pull/11254#issuecomment-1196500225 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau opened a new pull request, #11260: [feature-wip](multi-catalog)(fix) partition value error when a block contains multiple splits
AshinGau opened a new pull request, #11260: URL: https://github.com/apache/doris/pull/11260 # Proposed changes `FileArrowScanner::get_next` returns a block when full, so it maybe contains multiple splits in small files or crosses two splits in large files. However, a block can only fill the partition values from one file. Different splits may be from different files, causing the error of embed partition values. ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [x] Fix - [x] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [x] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau closed pull request #11237: [feature-wip](multi-catalog)(fix) partition value error when a block contains multiple splits.
AshinGau closed pull request #11237: [feature-wip](multi-catalog)(fix) partition value error when a block contains multiple splits. URL: https://github.com/apache/doris/pull/11237 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #11261: [Improvement][chore] replace from_decv2_to_packed128 to decv2.value
BiteThet opened a new pull request, #11261: URL: https://github.com/apache/doris/pull/11261 # Proposed changes part of work with https://github.com/apache/doris/issues/9312 1. use vdatatime.value() to replace binary_cast. 2. change some Irregular usage like `ColumnVector`,`ColumnVector`. ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 3. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 6. Does it need to update dependencies: - [ ] Yes - [ ] No 7. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #11257: [improvement]Use phmap::flat_hash_set in AggregateFunctionUniq
BiteThet commented on code in PR #11257:
URL: https://github.com/apache/doris/pull/11257#discussion_r930867953
##
be/src/vec/aggregate_functions/aggregate_function.h:
##
@@ -224,6 +228,9 @@ class IAggregateFunctionHelper : public IAggregateFunction {
arena);
}
}
+
+virtual void deserialize_and_merge(AggregateDataPtr __restrict place,
BufferReadable& buf,
+ Arena* arena) const override {}
Review Comment:
Does we need add some check at here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11187: [Vectorized] Support order by aggregate function
github-actions[bot] commented on PR #11187: URL: https://github.com/apache/doris/pull/11187#issuecomment-1196526315 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11187: [Vectorized] Support order by aggregate function
github-actions[bot] commented on PR #11187: URL: https://github.com/apache/doris/pull/11187#issuecomment-1196526360 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #11257: [improvement]Use phmap::flat_hash_set in AggregateFunctionUniq
BiteThet commented on code in PR #11257:
URL: https://github.com/apache/doris/pull/11257#discussion_r930881724
##
be/src/vec/aggregate_functions/aggregate_function_uniq.h:
##
@@ -111,16 +110,82 @@ class AggregateFunctionUniq final
void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
Arena*) const override {
-this->data(place).set.merge(this->data(rhs).set);
+auto& rhs_set = this->data(rhs).set;
+if (rhs_set.size() == 0) return;
+
+auto& set = this->data(place).set;
+set.rehash(set.size() + rhs_set.size());
+
+for (auto elem : rhs_set) {
+set.insert(elem);
+}
+}
+
+void add_batch_single_place(size_t batch_size, AggregateDataPtr place,
const IColumn** columns,
+Arena* arena) const override {
+auto& column = *columns[0];
+std::vector keys(batch_size);
+for (size_t i = 0; i != batch_size; ++i) {
+if constexpr (std::is_same_v) {
+StringRef value = column.get_data_at(i);
+
+UInt128 key;
+SipHash hash;
+hash.update(value.data, value.size);
+hash.get128(key.low, key.high);
+
+keys[i] = key;
+} else if constexpr (std::is_same_v) {
+keys[i] = assert_cast&>(column).get_data()[i];
+} else {
+keys[i] = assert_cast&>(column).get_data()[i];
+}
+}
+
+auto& set = this->data(place).set;
+for (size_t i = 0; i != batch_size; ++i) {
+if (i + 16 < batch_size) {
+set.prefetch(keys[i + 16]);
+}
+set.insert(keys[i]);
+}
}
void serialize(ConstAggregateDataPtr __restrict place, BufferWritable&
buf) const override {
-this->data(place).set.write(buf);
+auto& set = this->data(place).set;
+write_var_uint(set.size(), buf);
+for (const auto& elem : set) {
+write_pod_binary(elem, buf);
+}
+}
+
+void deserialize_and_merge(AggregateDataPtr __restrict place,
BufferReadable& buf,
+ Arena* arena) const override {
+auto& set = this->data(place).set;
+size_t size;
+read_var_uint(size, buf);
+
+set.rehash(size + set.size());
+
+for (size_t i = 0; i < size; ++i) {
+KeyType ref;
+read_pod_binary(ref, buf);
+set.insert(ref);
+}
}
void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
Review Comment:
Seems only diffrent with `deserialize_and_merge` is set.rehash, maybe we can
do some abstract here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11054: [feature-wip](array-type) add function array_slice
github-actions[bot] commented on PR #11054: URL: https://github.com/apache/doris/pull/11054#issuecomment-1196563413 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11054: [feature-wip](array-type) add function array_slice
github-actions[bot] commented on PR #11054: URL: https://github.com/apache/doris/pull/11054#issuecomment-1196563453 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 merged pull request #11054: [feature-wip](array-type) add function array_slice
xy720 merged PR #11054: URL: https://github.com/apache/doris/pull/11054 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](array-type) add function array_slice (#11054)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5913c7c52c [feature-wip](array-type) add function array_slice (#11054)
5913c7c52c is described below
commit 5913c7c52c7006e2e0075bd06dbde68fe46e7fd2
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Wed Jul 27 18:43:52 2022 +0800
[feature-wip](array-type) add function array_slice (#11054)
array_slice function returns a slice of the array.
---
be/src/vec/CMakeLists.txt | 1 +
.../functions/array/function_array_register.cpp| 2 +
.../vec/functions/array/function_array_slice.cpp | 29
be/src/vec/functions/array/function_array_slice.h | 134 ++
.../sql-functions/array-functions/array_slice.md | 157 +
.../sql-functions/array-functions/array_slice.md | 157 +
fe/fe-core/src/main/cup/sql_parser.cup | 19 +++
fe/fe-core/src/main/jflex/sql_scanner.flex | 2 +
gensrc/script/doris_builtins_functions.py | 28
.../array_functions/test_array_functions.out | 18 +++
.../array_functions/test_array_functions.groovy| 2 +
11 files changed, 549 insertions(+)
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index 01804896be..9ee5a5066f 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -143,6 +143,7 @@ set(VEC_FILES
functions/array/function_array_union.cpp
functions/array/function_array_except.cpp
functions/array/function_array_intersect.cpp
+ functions/array/function_array_slice.cpp
exprs/table_function/vexplode_json_array.cpp
functions/math.cpp
functions/function_bitmap.cpp
diff --git a/be/src/vec/functions/array/function_array_register.cpp
b/be/src/vec/functions/array/function_array_register.cpp
index f87f28c4c1..39d4902e83 100644
--- a/be/src/vec/functions/array/function_array_register.cpp
+++ b/be/src/vec/functions/array/function_array_register.cpp
@@ -33,6 +33,7 @@ void register_function_arrays_overlap(SimpleFunctionFactory&);
void register_function_array_union(SimpleFunctionFactory&);
void register_function_array_except(SimpleFunctionFactory&);
void register_function_array_intersect(SimpleFunctionFactory&);
+void register_function_array_slice(SimpleFunctionFactory&);
void register_function_array(SimpleFunctionFactory& factory) {
register_function_array_element(factory);
@@ -46,6 +47,7 @@ void register_function_array(SimpleFunctionFactory& factory) {
register_function_array_union(factory);
register_function_array_except(factory);
register_function_array_intersect(factory);
+register_function_array_slice(factory);
}
} // namespace doris::vectorized
diff --git a/be/src/vec/functions/array/function_array_slice.cpp
b/be/src/vec/functions/array/function_array_slice.cpp
new file mode 100644
index 00..5828351773
--- /dev/null
+++ b/be/src/vec/functions/array/function_array_slice.cpp
@@ -0,0 +1,29 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/functions/array/function_array_slice.h"
+
+#include "vec/functions/simple_function_factory.h"
+
+namespace doris::vectorized {
+
+void register_function_array_slice(SimpleFunctionFactory& factory) {
+factory.register_function();
+factory.register_alias(FunctionArraySlice::name, "%element_slice%");
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/functions/array/function_array_slice.h
b/be/src/vec/functions/array/function_array_slice.h
new file mode 100644
index 00..0a60135637
--- /dev/null
+++ b/be/src/vec/functions/array/function_array_slice.h
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may
[GitHub] [doris] github-actions[bot] commented on pull request #10565: [feature](multi-catalog) Support es datasource
github-actions[bot] commented on PR #10565: URL: https://github.com/apache/doris/pull/10565#issuecomment-1196570590 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10565: [feature](multi-catalog) Support es datasource
github-actions[bot] commented on PR #10565: URL: https://github.com/apache/doris/pull/10565#issuecomment-1196570551 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
