[GitHub] [doris] liaoxin01 opened a new pull request, #13410: [feature](alter) support rename column for table with unique column id
liaoxin01 opened a new pull request, #13410: URL: https://github.com/apache/doris/pull/13410 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13259: test benchmark pipeline
hello-stephen commented on PR #13259: URL: https://github.com/apache/doris/pull/13259#issuecomment-1280397741 TeamCity pipeline, clickbench performance test result: total run time:30.2s load time: 520s storage size: 17123069872 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/clickbench_pr_29174.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13406: Add support for parquet and orc file into spark load. issue(#13099)
github-actions[bot] commented on PR #13406: URL: https://github.com/apache/doris/pull/13406#issuecomment-1280399388 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13406: Add support for parquet and orc file into spark load. issue(#13099)
github-actions[bot] commented on PR #13406: URL: https://github.com/apache/doris/pull/13406#issuecomment-1280399347 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhanghengdashuaibi opened a new issue, #13411: [Feature] 运行一段时间的flink算子后，be就会宕机，谁能知道这个问题咋解决
zhanghengdashuaibi opened a new issue, #13411: URL: https://github.com/apache/doris/issues/13411 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description WARNING: Logging before InitGoogleLogging() is written to STDERR I1017 09:30:42.672709 8381 env.cpp:46] Env init successfully. *** Aborted at 1665971351 (unix time) try "date -d @1665971351" if you are using GNU date *** *** SIGSEGV unkown detail explain (@0x0) received by PID 8381 (TID 0x7facd1d47700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/apache-doris/be/src/common/signal_handler.h:420 1# 0x7FAD2F82D400 in /lib64/[libc.so](http://libc.so/).6 2# doris::BaseFieldtypeTraits<(doris::FieldType)9>::equal(void const*, void const*) at /mnt/disk2/apache-doris/be/src/olap/types.h:491 3# doris::TupleReader::_unique_key_next_row(doris::RowCursor*, doris::MemPool*, doris::ObjectPool*, bool*) at /mnt/disk2/apache-doris/be/src/olap/tuple_reader.cpp:197 4# doris::Merger::merge_rowsets(std::shared_ptr, doris::ReaderType, std::vector, std::allocator > > const&, doris::RowsetWriter*, doris::Merger::Statistics*) in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 5# doris::Compaction::do_compaction_impl(long) in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 6# doris::Compaction::do_compaction(long) at /mnt/disk2/apache-doris/be/src/olap/compaction.cpp:112 7# doris::CumulativeCompaction::execute_compact_impl() in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 8# doris::Compaction::execute_compact() at /mnt/disk2/apache-doris/be/src/olap/compaction.cpp:50 9# doris::Tablet::execute_compaction(doris::CompactionType) in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 10# std::_Function_handler, doris::CompactionType)::$_12>::_M_invoke(std::_Any_data const&) at /mnt/disk2/ygl/installs/ldbtools/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 11# doris::ThreadPool::dispatch_thread() at /mnt/disk2/apache-doris/be/src/util/threadpool.cpp:578 12# doris::Thread::supervise_thread(void*) at /mnt/disk2/apache-doris/be/src/util/thread.cpp:407 13# start_thread in /lib64/[libpthread.so](http://libpthread.so/).0 14# clone in /lib64/[libc.so](http://libc.so/).6 ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13259: test benchmark pipeline
hello-stephen commented on PR #13259: URL: https://github.com/apache/doris/pull/13259#issuecomment-1280411911 TeamCity -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #13067: [feature-wip](recover) new recover ddl and support show dropped
nextdreamblue commented on code in PR #13067: URL: https://github.com/apache/doris/pull/13067#discussion_r996698985 ## fe/fe-core/src/main/cup/sql_parser.cup: ## @@ -2743,17 +2745,17 @@ drop_stmt ::= // Recover statement recover_stmt ::= -KW_RECOVER KW_DATABASE ident:dbName +KW_RECOVER KW_DATABASE ident:dbName opt_id:dbId opt_table_alias:alias Review Comment: 这块我建议保留现在的设计，让用户自己去输入name和id。只是通过id的话，改动的代码有点多，而且很绕 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13259: test benchmark pipeline
hello-stephen commented on PR #13259: URL: https://github.com/apache/doris/pull/13259#issuecomment-1280414002 TeamCity -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13259: test benchmark pipeline
hello-stephen commented on PR #13259: URL: https://github.com/apache/doris/pull/13259#issuecomment-1280416096 TeamCity pipeline, clickbench performance test result: total run time: 30.2 s load time: 123 s storage size: 112211221122 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13259: test benchmark pipeline
hello-stephen commented on PR #13259: URL: https://github.com/apache/doris/pull/13259#issuecomment-1280417153 $comment_body -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate when push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r996606953
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EffectivePredicatesExtractor.java:
##
@@ -0,0 +1,157 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * extract effective predicates.
+ */
+public class EffectivePredicatesExtractor extends PlanVisitor,
Void> {
+
+PredicatePropagation propagation = new PredicatePropagation();
+
+@Override
+public Set visit(Plan plan, Void context) {
+return Sets.newHashSet();
+}
+
+@Override
+public Set visitLogicalFilter(LogicalFilter
filter, Void context) {
+List predicates =
ExpressionUtils.extractConjunction(filter.getPredicates()).stream()
+.filter(p -> {
+if (p instanceof SubqueryExpr) {
+SubqueryExpr subqueryExpr = (SubqueryExpr) p;
+return subqueryExpr.getCorrelateSlots().isEmpty();
+}
Review Comment:
We should keep the logical orthogonality(don't handle the logic that other
rules will handle
)?
SubqueryExpr will be replace to some join by UnnestSubquery rule.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EffectivePredicatesExtractor.java:
##
@@ -0,0 +1,157 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apac
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate when push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r996610488
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EffectivePredicatesExtractor.java:
##
@@ -0,0 +1,157 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * extract effective predicates.
+ */
+public class EffectivePredicatesExtractor extends PlanVisitor,
Void> {
+
+PredicatePropagation propagation = new PredicatePropagation();
+
+@Override
+public Set visit(Plan plan, Void context) {
+return Sets.newHashSet();
+}
+
+@Override
+public Set visitLogicalFilter(LogicalFilter
filter, Void context) {
+List predicates =
ExpressionUtils.extractConjunction(filter.getPredicates()).stream()
+.filter(p -> {
+if (p instanceof SubqueryExpr) {
+SubqueryExpr subqueryExpr = (SubqueryExpr) p;
+return subqueryExpr.getCorrelateSlots().isEmpty();
+}
+return true;
+}).collect(Collectors.toList());
+predicates.addAll(filter.child().accept(this, context));
+return getAvailableExpressions(Sets.newHashSet(predicates), filter);
+}
+
+@Override
+public Set visitLogicalJoin(LogicalJoin join, Void context) {
+Set predicates = Sets.newHashSet();
+Set leftPredicates = join.left().accept(this, context);
+Set rightPredicates = join.right().accept(this, context);
+switch (join.getJoinType()) {
+case INNER_JOIN:
+case CROSS_JOIN:
+predicates.addAll(leftPredicates);
+predicates.addAll(rightPredicates);
+join.getOnClauseCondition().map(on ->
predicates.addAll(ExpressionUtils.extractConjunction(on)));
+break;
+case LEFT_SEMI_JOIN:
+predicates.addAll(leftPredicates);
+join.getOnClauseCondition().map(on ->
predicates.addAll(ExpressionUtils.extractConjunction(on)));
+break;
+case RIGHT_SEMI_JOIN:
+predicates.addAll(rightPredicates);
+join.getOnClauseCondition().map(on ->
predicates.addAll(ExpressionUtils.extractConjunction(on)));
+break;
+case LEFT_OUTER_JOIN:
+case LEFT_ANTI_JOIN:
+predicates.addAll(leftPredicates);
+break;
+case RIGHT_OUTER_JOIN:
+case RIGHT_ANTI_JOIN:
+predicates.addAll(rightPredicates);
+break;
+default:
+}
+return getAvailableExpressions(predicates, join);
+}
+
+@Override
+public Set visitLogicalProject(LogicalProject
project, Void context) {
+Set childPredicates = project.child().accept(this,
context);
+Map expressionSlotMap = project.getAliasToProducer()
+.entrySet()
+.stream()
+.collect(Collectors.toMap(Entry::getValue,
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate when push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r996729772
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicates.java:
##
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.PlanRuleFactory;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RulePromise;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * infer additional predicates for filter and join.
+ */
+public class InferPredicates implements PlanRuleFactory {
+PredicatePropagation propagation = new PredicatePropagation();
+EffectivePredicatesExtractor predicatesExtractor = new
EffectivePredicatesExtractor();
+
+@Override
+public List buildRules() {
+return ImmutableList.of(
+inferWhere(),
+inferOn()
+);
+}
+
+private Rule inferWhere() {
+return logicalFilter(any()).thenApply(ctx -> {
+LogicalFilter root = ctx.root;
+Plan filter =
ctx.cascadesContext.getMemo().copyOut(root.getGroupExpression().get(), false);
+Set filterPredicates =
filter.accept(predicatesExtractor, null);
+Set filterChildPredicates =
filter.child(0).accept(predicatesExtractor, null);
+filterPredicates.removeAll(filterChildPredicates);
+
ExpressionUtils.extractConjunction(root.getPredicates()).forEach(filterPredicates::remove);
Review Comment:
I think this rule should process the current predicate, and not need to
remove the predicate which exist in the child, this is the responsibility of
another rule.
keep the logical orthogonality too, and this rule will become more clarity,
so you not need to copy out from the memo in the EffectivePredicateExtractor.
and then, EffectivePredicateExtractor can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13393: [feature-wip](unique-key-merge-on-write) fix that delete the bitmap of stale rowset
github-actions[bot] commented on PR #13393: URL: https://github.com/apache/doris/pull/13393#issuecomment-1280464173 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hf200012 commented on pull request #13364: fix(sec): upgrade org.apache.hive:hive-exec to 3.1.3
hf200012 commented on PR #13364: URL: https://github.com/apache/doris/pull/13364#issuecomment-1280483906 Incompatible version, compile error @zhoumengyks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #12996: [Nereids][Improve] infer predicate when push down predicate
hello-stephen commented on PR #12996: URL: https://github.com/apache/doris/pull/12996#issuecomment-1280487489 https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221017163314_clickbench_pr_30045.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #13414: [fix](mem) failure of allocating memory(ASAN)
mrhhsg opened a new pull request, #13414:
URL: https://github.com/apache/doris/pull/13414
# Proposed changes
When the target size to allocate is 8164, MemPool will return nullptr.
```bash
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_master/be/src/common/signal_handler.h:420
1# 0x7F44CF126090 in /lib/x86_64-linux-gnu/libc.so.6
2# doris::vectorized::ColumnNullable::serialize_vec(std::vector >&, unsigned long, unsigned long) const at
/home/zcp/repo_center/doris_maste
r/be/src/vec/columns/column_nullable.cpp:216
3# doris::vectorized::AggregationMethodSerialized, false>
>::serialize_keys(std::vector > const&, unsigned
long) at
/home/zcp/repo_center/doris_master/be/src/vec/exec/vaggregation_node.h:86
4# void
doris::vectorized::AggregationNode::_pre_serialize_key_if_need, char*, true>, d
oris::vectorized::AggregationMethodSerialized, false> >
>(doris::vectorized::ColumnsHashing::HashMethodSerialized, char*, true>&,
doris::vectorized::AggregationMethodSerialized, false> >&, std::vector > const&, unsigned long) at
/home/zcp/repo_center/doris_master/be/src/vec/exec/vaggregation_node.h:8
51
5# void
doris::vectorized::AggregationNode::_emplace_into_hash_table(char**,
std::vector >&,
unsigned
long)::{lambda(auto:1&&)#1}::operator(), false> >&>(doris::vecto
rized::AggregationMethodSerialized, false> >&) const at
/home/zcp/repo_center/doris_master/be/src/vec/exec/vaggregation_n
ode.cpp:792
6# void std::__invoke_impl >&, unsigned long)::{lambda(auto:1&&)#1},
doris::vectorized::AggregationMethodSerialized, false> >&
>(std::__invoke_other,
doris::vectorized::AggregationNode::_emplace_into_hash_table(char**,
std::vector >&, unsigned long)::{lambda(auto:1&&)#1}&&,
doris::vectorized::AggregationMethodSerialized, false> >&) at /var
/local/ldb_toolchain/include/c++/11/bits/invoke.h:61
7#
std::__invoke_result >&, unsigned long)::{lambda(auto:1&&)#1},
doris::vectorized::AggregationMethodSerialized, false> >&>::type s
td::__invoke
>&, unsigned long)::{lambda(auto:1&&)#1},
doris::vectorized::AggregationMethodSerialized, false> >&>(doris::vectorized::
AggregationNode::_emplace_into_hash_table(char**,
std::vector >&, unsigned
long)::{lambda(aut
o:1&&)#1}&&,
doris::vectorized::AggregationMethodSerialized, false> >&) at
/var/local/ldb_toolchain/include/c++/11/bits/i
nvoke.h:97
8#
_ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFNS0_21__deduce_visit_resultIvEEOZN5doris10vectorized15AggregationNode24_emplace_into_hash_tableEPPcRSt6vector
IPKNS6_7IColumnESaISD_EEmEUlOT_E_RSt7variantIJNS6_27AggregationMethodSerializedI9PHHashMapI9StringRefS8_11DefaultHashISO_vELb0NS6_26AggregationMethodOneNumberIh12FixedHashMap
IhS8_28FixedHashMapImplicitZeroCellIhS8_16HashTableNoStateE28FixedHashTableCalculatedSizeISX_E9AllocatorILb1ELb1EEELb0EEENST_ItSU_ItS8_SV_ItS8_SW_E24FixedHashTableStoredSizeIS14_
ES11_ELb0EEENST_IjSN_IjS8_9HashCRC32IjELb0EELb0EEENST_ImSN_ImS8_S19_ImELb0EELb0EEENS6_30AggregationMethodStringNoCacheI13StringHashMapIS8_S11_EEENST_INS6_7UInt128ESN_IS1K_S8_S19_
IS1K_ELb0EELb0EEENST_IjSN_IjS8_14HashMixWrapperIjS1A_ELb0EELb0EEENST_ImSN_ImS8_S1O_ImS1D_ELb0EELb0EEENST_IS1K_SN_IS1K_S8_S1O_IS1K_S1L_ELb0EELb0EEENS6_37AggregationMethodSingleNul
lableColumnINST_IhNS6_26AggregationDataWithNullKeyIS12_EELb0ENS1Y_INST_ItNS1Z_IS17_EELb0ENS1Y_INST_IjNS1Z_IS1B_EELb0ENS1Y_INST_ImNS1Z_IS1E_EELb0ENS1Y_INST_IjNS1Z_
IS1Q_EELb0ENS1Y_INST_ImNS1Z_IS1T_EELb0ENS1Y_INST_IS1K_NS1Z_IS1M_EELb0ENS1Y_INST_IS1K_NS1Z_IS1W_EELb0ENS1Y_INS1G_INS1Z_IS1I_EENS6_26AggregationMethodKeysFixedI
S1E_Lb0EEENS2R_IS1E_Lb1EEENS2R_IS1M_Lb0EEENS2R_IS1M_Lb1EEENS2R_ISN_INS6_7UInt256ES8_S19_IS2W_ELb0EELb0EEENS2R_IS2Y_Lb1EEENS2R_IS1T_Lb0EEENS2R_IS1T_Lb1EEENS2R_IS1W_Lb0EEENS2R_IS1W
_Lb1EEENS2R_ISN_IS2W_S8_S1O_IS2W_S2X_ELb0EELb0EEENS2R_IS36_Lb1EEJEEESt16integer_sequenceImJLm014__visit_invokeESK_S3A_
at /var/local/ldb_toolchain/include/c++/11/variant:
1015
9#
_ZSt10__do_visitINSt8__detail9__variant21__deduce_visit_resultIvEEZN5doris10vectorized15AggregationNode24_emplace_into_hash_tableEPPcRSt6vectorIPKNS5_7IColumnESaISC_EEmEUlOT_
E_JRSt7variantIJNS5_27AggregationMethodSerializedI9PHHashMapI9StringRefS7_11DefaultHashISM_vELb0NS5_26AggregationMethodOneNumberIh12FixedHashMapIhS7_28FixedHashMapImplicitZer
oCellIhS7_16HashTableNoStateE28FixedHashTableCalculatedSizeISV_E9AllocatorILb1ELb1EEELb0EEENSR_ItSS_ItS7_ST_ItS7_SU_E24FixedHashTableStoredSizeIS12_ESZ_ELb0EEENSR_IjSL_IjS7_9Hash
CRC32IjELb0EELb0EEENSR_ImSL_ImS7_S17_ImELb0EELb0EEENS5_30AggregationMethodStringNoCacheI13StringHashMapIS7_SZ_EEENSR_INS5_7UInt128ESL_IS1I_S7_S17_IS1I_ELb0EELb0EEENSR_IjSL_IjS7_1
4HashMixWrapperIjS18_ELb0EELb0EEENSR_ImSL_ImS7_S1M_ImS1B_ELb0EELb0EEENSR_IS1I_SL_IS1I_S7_S1M_IS1I_S1J_ELb0EELb0EEENS5_37Aggreg
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate when push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r996698062
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PredicatePropagation.java:
##
@@ -0,0 +1,100 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * derive additional predicates.
+ * for example:
+ * a = b and a = 1 => b = 1
+ */
+public class PredicatePropagation {
+
+/**
+ * infer additional predicates.
+ */
+public Set infer(List predicates) {
+Set inferred = Sets.newHashSet();
+for (Expression predicate : predicates) {
+if (canEquivalentDeduce(predicate)) {
+List candidates = subtract(predicates, predicate);
+candidates.forEach(candidate -> {
+inferred.add(transform(candidate, predicate.child(0),
predicate.child(1)));
+});
+}
+}
+predicates.forEach(inferred::remove);
+return inferred;
+}
+
+private Expression transform(Expression expression, Expression source,
Expression target) {
+Expression rewritten = replace(expression, source, target);
+if (expression.equals(rewritten)) {
+rewritten = mapChildren(expression, source, target);
+if (expression.equals(rewritten)) {
+return expression;
+} else {
+return rewritten;
+}
+} else {
+return mapChildren(rewritten, source, target);
+}
+}
+
+private Expression mapChildren(Expression expression, Expression source,
Expression target) {
+if (!expression.children().isEmpty()) {
+List children = Lists.newArrayList();
+for (Expression child : expression.children()) {
+children.add(transform(child, source, target));
+}
+return expression.withChildren(children);
+} else {
+return expression;
+}
+}
+
+private Expression replace(Expression expression, Expression source,
Expression target) {
+if (expression.equals(source)) {
+return target;
+}
+if (expression.equals(target)) {
+return source;
+}
+return expression;
+}
+
+private boolean canEquivalentDeduce(Expression predicate) {
+return predicate instanceof EqualTo &&
predicate.children().stream().allMatch(e -> e instanceof SlotReference);
+}
+
+private List subtract(List expressions, Expression
target) {
+ArrayList cloneList = Lists.newArrayList(expressions);
+cloneList.remove(target);
+return cloneList;
+}
+}
Review Comment:
Do not recursively traverse/rewrite manually, too.
Use DefaultExpressionRewriter:
```java
class PredicatePropagation {
public Set infer(List predicates) {
Set inferred = Sets.newHashSet();
for (Expression predicate : predicates) {
if (canEquivalentDeduce(predicate)) {
Expression leftSlotEqualToRightSlot = predicate;
List newInferred = predicates.stream()
.filter(p -> !p.equals(leftSlotEqualToRightSlot))
.map(p -> doInfer(leftSlotEqualToRightSlot, p))
.collect(Collectors.toList());
inferred.addAll(newInferred);
}
}
predicates.forEach(inferred::remove);
return inferred;
}
private Expression doInfer(Expression leftSlotEqualToRightSlot,
Expression expression) {
return expression.accept(new DefaultExpressionRewriter() {
@Override
public Expression visit(Expression expr, V
[GitHub] [doris] github-actions[bot] commented on pull request #13364: fix(sec): upgrade org.apache.hive:hive-exec to 3.1.3
github-actions[bot] commented on PR #13364: URL: https://github.com/apache/doris/pull/13364#issuecomment-1280518197 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13364: fix(sec): upgrade org.apache.hive:hive-exec to 3.1.3
github-actions[bot] commented on PR #13364: URL: https://github.com/apache/doris/pull/13364#issuecomment-1280518243 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #13415: [fix](memory) Fix USE_JEMALLOC=true compilation error
xinyiZzz opened a new pull request, #13415: URL: https://github.com/apache/doris/pull/13415 # Proposed changes Issue Number: close #xxx ## Problem summary fix #13398 ``` *** Query id: 0-0 *** *** Aborted at 1665994546 (unix time) try "date -d @1665994546" if you are using GNU date *** *** Current BE git commitID: e84d9a6c8 *** *** SIGSEGV address not mapped to object (@0x60) received by PID 3907460 (TID 0x7f8f9487e700) from PID 96; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/liyifan/doris/core/be/src/common/signal_handler.h:420 1# 0x7F8F97CAEB20 in /lib64/libc.so.6 2# void doris::destroy(void*) at /mnt/disk1/liyifan/doris/core/be/src/runtime/threadlocal.h:122 3# doris::invoke_destructors(void*) at /mnt/disk1/liyifan/doris/core/be/src/runtime/threadlocal.cc:53 4# __nptl_deallocate_tsd.part.8 in /lib64/libpthread.so.0 5# start_thread in /lib64/libpthread.so.0 6# clone in /lib64/libc.so.6 ``` ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate when push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r996698062
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PredicatePropagation.java:
##
@@ -0,0 +1,100 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * derive additional predicates.
+ * for example:
+ * a = b and a = 1 => b = 1
+ */
+public class PredicatePropagation {
+
+/**
+ * infer additional predicates.
+ */
+public Set infer(List predicates) {
+Set inferred = Sets.newHashSet();
+for (Expression predicate : predicates) {
+if (canEquivalentDeduce(predicate)) {
+List candidates = subtract(predicates, predicate);
+candidates.forEach(candidate -> {
+inferred.add(transform(candidate, predicate.child(0),
predicate.child(1)));
+});
+}
+}
+predicates.forEach(inferred::remove);
+return inferred;
+}
+
+private Expression transform(Expression expression, Expression source,
Expression target) {
+Expression rewritten = replace(expression, source, target);
+if (expression.equals(rewritten)) {
+rewritten = mapChildren(expression, source, target);
+if (expression.equals(rewritten)) {
+return expression;
+} else {
+return rewritten;
+}
+} else {
+return mapChildren(rewritten, source, target);
+}
+}
+
+private Expression mapChildren(Expression expression, Expression source,
Expression target) {
+if (!expression.children().isEmpty()) {
+List children = Lists.newArrayList();
+for (Expression child : expression.children()) {
+children.add(transform(child, source, target));
+}
+return expression.withChildren(children);
+} else {
+return expression;
+}
+}
+
+private Expression replace(Expression expression, Expression source,
Expression target) {
+if (expression.equals(source)) {
+return target;
+}
+if (expression.equals(target)) {
+return source;
+}
+return expression;
+}
+
+private boolean canEquivalentDeduce(Expression predicate) {
+return predicate instanceof EqualTo &&
predicate.children().stream().allMatch(e -> e instanceof SlotReference);
+}
+
+private List subtract(List expressions, Expression
target) {
+ArrayList cloneList = Lists.newArrayList(expressions);
+cloneList.remove(target);
+return cloneList;
+}
+}
Review Comment:
Do not recursively traverse/rewrite manually, too.
Use DefaultExpressionRewriter:
```java
class PredicatePropagation {
public Set infer(List predicates) {
Set inferred = Sets.newHashSet();
for (Expression predicate : predicates) {
if (canEquivalentDeduce(predicate)) {
Expression leftSlotEqualToRightSlot = predicate;
List newInferred = predicates.stream()
.filter(p -> !p.equals(leftSlotEqualToRightSlot))
.map(p -> doInfer(leftSlotEqualToRightSlot, p))
.collect(Collectors.toList());
inferred.addAll(newInferred);
}
}
predicates.forEach(inferred::remove);
return inferred;
}
private Expression doInfer(Expression leftSlotEqualToRightSlot,
Expression expression) {
return expression.accept(new DefaultExpressionRewriter() {
@Override
public Expression visit(Expression expr, V
[GitHub] [doris] wangshuo128 opened a new pull request, #13416: [Feature](Nereids) Support materialized index selection.
wangshuo128 opened a new pull request, #13416: URL: https://github.com/apache/doris/pull/13416 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] bleachkinsin opened a new issue, #74: [Bug] StreamLoadException: stream load error: failed to stream load data with label: flink_connector_20221017_170854_ef4f558e2bf4
bleachkinsin opened a new issue, #74: URL: https://github.com/apache/doris-flink-connector/issues/74 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.13.6 ### What's Wrong? 2022-10-17 17:09:08.886 WARN 19180 --- [Unnamed (1/1)#0] o.a.doris.flink.table.DorisStreamLoad: failed to stream load data with label: flink_connector_20221017_170905_802f8e174c884bd9805c8bc60c72ac50 org.apache.http.conn.HttpHostConnectException: Connect to 172.20.0.3:8040 [/172.20.0.3] failed: Connection refused: connect at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.apache.doris.flink.table.DorisStreamLoad.loadBatch(DorisStreamLoad.java:136) at org.apache.doris.flink.table.DorisStreamLoad.load(DorisStreamLoad.java:99) at org.apache.doris.flink.table.DorisDynamicOutputFormat.flush(DorisDynamicOutputFormat.java:309) at org.apache.doris.flink.table.DorisDynamicOutputFormat.writeRecord(DorisDynamicOutputFormat.java:223) at org.apache.doris.flink.cfg.GenericDorisSinkFunction.invoke(GenericDorisSinkFunction.java:51) at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:188) at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110) at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:101) at com.ververica.cdc.connectors.mysql.source.reader.MySqlRecordEmitter$OutputCollector.collect(MySqlRecordEmitter.java:143) at com.ververica.cdc.debezium.JsonDebeziumDeserializationSchema.deserialize(JsonDebeziumDeserializationSchema.java:74) at com.ververica.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitElement(MySqlRecordEmitter.java:118) at com.ververica.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:100) at com.ververica.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:54) at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:128) at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:305) at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:69) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) at org.apache.flink.stre
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #13417: [Bug](predicate) fix core dump on bool type runtime filter
BiteThet opened a new pull request, #13417: URL: https://github.com/apache/doris/pull/13417 # Proposed changes ```cpp 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/pxl/doris/be/src/common/signal_handler.h:420 1# 0x7F848CB99400 in /lib64/libc.so.6 2# gsignal in /lib64/libc.so.6 3# abort in /lib64/libc.so.6 4# 0x5602DF33A909 in /mnt/disk1/pxl/doris/output/be/lib/doris_be 5# 0x5602DF32FF1D in /mnt/disk1/pxl/doris/output/be/lib/doris_be 6# google::LogMessage::SendToLog() in /mnt/disk1/pxl/doris/output/be/lib/doris_be 7# google::LogMessage::Flush() in /mnt/disk1/pxl/doris/output/be/lib/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /mnt/disk1/pxl/doris/output/be/lib/doris_be 9# doris::ColumnPredicate* doris::create_list_predicate<(doris::PredicateType)8>(doris::TabletColumn const&, int, std::vector, std::allocator >, std::allocator, std::allocator > > > const&, bool, doris::MemPool*) in /mnt/disk1/pxl/doris/output/be/lib/doris_be 10# doris::parse_to_predicate(std::shared_ptr, doris::TCondition const&, doris::MemPool*, bool) at /mnt/disk1/pxl/doris/be/src/olap/predicate_creator.h:273 11# doris::TabletReader::_init_conditions_param(doris::TabletReader::ReaderParams const&) at /mnt/disk1/pxl/doris/be/src/olap/reader.cpp:444 12# doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) at /mnt/disk1/pxl/doris/be/src/olap/reader.cpp:240 13# doris::TabletReader::init(doris::TabletReader::ReaderParams const&) at /mnt/disk1/pxl/doris/be/src/olap/reader.cpp:98 14# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) at /mnt/disk1/pxl/doris/be/src/vec/olap/block_reader.cpp:109 15# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at /mnt/disk1/pxl/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:116 16# doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, doris::vectorized::VScanner*) at /mnt/disk1/pxl/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:203 17# doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const at /mnt/disk1/pxl/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:146 18# void std::__invoke_impl(std::__invoke_other, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1&) at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 19# std::enable_if, void>::type std::__invoke_r(doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1&) at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:117 20# std::_Function_handler::_M_invoke(std::_Any_data const&) at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 21# std::function::operator()() const at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 22# doris::FunctionRunnable::run() at /mnt/disk1/pxl/doris/be/src/util/threadpool.cpp:45 23# doris::ThreadPool::dispatch_thread() at /mnt/disk1/pxl/doris/be/src/util/threadpool.cpp:540 24# void std::__invoke_impl(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74 25# std::__invoke_result::type std::__invoke(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96 26# void std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420 27# void std::_Bind::operator()<, void>() at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503 28# void std::__invoke_impl&>(std::__invoke_other, std::_Bind&) at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 29# std::enable_if&>, void>::type std::__invoke_r&>(std::_Bind&) at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:117 30# std::_Function_handler >::_M_invoke(std::_Any_data const&) at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 31# std::function::operator()() const at /mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 32# doris::Thread::supervi
[GitHub] [doris] xinyiZzz closed pull request #13415: [fix](memory) Fix USE_JEMALLOC=true compilation error
xinyiZzz closed pull request #13415: [fix](memory) Fix USE_JEMALLOC=true compilation error URL: https://github.com/apache/doris/pull/13415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #13415: [fix](memory) Fix USE_JEMALLOC=true compilation error
xinyiZzz commented on PR #13415: URL: https://github.com/apache/doris/pull/13415#issuecomment-1280594725 abandoned -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate when push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r996606953
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EffectivePredicatesExtractor.java:
##
@@ -0,0 +1,157 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * extract effective predicates.
+ */
+public class EffectivePredicatesExtractor extends PlanVisitor,
Void> {
+
+PredicatePropagation propagation = new PredicatePropagation();
+
+@Override
+public Set visit(Plan plan, Void context) {
+return Sets.newHashSet();
+}
+
+@Override
+public Set visitLogicalFilter(LogicalFilter
filter, Void context) {
+List predicates =
ExpressionUtils.extractConjunction(filter.getPredicates()).stream()
+.filter(p -> {
+if (p instanceof SubqueryExpr) {
+SubqueryExpr subqueryExpr = (SubqueryExpr) p;
+return subqueryExpr.getCorrelateSlots().isEmpty();
+}
Review Comment:
We should keep the logical orthogonality(don't handle the logic that other
rules will handle
)?
SubqueryExpr will be replace to some join by UnnestSubquery rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate when push down predicate
924060929 commented on code in PR #12996: URL: https://github.com/apache/doris/pull/12996#discussion_r996612117 ## fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EffectivePredicatesExtractor.java: ## @@ -0,0 +1,157 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.rules.rewrite.logical; + +import org.apache.doris.nereids.trees.expressions.Alias; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.NamedExpression; +import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.SubqueryExpr; +import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction; +import org.apache.doris.nereids.trees.plans.Plan; +import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate; +import org.apache.doris.nereids.trees.plans.logical.LogicalFilter; +import org.apache.doris.nereids.trees.plans.logical.LogicalJoin; +import org.apache.doris.nereids.trees.plans.logical.LogicalProject; +import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor; +import org.apache.doris.nereids.util.ExpressionUtils; + +import com.google.common.collect.Lists; +import com.google.common.collect.Sets; + +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; +import java.util.stream.Collectors; + +/** + * extract effective predicates. Review Comment: What does effective mean? Please add comment to sum up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zy-kkk opened a new pull request, #13418: [function](string_function) add new string function 'not_null_or_empty'
zy-kkk opened a new pull request, #13418: URL: https://github.com/apache/doris/pull/13418 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #13355: [improvement](olap) try to read enough filtered rows in segment_iterator
HappenLee commented on code in PR #13355:
URL: https://github.com/apache/doris/pull/13355#discussion_r996702548
##
be/src/vec/columns/column_string.cpp:
##
@@ -444,6 +445,7 @@ void ColumnString::resize(size_t n) {
auto origin_size = size();
if (origin_size > n) {
offsets.resize(n);
+chars.resize(offsets.back());
Review Comment:
offsets.back() may contain value do not init? so maybe here may cause mem
problem here
##
be/src/olap/rowset/segment_v2/segment_iterator.cpp:
##
@@ -907,6 +907,10 @@ void SegmentIterator::_init_current_block(
if (_is_pred_column[cid] ||
i >= block->columns()) { //todo(wb) maybe we can release it after
output block
current_columns[cid]->clear();
+if (i < block->columns()) {
+block->get_by_position(i).column->assume_mutable()->reserve(
Review Comment:
this code seems should out of the `if else` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg commented on a diff in pull request #13355: [improvement](olap) try to read enough filtered rows in segment_iterator
mrhhsg commented on code in PR #13355:
URL: https://github.com/apache/doris/pull/13355#discussion_r996881116
##
be/src/vec/columns/column_string.cpp:
##
@@ -444,6 +445,7 @@ void ColumnString::resize(size_t n) {
auto origin_size = size();
if (origin_size > n) {
offsets.resize(n);
+chars.resize(offsets.back());
Review Comment:
It should not be since n is smaller than origin_size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13414: [fix](mem) failure of allocating memory(ASAN)
github-actions[bot] commented on PR #13414: URL: https://github.com/apache/doris/pull/13414#issuecomment-1280640830 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13414: [fix](mem) failure of allocating memory(ASAN)
github-actions[bot] commented on PR #13414: URL: https://github.com/apache/doris/pull/13414#issuecomment-1280640907 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13368: [enhancement](memtracker) Fix bthread local consume mem tracker
yiguolei merged PR #13368: URL: https://github.com/apache/doris/pull/13368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (3b5b7ae12b -> 87a6b1a13b)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3b5b7ae12b [improvement](config) let default value of alter and load timeout suitable for most cases (#13370) add 87a6b1a13b [enhancement](memtracker) Fix bthread local consume mem tracker (#13368) No new revisions were added by this update. Summary of changes: be/src/runtime/exec_env.h | 7 +- be/src/runtime/exec_env_init.cpp | 3 +- be/src/runtime/memory/mem_tracker_limiter.cpp | 7 +- be/src/runtime/memory/mem_tracker_limiter.h| 6 ++ be/src/runtime/memory/thread_mem_tracker_mgr.h | 46 +++ be/src/runtime/thread_context.cpp | 32 +--- be/src/runtime/thread_context.h| 105 + be/src/service/internal_service.cpp| 29 --- be/test/testutil/run_all_tests.cpp | 7 +- 9 files changed, 107 insertions(+), 135 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt merged pull request #13287: [feature](agg) support `group_bitmap_xor` agg function.
BiteThet merged PR #13287: URL: https://github.com/apache/doris/pull/13287 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (87a6b1a13b -> 207f4e559e)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 87a6b1a13b [enhancement](memtracker) Fix bthread local consume mem
tracker (#13368)
add 207f4e559e [feature](agg) support `group_bitmap_xor` agg function.
(#13287)
No new revisions were added by this update.
Summary of changes:
be/src/exprs/bitmap_function.cpp | 13 +++
be/src/exprs/bitmap_function.h | 1 +
.../aggregate_function_bitmap.cpp | 9 +
.../aggregate_function_bitmap.h| 22
.../aggregate-functions/group_bitmap_xor.md} | 41 +++---
docs/sidebars.json | 1 +
.../group_bitmap_xor.md} | 41 +++---
.../java/org/apache/doris/catalog/FunctionSet.java | 19 ++
.../test_aggregate_all_functions.out | 5 +++
.../test_aggregate_all_functions.groovy| 4 ++-
10 files changed, 115 insertions(+), 41 deletions(-)
copy
docs/{zh-CN/docs/sql-manual/sql-functions/bitmap-functions/intersect_count.md
=> en/docs/sql-manual/sql-functions/aggregate-functions/group_bitmap_xor.md}
(50%)
copy
docs/zh-CN/docs/sql-manual/sql-functions/{bitmap-functions/intersect_count.md
=> aggregate-functions/group_bitmap_xor.md} (50%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #13418: [function](string_function) add new string function 'not_null_or_empty'
BiteThet commented on code in PR #13418:
URL: https://github.com/apache/doris/pull/13418#discussion_r996905606
##
be/src/vec/functions/function_string.h:
##
@@ -420,6 +420,43 @@ class FunctionNullOrEmpty : public IFunction {
}
};
+class FunctionNotNullOrEmpty : public IFunction {
+public:
+static constexpr auto name = "not_null_or_empty";
+static FunctionPtr create() { return
std::make_shared(); }
+String get_name() const override { return name; }
+size_t get_number_of_arguments() const override { return 1; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+return std::make_shared();
+}
+
+bool use_default_implementation_for_nulls() const override { return false;
}
+bool use_default_implementation_for_constants() const override { return
true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+auto res_map = ColumnUInt8::create(input_rows_count, 0);
+
+auto column = block.get_by_position(arguments[0]).column;
+if (auto* nullable = check_and_get_column(*column)) {
+column = nullable->get_nested_column_ptr();
+res_map->get_data();
Review Comment:
We better check nullmap when input column is nullable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #13419: [fix](test) let each case uses its own table name
dataroaring opened a new pull request, #13419: URL: https://github.com/apache/doris/pull/13419 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #13418: [function](string_function) add new string function 'not_null_or_empty'
BiteThet commented on code in PR #13418:
URL: https://github.com/apache/doris/pull/13418#discussion_r996906315
##
be/src/vec/functions/function_string.h:
##
@@ -420,6 +420,43 @@ class FunctionNullOrEmpty : public IFunction {
}
};
+class FunctionNotNullOrEmpty : public IFunction {
+public:
+static constexpr auto name = "not_null_or_empty";
+static FunctionPtr create() { return
std::make_shared(); }
+String get_name() const override { return name; }
+size_t get_number_of_arguments() const override { return 1; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+return std::make_shared();
+}
+
+bool use_default_implementation_for_nulls() const override { return false;
}
+bool use_default_implementation_for_constants() const override { return
true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+auto res_map = ColumnUInt8::create(input_rows_count, 0);
+
+auto column = block.get_by_position(arguments[0]).column;
+if (auto* nullable = check_and_get_column(*column)) {
+column = nullable->get_nested_column_ptr();
+res_map->get_data();
Review Comment:
line 444 seems unused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #13418: [function](string_function) add new string function 'not_null_or_empty'
BiteThet commented on code in PR #13418:
URL: https://github.com/apache/doris/pull/13418#discussion_r996907313
##
docs/zh-CN/docs/sql-manual/sql-functions/string-functions/not_null_or_empty.md:
##
@@ -0,0 +1,60 @@
+---
+{
+"title": "not_null_or_empty",
+"language": "zh-CN"
+}
+---
+
+
+
+## not_null_or_empty
+### description
+ Syntax
+
+`BOOLEAN NOT_NULL_OR_EMPTY (VARCHAR str)`
+
+如果字符串为空字符串或者NULL，返回false。否则，返回true。
+
+### example
+
+```
+MySQL [(none)]> select not_null_or_empty(null);
++-+
+| null_or_empty(NULL) |
Review Comment:
```suggestion
| not_null_or_empty(NULL) |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (207f4e559e -> c114d87d13)
This is an automated email from the ASF dual-hosted git repository. xuyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 207f4e559e [feature](agg) support `group_bitmap_xor` agg function. (#13287) add c114d87d13 [Enhancement](array-type) Tuple is null predicate support array type (#13307) No new revisions were added by this update. Summary of changes: be/src/vec/exec/join/vhash_join_node.cpp | 2 +- gensrc/script/doris_builtins_functions.py | 15 +++ .../data/query_p0/join/sql/issue_12689.out| 3 +++ regression-test/suites/query_p0/join/ddl/table_3.sql | 19 +++ regression-test/suites/query_p0/join/load.groovy | 4 +++- .../suites/query_p0/join/sql/issue_12689.sql | 1 + 6 files changed, 42 insertions(+), 2 deletions(-) create mode 100644 regression-test/data/query_p0/join/sql/issue_12689.out create mode 100644 regression-test/suites/query_p0/join/ddl/table_3.sql create mode 100644 regression-test/suites/query_p0/join/sql/issue_12689.sql - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 merged pull request #13307: [Enhancement](array-type) Tuple is null predicate support array type
xy720 merged PR #13307: URL: https://github.com/apache/doris/pull/13307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #13400: [function](date function)add new date function 'last_day'
Gabriel39 commented on code in PR #13400:
URL: https://github.com/apache/doris/pull/13400#discussion_r996914151
##
be/src/vec/functions/function_timestamp.cpp:
##
@@ -569,6 +569,164 @@ class FunctionUnixTimestamp : public IFunction {
}
};
+template
+class FunctionDateOrDateTimeToDate : public IFunction {
+public:
+static constexpr auto name = Impl::name;
+static FunctionPtr create() {
+return std::make_shared>();
+}
+
+String get_name() const override { return name; }
+
+bool use_default_implementation_for_nulls() const override { return true; }
Review Comment:
`use_default_implementation_for_constants` also need to override and set to
true
##
be/src/vec/functions/function_timestamp.cpp:
##
@@ -569,6 +569,164 @@ class FunctionUnixTimestamp : public IFunction {
}
};
+template
+class FunctionDateOrDateTimeToDate : public IFunction {
+public:
+static constexpr auto name = Impl::name;
+static FunctionPtr create() {
+return std::make_shared>();
+}
+
+String get_name() const override { return name; }
+
+bool use_default_implementation_for_nulls() const override { return true; }
+
+size_t get_number_of_arguments() const override { return 1; }
+
+bool is_variadic() const override { return true; }
+
+// input DateTime and Date, return Date
+// input DateTimeV2 and DateV2, return DateV2
+DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName& arguments)
const override {
+if constexpr (std::is_same_v) {
Review Comment:
Use `or` to collapse these two conditions
##
regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy:
##
@@ -424,4 +424,30 @@ suite("test_date_function") {
qt_sql """ select minutes_sub(test_time2,1) result from ${tableName}; """
//seconds_sub
qt_sql """ select seconds_sub(test_time2,1) result from ${tableName}; """
+
+// test last_day
+sql """ SET enable_vectorized_engine = TRUE; """
Review Comment:
also add a case for row-based engine
##
be/src/vec/functions/function_timestamp.cpp:
##
@@ -569,6 +569,164 @@ class FunctionUnixTimestamp : public IFunction {
}
};
+template
+class FunctionDateOrDateTimeToDate : public IFunction {
+public:
+static constexpr auto name = Impl::name;
+static FunctionPtr create() {
+return std::make_shared>();
+}
+
+String get_name() const override { return name; }
+
+bool use_default_implementation_for_nulls() const override { return true; }
+
+size_t get_number_of_arguments() const override { return 1; }
+
+bool is_variadic() const override { return true; }
+
+// input DateTime and Date, return Date
+// input DateTimeV2 and DateV2, return DateV2
+DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName& arguments)
const override {
+if constexpr (std::is_same_v) {
+return make_nullable(std::make_shared());
+} else if constexpr (std::is_same_v) {
+return make_nullable(std::make_shared());
+} else if constexpr (std::is_same_v) {
+return make_nullable(std::make_shared());
+} else {
+return make_nullable(std::make_shared());
+}
+}
+
+DataTypes get_variadic_argument_types_impl() const override {
+if constexpr (std::is_same_v) {
Review Comment:
`DataTypeDate` is also possible as input argument
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on a diff in pull request #13400: [function](date function)add new date function 'last_day'
zhangstar333 commented on code in PR #13400:
URL: https://github.com/apache/doris/pull/13400#discussion_r996918671
##
be/src/vec/functions/function_timestamp.cpp:
##
@@ -569,6 +569,164 @@ class FunctionUnixTimestamp : public IFunction {
}
};
+template
+class FunctionDateOrDateTimeToDate : public IFunction {
+public:
+static constexpr auto name = Impl::name;
+static FunctionPtr create() {
+return std::make_shared>();
+}
+
+String get_name() const override { return name; }
+
+bool use_default_implementation_for_nulls() const override { return true; }
+
+size_t get_number_of_arguments() const override { return 1; }
+
+bool is_variadic() const override { return true; }
+
+// input DateTime and Date, return Date
+// input DateTimeV2 and DateV2, return DateV2
+DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName& arguments)
const override {
+if constexpr (std::is_same_v) {
+return make_nullable(std::make_shared());
+} else if constexpr (std::is_same_v) {
+return make_nullable(std::make_shared());
+} else if constexpr (std::is_same_v) {
+return make_nullable(std::make_shared());
+} else {
+return make_nullable(std::make_shared());
+}
+}
+
+DataTypes get_variadic_argument_types_impl() const override {
+if constexpr (std::is_same_v) {
+return {std::make_shared()};
+} else if constexpr (std::is_same_v) {
+return {std::make_shared()};
+} else {
+return {std::make_shared()};
+}
+}
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+return Impl::execute_impl(context, block, arguments, result,
input_rows_count);
+}
+};
+
+template
+struct LastDayImpl {
+static constexpr auto name = "last_day";
+
+static Status execute_impl(FunctionContext* context, Block& block,
+ const ColumnNumbers& arguments, size_t result,
+ size_t input_rows_count) {
+auto null_map = ColumnUInt8::create(input_rows_count, 0);
+ColumnPtr res_column;
+ColumnPtr argument_column =
+
block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();
+if constexpr (std::is_same_v) {
+auto data_col = assert_cast*>(argument_column.get());
+res_column = ColumnInt64::create(input_rows_count);
+execute_straight(
+input_rows_count, null_map->get_data(),
data_col->get_data(),
+
static_cast*>(res_column->assume_mutable().get())
+->get_data());
+
+} else if constexpr (std::is_same_v) {
+auto data_col = assert_cast*>(argument_column.get());
+res_column = ColumnVector::create(input_rows_count);
+execute_straight, UInt32, UInt32>(
+input_rows_count, null_map->get_data(),
data_col->get_data(),
+
static_cast*>(res_column->assume_mutable().get())
+->get_data());
+
+} else if constexpr (std::is_same_v) {
+auto data_col = assert_cast*>(argument_column.get());
+res_column = ColumnVector::create(input_rows_count);
+execute_straight, UInt32, UInt64>(
+input_rows_count, null_map->get_data(),
data_col->get_data(),
+
static_cast*>(res_column->assume_mutable().get())
+->get_data());
+} else {
+// neither DateTime nor DateTimeV2/DateV2, return null
+if constexpr (std::is_same_v) {
+res_column = ColumnInt64::create(input_rows_count);
Review Comment:
this check seems to have been done at the above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju opened a new pull request, #13420: [bugfix](array-type) ColumnDate lost is_date_type after cloned
cambyzju opened a new pull request, #13420: URL: https://github.com/apache/doris/pull/13420 # Proposed changes Issue Number: close #xxx Describe your changes. Problem: IColumn::is_date property will lost after ColumnDate::clone called. Fix: After ColumnDate created, also set IColumn::is_date. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zy-kkk commented on a diff in pull request #13418: [function](string_function) add new string function 'not_null_or_empty'
zy-kkk commented on code in PR #13418:
URL: https://github.com/apache/doris/pull/13418#discussion_r996925807
##
docs/zh-CN/docs/sql-manual/sql-functions/string-functions/not_null_or_empty.md:
##
@@ -0,0 +1,60 @@
+---
+{
+"title": "not_null_or_empty",
+"language": "zh-CN"
+}
+---
+
+
+
+## not_null_or_empty
+### description
+ Syntax
+
+`BOOLEAN NOT_NULL_OR_EMPTY (VARCHAR str)`
+
+如果字符串为空字符串或者NULL，返回false。否则，返回true。
+
+### example
+
+```
+MySQL [(none)]> select not_null_or_empty(null);
++-+
+| null_or_empty(NULL) |
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zy-kkk commented on pull request #13418: [function](string_function) add new string function 'not_null_or_empty'
zy-kkk commented on PR #13418: URL: https://github.com/apache/doris/pull/13418#issuecomment-1280687878 > Need add test done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #13402: [Nereids][Improve] refactor eliminate outer join
wangshuo128 commented on code in PR #13402:
URL: https://github.com/apache/doris/pull/13402#discussion_r996929481
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EliminateOutJoin.java:
##
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import
org.apache.doris.nereids.rules.expression.rewrite.rules.FoldConstantRule;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Eliminate out join.
+ */
+public class EliminateOutJoin extends OneRewriteRuleFactory {
+
+@Override
+public Rule build() {
+return logicalFilter(logicalJoin())
+.when(filter -> filter.child().getJoinType().isOuterJoin())
+.then(filter -> {
+LogicalJoin join = filter.child();
+List leftPredicates = Lists.newArrayList();
+List rightPredicates = Lists.newArrayList();
+for (Expression predicate :
ExpressionUtils.extractConjunction(filter.getPredicates())) {
Review Comment:
```suggestion
for (Expression predicate : filter.getConjuncts) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #13402: [Nereids][Improve] refactor eliminate outer join
wangshuo128 commented on code in PR #13402:
URL: https://github.com/apache/doris/pull/13402#discussion_r996929481
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EliminateOutJoin.java:
##
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import
org.apache.doris.nereids.rules.expression.rewrite.rules.FoldConstantRule;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Eliminate out join.
+ */
+public class EliminateOutJoin extends OneRewriteRuleFactory {
+
+@Override
+public Rule build() {
+return logicalFilter(logicalJoin())
+.when(filter -> filter.child().getJoinType().isOuterJoin())
+.then(filter -> {
+LogicalJoin join = filter.child();
+List leftPredicates = Lists.newArrayList();
+List rightPredicates = Lists.newArrayList();
+for (Expression predicate :
ExpressionUtils.extractConjunction(filter.getPredicates())) {
Review Comment:
```suggestion
for (Expression predicate : filter.getConjuncts()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zy-kkk commented on a diff in pull request #13418: [function](string_function) add new string function 'not_null_or_empty'
zy-kkk commented on code in PR #13418:
URL: https://github.com/apache/doris/pull/13418#discussion_r996937312
##
be/src/vec/functions/function_string.h:
##
@@ -420,6 +420,43 @@ class FunctionNullOrEmpty : public IFunction {
}
};
+class FunctionNotNullOrEmpty : public IFunction {
+public:
+static constexpr auto name = "not_null_or_empty";
+static FunctionPtr create() { return
std::make_shared(); }
+String get_name() const override { return name; }
+size_t get_number_of_arguments() const override { return 1; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+return std::make_shared();
+}
+
+bool use_default_implementation_for_nulls() const override { return false;
}
+bool use_default_implementation_for_constants() const override { return
true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+auto res_map = ColumnUInt8::create(input_rows_count, 0);
+
+auto column = block.get_by_position(arguments[0]).column;
+if (auto* nullable = check_and_get_column(*column)) {
+column = nullable->get_nested_column_ptr();
+res_map->get_data();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zy-kkk commented on a diff in pull request #13418: [function](string_function) add new string function 'not_null_or_empty'
zy-kkk commented on code in PR #13418:
URL: https://github.com/apache/doris/pull/13418#discussion_r996938812
##
be/src/vec/functions/function_string.h:
##
@@ -420,6 +420,43 @@ class FunctionNullOrEmpty : public IFunction {
}
};
+class FunctionNotNullOrEmpty : public IFunction {
+public:
+static constexpr auto name = "not_null_or_empty";
+static FunctionPtr create() { return
std::make_shared(); }
+String get_name() const override { return name; }
+size_t get_number_of_arguments() const override { return 1; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+return std::make_shared();
+}
+
+bool use_default_implementation_for_nulls() const override { return false;
}
+bool use_default_implementation_for_constants() const override { return
true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+auto res_map = ColumnUInt8::create(input_rows_count, 0);
+
+auto column = block.get_by_position(arguments[0]).column;
+if (auto* nullable = check_and_get_column(*column)) {
+column = nullable->get_nested_column_ptr();
+res_map->get_data();
Review Comment:
I set all the null's to 0 at the beginning, and what I want is for null to
return 0, so I don't think it needs to be handled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12996: [Nereids][Improve] infer predicate when push down predicate
924060929 commented on code in PR #12996:
URL: https://github.com/apache/doris/pull/12996#discussion_r996610488
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EffectivePredicatesExtractor.java:
##
@@ -0,0 +1,157 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * extract effective predicates.
+ */
+public class EffectivePredicatesExtractor extends PlanVisitor,
Void> {
+
+PredicatePropagation propagation = new PredicatePropagation();
+
+@Override
+public Set visit(Plan plan, Void context) {
+return Sets.newHashSet();
+}
+
+@Override
+public Set visitLogicalFilter(LogicalFilter
filter, Void context) {
+List predicates =
ExpressionUtils.extractConjunction(filter.getPredicates()).stream()
+.filter(p -> {
+if (p instanceof SubqueryExpr) {
+SubqueryExpr subqueryExpr = (SubqueryExpr) p;
+return subqueryExpr.getCorrelateSlots().isEmpty();
+}
+return true;
+}).collect(Collectors.toList());
+predicates.addAll(filter.child().accept(this, context));
+return getAvailableExpressions(Sets.newHashSet(predicates), filter);
+}
+
+@Override
+public Set visitLogicalJoin(LogicalJoin join, Void context) {
+Set predicates = Sets.newHashSet();
+Set leftPredicates = join.left().accept(this, context);
+Set rightPredicates = join.right().accept(this, context);
+switch (join.getJoinType()) {
+case INNER_JOIN:
+case CROSS_JOIN:
+predicates.addAll(leftPredicates);
+predicates.addAll(rightPredicates);
+join.getOnClauseCondition().map(on ->
predicates.addAll(ExpressionUtils.extractConjunction(on)));
+break;
+case LEFT_SEMI_JOIN:
+predicates.addAll(leftPredicates);
+join.getOnClauseCondition().map(on ->
predicates.addAll(ExpressionUtils.extractConjunction(on)));
+break;
+case RIGHT_SEMI_JOIN:
+predicates.addAll(rightPredicates);
+join.getOnClauseCondition().map(on ->
predicates.addAll(ExpressionUtils.extractConjunction(on)));
+break;
+case LEFT_OUTER_JOIN:
+case LEFT_ANTI_JOIN:
+predicates.addAll(leftPredicates);
+break;
+case RIGHT_OUTER_JOIN:
+case RIGHT_ANTI_JOIN:
+predicates.addAll(rightPredicates);
+break;
+default:
+}
+return getAvailableExpressions(predicates, join);
+}
+
+@Override
+public Set visitLogicalProject(LogicalProject
project, Void context) {
+Set childPredicates = project.child().accept(this,
context);
+Map expressionSlotMap = project.getAliasToProducer()
+.entrySet()
+.stream()
+.collect(Collectors.toMap(Entry::getValue,
[GitHub] [doris] nextdreamblue opened a new pull request, #13421: [Enhancement](profile) support configure the number of query profile
nextdreamblue opened a new pull request, #13421: URL: https://github.com/apache/doris/pull/13421 Signed-off-by: nextdreamblue # Proposed changes Issue Number: close #13352 ## Problem summary support configure the number of query profile ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liujinhui1994 commented on pull request #13406: Add support for parquet and orc file into spark load. issue(#13099)
liujinhui1994 commented on PR #13406: URL: https://github.com/apache/doris/pull/13406#issuecomment-1280775468 Don't merge This PR，because has some problems, I'll deal with it tomorrow morning if allowed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #13424: [fix](agg)the intermediate slots should be materialized as output slots
starocean999 opened a new pull request, #13424: URL: https://github.com/apache/doris/pull/13424 # Proposed changes Issue Number: close #xxx ## Problem summary pick https://github.com/apache/doris/pull/12441 from master ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin opened a new issue, #13425: [Bug] `year()` results are not as expected
luozenglin opened a new issue, #13425: URL: https://github.com/apache/doris/issues/13425 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? `year()` results are not as expected ### What You Expected? ``` MySQL [testdb]> select date, year(date) from test_year; +++ | date | year(date) | +++ | 2021-01-01 | 2021 | | -01-01 | 0 | | -01-01 | | +++ ``` ### How to Reproduce? ``` CREATE TABLE `test_year` ( `id` int null, `date` date NULL ) ENGINE=OLAP UNIQUE KEY(`id`) COMMENT 'OLAP' DISTRIBUTED BY HASH(`id`) BUCKETS 2 PROPERTIES ( "replication_allocation" = "tag.location.default: 1" ); insert into test_year values(1, '2021-01-01'), (2, '0-01-01'),(3, '-01-01'); mysql> select date, year(date) from test_year; ++--+ | date | year(`date`) | ++--+ | -01-01 | NULL | | 2021-01-01 | 2021 | | -01-01 | NULL | ++--+ ``` ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin opened a new pull request, #13426: [fix](year) fix `year()` results are not as expected
luozenglin opened a new pull request, #13426: URL: https://github.com/apache/doris/pull/13426 # Proposed changes Issue Number: close #13425 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #13418: [function](string_function) add new string function 'not_null_or_empty'
BiteThet commented on code in PR #13418:
URL: https://github.com/apache/doris/pull/13418#discussion_r997105937
##
be/src/vec/functions/function_string.h:
##
@@ -420,6 +420,43 @@ class FunctionNullOrEmpty : public IFunction {
}
};
+class FunctionNotNullOrEmpty : public IFunction {
+public:
+static constexpr auto name = "not_null_or_empty";
+static FunctionPtr create() { return
std::make_shared(); }
+String get_name() const override { return name; }
+size_t get_number_of_arguments() const override { return 1; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+return std::make_shared();
+}
+
+bool use_default_implementation_for_nulls() const override { return false;
}
+bool use_default_implementation_for_constants() const override { return
true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+auto res_map = ColumnUInt8::create(input_rows_count, 0);
+
+auto column = block.get_by_position(arguments[0]).column;
+if (auto* nullable = check_and_get_column(*column)) {
+column = nullable->get_nested_column_ptr();
+res_map->get_data();
Review Comment:
I mean, we can't guarantee that when the xth element of a ColumnNullable is
null, it must have a length of 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 commented on a diff in pull request #13405: [fix](docs) fix error description of `LDAP_ADMIN_PASSWORD` in the document
xy720 commented on code in PR #13405:
URL: https://github.com/apache/doris/pull/13405#discussion_r997131152
##
docs/zh-CN/docs/sql-manual/sql-reference/Account-Management-Statements/LDAP.md:
##
@@ -44,12 +44,12 @@ SET LDAP_ADMIN_PASSWORD
1. 设置LDAP管理员密码
```sql
-SET LDAP_ADMIN_PASSWORD = '123456'
+SET LDAP_ADMIN_PASSWORD = PASSWORD('123456')
```
### Keywords
-LDAP
Review Comment:
how about use LDAP,ADMIN,PASSWORD ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13402: [Nereids][Improve] refactor eliminate outer join
hello-stephen commented on PR #13402: URL: https://github.com/apache/doris/pull/13402#issuecomment-1280970501 https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221017223936_clickbench_pr_30091.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #13427: [fix](conf) aggressive_memory_decommit and chunk_reserve_limits can n…
dataroaring opened a new pull request, #13427: URL: https://github.com/apache/doris/pull/13427 …ot be changed when running # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #13399: [feature-wip](statistics) Collect statistics by sampling sql-tasks
morrySnow commented on PR #13399: URL: https://github.com/apache/doris/pull/13399#issuecomment-1281043160 @Kikyou1997 PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #13354: [feature-wip](new-scan) Support stream load with csv in new scan framework
morningman merged PR #13354: URL: https://github.com/apache/doris/pull/13354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #13428: [feature-wip](array) remove array config and check array nested depth
morningman opened a new pull request, #13428: URL: https://github.com/apache/doris/pull/13428 # Proposed changes Issue Number: close #xxx ## Problem summary 1. remove FE config `enable_array_type` 2. limit the nested depth of array in FE side. 3. Fix some bug when loading array from parquet. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 6. Does it need to update dependencies: - [ ] Yes - [ ] No 7. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 merged pull request #13407: [optimization](array-type) array_min/array_max function support the d…
xy720 merged PR #13407: URL: https://github.com/apache/doris/pull/13407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [optimization](array-type) array_min/array_max function support the date/datetime type (#13407)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 49b060418a [optimization](array-type) array_min/array_max function
support the date/datetime type (#13407)
49b060418a is described below
commit 49b060418ad6a6a0d182f483d5ab0eacc414a563
Author: carlvinhust2012
AuthorDate: Mon Oct 17 23:38:20 2022 +0800
[optimization](array-type) array_min/array_max function support the
date/datetime type (#13407)
This pr is used to expand the supported data type for array_min/array_max
function.
Before the change , the array_min/array_max function can't support the
date/datetime type.
After the change, array_min/array_max function can support the
date/datetime type.
Co-authored-by: hucheng01
---
.../functions/array/function_array_aggregation.cpp | 6 -
gensrc/script/doris_builtins_functions.py | 8 ++
.../test_array_aggregation_functions.out | 31 +++---
.../test_array_aggregation_functions.groovy| 6 ++---
4 files changed, 32 insertions(+), 19 deletions(-)
diff --git a/be/src/vec/functions/array/function_array_aggregation.cpp
b/be/src/vec/functions/array/function_array_aggregation.cpp
index f09a0a72fa..aed53bc25b 100644
--- a/be/src/vec/functions/array/function_array_aggregation.cpp
+++ b/be/src/vec/functions/array/function_array_aggregation.cpp
@@ -166,7 +166,11 @@ struct ArrayAggregateImpl {
execute_type(res, type, data, offsets) ||
execute_type(res, type, data, offsets) ||
execute_type(res, type, data, offsets) ||
-execute_type(res, type, data, offsets)) {
+execute_type(res, type, data, offsets) ||
+execute_type(res, type, data, offsets) ||
+execute_type(res, type, data, offsets) ||
+execute_type(res, type, data, offsets) ||
+execute_type(res, type, data, offsets)) {
block.replace_by_position(result, std::move(res));
return Status::OK();
} else {
diff --git a/gensrc/script/doris_builtins_functions.py
b/gensrc/script/doris_builtins_functions.py
index c27212fc8d..01fbda64d0 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -272,6 +272,10 @@ visible_functions = [
[['array_min'], 'FLOAT',['ARRAY_FLOAT'],'', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_min'], 'DOUBLE', ['ARRAY_DOUBLE'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_min'], 'DECIMALV2',['ARRAY_DECIMALV2'],'', '', '', 'vec',
'ALWAYS_NULLABLE'],
+[['array_min'], 'DATE', ['ARRAY_DATE'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
+[['array_min'], 'DATETIME', ['ARRAY_DATETIME'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
+[['array_min'], 'DATEV2', ['ARRAY_DATEV2'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
+[['array_min'], 'DATETIMEV2', ['ARRAY_DATETIMEV2'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_max'], 'TINYINT', ['ARRAY_TINYINT'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_max'], 'SMALLINT', ['ARRAY_SMALLINT'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_max'], 'INT', ['ARRAY_INT'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
@@ -280,6 +284,10 @@ visible_functions = [
[['array_max'], 'FLOAT',['ARRAY_FLOAT'],'', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_max'], 'DOUBLE', ['ARRAY_DOUBLE'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_max'], 'DECIMALV2',['ARRAY_DECIMALV2'],'', '', '', 'vec',
'ALWAYS_NULLABLE'],
+[['array_max'], 'DATE', ['ARRAY_DATE'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
+[['array_max'], 'DATETIME', ['ARRAY_DATETIME'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
+[['array_max'], 'DATEV2', ['ARRAY_DATEV2'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
+[['array_max'], 'DATETIMEV2', ['ARRAY_DATETIMEV2'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_sum'], 'BIGINT', ['ARRAY_TINYINT'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_sum'], 'BIGINT', ['ARRAY_SMALLINT'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
[['array_sum'], 'BIGINT', ['ARRAY_INT'], '', '', '', 'vec',
'ALWAYS_NULLABLE'],
diff --git
a/regression-test/data/query_p0/sql_functions/array_functions/test_array_aggregation_functions.out
b/regression-test/data/query_p0/sql_functions/array_functions/test_array_aggregation_functions.out
index 3564d7eb6e..6424664700 100644
---
a/regression-test/data/query_p0/sql_functions/array_functions/test_array_aggregation_functions.out
+++
b/regression-test/data/query_p0/sql_functions/array_functions/test_array_aggregation_functions.out
@@ -1,30 +1,31 @@
-- This file is automatically generated. You should
[GitHub] [doris] dataalive commented on issue #13411: [Feature] 运行一段时间的flink算子后，be就会宕机，谁能知道这个问题咋解决
dataalive commented on issue #13411: URL: https://github.com/apache/doris/issues/13411#issuecomment-1281073849 which doris version ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive commented on issue #13403: [Feature] be很脆弱，动不动就宕机
dataalive commented on issue #13403: URL: https://github.com/apache/doris/issues/13403#issuecomment-1281076860 Doris的版本是？你可以给更多的信息看看？这种反复挂的应该是某一个特殊的SQL或者什么触发了。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive commented on issue #13389: [Bug] cpu 100%
dataalive commented on issue #13389: URL: https://github.com/apache/doris/issues/13389#issuecomment-1281084386 type `top -H` when cpu 100% -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive commented on issue #13343: [Bug] 1.1.1 be crash
dataalive commented on issue #13343: URL: https://github.com/apache/doris/issues/13343#issuecomment-1281086650 这个问题解决了吗？没有解决的话可以继续反馈下，或者加微信：warpdrive1987 来定位下。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13420: [bugfix](array-type) ColumnDate lost is_date_type after cloned
github-actions[bot] commented on PR #13420: URL: https://github.com/apache/doris/pull/13420#issuecomment-1281119651 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13420: [bugfix](array-type) ColumnDate lost is_date_type after cloned
github-actions[bot] commented on PR #13420: URL: https://github.com/apache/doris/pull/13420#issuecomment-1281119716 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated (d63a80eaba -> 49b060418a)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git from d63a80eaba [fix](bitmap_intersect) fix `bitmap_intersect` result error (#13298) add a77808e103 [Enhancement](function) optimize decimal minus and plus #13320 add 0732ccbf0f [fix](regression) rowsDisjointWindows output is not ordered #1 add 9b0978fad9 [fix](test) make output of test stable (#13317) add f80ac34786 [regression](bloom filter) add test case for is not null (#13310) add a27bcbe882 [bugfix](compatibility) Fix a compatibility problem case by emun value changed (#13305) add d430aec3ae [Bug](bloomfilter) fix concurrency bug caused by bloom filter (#13306) add c494ca0ed4 [enhancement](memtracker) Print query memory usage log every second when `memory_verbose_track` is enabled (#13302) add 9b590ac4cb [improvement](olap) cache value of has_null in ColumnNullable (#13289) add 7147c77f22 [Enhancement](broker)Doris support obs broker load (#12781) add e08ba8d573 [feature](restore) Add new property 'reserve_dynamic_partition_enable' to restore statement (#12498) add 3e84c04195 [Bug](predicate) fix nullptr in scan node (#13316) add 4248c6f37c [improve](Nereids): avoid duplicated stats derive. (#13293) add db7f955a70 [improve](Nereids): split otherJoinCondition with List. (#13216) add 830183984a [fix](hash)update_hashes_with_value method should handle if input value is null (#13332) add bdb8e08bd3 [fix](ci) rename the checks name for branch-1.1 (#13342) add c1ed7d4d7d [Bug](function) fix core dump on case when have 1000 condition #13315 add 0ff04e81bc [fix](DynamicPartition) Not check max_dynamic_partition_num when disable DynamicPartition (#13267) add 87793b7c00 [bugfix](datatimev2) fix value column loss precision and scale (#13233) add baf2689610 [Improvement](join) compute hash values by vectorized way (#13335) add 4a6eb01ccb [refactor](Nereids): refactor UT by using Pattern and rename to remove consecutive (#13337) add fe1524a287 [Enhancement](load) remove load mem limit (#13111) add 71d2d61d33 [chore](build release) remove doris home and user info from doris_be --version output (#13344) add cb300b0b39 [feature](agg) support `any`,`any_value` agg functions. (#13228) add de4315c1c5 [feature](function) support `initcap` string function (#13193) add 87e5e2b48b [Fix](array-type) Disable schema change between array type columns (#13261) add 88e08a92d8 [fix](array-type) fix the wrong result when import array element with double quotes (#12786) add 5e0c34b35a [fix](join) should call getOutputTblRefIds to get child's tuple info (#13227) add 8dc09ad05c [enhancement](memory) Default Jemalloc as generic memory allocator #13367 add b58ae34d1b [Doc](Readme)Update the 1.1.3 release note. (#13358) add ed73096f19 [improvemnt][doc] refine doc for unique key data model (#13319) add 8d729f9386 [fix](error-code) fix misuse fo OLAP_ERR_WRITE_PROTOBUF_ERROR (#13347) add a2e513720e [feature](Nereids) auto fallback to legacy planner if analyze failed (#13351) add 71f167ac51 [fix](sort) fix nullable column sorting incorrectly (#13125) add a2a2be22a5 [ResourceTag](tag) Unified tag format verification (#13312) add 50ae9e6b19 [enhancement](planner) support select table sample (#10170) add 005c2cd43b [feature](remote) support local cache GC by disk usage (#12897) add b82e54a525 [feature](statistics) support to drop table or partition statistics (#13303) add 6746434770 [improvement](schema change) avoid using column ptr swap (#13273) add 5bc8858571 [fix](jsonreader) teach jsonreader to release memory (#13336) add 993f38fe3c [feature](Nereids): use Multi join to rearrange join to eliminate cross join by using predicate. (#13353) add 79a5125eff [Improvement](predicates) Use datev2 as the compatible type between string and datev2 (#13348) add 8218cfed40 [Bug](function) Fix constant predicate evaluation (#13346) add 4bc33a54a1 [Fix](agg) fix bitmap agg core dump when phmap pointer assert alignment (#13381) add f2fa9606c9 [fix](agg)count function should return 0 for null value (#13247) add 52397df9f0 [thirdparty](update) zstd 1.5.0 to 1.5.2 #13378 add bf2e20c4c4 [fix](agg) reset the content of grouping exprs instead of replace it with original exprs (#13376) add a5f3880649 [improvement](memory) disable page cache and chunk allocator, optimize memory allocate size (#13285) add 144486e220 [Opt](fun) simd the substring function and use stack buf to speed up (#13338) add 632670a49c [Enhancement](function) refactor of date function (#13362) add 1d5ba9cbcc [Improvement](like) Change `like` function to batch call (#13314) add a83eaddfcf [test](cache)Add remote cache
[GitHub] [doris] morningman commented on issue #13403: [Feature] be很脆弱，动不动就宕机
morningman commented on issue #13403: URL: https://github.com/apache/doris/issues/13403#issuecomment-1281154823 这不是一个好的提问题的方式 @zhanghengdashuaibi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13357: [Improvement](join) optimize join probing phase
github-actions[bot] commented on PR #13357: URL: https://github.com/apache/doris/pull/13357#issuecomment-1281276440 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13357: [Improvement](join) optimize join probing phase
github-actions[bot] commented on PR #13357: URL: https://github.com/apache/doris/pull/13357#issuecomment-1281276529 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13399: [feature-wip](statistics) Collect statistics by sampling sql-tasks
github-actions[bot] commented on PR #13399: URL: https://github.com/apache/doris/pull/13399#issuecomment-1281305703 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13399: [feature-wip](statistics) Collect statistics by sampling sql-tasks
github-actions[bot] commented on PR #13399: URL: https://github.com/apache/doris/pull/13399#issuecomment-1281305762 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13195: [enhancement](macOS M1) Support building from source on macOS (M1)
github-actions[bot] commented on PR #13195: URL: https://github.com/apache/doris/pull/13195#issuecomment-1281642098 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13195: [enhancement](macOS M1) Support building from source on macOS (M1)
github-actions[bot] commented on PR #13195: URL: https://github.com/apache/doris/pull/13195#issuecomment-1281642126 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13374: [Improvement](bf) acclocate memory in open phase
hello-stephen commented on PR #13374: URL: https://github.com/apache/doris/pull/13374#issuecomment-1281662380 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.76 seconds load time: 586 seconds storage size: 17154629339 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221018083207_clickbench_pr_30347.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13419: [fix](test) let each case uses its own table name
github-actions[bot] commented on PR #13419: URL: https://github.com/apache/doris/pull/13419#issuecomment-1281681784 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13419: [fix](test) let each case uses its own table name
github-actions[bot] commented on PR #13419: URL: https://github.com/apache/doris/pull/13419#issuecomment-1281681803 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhanghengdashuaibi commented on issue #13411: [Feature] 运行一段时间的flink算子后，be就会宕机，谁能知道这个问题咋解决
zhanghengdashuaibi commented on issue #13411: URL: https://github.com/apache/doris/issues/13411#issuecomment-1281689253 > 哪个多丽丝版本？ 1.1.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhanghengdashuaibi commented on issue #13403: [Feature] be很脆弱，动不动就宕机
zhanghengdashuaibi commented on issue #13403: URL: https://github.com/apache/doris/issues/13403#issuecomment-1281692768 > Doris的版本是？你可以给更多的信息看看？这种反复挂的应该是某一个特殊的SQL或者什么触发了。 1.1.2的版本 flinkcdc同步mysql的数据到doris，没有多大数据量就是为了测试，跑了一个星期，宕机了5 6次 WARNING: Logging before InitGoogleLogging() is written to STDERR I1017 09:30:42.672709 8381 env.cpp:46] Env init successfully. *** Aborted at 1665971351 (unix time) try "date -d @1665971351" if you are using GNU date *** *** SIGSEGV unkown detail explain (@0x0) received by PID 8381 (TID 0x7facd1d47700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/apache-doris/be/src/common/signal_handler.h:420 1# 0x7FAD2F82D400 in /lib64/[libc.so](http://libc.so/).6 2# doris::BaseFieldtypeTraits<(doris::FieldType)9>::equal(void const*, void const*) at /mnt/disk2/apache-doris/be/src/olap/types.h:491 3# doris::TupleReader::_unique_key_next_row(doris::RowCursor*, doris::MemPool*, doris::ObjectPool*, bool*) at /mnt/disk2/apache-doris/be/src/olap/tuple_reader.cpp:197 4# doris::Merger::merge_rowsets(std::shared_ptr, doris::ReaderType, std::vector, std::allocator > > const&, doris::RowsetWriter*, doris::Merger::Statistics*) in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 5# doris::Compaction::do_compaction_impl(long) in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 6# doris::Compaction::do_compaction(long) at /mnt/disk2/apache-doris/be/src/olap/compaction.cpp:112 7# doris::CumulativeCompaction::execute_compact_impl() in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 8# doris::Compaction::execute_compact() at /mnt/disk2/apache-doris/be/src/olap/compaction.cpp:50 9# doris::Tablet::execute_compaction(doris::CompactionType) in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 10# std::_Function_handler, doris::CompactionType)::$_12>::_M_invoke(std::_Any_data const&) at /mnt/disk2/ygl/installs/ldbtools/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 11# doris::ThreadPool::dispatch_thread() at /mnt/disk2/apache-doris/be/src/util/threadpool.cpp:578 12# doris::Thread::supervise_thread(void*) at /mnt/disk2/apache-doris/be/src/util/thread.cpp:407 13# start_thread in /lib64/[libpthread.so](http://libpthread.so/).0 14# clone in /lib64/[libc.so](http://libc.so/).6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhanghengdashuaibi commented on issue #13403: [Feature] be很脆弱，动不动就宕机
zhanghengdashuaibi commented on issue #13403: URL: https://github.com/apache/doris/issues/13403#issuecomment-1281692927 > 这不是一个好的提问题的方式 @zhanghengdashuaibi 抱歉抱歉 有点着急了 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhanghengdashuaibi commented on issue #13403: [Feature] be很脆弱，动不动就宕机
zhanghengdashuaibi commented on issue #13403: URL: https://github.com/apache/doris/issues/13403#issuecomment-1281693833 > 机型不支持某些指令吧.. 一个测试的算子任务跑了一个星期，没大多数据量，宕机5 6次 用的1.1.2版本的 WARNING: Logging before InitGoogleLogging() is written to STDERR I1017 09:30:42.672709 8381 env.cpp:46] Env init successfully. *** Aborted at 1665971351 (unix time) try "date -d @1665971351" if you are using GNU date *** *** SIGSEGV unkown detail explain (@0x0) received by PID 8381 (TID 0x7facd1d47700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/apache-doris/be/src/common/signal_handler.h:420 1# 0x7FAD2F82D400 in /lib64/[libc.so](http://libc.so/).6 2# doris::BaseFieldtypeTraits<(doris::FieldType)9>::equal(void const*, void const*) at /mnt/disk2/apache-doris/be/src/olap/types.h:491 3# doris::TupleReader::_unique_key_next_row(doris::RowCursor*, doris::MemPool*, doris::ObjectPool*, bool*) at /mnt/disk2/apache-doris/be/src/olap/tuple_reader.cpp:197 4# doris::Merger::merge_rowsets(std::shared_ptr, doris::ReaderType, std::vector, std::allocator > > const&, doris::RowsetWriter*, doris::Merger::Statistics*) in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 5# doris::Compaction::do_compaction_impl(long) in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 6# doris::Compaction::do_compaction(long) at /mnt/disk2/apache-doris/be/src/olap/compaction.cpp:112 7# doris::CumulativeCompaction::execute_compact_impl() in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 8# doris::Compaction::execute_compact() at /mnt/disk2/apache-doris/be/src/olap/compaction.cpp:50 9# doris::Tablet::execute_compaction(doris::CompactionType) in /opt/apache-doris-be-1.1.2-bin-x86_64/be/lib/doris_be 10# std::_Function_handler, doris::CompactionType)::$_12>::_M_invoke(std::_Any_data const&) at /mnt/disk2/ygl/installs/ldbtools/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 11# doris::ThreadPool::dispatch_thread() at /mnt/disk2/apache-doris/be/src/util/threadpool.cpp:578 12# doris::Thread::supervise_thread(void*) at /mnt/disk2/apache-doris/be/src/util/thread.cpp:407 13# start_thread in /lib64/[libpthread.so](http://libpthread.so/).0 14# clone in /lib64/[libc.so](http://libc.so/).6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #13099: [improvement](spark-load) support parquet and orc file
morningman closed pull request #13099: [improvement](spark-load) support parquet and orc file URL: https://github.com/apache/doris/pull/13099 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary opened a new pull request, #13429: [Doc](Multicatalog): Add multicatalog es doc
stalary opened a new pull request, #13429: URL: https://github.com/apache/doris/pull/13429 # Proposed changes Issue Number: close #xxx ## Problem summary 1. Add multicatalog es doc 2. Modify es unsigned_long mapping to largeint. 3. getHost add pre judge logic. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 6. Does it need to update dependencies: - [ ] Yes - [ ] No 7. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13382: [feature](Nereids) generate ExprId from 0 for each statement
hello-stephen commented on PR #13382: URL: https://github.com/apache/doris/pull/13382#issuecomment-1281705993 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.75 seconds load time: 583 seconds storage size: 17154562352 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221018095243_clickbench_pr_30408.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #13408: [Bug](funciton) fix repeat coredump when step is to long
HappenLee merged PR #13408: URL: https://github.com/apache/doris/pull/13408 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (49b060418a -> f0dbbe5b46)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 49b060418a [optimization](array-type) array_min/array_max function support the date/datetime type (#13407) add f0dbbe5b46 [Bug](funciton) fix repeat coredump when step is to long (#13408) No new revisions were added by this update. Summary of changes: be/src/vec/functions/function_string.h| 79 ++- be/test/vec/function/function_string_test.cpp | 1 + gensrc/script/doris_builtins_functions.py | 4 +- 3 files changed, 67 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13429: [Doc](Multicatalog): Add multicatalog es doc
github-actions[bot] commented on PR #13429: URL: https://github.com/apache/doris/pull/13429#issuecomment-1281715690 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13429: [Doc](Multicatalog): Add multicatalog es doc
github-actions[bot] commented on PR #13429: URL: https://github.com/apache/doris/pull/13429#issuecomment-1281715655 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive commented on issue #13411: [Feature] 运行一段时间的flink算子后，be就会宕机，谁能知道这个问题咋解决
dataalive commented on issue #13411: URL: https://github.com/apache/doris/issues/13411#issuecomment-1281716403 this bug has been fixed, use 1.1.3 release binary to have a try. [https://github.com/apache/doris/releases/tag/1.1.3-rc02](https://github.com/apache/doris/releases/tag/1.1.3-rc02 ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive commented on issue #13403: [Feature] be很脆弱，动不动就宕机
dataalive commented on issue #13403: URL: https://github.com/apache/doris/issues/13403#issuecomment-1281716991 我在 https://github.com/apache/doris/issues/13411 中已经回复，这个是一个在1.1.3中修复的bug，请升级下看看。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive commented on issue #13411: [Feature] 运行一段时间的flink算子后，be就会宕机，谁能知道这个问题咋解决
dataalive commented on issue #13411: URL: https://github.com/apache/doris/issues/13411#issuecomment-1281718990 the pr https://github.com/apache/doris/pull/10094 fixed this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #13363: [Improvement](multi-catalog)Support refresh external catalog.
morningman commented on code in PR #13363:
URL: https://github.com/apache/doris/pull/13363#discussion_r997623662
##
fe/fe-core/src/main/java/org/apache/doris/analysis/RefreshDbStmt.java:
##
@@ -34,16 +34,26 @@
public class RefreshDbStmt extends DdlStmt {
private static final Logger LOG =
LogManager.getLogger(RefreshDbStmt.class);
+private String catalogName;
Review Comment:
fill the `catalogName` field in `analyze()` method.
To avoid return null from `getCatalogName()`
##
fe/fe-core/src/main/java/org/apache/doris/catalog/external/HMSExternalDatabase.java:
##
@@ -66,14 +66,32 @@ private synchronized void makeSureInitialized() {
private void init() {
List tableNames = extCatalog.listTableNames(null, name);
if (tableNames != null) {
+Map tmpTableNameToId = Maps.newConcurrentMap();
+Map tmpIdToTbl = Maps.newHashMap();
for (String tableName : tableNames) {
-long tblId = Env.getCurrentEnv().getNextId();
-tableNameToId.put(tableName, tblId);
-idToTbl.put(tblId, new HMSExternalTable(tblId, tableName,
name, (HMSExternalCatalog) extCatalog));
+long tblId;
+if (tableNameToId != null &&
tableNameToId.containsKey(tableName)) {
Review Comment:
`tableNameToId != null` is always true?
##
fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java:
##
@@ -84,10 +83,21 @@ private void init() {
return;
}
for (String dbName : allDatabases) {
-long dbId = Env.getCurrentEnv().getNextId();
-dbNameToId.put(dbName, dbId);
-idToDb.put(dbId, new HMSExternalDatabase(this, dbId, dbName));
+long dbId;
+if (dbNameToId != null && dbNameToId.containsKey(dbName)) {
Review Comment:
`dbNameToId != null` is always true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
