[GitHub] [doris] Gabriel39 opened a new pull request, #11618: [Bug](datev2) Support to use datev2 as partition column
Gabriel39 opened a new pull request, #11618: URL: https://github.com/apache/doris/pull/11618 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li commented on a diff in pull request #11582: [feature-wip](new-scan) add scanner scheduling framework
Jibing-Li commented on code in PR #11582:
URL: https://github.com/apache/doris/pull/11582#discussion_r940998214
##
be/src/vec/exec/scan/scanner_scheduler.h:
##
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "common/status.h"
+#include "util/blocking_queue.hpp"
+#include "vec/exec/scan/scanner_context.h"
+
+namespace doris {
+
+// Responsible for the scheduling and execution of all Scanners of a BE node.
+// ScannerScheduler has two types of thread pools:
+// 1. Scheduling thread pool
+// Responsible for Scanner scheduling.
+// A set of Scanners for a query will be encapsulated into a ScannerContext
+// and submitted to the ScannerScheduler's scheduling queue.
+// There are multiple scheduling queues in ScannerScheduler, and each
scheduling queue
+// is handled by a scheduling thread.
+// The scheduling thread is scheduled in granularity of ScannerContext,
+// that is, a group of Scanners in a ScannerContext are scheduled at a
time.
+//
+//2. Execution thread pool
+// The scheduling thread will submit the Scanners selected from the
ScannerContext
+// to the execution thread pool to do the actual scan task.
+// Each Scanner will act as a producer, read a group of blocks and put
them into
+// the corresponding block queue.
+// The corresponding ScanNode will act as a consumer to consume blocks
from the block queue.
+class Env;
+class ScannerScheduler {
+public:
+ScannerScheduler();
+~ScannerScheduler();
+
+Status init(ExecEnv* env);
+
+Status submit(ScannerContext* ctx);
+
+private:
+// scheduling thread function
+void _schedule_thread(int queue_id);
+// schedule scanners in a certain ScannerContext
+void _schedule_scanners(ScannerContext* ctx);
+// execution thread function
+void _scanner_scan(ScannerContext* ctx, VScanner* scanner);
+
+private:
+// scheduling queue number.
+// TODO: make it configurable.
+static const int QUEUE_NUM = 4;
+// the ScannerContext will be submitted to the scheduling queue roundrobin.
+// _queue_idx pointer to the current queue.
+std::atomic_int _queue_idx = {0};
+BlockingQueue** _scheduling_queues;
Review Comment:
Why do we need multiple scheduling queues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 opened a new pull request, #11619: [Feature](Nereids)Add plan checker
wangshuo128 opened a new pull request, #11619: URL: https://github.com/apache/doris/pull/11619 # Proposed changes This PR proposes to add a plan checker to facilitate plan checking in unit tests. Usage of plan checker is like below: ```java new PlanChecker() .plan(myPlan) .applyBottomUp(myRule) .matches(expectedPattern); ``` ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #11577: [Chore](thirdparty) upgrade phmap and s2geo
morningman commented on PR #11577: URL: https://github.com/apache/doris/pull/11577#issuecomment-1209026575 Hi @BiteThet , how about do this after we releasing the v1.2? or it may hard to maintain the version of the thirdparties, cause they are not compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #11620: [minor](*): remove redundant log and unused code.
jackwener opened a new pull request, #11620: URL: https://github.com/apache/doris/pull/11620 # Proposed changes Issue Number: close #xxx ## Problem summary Don't print config during compilation. BTW, Remove unused code. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 opened a new pull request, #11621: [Fix](planner) Fix wrong planner with count(*) optmizer for cross join optimization
Kikyou1997 opened a new pull request, #11621: URL: https://github.com/apache/doris/pull/11621 # Proposed changes Issue Number: noissue ## Problem summary 1. Set source expr for the slotDescrptor of inlineView, so that we could mark the slot in the lower level as materialzied when we do the column prune in the logic of `materializeInlineViewResultExprForCrossJoinOrCountStar` 2. Delay the deletion for not materialized slot in the resolvedTupleExprs in the SortNode, so that we could make sure those slot which would be materialized in the logic of `materializeInlineViewResultExprForCrossJoinOrCountStar` could be add to the resolvedTupleExprs as the output of the SortNode 3. Remove some useless field in the AggregateInfo which always bother and make developer confused when reading code Besides, this PR will also change the nullability inference of the CreateTableAsSelectStmt, since the nullability of created table is determined by the srcExpr first, if there doesn't exist srcExpr, then will use the nullability of the SlotRef itself, in the past, this srcExpr of tuple created for InlineView is always null, but now we set it. You can learn these affected logic in the below methods: * InternalDataSource# createTableAsSelect * Expr#getSrcSlotRef ## Checklist(Required) 1. Does it affect the original behavior: - [x] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 6. Does it need to update dependencies: - [ ] Yes - [x] No 7. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #11622: [dev-1.1.2](doc) fix help doc invalid format
morningman opened a new pull request, #11622: URL: https://github.com/apache/doris/pull/11622 # Proposed changes Issue Number: close #xxx ## Problem summary This will cause FE start with error in log: ``` org.apache.doris.common.UserException: errCode = 2, detailMessage = Head first read is not h1. ``` And can not use `help` command ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on pull request #11515: [fix] infer equation expression bug
englefly commented on PR #11515: URL: https://github.com/apache/doris/pull/11515#issuecomment-1209044710 I already add ut to check this incorrectly inference. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LemonLiTree commented on pull request #11558: [regression-test](time_add/sub)add time_add and time_sub funcation case
LemonLiTree commented on PR #11558: URL: https://github.com/apache/doris/pull/11558#issuecomment-1209063096 > FYI: I think it's better to put those cases in test_date_function.groovy Thank you for your reply,I've already put those cases in test_date_function.groovy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen opened a new pull request, #11623: [Feature](unique-key-merge-on-write) some fix on delete bitmap usage
zhannngchen opened a new pull request, #11623: URL: https://github.com/apache/doris/pull/11623 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11622: [dev-1.1.2](doc) fix help doc invalid format
yiguolei merged PR #11622: URL: https://github.com/apache/doris/pull/11622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [dev-1.1.2](doc) fix help doc invalid format (#11622)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch dev-1.1.2 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/dev-1.1.2 by this push: new 10e5bd6234 [dev-1.1.2](doc) fix help doc invalid format (#11622) 10e5bd6234 is described below commit 10e5bd6234e7497a93f795a45441eaba6c94db05 Author: Mingyu Chen AuthorDate: Tue Aug 9 16:22:33 2022 +0800 [dev-1.1.2](doc) fix help doc invalid format (#11622) Co-authored-by: morningman --- .../sql-functions/bitmap-functions/intersect_count.md | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/zh-CN/sql-reference/sql-functions/bitmap-functions/intersect_count.md b/docs/zh-CN/sql-reference/sql-functions/bitmap-functions/intersect_count.md index 41f58f6da0..593ae921bf 100644 --- a/docs/zh-CN/sql-reference/sql-functions/bitmap-functions/intersect_count.md +++ b/docs/zh-CN/sql-reference/sql-functions/bitmap-functions/intersect_count.md @@ -22,15 +22,16 @@ specific language governing permissions and limitations under the License. --> -## intersect_count -### description - Syntax +# intersect_count + +## description +### Syntax `BITMAP INTERSECT_COUNT(bitmap_column, column_to_filter, filter_values)` 聚合函数，求bitmap交集大小的函数, 不要求数据分布正交 第一个参数是Bitmap列，第二个参数是用来过滤的维度列，第三个参数是变长参数，含义是过滤维度列的不同取值 -### example +## example ``` MySQL [test_query_qa]> select dt,bitmap_to_string(user_id) from pv_bitmap where dt in (3,4); @@ -51,6 +52,6 @@ MySQL [test_query_qa]> select intersect_count(user_id,dt,3,4) from pv_bitmap; 1 row in set (0.014 sec) ``` -### keywords +## keywords INTERSECT_COUNT,BITMAP - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11604: [feature](Nereids): polish enforcer
jackwener commented on code in PR #11604:
URL: https://github.com/apache/doris/pull/11604#discussion_r941047286
##
fe/fe-core/src/test/java/org/apache/doris/nereids/jobs/CostAndEnforcerJobTest.java:
##
@@ -55,60 +49,69 @@ public class CostAndEnforcerJobTest {
*/\
* A B
*/
+
+private static List scans = Lists.newArrayList();
+private static List> outputs = Lists.newArrayList();
+
+@BeforeAll
+public static void init() {
+LogicalOlapScan scan1 = PlanConstructor.newLogicalOlapScan(0, "a", 0);
+LogicalOlapScan scan2 = PlanConstructor.newLogicalOlapScan(1, "b", 1);
+LogicalOlapScan scan3 = PlanConstructor.newLogicalOlapScan(2, "c", 0);
+
+scans.add(scan1);
+scans.add(scan2);
+scans.add(scan3);
+
+List t1Output = scan1.getOutput().stream().map(slot ->
(SlotReference) slot)
+.collect(Collectors.toList());
+List t2Output = scan2.getOutput().stream().map(slot ->
(SlotReference) slot)
+.collect(Collectors.toList());
+List t3Output = scan3.getOutput().stream().map(slot ->
(SlotReference) slot)
+.collect(Collectors.toList());
+outputs.add(t1Output);
+outputs.add(t2Output);
+outputs.add(t3Output);
+}
+
@Test
-public void testExecute(@Mocked LogicalProperties logicalProperties) {
+public void testExecute() {
Review Comment:
Clear this UT
##
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/CostAndEnforcerJob.java:
##
@@ -66,18 +67,18 @@ public CostAndEnforcerJob(GroupExpression groupExpression,
JobContext context) {
this.groupExpression = groupExpression;
}
-@Override
-public void execute() {
-for (Group childGroup : groupExpression.children()) {
-if (!childGroup.isHasCost()) {
-// TODO: interim solution
-pushTask(new CostAndEnforcerJob(this.groupExpression,
context));
-pushTask(new OptimizeGroupJob(childGroup, context));
-childGroup.setHasCost(true);
-return;
-}
-}
-}
+//@Override
Review Comment:
Should remove these code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11621: [Fix](planner) Fix wrong planner with count(*) optmizer for cross join optimization
yiguolei merged PR #11621: URL: https://github.com/apache/doris/pull/11621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [cherry-pick 1.1.2](planner) Fix wrong planner with count(*) optmizer for cross join optimization (#11621)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new a746b2675e [cherry-pick 1.1.2](planner) Fix wrong planner with
count(*) optmizer for cross join optimization (#11621)
a746b2675e is described below
commit a746b2675ead5cd4b24e2dd2f3893ba1c7946677
Author: Kikyou1997 <33112463+kikyou1...@users.noreply.github.com>
AuthorDate: Tue Aug 9 16:23:50 2022 +0800
[cherry-pick 1.1.2](planner) Fix wrong planner with count(*) optmizer for
cross join optimization (#11621)
---
.../org/apache/doris/analysis/AggregateInfo.java | 6 --
.../apache/doris/analysis/FunctionCallExpr.java| 17 -
.../org/apache/doris/analysis/InlineViewRef.java | 2 +-
.../org/apache/doris/analysis/SlotDescriptor.java | 15
.../apache/doris/planner/SingleNodePlanner.java| 30
.../java/org/apache/doris/planner/SortNode.java| 5 +-
.../data/aggregate/aggregate_count1.out| 4 ++
.../suites/query/aggregate/aggregate_count1.groovy | 82 ++
8 files changed, 134 insertions(+), 27 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/AggregateInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/AggregateInfo.java
index b599f36909..128e60df7a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/AggregateInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/AggregateInfo.java
@@ -104,11 +104,6 @@ public final class AggregateInfo extends AggregateInfoBase
{
// if set, a subset of groupingExprs_; set and used during planning
private List partitionExprs_;
-// indices into aggregateExprs for those that need to be materialized;
-// shared between this, mergeAggInfo and secondPhaseDistinctAggInfo
-private ArrayList materializedAggregateSlots_ =
Lists.newArrayList();
-// if true, this AggregateInfo is the first phase of a 2-phase DISTINCT
computation
-private boolean isDistinctAgg = false;
// If true, the sql has MultiDistinct
private boolean isMultiDistinct_;
@@ -309,7 +304,6 @@ public final class AggregateInfo extends AggregateInfoBase {
}
this.isMultiDistinct_=
estimateIfContainsMultiDistinct(distinctAggExprs);
-isDistinctAgg = true;
// add DISTINCT parameters to grouping exprs
if (!isMultiDistinct_) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
index 80a7d631f7..0021a1cf9b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
@@ -330,23 +330,6 @@ public class FunctionCallExpr extends Expr {
return fnParams.isDistinct();
}
-public boolean isCountStar() {
-if (fnName.getFunction().equalsIgnoreCase(FunctionSet.COUNT)) {
-if (fnParams.isStar()) {
-return true;
-} else if (fnParams.exprs() == null || fnParams.exprs().isEmpty())
{
-return true;
-} else {
-for (Expr expr : fnParams.exprs()) {
-if (expr.isConstant()) {
-return true;
-}
-}
-}
-}
-return false;
-}
-
public boolean isCountDistinctBitmapOrHLL() {
if (!fnParams.isDistinct()) {
return false;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/InlineViewRef.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/InlineViewRef.java
index 01791ed17f..ecc5c4360d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/InlineViewRef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/InlineViewRef.java
@@ -198,7 +198,6 @@ public class InlineViewRef extends TableRef {
desc.setIsMaterialized(true);
materializedTupleIds.add(desc.getId());
}
-
// create sMap and baseTblSmap and register auxiliary eq predicates
between our
// tuple descriptor's slots and our *unresolved* select list exprs;
// we create these auxiliary predicates so that the analyzer can
compute the value
@@ -214,6 +213,7 @@ public class InlineViewRef extends TableRef {
String colName = getColLabels().get(i);
SlotDescriptor slotDesc =
analyzer.registerColumnRef(getAliasAsName(), colName);
Expr colExpr = queryStmt.getResultExprs().get(i);
+slotDesc.setSourceExpr(colExpr);
SlotRef slotRef = new SlotRef(slotDesc);
sMap.put(slotRef, colExpr);
baseTblSmap.put(slotRef, queryStmt.getBaseTblResultExprs().get(i));
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/
[GitHub] [doris] morningman merged pull request #10699: [enhancement] Improve the availability of broker load
morningman merged PR #10699: URL: https://github.com/apache/doris/pull/10699 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](broker) Improve the availability of broker load (#10699)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 583b44dfa8 [enhancement](broker) Improve the availability of broker
load (#10699)
583b44dfa8 is described below
commit 583b44dfa83dc49bc743a5c217956077d951d240
Author: HB
AuthorDate: Tue Aug 9 17:00:48 2022 +0800
[enhancement](broker) Improve the availability of broker load (#10699)
---
.../conf/apache_hdfs_broker.conf | 2 +-
.../org/apache/doris/broker/hdfs/BrokerConfig.java | 2 +-
.../doris/broker/hdfs/FileSystemManager.java | 97 ++
3 files changed, 27 insertions(+), 74 deletions(-)
diff --git a/fs_brokers/apache_hdfs_broker/conf/apache_hdfs_broker.conf
b/fs_brokers/apache_hdfs_broker/conf/apache_hdfs_broker.conf
index 92a30ac24e..8466de02cd 100644
--- a/fs_brokers/apache_hdfs_broker/conf/apache_hdfs_broker.conf
+++ b/fs_brokers/apache_hdfs_broker/conf/apache_hdfs_broker.conf
@@ -27,7 +27,7 @@
broker_ipc_port = 8000
# client session will be deleted if not receive ping after this time
-client_expire_seconds = 300
+client_expire_seconds = 1800
# Advanced configurations
# sys_log_dir = ${BROKER_HOME}/log
diff --git
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerConfig.java
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerConfig.java
index c5a7b66acd..6381f16db2 100644
---
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerConfig.java
+++
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerConfig.java
@@ -29,7 +29,7 @@ public class BrokerConfig extends ConfigBase {
public static int hdfs_write_buffer_size_kb = 1024;
@ConfField
-public static int client_expire_seconds = 300;
+public static int client_expire_seconds = 1800;
@ConfField
public static int broker_ipc_port = 8000;
diff --git
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
index b522d21819..262875de25 100644
---
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
+++
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
@@ -120,7 +120,6 @@ public class FileSystemManager {
clientContextManager = new ClientContextManager(handleManagementPool);
readBufferSize = BrokerConfig.hdfs_read_buffer_size_kb << 10;
writeBufferSize = BrokerConfig.hdfs_write_buffer_size_kb << 10;
-handleManagementPool.schedule(new FileSystemExpirationChecker(), 0,
TimeUnit.SECONDS);
}
private static String preparePrincipal(String originalPrincipal) throws
UnknownHostException {
@@ -212,7 +211,6 @@ public class FileSystemManager {
}
String hdfsUgi = username + "," + password;
FileSystemIdentity fileSystemIdentity = null;
-BrokerFileSystem fileSystem = null;
if (authentication.equals(AUTHENTICATION_SIMPLE)) {
fileSystemIdentity = new FileSystemIdentity(host, hdfsUgi);
} else {
@@ -242,19 +240,9 @@ public class FileSystemManager {
e.getMessage());
}
}
-cachedFileSystem.putIfAbsent(fileSystemIdentity, new
BrokerFileSystem(fileSystemIdentity));
-fileSystem = cachedFileSystem.get(fileSystemIdentity);
-if (fileSystem == null) {
-// it means it is removed concurrently by checker thread
-return null;
-}
+BrokerFileSystem fileSystem =
updateCachedFileSystem(fileSystemIdentity);
fileSystem.getLock().lock();
try {
-if (!cachedFileSystem.containsKey(fileSystemIdentity)) {
-// this means the file system is closed by file system checker
thread
-// it is a corner case
-return null;
-}
if (fileSystem.getDFSFileSystem() == null) {
logger.info("create file system for new path: " + path);
UserGroupInformation ugi = null;
@@ -406,20 +394,10 @@ public class FileSystemManager {
String disableCache =
properties.getOrDefault(FS_S3A_IMPL_DISABLE_CACHE, "true");
String s3aUgi = accessKey + "," + secretKey;
FileSystemIdentity fileSystemIdentity = new FileSystemIdentity(host,
s3aUgi);
-BrokerFileSystem fileSystem = null;
cachedFileSystem.putIfAbsent(fileSystemIdentity, new
BrokerFileSystem(fileSystemIdentity));
-fileSystem = cachedFileSystem.get(fileSystemIdentity);
-if (fileSystem == null) {
-// it means it is removed concurrently by checker thread
-
[GitHub] [doris] github-actions[bot] commented on pull request #11623: [Feature](unique-key-merge-on-write) some fix on delete bitmap usage
github-actions[bot] commented on PR #11623: URL: https://github.com/apache/doris/pull/11623#issuecomment-1209119665 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LizZhang315 opened a new pull request, #11624: [regressiont-test] add p0 test cases from Baidu
LizZhang315 opened a new pull request, #11624: URL: https://github.com/apache/doris/pull/11624 # Proposed changes Add p0 test cases from Baidu, including: aggregate join union order by group by keyword arithmetic operators logical operators case function coalesce between in like limit where regexp window function runtime filter schema change ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - No ## Further comments This is only part of the regression test cases of Baidu. If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 merged pull request #11619: [Feature](Nereids)Add plan checker
924060929 merged PR #11619: URL: https://github.com/apache/doris/pull/11619 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (583b44dfa8 -> 7b67661262)
This is an automated email from the ASF dual-hosted git repository.
huajianlan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 583b44dfa8 [enhancement](broker) Improve the availability of broker
load (#10699)
add 7b67661262 add plan checker (#11619)
No new revisions were added by this update.
Summary of changes:
.../apache/doris/nereids/rules/RulePromise.java| 2 +-
.../nereids/trees/expressions/SlotReference.java | 4 +-
.../doris/nereids/trees/plans/GroupPlan.java | 9 +-
.../trees/plans/logical/LogicalOlapScan.java | 18 +-
.../rules/rewrite/logical/ColumnPruningTest.java | 248 ++---
.../doris/nereids/util/PatternMatchSupported.java} | 12 +-
.../org/apache/doris/nereids/util/PlanChecker.java | 67 ++
7 files changed, 214 insertions(+), 146 deletions(-)
copy
fe/fe-core/src/{main/java/org/apache/doris/nereids/rules/rewrite/RewriteRuleFactory.java
=> test/java/org/apache/doris/nereids/util/PatternMatchSupported.java} (78%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanChecker.java
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11325: [Improvement] start|stop script files improvements
github-actions[bot] commented on PR #11325: URL: https://github.com/apache/doris/pull/11325#issuecomment-1209154333 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #11325: [Improvement] start|stop script files improvements
hello-stephen commented on PR #11325: URL: https://github.com/apache/doris/pull/11325#issuecomment-1209159194 start_fe may need to rebase on master since it has added the '--version' option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] link3280 opened a new issue, #53: [Bug] Stream load fails when there's no data
link3280 opened a new issue, #53: URL: https://github.com/apache/doris-flink-connector/issues/53 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.0 ### What's Wrong? Stream load fails if there's no data to flush. https://user-images.githubusercontent.com/13482005/183617882-8744e90d-0333-4c30-b808-024f8d081aca.png";> We could fix this by skip the flush if there's no pending data. ### What You Expected? No failure when there's no input data. ### How to Reproduce? Submit a job with an empty data source and Doris sink. ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt opened a new pull request, #11625: [Enhancement](load) add hidden_columns in stream load param
yixiutt opened a new pull request, #11625:
URL: https://github.com/apache/doris/pull/11625
Stream load will ignore invisible columns if no http header columns
specified, but in some case user cannot get all columns if columns
changed frequently。
Add a hidden_columns header to support hidden columns import。User can
set hidden_columns such as __DORIS_DELETE_SIGN__ and add this column
in stream load data so we can delete this line.
For example:
curl -u root -v --location-trusted -H "hidden_columns:
__DORIS_DELETE_SIGN__" -H
"format: json" -H "strip_outer_array: true" -H "jsonpaths: [\"$.id\",
\"$.name\",\"$.__DORIS_DELETE_SIGN__\"]" -T 1.json
http://{beip}:{be_port}/api/test/test1/_stream_load
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] link3280 opened a new pull request, #54: [fix] Fix job failure caused by stream load no data exception
link3280 opened a new pull request, #54: URL: https://github.com/apache/doris-flink-connector/pull/54 # Proposed changes Issue Number: close #53 ## Problem Summary: Avoid ending recordStream in DorisStreamLoad when there's no data to flush. ## Checklist(Required) 1. Does it affect the original behavior: Yes 2. Has unit tests been added: No 3. Has document been added or modified: No Need 4. Does it need to update dependencies: No 5. Are there any changes that cannot be rolled back: No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangy5 commented on a diff in pull request #11624: [regression-test] add p0 test cases from Baidu
zhangy5 commented on code in PR #11624:
URL: https://github.com/apache/doris/pull/11624#discussion_r941143348
##
regression-test/plugins/plugin_example.groovy:
##
@@ -32,3 +32,23 @@ Suite.metaClass.testPlugin = { String info /* param */ ->
}
logger.info("Added 'testPlugin' function to Suite")
+
+Suite.metaClass.check2_palo = { Object res1, Object res2 /* param */ ->
Review Comment:
avoid using palo, use doris instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11266: [feature](information_schema) add `rowsets` table into information_s…
yiguolei merged PR #11266: URL: https://github.com/apache/doris/pull/11266 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (7b67661262 -> 169996d8e4)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 7b67661262 add plan checker (#11619)
add 169996d8e4 [feature](information_schema) add `rowsets` table into
information_s… (#11266)
No new revisions were added by this update.
Summary of changes:
be/src/exec/CMakeLists.txt | 1 +
be/src/exec/schema_scanner.cpp | 3 +
.../exec/schema_scanner/schema_rowsets_scanner.cpp | 176 +
...ivileges_scanner.h => schema_rowsets_scanner.h} | 33 ++--
be/src/olap/tablet_manager.cpp | 17 ++
be/src/olap/tablet_manager.h | 2 +
.../org/apache/doris/analysis/SchemaTableType.java | 3 +-
.../java/org/apache/doris/catalog/SchemaTable.java | 17 +-
.../planner/BackendPartitionedSchemaScanNode.java | 169
.../apache/doris/planner/DistributedPlanner.java | 2 +-
.../apache/doris/planner/SingleNodePlanner.java| 7 +-
gensrc/thrift/Descriptors.thrift | 3 +-
12 files changed, 415 insertions(+), 18 deletions(-)
create mode 100644 be/src/exec/schema_scanner/schema_rowsets_scanner.cpp
copy be/src/exec/schema_scanner/{schema_table_privileges_scanner.h =>
schema_rowsets_scanner.h} (60%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/BackendPartitionedSchemaScanNode.java
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #11624: [regression-test] add p0 test cases from Baidu
yiguolei commented on PR #11624: URL: https://github.com/apache/doris/pull/11624#issuecomment-1209192090 wow, its great -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11595: [fix](regexpr)regexpr functions' contexts should be THREAD_LOCAL
github-actions[bot] commented on PR #11595: URL: https://github.com/apache/doris/pull/11595#issuecomment-1209251189 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11598: [fix](scanner) delete predicates might be inconsistent with rowset readers
github-actions[bot] commented on PR #11598: URL: https://github.com/apache/doris/pull/11598#issuecomment-1209258370 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] catpineapple opened a new pull request, #11626: [fix](dbt-doris) avoid the restriction of having to add label_id when…
catpineapple opened a new pull request, #11626: URL: https://github.com/apache/doris/pull/11626 Avoid the restriction of having to add label_id when sql is insert into + CTE When building a dbt project, the "with ... as(select)" sql expression model is often used. However, due to the syntax limitation of Doris here, label_id or column_list must be given. The original dbt-doris code is not very compatible with this. Special SQL. Therefore, I added the "label_id" macro function to the config. When the user gives his own label_id, the user's own "label_id" is used. If not given, a time-related "label_id" is automatically generated, prefixed with: "dbt_doris_label_". ## Checklist(Required) 1. Does it affect the original behavior: - [✔️] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [✔️] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [✔️] No Need 4. Does it need to update dependencies: - [ ] Yes - [✔️] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [✔️] No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] GoGoWen opened a new pull request, #7: support crc32 checksum
GoGoWen opened a new pull request, #7: URL: https://github.com/apache/doris-thirdparty/pull/7 support crc32 checksum for hdfs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] caiconghui merged pull request #7: support crc32 checksum
caiconghui merged PR #7: URL: https://github.com/apache/doris-thirdparty/pull/7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] branch libhdfs3 updated: [enhancement](libhdfs3) Support crc32 checksum (#7)
This is an automated email from the ASF dual-hosted git repository.
caiconghui pushed a commit to branch libhdfs3
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git
The following commit(s) were added to refs/heads/libhdfs3 by this push:
new dad0008 [enhancement](libhdfs3) Support crc32 checksum (#7)
dad0008 is described below
commit dad00080bdf065ee1a1b493f56e5a8024c961044
Author: GoGoWen <82132356+gogo...@users.noreply.github.com>
AuthorDate: Tue Aug 9 20:30:13 2022 +0800
[enhancement](libhdfs3) Support crc32 checksum (#7)
---
src/client/LocalBlockReader.cpp | 5 +--
src/client/RemoteBlockReader.cpp | 5 +--
src/common/Crc32.h | 74
3 files changed, 80 insertions(+), 4 deletions(-)
diff --git a/src/client/LocalBlockReader.cpp b/src/client/LocalBlockReader.cpp
index 2455ad1..4080b72 100644
--- a/src/client/LocalBlockReader.cpp
+++ b/src/client/LocalBlockReader.cpp
@@ -20,6 +20,7 @@
* limitations under the License.
*/
#include "BigEndian.h"
+#include "Crc32.h"
#include "datatransfer.pb.h"
#include "Exception.h"
#include "ExceptionInternal.h"
@@ -76,8 +77,8 @@ LocalBlockReader::LocalBlockReader(const
shared_ptr& info,
break;
case ChecksumTypeProto::CHECKSUM_CRC32:
-THROW(HdfsIOException,
- "LocalBlockReader does not support CRC32 checksum.");
+checksum = std::make_shared();
+checksumSize = sizeof(int32_t);
break;
case ChecksumTypeProto::CHECKSUM_CRC32C:
diff --git a/src/client/RemoteBlockReader.cpp b/src/client/RemoteBlockReader.cpp
index d7faf6e..508dfdd 100644
--- a/src/client/RemoteBlockReader.cpp
+++ b/src/client/RemoteBlockReader.cpp
@@ -20,6 +20,7 @@
* limitations under the License.
*/
#include "BigEndian.h"
+#include "Crc32.h"
#include "DataTransferProtocolSender.h"
#include "Exception.h"
#include "ExceptionInternal.h"
@@ -146,8 +147,8 @@ void RemoteBlockReader::checkResponse() {
break;
case ChecksumTypeProto::CHECKSUM_CRC32:
-THROW(HdfsIOException, "RemoteBlockReader does not support CRC32
checksum, Block: %s, from Datanode: %s",
- binfo.toString().c_str(), datanode.formatAddress().c_str());
+checksum = std::make_shared();
+checksumSize = sizeof(int32_t);
break;
case ChecksumTypeProto::CHECKSUM_CRC32C:
diff --git a/src/common/Crc32.h b/src/common/Crc32.h
new file mode 100644
index 000..2d5cfec
--- /dev/null
+++ b/src/common/Crc32.h
@@ -0,0 +1,74 @@
+/
+ * 2014 -
+ * open source under Apache License Version 2.0
+ /
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef _HDFS_LIBHDFS3_COMMON_CRC32_H_
+#define _HDFS_LIBHDFS3_COMMON_CRC32_H_
+
+#include "Checksum.h"
+
+#include
+
+namespace Hdfs {
+namespace Internal {
+
+/**
+ * Calculate CRC with boost::crc_32_type.
+ */
+class Crc32: public Checksum {
+public:
+/**
+ * Constructor.
+ */
+Crc32() {
+}
+
+uint32_t getValue() {
+return crc.checksum();
+}
+
+/**
+ * @ref Checksum#reset()
+ */
+void reset() {
+crc.reset();
+}
+
+/**
+ * @ref Checksum#update(const void *, int)
+ */
+void update(const void * b, int len) {
+crc.process_bytes((const char*) b, len);
+}
+
+/**
+ * Destory an Crc32 instance.
+ */
+~Crc32() {
+}
+
+private:
+boost::crc_32_type crc;
+};
+
+}
+}
+
+#endif /* _HDFS_LIBHDFS3_COMMON_CRC32_H_ */
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11604: [feature](Nereids): polish enforcer
morrySnow commented on code in PR #11604:
URL: https://github.com/apache/doris/pull/11604#discussion_r941116524
##
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/CostAndEnforcerJob.java:
##
@@ -66,18 +67,18 @@ public CostAndEnforcerJob(GroupExpression groupExpression,
JobContext context) {
this.groupExpression = groupExpression;
}
-@Override
-public void execute() {
-for (Group childGroup : groupExpression.children()) {
-if (!childGroup.isHasCost()) {
-// TODO: interim solution
-pushTask(new CostAndEnforcerJob(this.groupExpression,
context));
-pushTask(new OptimizeGroupJob(childGroup, context));
-childGroup.setHasCost(true);
-return;
-}
-}
-}
+//@Override
+//public void execute1() {
Review Comment:
remove it directly?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java:
##
@@ -43,10 +43,11 @@ public class CostCalculator {
* Constructor.
*/
public static double calculateCost(GroupExpression groupExpression) {
-PlanContext planContext = new PlanContext(groupExpression);
-CostEstimator costCalculator = new CostEstimator();
-CostEstimate costEstimate =
groupExpression.getPlan().accept(costCalculator, planContext);
-return costFormula(costEstimate);
+// PlanContext planContext = new PlanContext(groupExpression);
Review Comment:
add a comment to explain why comment these code.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildOutputPropertyDeriver.java:
##
@@ -74,20 +74,13 @@ public PhysicalProperties
visitPhysicalHashJoin(PhysicalHashJoin has
PhysicalProperties rightOutputProperty =
childrenOutputProperties.get(1);
// broadcast
+// TODO
Review Comment:
TODO what?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java:
##
@@ -112,6 +113,7 @@ public static Pair,
List> getOnClauseUsedSlot
}
}
+Preconditions.checkArgument(childSlots.first.size() ==
childSlots.second.size());
Review Comment:
```suggestion
Preconditions.checkState(childSlots.first.size() ==
childSlots.second.size());
```
https://guava.dev/releases/19.0/api/docs/com/google/common/base/Preconditions.html
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java:
##
@@ -71,21 +71,21 @@ public Void visit(Plan plan, PlanContext context) {
@Override
public Void visitPhysicalHashJoin(PhysicalHashJoin hashJoin,
PlanContext context) {
-// for broadcast join
-List propertiesForBroadcast = Lists.newArrayList(
-new PhysicalProperties(),
-new PhysicalProperties(new DistributionSpecReplicated())
-);
// for shuffle join
-Pair, List> onClauseUsedSlots =
JoinUtils.getOnClauseUsedSlots(hashJoin);
-List propertiesForShuffle = Lists.newArrayList(
-new PhysicalProperties(new
DistributionSpecHash(onClauseUsedSlots.first, ShuffleType.JOIN)),
-new PhysicalProperties(new
DistributionSpecHash(onClauseUsedSlots.second, ShuffleType.JOIN)));
-
if (!JoinUtils.onlyBroadcast(hashJoin)) {
+Pair, List> onClauseUsedSlots =
JoinUtils.getOnClauseUsedSlots(hashJoin);
+List propertiesForShuffle = Lists.newArrayList(
+new PhysicalProperties(new
DistributionSpecHash(onClauseUsedSlots.first, ShuffleType.JOIN)),
+new PhysicalProperties(new
DistributionSpecHash(onClauseUsedSlots.second, ShuffleType.JOIN)));
+
requestPropertyToChildren.add(propertiesForShuffle);
}
+// for broadcast join
if (!JoinUtils.onlyShuffle(hashJoin)) {
+List propertiesForBroadcast =
Lists.newArrayList(
+new PhysicalProperties(),
Review Comment:
i think it is better to create any distribution properties explicitly. such
as
```
new PhysicalProperties(DistributionSpecAny.getInstance())
```
or
```
PhysicalProperties.CreateDistributionSpecAnyInstance();
```
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildOutputPropertyDeriver.java:
##
@@ -74,20 +74,13 @@ public PhysicalProperties
visitPhysicalHashJoin(PhysicalHashJoin has
PhysicalProperties rightOutputProperty =
childrenOutputProperties.get(1);
// broadcast
+// TODO
if (rightOutputProperty.getDistributionSpec() instanceof
DistributionSpecReplicated) {
-// TODO
return leftOutputProperty;
}
// shuffle
-// List leftSlotRefs =
hashJoin.left().getOutput().stream().map(slot -> (SlotReference) slot)
-//.collect(Coll
[GitHub] [doris] pengxiangyu opened a new issue, #11627: [Bug] Fix bug for SubFileCache
pengxiangyu opened a new issue, #11627: URL: https://github.com/apache/doris/issues/11627 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1 ### What's Wrong? FileCache can't write files for Remote storage. ### What You Expected? FileCache can write files on local disk ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #9983: [feature-wip](statistics) FE queries BE by SQL
morrySnow commented on code in PR #9983:
URL: https://github.com/apache/doris/pull/9983#discussion_r941285015
##
fe/fe-core/src/main/java/org/apache/doris/statistics/util/Packet.java:
##
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics.util;
+
+import java.nio.ByteBuffer;
+
+public abstract class Packet {
Review Comment:
please add some comments to explain what is Packet and where to use it
##
fe/fe-core/src/main/java/org/apache/doris/statistics/util/Connection.java:
##
@@ -0,0 +1,424 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics.util;
+
+import org.apache.doris.common.Config;
+import org.apache.doris.mysql.MysqlCapability;
+import org.apache.doris.mysql.MysqlCommand;
+import org.apache.doris.mysql.privilege.PaloAuth;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.nio.ByteBuffer;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+
+public class Connection implements AutoCloseable {
Review Comment:
please add usage in commit msg into this class's comment directly
##
fe/fe-core/src/main/java/org/apache/doris/statistics/util/Connection.java:
##
@@ -0,0 +1,424 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics.util;
+
+import org.apache.doris.common.Config;
+import org.apache.doris.mysql.MysqlCapability;
+import org.apache.doris.mysql.MysqlCommand;
+import org.apache.doris.mysql.privilege.PaloAuth;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.nio.ByteBuffer;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+
+public class Connection implements AutoCloseable {
+private static final Logger LOG = LogManager.getLogger(Connection.class);
+
+private static final int BUFFER_SIZE = 1024;
+private static final int ERROR_STATUS = 0xff;
+private static final int OK_STATUS = 0x00;
+
+private final StringBuilder sb = new StringBuilder();
+
+private String host;
+private int port;
+private String username;
+private String password;
+private String database;
+
+private Socket socket;
+
+private InputStream in;
+private OutputStream out;
+
[GitHub] [doris] hello-stephen commented on pull request #11624: [regression-test](p0) add p0 test cases
hello-stephen commented on PR #11624: URL: https://github.com/apache/doris/pull/11624#issuecomment-1209333813 great job~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 closed issue #11528: [Bug] collect_list without groupby make be crashed
xy720 closed issue #11528: [Bug] collect_list without groupby make be crashed URL: https://github.com/apache/doris/issues/11528 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 merged pull request #11213: [feature-wip](array-type) support the array type in reverse function
xy720 merged PR #11213: URL: https://github.com/apache/doris/pull/11213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](array-type) support the array type in reverse function (#11213)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new df47b6941d [feature-wip](array-type) support the array type in reverse
function (#11213)
df47b6941d is described below
commit df47b6941dfa574f5f29c03eb2e6961d3a306465
Author: carlvinhust2012
AuthorDate: Tue Aug 9 20:49:09 2022 +0800
[feature-wip](array-type) support the array type in reverse function
(#11213)
Co-authored-by: hucheng01
---
.../vec/functions/array/function_array_reverse.h | 92 ++
be/src/vec/functions/function_reverse.h| 66
be/src/vec/functions/function_string.cpp | 28 +--
be/src/vec/functions/function_string.h | 19 +
.../sql-functions/string-functions/reverse.md | 37 -
.../sql-functions/string-functions/reverse.md | 37 -
gensrc/script/doris_builtins_functions.py | 24 +-
.../array_functions/test_array_functions.out | 10 +++
.../array_functions/test_array_functions.groovy| 11 +--
9 files changed, 285 insertions(+), 39 deletions(-)
diff --git a/be/src/vec/functions/array/function_array_reverse.h
b/be/src/vec/functions/array/function_array_reverse.h
new file mode 100644
index 00..bc6891a29b
--- /dev/null
+++ b/be/src/vec/functions/array/function_array_reverse.h
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+//
https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/array/arrayReverse.cpp
+// and modified by Doris
+#pragma once
+
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_const.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/functions/array/function_array_utils.h"
+
+namespace doris::vectorized {
+
+struct ArrayReverseImpl {
+static Status _execute(Block& block, const ColumnNumbers& arguments,
size_t result,
+ size_t input_rows_count) {
+ColumnPtr src_column =
+
block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();
+ColumnArrayExecutionData src;
+if (!extract_column_array_info(*src_column, src)) {
+return Status::RuntimeError(
+fmt::format("execute failed, unsupported types for
function {}({})", "reverse",
+
block.get_by_position(arguments[0]).type->get_name()));
+}
+
+bool is_nullable = src.nested_nullmap_data ? true : false;
+ColumnArrayMutableData dst = create_mutable_data(src.nested_col,
is_nullable);
+dst.offsets_ptr->reserve(input_rows_count);
+
+auto res_val = _execute_internal(*src.nested_col, *src.offsets_ptr,
*dst.nested_col,
+ *dst.offsets_ptr,
src.nested_nullmap_data,
+ dst.nested_nullmap_data);
+if (!res_val) {
+return Status::RuntimeError(
+fmt::format("execute failed or unsupported types for
function {}({})",
+"reverse",
block.get_by_position(arguments[0]).type->get_name()));
+}
+
+ColumnPtr res_column = assemble_column_array(dst);
+block.replace_by_position(result, std::move(res_column));
+return Status::OK();
+}
+
+static bool _execute_internal(const IColumn& src_column,
+ const ColumnArray::Offsets& src_offsets,
IColumn& dest_column,
+ ColumnArray::Offsets& dest_offsets, const
UInt8* src_null_map,
+ ColumnUInt8::Container* dest_null_map) {
+ColumnArray::Offset prev_src_offset = 0;
+
+for (auto curr_src_offset : src_offsets) {
+size_t array_size = curr_src_offset - prev_src_offset;
+for (size_t j = 0; j < array_size; ++j) {
+size_t j_reverse = curr_src_offset - j - 1;
+if (src_null_map && src_null_map[j_reverse
[doris] branch master updated: [fix]collect_list/collect_set without GROUP BY for NOT NULL column (#11529)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 01e4522612 [fix]collect_list/collect_set without GROUP BY for NOT NULL
column (#11529)
01e4522612 is described below
commit 01e4522612a582f0c3021c42254f0b0e1db2abd1
Author: camby <104178...@qq.com>
AuthorDate: Tue Aug 9 20:49:37 2022 +0800
[fix]collect_list/collect_set without GROUP BY for NOT NULL column (#11529)
Co-authored-by: cambyzju
---
be/src/vec/exec/vaggregation_node.cpp | 11 ++-
.../aggregate_functions/test_aggregate_collect.out| 6 ++
.../aggregate_functions/test_aggregate_collect.groovy | 11 ---
3 files changed, 20 insertions(+), 8 deletions(-)
diff --git a/be/src/vec/exec/vaggregation_node.cpp
b/be/src/vec/exec/vaggregation_node.cpp
index 1bdccbd614..e488e6d9a6 100644
--- a/be/src/vec/exec/vaggregation_node.cpp
+++ b/be/src/vec/exec/vaggregation_node.cpp
@@ -531,11 +531,12 @@ Status
AggregationNode::_get_without_key_result(RuntimeState* state, Block* bloc
for (int i = 0; i < block_schema.size(); ++i) {
const auto column_type = block_schema[i].type;
if (!column_type->equals(*data_types[i])) {
-DCHECK(column_type->is_nullable());
-DCHECK(((DataTypeNullable*)column_type.get())
- ->get_nested_type()
- ->equals(*data_types[i]));
-DCHECK(!data_types[i]->is_nullable());
+if (!is_array(remove_nullable(column_type))) {
+DCHECK(column_type->is_nullable());
+DCHECK(!data_types[i]->is_nullable());
+DCHECK(remove_nullable(column_type)->equals(*data_types[i]));
+}
+
ColumnPtr ptr = std::move(columns[i]);
// unless `count`, other aggregate function dispose empty set
should be null
// so here check the children row return
diff --git
a/regression-test/data/query/sql_functions/aggregate_functions/test_aggregate_collect.out
b/regression-test/data/query/sql_functions/aggregate_functions/test_aggregate_collect.out
index cec8df6151..b18f47128c 100644
---
a/regression-test/data/query/sql_functions/aggregate_functions/test_aggregate_collect.out
+++
b/regression-test/data/query/sql_functions/aggregate_functions/test_aggregate_collect.out
@@ -7,3 +7,9 @@
1 ['hello'] [2022-07-04][1.23]
2 \N \N \N
+-- !select --
+['hello', 'hello'] ['hello', 'hello', 'hello', 'hello']
+
+-- !select --
+['hello'] ['hello']
+
diff --git
a/regression-test/suites/query/sql_functions/aggregate_functions/test_aggregate_collect.groovy
b/regression-test/suites/query/sql_functions/aggregate_functions/test_aggregate_collect.groovy
index a1c1ff260e..064b3d9fac 100644
---
a/regression-test/suites/query/sql_functions/aggregate_functions/test_aggregate_collect.groovy
+++
b/regression-test/suites/query/sql_functions/aggregate_functions/test_aggregate_collect.groovy
@@ -26,16 +26,21 @@ suite("test_aggregate_collect", "query") {
c_int INT,
c_string VARCHAR(10),
c_date Date,
- c_decimal DECIMAL(10, 2)
+ c_decimal DECIMAL(10, 2),
+ c_string_not_null VARCHAR(10) NOT NULL
)
DISTRIBUTED BY HASH(c_int) BUCKETS 1
PROPERTIES (
"replication_num" = "1"
)
"""
-sql "INSERT INTO ${tableName} values(1,'hello','2022-07-04',1.23),
(2,NULL,NULL,NULL)"
-sql "INSERT INTO ${tableName} values(1,'hello','2022-07-04',1.23),
(2,NULL,NULL,NULL)"
+sql "INSERT INTO ${tableName} values(1,'hello','2022-07-04',1.23,'hello'),
(2,NULL,NULL,NULL,'hello')"
+sql "INSERT INTO ${tableName} values(1,'hello','2022-07-04',1.23,'hello'),
(2,NULL,NULL,NULL,'hello')"
qt_select "select
c_int,collect_list(c_string),collect_list(c_date),collect_list(c_decimal) from
${tableName} group by c_int order by c_int"
qt_select "select
c_int,collect_set(c_string),collect_set(c_date),collect_set(c_decimal) from
${tableName} group by c_int order by c_int"
+
+// test without GROUP BY
+qt_select "select collect_list(c_string),collect_list(c_string_not_null)
from ${tableName}"
+qt_select "select collect_set(c_string),collect_set(c_string_not_null)
from ${tableName}"
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 merged pull request #11529: [fix](array-type) collect_list without GROUP BY for NOT NULL column
xy720 merged PR #11529: URL: https://github.com/apache/doris/pull/11529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new pull request, #11628: [feature-wip](unique-key-merge-on-write) fix rowid conversion ut that may create a directory under an incorrect path
liaoxin01 opened a new pull request, #11628: URL: https://github.com/apache/doris/pull/11628 # Proposed changes Issue Number: close #xxx ## Problem summary The path missing backslash，so create a directory under an incorrect path. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu opened a new pull request, #11629: [fix] (remote) Fix bug for Cache Reader
pengxiangyu opened a new pull request, #11629: URL: https://github.com/apache/doris/pull/11629 # Proposed changes Issue Number: close #11627 ## Problem summary Fix bug for Cache Reader ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] branch revert-5-libhdfs3 created (now c44bda4)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch revert-5-libhdfs3 in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git at c44bda4 Revert "[Enhancement] Make libhdfs3 be compatible with old linux kernel" This branch includes the following new commits: new c44bda4 Revert "[Enhancement] Make libhdfs3 be compatible with old linux kernel" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] 01/01: Revert "[Enhancement] Make libhdfs3 be compatible with old linux kernel"
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch revert-5-libhdfs3
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git
commit c44bda4c577df4817dea3a2d7ad8d723c7fc894a
Author: Mingyu Chen
AuthorDate: Tue Aug 9 21:55:09 2022 +0800
Revert "[Enhancement] Make libhdfs3 be compatible with old linux kernel"
---
src/CMakeLists.txt | 2 --
1 file changed, 2 deletions(-)
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 269ef51..0d125cf 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -45,8 +45,6 @@ SET(HEADER
common/Exception.h
common/XmlConfig.h)
-set_source_files_properties(${libhdfs3_ROOT_SOURCES_DIR}/rpc/RpcClient.cpp
PROPERTIES COMPILE_FLAGS "-DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX=1")
-
INCLUDE_DIRECTORIES(${libhdfs3_ROOT_SOURCES_DIR})
INCLUDE_DIRECTORIES(${libhdfs3_COMMON_SOURCES_DIR})
INCLUDE_DIRECTORIES(${CMAKE_CURRENT_BINARY_DIR})
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] morningman opened a new pull request, #9: [github] add branch protection
morningman opened a new pull request, #9: URL: https://github.com/apache/doris-thirdparty/pull/9 protect following branches: 1. main 2. bdbje 3. libhdfs3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] caiconghui merged pull request #8: [release] update version to 2.3.3 and add CHANGELOG
caiconghui merged PR #8: URL: https://github.com/apache/doris-thirdparty/pull/8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] branch libhdfs3 updated: [release] update version to 2.3.3 and add CHANGELOG (#8)
This is an automated email from the ASF dual-hosted git repository.
caiconghui pushed a commit to branch libhdfs3
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git
The following commit(s) were added to refs/heads/libhdfs3 by this push:
new e8a64b8 [release] update version to 2.3.3 and add CHANGELOG (#8)
e8a64b8 is described below
commit e8a64b8eca25f02e6403d1eb316318439ac4b94d
Author: Mingyu Chen
AuthorDate: Tue Aug 9 22:16:51 2022 +0800
[release] update version to 2.3.3 and add CHANGELOG (#8)
---
CHANGELOG.md | 10 ++
src/CMakeLists.txt | 2 +-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 000..3247a15
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,10 @@
+# CHANGELOG
+
+## v2.3.3
+
+1. Support CRC32 checksum
+2. Support old linux kernel
+
+## v2.3.2
+
+1. Improving seek performence when seeking a wide range
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 269ef51..75d4607 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -2,7 +2,7 @@ CMAKE_MINIMUM_REQUIRED(VERSION 2.8)
SET(libhdfs3_VERSION_MAJOR 2)
SET(libhdfs3_VERSION_MINOR 3)
-SET(libhdfs3_VERSION_PATCH 2)
+SET(libhdfs3_VERSION_PATCH 3)
SET(libhdfs3_VERSION_STRING
"${libhdfs3_VERSION_MAJOR}.${libhdfs3_VERSION_MINOR}.${libhdfs3_VERSION_PATCH}")
SET(libhdfs3_VERSION_API 1)
SET(libhdfs3_ROOT_SOURCES_DIR ${CMAKE_SOURCE_DIR}/src)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] caiconghui merged pull request #9: [github] add branch protection
caiconghui merged PR #9: URL: https://github.com/apache/doris-thirdparty/pull/9 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] branch main updated: [github] add branch protection (#9)
This is an automated email from the ASF dual-hosted git repository. caiconghui pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git The following commit(s) were added to refs/heads/main by this push: new 20fe30b [github] add branch protection (#9) 20fe30b is described below commit 20fe30bdbfe92c6ee1563ff1e4ee51fb8804da83 Author: Mingyu Chen AuthorDate: Tue Aug 9 22:17:26 2022 +0800 [github] add branch protection (#9) protect following branches: 1.main 2.bdbje 3.libhdfs3 --- .asf.yaml | 13 + README.md | 8 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/.asf.yaml b/.asf.yaml index 7aee5a7..a88105c 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -42,5 +42,18 @@ github: squash: true merge: false rebase: false + protected_branches: +main: + required_pull_request_reviews: +dismiss_stale_reviews: true +required_approving_review_count: 1 +bdbje: + required_pull_request_reviews: +dismiss_stale_reviews: true +required_approving_review_count: 1 +libhdfs3: + required_pull_request_reviews: +dismiss_stale_reviews: true +required_approving_review_count: 1 notifications: pullrequests_status: commits@doris.apache.org diff --git a/README.md b/README.md index 930c291..4d9410d 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ This repository is used to manage third-party libraries used in Apache Doris. So # Current Libs -| Lib Name | Branch | Description | Base version | Source URL | Latest Tag | -| | | | | | -- | -| libhdfs3 | libhdfs3 | designed as an alternative implementation of libhdfs, is implemented based on native Hadoop RPC protocol and HDFS data transfer protocol. It gets rid of the drawbacks of JNI, and it has a lightweight, small memory footprint code base. In addition, it is easy to use and deploy. | Master | [HAWQ_depends](https://github.com/apache/hawq/tree/master/depends/libhdfs3) | libhdfs3-v2.3.2 | -| bdbje| bdbje| Berkley Database Java Edition - build and runtime support. | 18.3.12 | [bdbje Maven src](https://search.maven.org/artifact/com.sleepycat/je/18.3.12/jar) | bdbje-18.3.13-doris-snapshot| +| Lib Name | Branch | Description | Base version | Source URL | Latest Tag | CHANGELOG| +| | | | | | -- | --- | +| libhdfs3 | libhdfs3 | designed as an alternative implementation of libhdfs, is implemented based on native Hadoop RPC protocol and HDFS data transfer protocol. It gets rid of the drawbacks of JNI, and it has a lightweight, small memory footprint code base. In addition, it is easy to use and deploy. | Master | [HAWQ_depends](https://github.com/apache/hawq/tree/master/depends/libhdfs3) | libhdfs3-v2.3.3 | [CHANGELOG](https://github.com/apache/doris-thirdparty/blob/libhdfs3/CHAN [...] +| bdbje| bdbje| Berkley Database Java Edition - build and runtime support. | 18.3.12 | [bdbje Maven src](https://search.maven.org/artifact/com.sleepycat/je/18.3.12/jar) | bdbje-18.3.13-doris-snapshot| [CHANGELOG](https://github.com/apache/doris-thirdparty/blob/bdbje/CHANGELOG.md) | - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] caiconghui merged pull request #10: [doc] add changelog
caiconghui merged PR #10: URL: https://github.com/apache/doris-thirdparty/pull/10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] branch bdbje updated: [doc] add changelog (#10)
This is an automated email from the ASF dual-hosted git repository. caiconghui pushed a commit to branch bdbje in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git The following commit(s) were added to refs/heads/bdbje by this push: new 5cd4d31 [doc] add changelog (#10) 5cd4d31 is described below commit 5cd4d31d0a48d070656f9537411e151ee4311969 Author: Mingyu Chen AuthorDate: Tue Aug 9 22:18:33 2022 +0800 [doc] add changelog (#10) --- CHANGELOG.md | 6 ++ pom.xml | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 000..9edb95b --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,6 @@ +# CHANGELOG + +## bdbje-18.3.13-doris-SNAPSHOT + +1. fix https://github.com/apache/doris/issues/10436 +2. support build with jdk8 and jdk11 diff --git a/pom.xml b/pom.xml index 5cc0e63..42ae591 100644 --- a/pom.xml +++ b/pom.xml @@ -10,7 +10,7 @@ org.apache.doris je -18.3.13-doris-SNAPSHOT +18.3.14-doris-SNAPSHOT bdb-je apache doris release https://doris.apache.org/ fork from bdb-je 18.3.12 from maven with starrocks bdbje patches - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #11630: [feature](nereids): SimplifyCastRule
jackwener opened a new pull request, #11630: URL: https://github.com/apache/doris/pull/11630 # Proposed changes Issue Number: close #xxx ## Problem summary - Remove redundant cast like - cast(1 as int) -> 1. - Merge cast like - cast(cast(1 as bigint) as string) -> cast(1 as string). ## Checklist(Required) 1. Does it affect the original behavior: - [x] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [x] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] annotated tag libhdfs3-v2.3.3 updated (e8a64b8 -> 2985e11)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to annotated tag libhdfs3-v2.3.3 in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git *** WARNING: tag libhdfs3-v2.3.3 was modified! *** from e8a64b8 (commit) to 2985e11 (tag) tagging e8a64b8eca25f02e6403d1eb316318439ac4b94d (commit) replaces libhdfs3-v2.3.2 by morningman on Tue Aug 9 22:20:54 2022 +0800 - Log - v2.3.3 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #11615: [chore](build) add apache snapshot maven repo to repositories
morningman commented on PR #11615: URL: https://github.com/apache/doris/pull/11615#issuecomment-1209472068 HI @siriume I think this is done by #11549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #11631: [fix](like-predicate) Add missing functions in LikeColumnPredicate
mrhhsg opened a new pull request, #11631: URL: https://github.com/apache/doris/pull/11631 # Proposed changes Issue Number: close #xxx ## Problem summary The `LikeColumnPredicater` in #10355 was incomplete. Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #11632: [chore](regression-test) Add drop table in aggregate_count1
mrhhsg opened a new pull request, #11632: URL: https://github.com/apache/doris/pull/11632 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11595: [fix](regexpr)regexpr functions' contexts should be THREAD_LOCAL
yiguolei merged PR #11595: URL: https://github.com/apache/doris/pull/11595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](regexpr)regexpr functions' contexts should be THREAD_LOCAL (#11595)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 601f28dd90 [fix](regexpr)regexpr functions' contexts should be
THREAD_LOCAL (#11595)
601f28dd90 is described below
commit 601f28dd9064554dae9cda0b134bd4be8fb57a8e
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Wed Aug 10 06:58:24 2022 +0800
[fix](regexpr)regexpr functions' contexts should be THREAD_LOCAL (#11595)
---
be/src/vec/functions/function_regexp.cpp | 48 ++--
1 file changed, 27 insertions(+), 21 deletions(-)
diff --git a/be/src/vec/functions/function_regexp.cpp
b/be/src/vec/functions/function_regexp.cpp
index 570c7f3399..c99bb84d31 100644
--- a/be/src/vec/functions/function_regexp.cpp
+++ b/be/src/vec/functions/function_regexp.cpp
@@ -49,24 +49,24 @@ public:
}
Status prepare(FunctionContext* context,
FunctionContext::FunctionStateScope scope) override {
-if (scope != FunctionContext::FRAGMENT_LOCAL) {
-return Status::OK();
-}
-
-if (context->is_col_constant(1)) {
-const auto pattern_col = context->get_constant_col(1)->column_ptr;
-const auto& pattern = pattern_col->get_data_at(0).to_string_val();
-if (pattern.is_null) {
-return Status::OK();
-}
+if (scope == FunctionContext::THREAD_LOCAL) {
+if (context->is_col_constant(1)) {
+DCHECK(!context->get_function_state(scope));
+const auto pattern_col =
context->get_constant_col(1)->column_ptr;
+const auto& pattern =
pattern_col->get_data_at(0).to_string_val();
+if (pattern.is_null) {
+return Status::OK();
+}
-std::string error_str;
-re2::RE2* re = StringFunctions::compile_regex(pattern, &error_str,
StringVal::null());
-if (re == nullptr) {
-context->set_error(error_str.c_str());
-return Status::InvalidArgument(error_str);
+std::string error_str;
+re2::RE2* re =
+StringFunctions::compile_regex(pattern, &error_str,
StringVal::null());
+if (re == nullptr) {
+context->set_error(error_str.c_str());
+return Status::InvalidArgument(error_str);
+}
+context->set_function_state(scope, re);
}
-context->set_function_state(scope, re);
}
return Status::OK();
}
@@ -101,9 +101,13 @@ public:
}
Status close(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
-if (scope == FunctionContext::FRAGMENT_LOCAL) {
-re2::RE2* re =
reinterpret_cast(context->get_function_state(scope));
-delete re;
+if (scope == FunctionContext::THREAD_LOCAL) {
+if (context->is_col_constant(1)) {
+re2::RE2* re =
reinterpret_cast(context->get_function_state(scope));
+DCHECK(re);
+delete re;
+context->set_function_state(scope, nullptr);
+}
}
return Status::OK();
}
@@ -124,8 +128,9 @@ struct RegexpReplaceImpl {
StringOP::push_null_string(i, result_data, result_offset,
null_map);
continue;
}
+
re2::RE2* re = reinterpret_cast(
-
context->get_function_state(FunctionContext::FRAGMENT_LOCAL));
+
context->get_function_state(FunctionContext::THREAD_LOCAL));
std::unique_ptr scoped_re; // destroys re if state->re
is nullptr
if (re == nullptr) {
std::string error_str;
@@ -171,8 +176,9 @@ struct RegexpExtractImpl {
StringOP::push_empty_string(i, result_data, result_offset);
continue;
}
+
re2::RE2* re = reinterpret_cast(
-
context->get_function_state(FunctionContext::FRAGMENT_LOCAL));
+
context->get_function_state(FunctionContext::THREAD_LOCAL));
std::unique_ptr scoped_re;
if (re == nullptr) {
std::string error_str;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] closed pull request #7951: add performance test framework for ci/cd
github-actions[bot] closed pull request #7951: add performance test framework for ci/cd URL: https://github.com/apache/doris/pull/7951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11628: [feature-wip](unique-key-merge-on-write) fix rowid conversion ut that may create a directory under an incorrect path
yiguolei merged PR #11628: URL: https://github.com/apache/doris/pull/11628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](unique-key-merge-on-write) fix rowid conversion ut that may create a directory under an incorrect path (#11628)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new aaaf6915e4 [feature-wip](unique-key-merge-on-write) fix rowid conversion ut that may create a directory under an incorrect path (#11628) aaaf6915e4 is described below commit aaaf6915e4644d1c248c3075bf7b72c3b21ce919 Author: Xin Liao AuthorDate: Wed Aug 10 08:17:47 2022 +0800 [feature-wip](unique-key-merge-on-write) fix rowid conversion ut that may create a directory under an incorrect path (#11628) --- be/test/olap/rowid_conversion_test.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/be/test/olap/rowid_conversion_test.cpp b/be/test/olap/rowid_conversion_test.cpp index 72c12d4899..1bae4f83e9 100644 --- a/be/test/olap/rowid_conversion_test.cpp +++ b/be/test/olap/rowid_conversion_test.cpp @@ -400,7 +400,7 @@ protected: } private: -const std::string kTestDir = "ut_dir/rowid_conversion_test"; +const std::string kTestDir = "/ut_dir/rowid_conversion_test"; string absolute_dir; std::unique_ptr _data_dir; }; - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #11614: [Bug] data skew should be 0.00% when partition is empty
yiguolei closed issue #11614: [Bug] data skew should be 0.00% when partition is empty URL: https://github.com/apache/doris/issues/11614 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug](show data skew)fix show data skew logic (#11616)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ae90d45594 [Bug](show data skew)fix show data skew logic (#11616)
ae90d45594 is described below
commit ae90d45594d6abe32b84dc75ab13ad6db0ca9538
Author: Henry2SS <45096548+henry...@users.noreply.github.com>
AuthorDate: Wed Aug 10 08:18:39 2022 +0800
[Bug](show data skew)fix show data skew logic (#11616)
Co-authored-by: wuhangze
---
.../src/main/java/org/apache/doris/catalog/MetadataViewer.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java
index 18172fd29d..a4182fd93e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java
@@ -245,7 +245,7 @@ public class MetadataViewer {
private static String graph(long num, long totalNum) {
StringBuilder sb = new StringBuilder();
-long normalized = num == totalNum ? 100 : (int) Math.ceil(num * 100 /
totalNum);
+long normalized = num == totalNum ? (totalNum == 0L ? 0 : 100) : (int)
Math.ceil(num * 100 / totalNum);
for (int i = 0; i < normalized; ++i) {
sb.append(">");
}
@@ -309,8 +309,8 @@ public class MetadataViewer {
row.add(rowCountTabletInfos.get(i).toString());
row.add(dataSizeTabletInfos.get(i).toString());
row.add(graph(dataSizeTabletInfos.get(i), totalSize));
-row.add(totalSize == dataSizeTabletInfos.get(i)
-? "100.00%" : df.format((double)
dataSizeTabletInfos.get(i) / totalSize));
+row.add(totalSize == dataSizeTabletInfos.get(i) ? (totalSize
== 0L ? "0.00%" : "100.00%") :
+df.format((double) dataSizeTabletInfos.get(i) /
totalSize));
result.add(row);
}
} finally {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11616: [Bug](show data skew)fix show data skew logic
yiguolei merged PR #11616: URL: https://github.com/apache/doris/pull/11616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11612: [fix](ui)source map files not included in production builds
yiguolei merged PR #11612: URL: https://github.com/apache/doris/pull/11612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](ui)source map files not included in production builds (#11612)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b3d476eebb [fix](ui)source map files not included in production builds
(#11612)
b3d476eebb is described below
commit b3d476eebb6014709eb8a27901f07d280849e38b
Author: wangyongfeng <943155...@qq.com>
AuthorDate: Wed Aug 10 08:19:07 2022 +0800
[fix](ui)source map files not included in production builds (#11612)
Co-authored-by: wangyf0555
---
ui/config/webpack.common.js | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ui/config/webpack.common.js b/ui/config/webpack.common.js
index f5a4b713df..3e5eea5619 100644
--- a/ui/config/webpack.common.js
+++ b/ui/config/webpack.common.js
@@ -49,7 +49,7 @@ module.exports = {
filename: '[name].[hash].js',
chunkFilename: '[name].[hash].js'
},
-devtool: 'source-map',
+devtool: 'none',
resolve: {
extensions: ['*', '.ts', '.tsx', 'jsx', '.js', 'json'],
alias: {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11598: [fix](scanner) delete predicates might be inconsistent with rowset readers
yiguolei commented on code in PR #11598: URL: https://github.com/apache/doris/pull/11598#discussion_r941909423 ## be/src/olap/merger.cpp: ## @@ -41,6 +41,10 @@ Status Merger::merge_rowsets(TabletSharedPtr tablet, ReaderType reader_type, reader_params.reader_type = reader_type; reader_params.rs_readers = src_rowset_readers; reader_params.version = dst_rowset_writer->version(); +reader_params.delete_predicates = tablet->delete_predicates(); Review Comment: line 44 set reader_params.delete_predicates = tablet->delete_predicates(); and then add it again? Are there duplicate predicates? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11598: [fix](scanner) delete predicates might be inconsistent with rowset readers
yiguolei commented on code in PR #11598: URL: https://github.com/apache/doris/pull/11598#discussion_r941909738 ## be/src/olap/merger.cpp: ## @@ -41,6 +41,10 @@ Status Merger::merge_rowsets(TabletSharedPtr tablet, ReaderType reader_type, reader_params.reader_type = reader_type; reader_params.rs_readers = src_rowset_readers; reader_params.version = dst_rowset_writer->version(); +reader_params.delete_predicates = tablet->delete_predicates(); +std::copy(tablet->delete_predicates().cbegin(), tablet->delete_predicates().cend(), + std::inserter(reader_params.delete_predicates, +reader_params.delete_predicates.begin())); Review Comment: Why not acquire tablet lock during init reader_params's delete predicate property? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11598: [fix](scanner) delete predicates might be inconsistent with rowset readers
yiguolei commented on code in PR #11598:
URL: https://github.com/apache/doris/pull/11598#discussion_r941912108
##
be/src/olap/reader.cpp:
##
@@ -613,15 +613,14 @@ Status TabletReader::_init_delete_condition(const
ReaderParams& read_params) {
}
auto delete_init = [&]() -> Status {
-return _delete_handler.init(_tablet_schema,
_tablet->delete_predicates(),
+return _delete_handler.init(_tablet_schema,
read_params.delete_predicates,
read_params.version.second, this);
};
if (read_params.reader_type == READER_ALTER_TABLE) {
return delete_init();
}
Review Comment:
If not need acquire tablet lock here, then I think we do not need
distinguish alter table and others. 620-623 could be deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary opened a new issue, #11633: [Feature](multi-catalog) Support refresh catalog metadata
stalary opened a new issue, #11633:
URL: https://github.com/apache/doris/issues/11633
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Description
Support refresh catalog metadata. Maunal refresh is first support
Syntax is as follows
1. Refresh the specified catalog manual.
```
refresh catalog catalog_name
```
2. Periodically refresh the specified catalog, interval timeunit is s.
```
create catalog catalog_name properties("auto_refresh.enable"="true",
"auto_refresh.interval"="60")
```
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] compasses commented on a diff in pull request #11631: [fix](like-predicate) Add missing functions in LikeColumnPredicate
compasses commented on code in PR #11631:
URL: https://github.com/apache/doris/pull/11631#discussion_r941925698
##
be/src/olap/like_column_predicate.h:
##
@@ -45,33 +47,115 @@ class LikeColumnPredicate : public ColumnPredicate {
return Status::OK();
}
+uint16_t evaluate(const vectorized::IColumn& column, uint16_t* sel,
+ uint16_t size) const override;
+
+void evaluate_and_vec(const vectorized::IColumn& column, uint16_t size,
+ bool* flags) const override;
+
private:
template
void _base_evaluate(const ColumnBlock* block, uint16_t* sel, uint16_t*
size) const {
uint16_t new_size = 0;
-for (uint16_t i = 0; i < *size; ++i) {
-uint16_t idx = sel[i];
-sel[new_size] = idx;
-const StringValue* cell_value =
-reinterpret_cast(block->cell(idx).cell_ptr());
-doris_udf::StringVal target;
-cell_value->to_string_val(&target);
-if constexpr (is_nullable) {
-new_size += _opposite ^ (!block->cell(idx).is_null() &&
- (_state->function)(_fn_ctx, target,
pattern).val);
-} else {
-new_size += _opposite ^ (_state->function)(_fn_ctx, target,
pattern).val;
+if constexpr (!is_vectorized) {
+for (uint16_t i = 0; i < *size; ++i) {
+uint16_t idx = sel[i];
+sel[new_size] = idx;
+const StringValue* cell_value =
+reinterpret_cast(block->cell(idx).cell_ptr());
+doris_udf::StringVal target;
+cell_value->to_string_val(&target);
+if constexpr (is_nullable) {
+new_size += _opposite ^ (!block->cell(idx).is_null() &&
+ (_state->function)(_fn_ctx,
target, pattern).val);
+} else {
+new_size += _opposite ^ (_state->function)(_fn_ctx,
target, pattern).val;
+}
}
}
*size = new_size;
}
+template
+void _evaluate_vec(const vectorized::IColumn& column, uint16_t size, bool*
flags) const {
+if constexpr (is_vectorized) {
+if (column.is_nullable()) {
+auto* nullable_col =
+
vectorized::check_and_get_column(column);
+auto& null_map_data =
nullable_col->get_null_map_column().get_data();
+auto& nested_col = nullable_col->get_nested_column();
+if (nested_col.is_column_dictionary()) {
+auto* nested_col_ptr = vectorized::check_and_get_column<
+
vectorized::ColumnDictionary>(nested_col);
+auto& data_array = nested_col_ptr->get_data();
+for (uint16_t i = 0; i < size; i++) {
+if (null_map_data[i]) {
+if constexpr (is_and) {
+flags[i] &= _opposite;
+} else {
+flags[i] = _opposite;
+}
+continue;
+}
+
+StringValue cell_value =
nested_col_ptr->get_value(data_array[i]);
+if constexpr (is_and) {
+unsigned char flag = 0;
+(_state->function)(
+
const_cast(&_like_state),
+cell_value, pattern, &flag);
+flags[i] &= _opposite ^ flag;
+} else {
+unsigned char flag = 0;
+(_state->function)(
+
const_cast(&_like_state),
+cell_value, pattern, &flag);
+flags[i] = _opposite ^ flag;
+}
+}
+} else {
+LOG(FATAL) << "vectorized (not) like predicates should be
dict column";
+}
+} else {
+if (column.is_column_dictionary()) {
+auto* nested_col_ptr = vectorized::check_and_get_column<
+
vectorized::ColumnDictionary>(column);
+auto& data_array = nested_col_ptr->get_data();
+for (uint16_t i = 0; i < size; i++) {
+StringValue cell_value =
nested_col_ptr->get_value(data_array[i]);
+if constexpr (is_and) {
+unsigned char flag = 0;
+(_state->function)(
+
const_cast(&_like_state),
+cell_value, pattern, &flag);
+
[GitHub] [doris] compasses commented on pull request #11631: [fix](like-predicate) Add missing functions in LikeColumnPredicate
compasses commented on PR #11631: URL: https://github.com/apache/doris/pull/11631#issuecomment-1210047061 Can you help add some regression test case for the like query? Need open the configuration ```enable_function_pushdown```, which default value is ```false```. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #11598: [fix](scanner) delete predicates might be inconsistent with rowset readers
zhannngchen commented on code in PR #11598: URL: https://github.com/apache/doris/pull/11598#discussion_r941926863 ## be/src/olap/merger.cpp: ## @@ -41,6 +41,10 @@ Status Merger::merge_rowsets(TabletSharedPtr tablet, ReaderType reader_type, reader_params.reader_type = reader_type; reader_params.rs_readers = src_rowset_readers; reader_params.version = dst_rowset_writer->version(); +reader_params.delete_predicates = tablet->delete_predicates(); Review Comment: forgot to delete this line, thanks for remind. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [minor](log) add a warn log to observer invalid query profile (#11588)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new a1483d0575 [minor](log) add a warn log to observer invalid query
profile (#11588)
a1483d0575 is described below
commit a1483d0575ec3e93c57b5ce59071ff7218eef594
Author: Mingyu Chen
AuthorDate: Tue Aug 9 14:10:03 2022 +0800
[minor](log) add a warn log to observer invalid query profile (#11588)
I try to fix the bug in #10095. the error occurred when I first create a
empty table and query it.
But I can't reproduced it again.
So I add a warn log here to observer
---
.../java/org/apache/doris/common/profile/ProfileTreeBuilder.java | 9 +
1 file changed, 9 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeBuilder.java
index d810e02fd1..da621b3d6a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeBuilder.java
@@ -31,6 +31,8 @@ import com.google.common.collect.Maps;
import org.apache.commons.lang3.tuple.ImmutableTriple;
import org.apache.commons.lang3.tuple.Triple;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
import java.util.Formatter;
import java.util.List;
@@ -48,6 +50,7 @@ import java.util.regex.Pattern;
* Each runtime profile of a query should be built once and be read every
where.
*/
public class ProfileTreeBuilder {
+private static final Logger LOG =
LogManager.getLogger(ProfileTreeBuilder.class);
private static final String PROFILE_NAME_DATA_STREAM_SENDER =
"DataStreamSender";
private static final String PROFILE_NAME_VDATA_STREAM_SENDER =
"VDataStreamSender";
@@ -227,6 +230,12 @@ public class ProfileTreeBuilder {
}
}
if (senderNode == null || execNode == null) {
+// TODO(cmy): This shouldn't happen, but there are sporadic
errors. So I add a log to observe this error.
+// Issue: https://github.com/apache/doris/issues/10095
+StringBuilder sb = new StringBuilder();
+instanceProfile.prettyPrint(sb, "");
+LOG.warn("Invalid instance profile, sender is null: {}, execNode
is null: {}, instance profile: {}",
+(senderNode == null), (execNode == null), sb.toString());
throw new UserException("Invalid instance profile, without sender
or exec node: " + instanceProfile);
}
senderNode.addChild(execNode);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #11598: [fix](scanner) delete predicates might be inconsistent with rowset readers
zhannngchen commented on code in PR #11598: URL: https://github.com/apache/doris/pull/11598#discussion_r941929060 ## be/src/olap/merger.cpp: ## @@ -41,6 +41,10 @@ Status Merger::merge_rowsets(TabletSharedPtr tablet, ReaderType reader_type, reader_params.reader_type = reader_type; reader_params.rs_readers = src_rowset_readers; reader_params.version = dst_rowset_writer->version(); +reader_params.delete_predicates = tablet->delete_predicates(); +std::copy(tablet->delete_predicates().cbegin(), tablet->delete_predicates().cend(), + std::inserter(reader_params.delete_predicates, +reader_params.delete_predicates.begin())); Review Comment: 1. compaction don't need to care new added delete predicates, it's useless for this compaction. 2. only compaction would remove delete predicates, so it also don't need to worry other thread to remove the delete predicates related with input rowset. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #11598: [fix](scanner) delete predicates might be inconsistent with rowset readers
zhannngchen commented on code in PR #11598: URL: https://github.com/apache/doris/pull/11598#discussion_r941929060 ## be/src/olap/merger.cpp: ## @@ -41,6 +41,10 @@ Status Merger::merge_rowsets(TabletSharedPtr tablet, ReaderType reader_type, reader_params.reader_type = reader_type; reader_params.rs_readers = src_rowset_readers; reader_params.version = dst_rowset_writer->version(); +reader_params.delete_predicates = tablet->delete_predicates(); +std::copy(tablet->delete_predicates().cbegin(), tablet->delete_predicates().cend(), + std::inserter(reader_params.delete_predicates, +reader_params.delete_predicates.begin())); Review Comment: 1. compaction don't need to care new added delete predicates, it's useless for this compaction. 2. only compaction would remove delete predicates, so it also don't need to worry other thread to concurrently remove the delete predicates related with input rowset. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg commented on a diff in pull request #11631: [fix](like-predicate) Add missing functions in LikeColumnPredicate
mrhhsg commented on code in PR #11631:
URL: https://github.com/apache/doris/pull/11631#discussion_r941930620
##
be/src/olap/like_column_predicate.cpp:
##
@@ -23,74 +23,131 @@
namespace doris {
-LikeColumnPredicate::LikeColumnPredicate(bool opposite, uint32_t column_id,
- doris_udf::FunctionContext* fn_ctx,
- doris_udf::StringVal val)
+template <>
+LikeColumnPredicate::LikeColumnPredicate(bool opposite, uint32_t
column_id,
+ doris_udf::FunctionContext*
fn_ctx,
+ doris_udf::StringVal val)
+: ColumnPredicate(column_id, opposite),
+ _fn_ctx(fn_ctx),
+ pattern(reinterpret_cast(val.ptr), val.len) {
+_state = reinterpret_cast(
+
_fn_ctx->get_function_state(doris_udf::FunctionContext::THREAD_LOCAL));
+_state->search_state.clone(_like_state);
Review Comment:
Yes, every concurrent caller of the Hyperscan API should use a separate
scratch. Here each LikeColumnPredicate only clones it once, so I think the
overload is acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #11598: [fix](scanner) delete predicates might be inconsistent with rowset readers
zhannngchen commented on code in PR #11598:
URL: https://github.com/apache/doris/pull/11598#discussion_r941930649
##
be/src/olap/reader.cpp:
##
@@ -613,15 +613,14 @@ Status TabletReader::_init_delete_condition(const
ReaderParams& read_params) {
}
auto delete_init = [&]() -> Status {
-return _delete_handler.init(_tablet_schema,
_tablet->delete_predicates(),
+return _delete_handler.init(_tablet_schema,
read_params.delete_predicates,
read_params.version.second, this);
};
if (read_params.reader_type == READER_ALTER_TABLE) {
return delete_init();
}
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #11598: [fix](scanner) delete predicates might be inconsistent with rowset readers
zhannngchen commented on code in PR #11598: URL: https://github.com/apache/doris/pull/11598#discussion_r941931065 ## be/src/olap/merger.cpp: ## @@ -41,6 +41,10 @@ Status Merger::merge_rowsets(TabletSharedPtr tablet, ReaderType reader_type, reader_params.reader_type = reader_type; reader_params.rs_readers = src_rowset_readers; reader_params.version = dst_rowset_writer->version(); +reader_params.delete_predicates = tablet->delete_predicates(); +std::copy(tablet->delete_predicates().cbegin(), tablet->delete_predicates().cend(), + std::inserter(reader_params.delete_predicates, +reader_params.delete_predicates.begin())); Review Comment: I added the lock finally, it's more easier to understand and cost very low. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg commented on pull request #11631: [fix](like-predicate) Add missing functions in LikeColumnPredicate
mrhhsg commented on PR #11631: URL: https://github.com/apache/doris/pull/11631#issuecomment-1210053456 > Can you help add some regression test case for the like query? Need open the configuration `enable_function_pushdown`, which default value is `false`. @compasses I think using one session variable is better than a configuration item. I will create another PR to do this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg commented on a diff in pull request #11631: [fix](like-predicate) Add missing functions in LikeColumnPredicate
mrhhsg commented on code in PR #11631:
URL: https://github.com/apache/doris/pull/11631#discussion_r941933100
##
be/src/olap/like_column_predicate.h:
##
@@ -45,33 +47,115 @@ class LikeColumnPredicate : public ColumnPredicate {
return Status::OK();
}
+uint16_t evaluate(const vectorized::IColumn& column, uint16_t* sel,
+ uint16_t size) const override;
+
+void evaluate_and_vec(const vectorized::IColumn& column, uint16_t size,
+ bool* flags) const override;
+
private:
template
void _base_evaluate(const ColumnBlock* block, uint16_t* sel, uint16_t*
size) const {
uint16_t new_size = 0;
-for (uint16_t i = 0; i < *size; ++i) {
-uint16_t idx = sel[i];
-sel[new_size] = idx;
-const StringValue* cell_value =
-reinterpret_cast(block->cell(idx).cell_ptr());
-doris_udf::StringVal target;
-cell_value->to_string_val(&target);
-if constexpr (is_nullable) {
-new_size += _opposite ^ (!block->cell(idx).is_null() &&
- (_state->function)(_fn_ctx, target,
pattern).val);
-} else {
-new_size += _opposite ^ (_state->function)(_fn_ctx, target,
pattern).val;
+if constexpr (!is_vectorized) {
+for (uint16_t i = 0; i < *size; ++i) {
+uint16_t idx = sel[i];
+sel[new_size] = idx;
+const StringValue* cell_value =
+reinterpret_cast(block->cell(idx).cell_ptr());
+doris_udf::StringVal target;
+cell_value->to_string_val(&target);
+if constexpr (is_nullable) {
+new_size += _opposite ^ (!block->cell(idx).is_null() &&
+ (_state->function)(_fn_ctx,
target, pattern).val);
+} else {
+new_size += _opposite ^ (_state->function)(_fn_ctx,
target, pattern).val;
+}
}
}
*size = new_size;
}
+template
+void _evaluate_vec(const vectorized::IColumn& column, uint16_t size, bool*
flags) const {
+if constexpr (is_vectorized) {
+if (column.is_nullable()) {
+auto* nullable_col =
+
vectorized::check_and_get_column(column);
+auto& null_map_data =
nullable_col->get_null_map_column().get_data();
+auto& nested_col = nullable_col->get_nested_column();
+if (nested_col.is_column_dictionary()) {
+auto* nested_col_ptr = vectorized::check_and_get_column<
+
vectorized::ColumnDictionary>(nested_col);
+auto& data_array = nested_col_ptr->get_data();
+for (uint16_t i = 0; i < size; i++) {
+if (null_map_data[i]) {
+if constexpr (is_and) {
+flags[i] &= _opposite;
+} else {
+flags[i] = _opposite;
+}
+continue;
+}
+
+StringValue cell_value =
nested_col_ptr->get_value(data_array[i]);
+if constexpr (is_and) {
+unsigned char flag = 0;
+(_state->function)(
+
const_cast(&_like_state),
+cell_value, pattern, &flag);
+flags[i] &= _opposite ^ flag;
+} else {
+unsigned char flag = 0;
+(_state->function)(
+
const_cast(&_like_state),
+cell_value, pattern, &flag);
+flags[i] = _opposite ^ flag;
+}
+}
+} else {
+LOG(FATAL) << "vectorized (not) like predicates should be
dict column";
+}
+} else {
+if (column.is_column_dictionary()) {
+auto* nested_col_ptr = vectorized::check_and_get_column<
+
vectorized::ColumnDictionary>(column);
+auto& data_array = nested_col_ptr->get_data();
+for (uint16_t i = 0; i < size; i++) {
+StringValue cell_value =
nested_col_ptr->get_value(data_array[i]);
+if constexpr (is_and) {
+unsigned char flag = 0;
+(_state->function)(
+
const_cast(&_like_state),
+cell_value, pattern, &flag);
+
[GitHub] [doris] qinchaofeng commented on issue #11591: [Bug] I want to create a new table from the query results of a table
qinchaofeng commented on issue #11591: URL: https://github.com/apache/doris/issues/11591#issuecomment-1210072218 @longzmkm just like [stalary](https://github.com/stalary) said : ``` create table information_schema.first_model as select * from ( with source_data as ( select * from information_schema.tables ) select * from source_data) as aa; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg merged pull request #11188: [doc][fix] Fix the duplicate partition name in example of CREATE TABLE
yangzhg merged PR #11188: URL: https://github.com/apache/doris/pull/11188 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [doc][fix] fix the duplicate partition name in example of CREATE TABLE (#11188)
This is an automated email from the ASF dual-hosted git repository.
yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a478e1d669 [doc][fix] fix the duplicate partition name in example of
CREATE TABLE (#11188)
a478e1d669 is described below
commit a478e1d6692f19ef642e9c0b214b8b9f529c2e37
Author: zxealous
AuthorDate: Wed Aug 10 10:26:12 2022 +0800
[doc][fix] fix the duplicate partition name in example of CREATE TABLE
(#11188)
---
.../sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md | 4 ++--
.../sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git
a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
index 0c38e29af9..0e99f10128 100644
---
a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
+++
b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
@@ -369,8 +369,8 @@ distribution_info
PARTITION BY RANGE(k1)
(
PARTITION p1 VALUES LESS THAN ("2020-02-01"),
-PARTITION p1 VALUES LESS THAN ("2020-03-01"),
-PARTITION p1 VALUES LESS THAN ("2020-04-01")
+PARTITION p2 VALUES LESS THAN ("2020-03-01"),
+PARTITION p3 VALUES LESS THAN ("2020-04-01")
)
DISTRIBUTED BY HASH(k1) BUCKETS 32
PROPERTIES (
diff --git
a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
index 34df1e14a3..362bedbf2c 100644
---
a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
+++
b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
@@ -370,8 +370,8 @@ distribution_info
PARTITION BY RANGE(k1)
(
PARTITION p1 VALUES LESS THAN ("2020-02-01"),
-PARTITION p1 VALUES LESS THAN ("2020-03-01"),
-PARTITION p1 VALUES LESS THAN ("2020-04-01")
+PARTITION p2 VALUES LESS THAN ("2020-03-01"),
+PARTITION p3 VALUES LESS THAN ("2020-04-01")
)
DISTRIBUTED BY HASH(k1) BUCKETS 32
PROPERTIES (
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #11535: [improvement](olapscan) Simplify data block queue of VOlapScanNode
carlvinhust2012 commented on code in PR #11535:
URL: https://github.com/apache/doris/pull/11535#discussion_r941952794
##
be/src/vec/exec/volap_scan_node.cpp:
##
@@ -204,22 +201,6 @@ Status VOlapScanNode::prepare(RuntimeState* state) {
return Status::InternalError("Failed to get tuple descriptor.");
}
-const std::vector& slots = _tuple_desc->slots();
Review Comment:
why this code is not needed ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 closed pull request #31: [doc](bloomfilter)fix-doc
hf200012 closed pull request #31: [doc](bloomfilter)fix-doc URL: https://github.com/apache/doris-website/pull/31 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11604: [feature](Nereids): polish enforcer
morrySnow commented on code in PR #11604:
URL: https://github.com/apache/doris/pull/11604#discussion_r941964427
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/PhysicalProperties.java:
##
@@ -17,35 +17,47 @@
package org.apache.doris.nereids.properties;
+import java.util.Objects;
+
/**
* Physical properties used in cascades.
- * TODO(wj): Do we need to `PhysicalPropertySpec` Interface like NoisePage?
*/
public class PhysicalProperties {
+
+public static PhysicalProperties ANY = new PhysicalProperties();
+
+public static PhysicalProperties REPLICATED = new
PhysicalProperties(DistributionSpecReplicated.INSTANCE);
+
+public static PhysicalProperties GATHER = new
PhysicalProperties(DistributionSpecGather.INSTANCE);
+
private final OrderSpec orderSpec;
private final DistributionSpec distributionSpec;
-public PhysicalProperties() {
+private PhysicalProperties() {
this.orderSpec = new OrderSpec();
-this.distributionSpec = DistributionSpecAny.getInstance();
+this.distributionSpec = DistributionSpecAny.INSTANCE;
}
-public PhysicalProperties(DistributionSpec distributionSpec) {
+private PhysicalProperties(DistributionSpec distributionSpec) {
this.distributionSpec = distributionSpec;
this.orderSpec = new OrderSpec();
}
public PhysicalProperties(OrderSpec orderSpec) {
this.orderSpec = orderSpec;
-this.distributionSpec = DistributionSpecAny.getInstance();
+this.distributionSpec = DistributionSpecAny.INSTANCE;
}
public PhysicalProperties(DistributionSpec distributionSpec, OrderSpec
orderSpec) {
this.distributionSpec = distributionSpec;
this.orderSpec = orderSpec;
}
+public static PhysicalProperties createHash(DistributionSpec
distributionSpec) {
Review Comment:
this function name is `createHash`, so i think the parameter should be
`DistributionSpecHash`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11630: [feature](nereids): SimplifyCastRule
morrySnow commented on code in PR #11630:
URL: https://github.com/apache/doris/pull/11630#discussion_r941967031
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/SimplifyCastRule.java:
##
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.StringLiteral;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.IntegralType;
+import org.apache.doris.nereids.types.NumericType;
+
+import java.util.Optional;
+
+
+/**
+ * Rewrite rule of simplify CAST expression.
+ * Remove redundant cast like
+ * - cast(1 as int) -> 1.
+ * Merge cast like
+ * - cast(cast(1 as bigint) as string) -> cast(1 as string).
+ */
+public class SimplifyCastRule extends AbstractExpressionRewriteRule {
+
+public static SimplifyCastRule INSTANCE = new SimplifyCastRule();
+
+@Override
+public Expression visitCast(Cast origin, ExpressionRewriteContext context)
{
+return simplify(origin)
+.map(simplifiedExpr -> {
+if (simplifiedExpr instanceof Cast) {
+return rewrite(simplifiedExpr, context);
+}
+return simplifiedExpr;
+})
+.orElse(origin);
+}
+
+private Optional simplify(Cast cast) {
+Expression source = cast.left();
+StringLiteral type = cast.right();
+
+// remove redundant cast
+// CAST(value as type), value is type
+if (cast.getDataType().equals(source.getDataType())) {
+return Optional.of(source);
+}
+// CAST(CAST()) -> CAST()
+if (source instanceof Cast) {
+Cast innerCast = (Cast) source;
+if (check(innerCast.getDataType(), cast.getDataType())
+&& check(innerCast.getDataType(), source.getDataType())) {
+return Optional.of(new Cast(innerCast.left(), type));
+}
+return simplify(innerCast).map(inner ->
+new Cast(inner, type)
+);
+}
+
+return Optional.empty();
+}
+
+// It after remove redundant cast. so left.class != right.class
+// We forbidden subclass of NumericType convert other subclass of
NumericType,
+// except subclass of IntegralType can mutual conversion.
+private boolean check(DataType left, DataType right) {
+// int, bigint ... can mutual conversion.
+if (left instanceof IntegralType && right instanceof IntegralType) {
+return true;
+}
+
+// We forbidden subclass of NumericType convert to other subclass of
NumericType,
+if (!(left instanceof NumericType) || (!(right instanceof
NumericType))) {
+return true;
+}
+
+// It after remove redundant cast. so left.class != right.class
+// So we don't need to return left.class == right.class
+return false;
Review Comment:
```suggestion
// We forbidden subclass of NumericType convert to other subclass of
NumericType,
if (left instanceof NumericType && right instanceof NumericType) {
return false;
}
// It after remove redundant cast. so left.class != right.class
// So we don't need to return left.class == right.class
return true;
```
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/SimplifyCastRule.java:
##
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Versi
[GitHub] [doris] compasses commented on a diff in pull request #11631: [fix](like-predicate) Add missing functions in LikeColumnPredicate
compasses commented on code in PR #11631:
URL: https://github.com/apache/doris/pull/11631#discussion_r941967819
##
be/src/olap/like_column_predicate.cpp:
##
@@ -23,74 +23,131 @@
namespace doris {
-LikeColumnPredicate::LikeColumnPredicate(bool opposite, uint32_t column_id,
- doris_udf::FunctionContext* fn_ctx,
- doris_udf::StringVal val)
+template <>
+LikeColumnPredicate::LikeColumnPredicate(bool opposite, uint32_t
column_id,
+ doris_udf::FunctionContext*
fn_ctx,
+ doris_udf::StringVal val)
+: ColumnPredicate(column_id, opposite),
+ _fn_ctx(fn_ctx),
+ pattern(reinterpret_cast(val.ptr), val.len) {
+_state = reinterpret_cast(
+
_fn_ctx->get_function_state(doris_udf::FunctionContext::THREAD_LOCAL));
+_state->search_state.clone(_like_state);
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] compasses commented on a diff in pull request #11631: [fix](like-predicate) Add missing functions in LikeColumnPredicate
compasses commented on code in PR #11631:
URL: https://github.com/apache/doris/pull/11631#discussion_r941967983
##
be/src/olap/like_column_predicate.h:
##
@@ -45,33 +47,115 @@ class LikeColumnPredicate : public ColumnPredicate {
return Status::OK();
}
+uint16_t evaluate(const vectorized::IColumn& column, uint16_t* sel,
+ uint16_t size) const override;
+
+void evaluate_and_vec(const vectorized::IColumn& column, uint16_t size,
+ bool* flags) const override;
+
private:
template
void _base_evaluate(const ColumnBlock* block, uint16_t* sel, uint16_t*
size) const {
uint16_t new_size = 0;
-for (uint16_t i = 0; i < *size; ++i) {
-uint16_t idx = sel[i];
-sel[new_size] = idx;
-const StringValue* cell_value =
-reinterpret_cast(block->cell(idx).cell_ptr());
-doris_udf::StringVal target;
-cell_value->to_string_val(&target);
-if constexpr (is_nullable) {
-new_size += _opposite ^ (!block->cell(idx).is_null() &&
- (_state->function)(_fn_ctx, target,
pattern).val);
-} else {
-new_size += _opposite ^ (_state->function)(_fn_ctx, target,
pattern).val;
+if constexpr (!is_vectorized) {
+for (uint16_t i = 0; i < *size; ++i) {
+uint16_t idx = sel[i];
+sel[new_size] = idx;
+const StringValue* cell_value =
+reinterpret_cast(block->cell(idx).cell_ptr());
+doris_udf::StringVal target;
+cell_value->to_string_val(&target);
+if constexpr (is_nullable) {
+new_size += _opposite ^ (!block->cell(idx).is_null() &&
+ (_state->function)(_fn_ctx,
target, pattern).val);
+} else {
+new_size += _opposite ^ (_state->function)(_fn_ctx,
target, pattern).val;
+}
}
}
*size = new_size;
}
+template
+void _evaluate_vec(const vectorized::IColumn& column, uint16_t size, bool*
flags) const {
+if constexpr (is_vectorized) {
+if (column.is_nullable()) {
+auto* nullable_col =
+
vectorized::check_and_get_column(column);
+auto& null_map_data =
nullable_col->get_null_map_column().get_data();
+auto& nested_col = nullable_col->get_nested_column();
+if (nested_col.is_column_dictionary()) {
+auto* nested_col_ptr = vectorized::check_and_get_column<
+
vectorized::ColumnDictionary>(nested_col);
+auto& data_array = nested_col_ptr->get_data();
+for (uint16_t i = 0; i < size; i++) {
+if (null_map_data[i]) {
+if constexpr (is_and) {
+flags[i] &= _opposite;
+} else {
+flags[i] = _opposite;
+}
+continue;
+}
+
+StringValue cell_value =
nested_col_ptr->get_value(data_array[i]);
+if constexpr (is_and) {
+unsigned char flag = 0;
+(_state->function)(
+
const_cast(&_like_state),
+cell_value, pattern, &flag);
+flags[i] &= _opposite ^ flag;
+} else {
+unsigned char flag = 0;
+(_state->function)(
+
const_cast(&_like_state),
+cell_value, pattern, &flag);
+flags[i] = _opposite ^ flag;
+}
+}
+} else {
+LOG(FATAL) << "vectorized (not) like predicates should be
dict column";
+}
+} else {
+if (column.is_column_dictionary()) {
+auto* nested_col_ptr = vectorized::check_and_get_column<
+
vectorized::ColumnDictionary>(column);
+auto& data_array = nested_col_ptr->get_data();
+for (uint16_t i = 0; i < size; i++) {
+StringValue cell_value =
nested_col_ptr->get_value(data_array[i]);
+if constexpr (is_and) {
+unsigned char flag = 0;
+(_state->function)(
+
const_cast(&_like_state),
+cell_value, pattern, &flag);
+
[GitHub] [doris] github-actions[bot] commented on pull request #11631: [fix](like-predicate) Add missing functions in LikeColumnPredicate
github-actions[bot] commented on PR #11631: URL: https://github.com/apache/doris/pull/11631#issuecomment-1210100922 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #34: [doc](add)add show query profile doc
hf200012 merged PR #34: URL: https://github.com/apache/doris-website/pull/34 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: add_doc (#34)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new ce726330302 add_doc (#34) ce726330302 is described below commit ce726330302bdf002820694fa65efba4cee1e533 Author: Liqf <109049295+lemonlit...@users.noreply.github.com> AuthorDate: Wed Aug 10 11:25:48 2022 +0800 add_doc (#34) add show query profile doc --- .../Show-Statements/SHOW-QUERY-PROFILE.md | 183 + .../Show-Statements/SHOW-QUERY-PROFILE.md | 181 2 files changed, 364 insertions(+) diff --git a/docs/sql-manual/sql-reference/Show-Statements/SHOW-QUERY-PROFILE.md b/docs/sql-manual/sql-reference/Show-Statements/SHOW-QUERY-PROFILE.md index d90a09a99ec..1a8a0595ec2 100644 --- a/docs/sql-manual/sql-reference/Show-Statements/SHOW-QUERY-PROFILE.md +++ b/docs/sql-manual/sql-reference/Show-Statements/SHOW-QUERY-PROFILE.md @@ -26,10 +26,193 @@ under the License. ## SHOW-QUERY-PROFILE +### Name + +SHOW QUERY PROFILE + ### Description +This statement is used to view the tree profile information of the query operation,this function requires the user to open profile settings. +Before versions 0.15, perform the following settings: + +```sql +SET is_report_success=true; +``` + +For versions 0.15 and later, perform the following settings: + +```sql +SET [GLOBAL] enable_profile=true; +``` + +grammar: + +```sql +show query profile "/"; +``` +This command will list the profiles of all currently saved query operations. + +```sql +show query profile "/queryId"\G +show query profile "/queryId/fragment_id/instance_id"\G +``` +Get the tree profile information of the specified query ID,Return to profile simple tree.Specify fragment_ ID and instance_ ID returns the corresponding detailed profile tree. + + ### Example +1. List all query Profile + + ```sql + mysql> show query profile "/"; + +---+--+-++---+-+-+---++ + | QueryId | User | DefaultDb | SQL | QueryType | StartTime | EndTime | TotalTime | QueryState | + +---+--+-++---+-+-+---++ + | 327167e0db4749a9-adce3b3d770b2bb1 | root | default_cluster:test_db | select * from test | Query | 2022-08-09 10:50:09 | 2022-08-09 10:50:09 | 19ms | EOF| + +---+--+-++---+-+-+---++ + 1 row in set (0.00 sec) + ``` + +2. List the query profile of the specified queryid + + ```sql + mysql> show query profile "/327167e0db4749a9-adce3b3d770b2bb1"\G + *** 1. row *** + Fragments: ┌┐ + │[-1: VDataBufferSender] │ + │Fragment: 0 │ + │MaxActiveTime: 783.263us│ + └┘ + ┌┘ + │ + ┌───┐ + │[1: VEXCHANGE_NODE]│ + │Fragment: 0│ + └───┘ + └┐ +│ + ┌┐ + │[1: VDataStreamSender] │ + │Fragment: 1 │ + │MaxActiveTime: 847.612us│ + └┘ +│ +│ + ┌┐ + │[0: VOLAP_SCAN_NODE]│ + │Fragment: 1 │ + └┘ + ┌┘ + │ +┌─┐ +│[OlapScanner]│ +│Fragment: 1 │ +└─┘ + │ + │ + ┌─┐ + │[SegmentIterator]│ + │Fragment: 1 │ + └─┘ + 1 row in set (0.00 sec) + ``` +3. Lists the instance profile of the specified fragment: + + ```sql + mysql> show query profile "/327167e0db4749a9-adce3b3d770b2bb1/1/"\G + *** 1. row *** +Instances: 327167e0db4749a9-adce3b3d770b2bb2 + Host: 172.26.0.1:9111 + ActiveTime: 847.612us + 1 row in set (0.01 sec) + ``` + +4. Continue to view the detailed profile of each operator on a specific instance: + + ```sql + mysql> show query profile "/327167e0db4749a9-adce3b3d770b2bb1/1/327167e0db4749a9-adce3b3d770b2bb2"\G + *** 1. row *** + Instance: ┌───┐ + │[1: VDataStreamSender] │ + │(Active: 36.944us, non-child: 0.20)│ + │ - Counters: │ + │ - Byt
[GitHub] [doris] caiconghui opened a new pull request, #11634: [Enhancement](hdfs) Support loading hdfs config for be from hdfs-site.xml
caiconghui opened a new pull request, #11634: URL: https://github.com/apache/doris/pull/11634 # Proposed changes Issue Number: close #11570 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue opened a new issue, #11636: [Enhancement] sort result by tablename when show tables like 'show data'
nextdreamblue opened a new issue, #11636: URL: https://github.com/apache/doris/issues/11636 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description when show tables from xxx，i get table names unsort，for example： MySQL [abc]> show tables; +---+ | Tables_in_abc | +---+ | ctable_hash | | atable_hash | | table_hash| | btable_hash | | ztable_hash | +---+ 5 rows in set (0.00 sec) but when show data, i can get table names which be sorted, for example： MySQL [abc]> show data; -- show data -- +-+-+--+ | TableName | Size| ReplicaCount | +-+-+--+ | atable_hash | 0.000 | 96 | | btable_hash | 0.000 | 96 | | ctable_hash | 0.000 | 96 | | table_hash | 790.000 B | 96 | | ztable_hash | 0.000 | 96 | | Total | 790.000 B | 480 | | Quota | 1024.000 TB | 1073741824 | | Left| 1024.000 TB | 1073741344 | +-+-+--+ i want to get result sorted by tablename when show tables like show data. ### Solution when show tables, sort result by tablenames. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
