[GitHub] [doris] hello-stephen commented on pull request #15155: [fix](executor) fix some npe and add some error info
hello-stephen commented on PR #15155: URL: https://github.com/apache/doris/pull/15155#issuecomment-1356732010 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.54 seconds load time: 716 seconds storage size: 17123130792 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221218082156_clickbench_pr_64709.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #11579: [Feature](NGram BloomFilter Index) add new ngram bloom filter index to speed up like query
github-actions[bot] commented on code in PR #11579:
URL: https://github.com/apache/doris/pull/11579#discussion_r1051554692
##
be/test/olap/itoken_extractor_test.cpp:
##
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/itoken_extractor.h"
+
+#include
+
+#include
+
+#include "common/logging.h"
+#include "util/utf8_check.h"
+
+namespace doris {
+
+class TestITokenExtractor : public testing::Test {
+public:
+void SetUp() {}
+void TearDown() {}
Review Comment:
warning: annotate this function with 'override' or (rarely) 'final'
[modernize-use-override]
```suggestion
void TearDown() override {}
```
##
be/test/olap/itoken_extractor_test.cpp:
##
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/itoken_extractor.h"
+
+#include
+
+#include
+
+#include "common/logging.h"
+#include "util/utf8_check.h"
+
+namespace doris {
+
+class TestITokenExtractor : public testing::Test {
+public:
+void SetUp() {}
Review Comment:
warning: annotate this function with 'override' or (rarely) 'final'
[modernize-use-override]
```suggestion
void SetUp() override {}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] maochongxin commented on pull request #15154: [Feature] Support function roundBankers
maochongxin commented on PR #15154: URL: https://github.com/apache/doris/pull/15154#issuecomment-1356741818 > Great job! Could you also add some documents for this new function? Ok, let me do this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on issue #11706: Good First Issue
TangSiyang2001 commented on issue #11706: URL: https://github.com/apache/doris/issues/11706#issuecomment-1356760148 > Good luck! If you have any questions, I am willing to help! Hi,I realized that `argmax` and `argmin` seem to act the same role of `max_by` and `min_by`.May I just treat `argmax` and `argmin` as aliases of the latter?Or plz point it out if I misunderstood.Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li opened a new pull request, #15156: [Regression](multi catalog)P2 regression case for external hms catalog on emr.
Jibing-Li opened a new pull request, #15156: URL: https://github.com/apache/doris/pull/15156 # Proposed changes Issue Number: close #xxx ## Problem summary Regression cases for external hms catalog. Cases are copied from doris p2 cases: yandex brown github_events ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15156: [Regression](multi catalog)P2 regression case for external hms catalog on emr.
hello-stephen commented on PR #15156: URL: https://github.com/apache/doris/pull/15156#issuecomment-1356768360 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.74 seconds load time: 628 seconds storage size: 17122844904 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221218103041_clickbench_pr_64735.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15040: [Pipeline](runtime filter) Support runtime filters on pipeline engine
github-actions[bot] commented on PR #15040: URL: https://github.com/apache/doris/pull/15040#issuecomment-1356771376 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15040: [Pipeline](runtime filter) Support runtime filters on pipeline engine
github-actions[bot] commented on PR #15040: URL: https://github.com/apache/doris/pull/15040#issuecomment-1356771363 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15154: [Feature] Support function roundBankers
github-actions[bot] commented on PR #15154: URL: https://github.com/apache/doris/pull/15154#issuecomment-1356774638 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15154: [Feature] Support function roundBankers
github-actions[bot] commented on PR #15154: URL: https://github.com/apache/doris/pull/15154#issuecomment-1356775155 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liutang123 commented on a diff in pull request #15124: [Pipeline] Add MLFQ when schedule
liutang123 commented on code in PR #15124:
URL: https://github.com/apache/doris/pull/15124#discussion_r1051589276
##
be/src/pipeline/task_queue.cpp:
##
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "task_queue.h"
+
+namespace doris {
+namespace pipeline {
+
+PipelineTask* SubWorkTaskQueue::try_take() {
+if (_queue.empty()) {
+return nullptr;
+}
+++_schedule_time;
+auto task = _queue.front();
+_queue.pop();
+return task;
+}
+
+ WorkTaskQueue
+
+WorkTaskQueue::WorkTaskQueue() : _closed(false) {
+double factor = 1;
+for (int i = 0; i < SUB_QUEUE_LEVEL; ++i) {
+_sub_queues[i].set_factor_for_normal(factor);
+factor *= LEVEL_QUEUE_TIME_FACTOR;
+}
+
+int i = 0;
+_task_schedule_limit[i] = BASE_LIMIT * (i + 1);
+for (i = 1; i < SUB_QUEUE_LEVEL - 1; ++i) {
+_task_schedule_limit[i] = _task_schedule_limit[i - 1] + BASE_LIMIT *
(i + 1);
+}
+}
+
+void WorkTaskQueue::close() {
+std::unique_lock lock(_work_size_mutex);
+_closed = true;
+_wait_task.notify_all();
+}
+
+PipelineTask* WorkTaskQueue::try_take_unprotected() {
+if (_total_task_size == 0 || _closed) {
+return nullptr;
+}
+double normal_schedule_times[SUB_QUEUE_LEVEL];
+double min_schedule_time = 0;
+int idx = -1;
+for (int i = 0; i < SUB_QUEUE_LEVEL; ++i) {
+normal_schedule_times[i] = _sub_queues[i].schedule_time_after_normal();
+if (!_sub_queues[i].empty()) {
+if (idx == -1 || normal_schedule_times[i] < min_schedule_time) {
+idx = i;
+min_schedule_time = normal_schedule_times[i];
+}
+}
+}
+DCHECK(idx != -1);
Review Comment:
if `_total_task_size == 0`, `nullptr` will return in the begin of this
method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15156: [Regression](multi catalog)P2 regression case for external hms catalog on emr.
github-actions[bot] commented on PR #15156: URL: https://github.com/apache/doris/pull/15156#issuecomment-1356788943 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15156: [Regression](multi catalog)P2 regression case for external hms catalog on emr.
github-actions[bot] commented on PR #15156: URL: https://github.com/apache/doris/pull/15156#issuecomment-1356788951 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [cherrypick](datev2-decimalv3) refine function expr of datev2 & compute accurate round value by decimal (#15103)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new b6738c1938 [cherrypick](datev2-decimalv3) refine function expr of
datev2 & compute accurate round value by decimal (#15103)
b6738c1938 is described below
commit b6738c193868d885dbdf0418f6e18a52afe09548
Author: Gabriel
AuthorDate: Sun Dec 18 20:35:52 2022 +0800
[cherrypick](datev2-decimalv3) refine function expr of datev2 & compute
accurate round value by decimal (#15103)
---
be/src/vec/functions/math.cpp | 119 +++--
be/src/vec/functions/round.h | 556 +
be/test/vec/function/function_math_test.cpp| 35 +-
.../apache/doris/analysis/FunctionCallExpr.java| 147 +++---
.../java/org/apache/doris/catalog/Function.java| 4 -
.../apache/doris/planner/ConstantExpressTest.java | 7 -
gensrc/script/doris_builtins_functions.py | 60 ++-
.../sql_functions/math_functions/test_round.out| 12 +-
.../sql_functions/math_functions/test_round.groovy | 22 +-
9 files changed, 777 insertions(+), 185 deletions(-)
diff --git a/be/src/vec/functions/math.cpp b/be/src/vec/functions/math.cpp
index 417f576a52..67552b0f18 100644
--- a/be/src/vec/functions/math.cpp
+++ b/be/src/vec/functions/math.cpp
@@ -24,6 +24,7 @@
#include "vec/functions/function_string.h"
#include "vec/functions/function_totype.h"
#include "vec/functions/function_unary_arithmetic.h"
+#include "vec/functions/round.h"
#include "vec/functions/simple_function_factory.h"
namespace doris::vectorized {
@@ -184,11 +185,6 @@ struct LogImpl {
};
using FunctionLog = FunctionBinaryArithmetic;
-struct CeilName {
-static constexpr auto name = "ceil";
-};
-using FunctionCeil = FunctionMathUnary>;
-
template
struct SignImpl {
using ResultType = Int8;
@@ -276,12 +272,6 @@ struct TanName {
};
using FunctionTan = FunctionMathUnary>;
-struct FloorName {
-static constexpr auto name = "floor";
-};
-using FunctionFloor =
-FunctionMathUnary>;
-
template
struct RadiansImpl {
using ResultType = A;
@@ -346,30 +336,6 @@ struct BinImpl {
using FunctionBin = FunctionUnaryToType;
-struct RoundName {
-static constexpr auto name = "round";
-};
-
-/// round(double)-->int64
-/// key_str:roundFloat64
-template
-struct RoundOneImpl {
-using Type = DataTypeInt64;
-static constexpr auto name = RoundName::name;
-static constexpr auto rows_per_iteration = 1;
-static constexpr bool always_returns_float64 = false;
-
-static DataTypes get_variadic_argument_types() {
-return {std::make_shared()};
-}
-
-template
-static void execute(const T* src, U* dst) {
-dst[0] = static_cast(std::round(static_cast(src[0])));
-}
-};
-using FunctionRoundOne = FunctionMathUnary>;
-
template
struct PowImpl {
using ResultType = double;
@@ -386,52 +352,83 @@ struct PowName {
};
using FunctionPow = FunctionBinaryArithmetic;
-template
-struct TruncateImpl {
-using ResultType = double;
-static const constexpr bool allow_decimal = false;
-
-template
-static inline double apply(A a, B b) {
-/// Next everywhere, static_cast - so that there is no wrong result in
expressions of the form Int64 c = UInt32(a) * Int32(-1).
-return static_cast(
-my_double_round(static_cast(a),
static_cast(b), false, true));
-}
-};
struct TruncateName {
static constexpr auto name = "truncate";
};
-using FunctionTruncate = FunctionBinaryArithmetic;
+
+struct CeilName {
+static constexpr auto name = "ceil";
+};
+
+struct FloorName {
+static constexpr auto name = "floor";
+};
+
+struct RoundName {
+static constexpr auto name = "round";
+};
/// round(double,int32)-->double
/// key_str:roundFloat64Int32
-template
-struct RoundTwoImpl {
-using ResultType = double;
-static const constexpr bool allow_decimal = false;
+template
+struct DoubleRoundTwoImpl {
+static constexpr auto name = Name::name;
static DataTypes get_variadic_argument_types() {
return {std::make_shared(),
std::make_shared()};
}
+};
-template
-static inline double apply(A a, B b) {
-/// Next everywhere, static_cast - so that there is no wrong result in
expressions of the form Int64 c = UInt32(a) * Int32(-1).
-return static_cast(
-my_double_round(static_cast(a),
static_cast(b), false, false));
+template
+struct DoubleRoundOneImpl {
+static constexpr auto name = Name::name;
+
+static DataTypes get_variadic_argument_types() {
+return {std::make_shared()};
+}
+};
+
+template
+struct DecimalRoundTwoImpl {
+static constexpr auto name = Name::name;
+
+static DataTypes get_variadic_argument_types() {
+retu
[GitHub] [doris] morningman merged pull request #15103: [cherrypick](datev2-decimalv3) refine function expr of datev2 & compute accurate round value by decimal
morningman merged PR #15103: URL: https://github.com/apache/doris/pull/15103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #15152: [Bug](decimalv3) Fix wrong argument for min_by/max_by
morningman merged PR #15152: URL: https://github.com/apache/doris/pull/15152 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [fix](decimalv3) Fix wrong argument for min_by/max_by (#15152)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new b569a3bbdb [fix](decimalv3) Fix wrong argument for min_by/max_by
(#15152)
b569a3bbdb is described below
commit b569a3bbdb80aa2085225583e529367b68c284ed
Author: Gabriel
AuthorDate: Sun Dec 18 20:36:41 2022 +0800
[fix](decimalv3) Fix wrong argument for min_by/max_by (#15152)
---
be/src/vec/aggregate_functions/aggregate_function_min_max_by.cpp | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/be/src/vec/aggregate_functions/aggregate_function_min_max_by.cpp
b/be/src/vec/aggregate_functions/aggregate_function_min_max_by.cpp
index ea513a26bb..8967f130f5 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_min_max_by.cpp
+++ b/be/src/vec/aggregate_functions/aggregate_function_min_max_by.cpp
@@ -105,22 +105,22 @@ static IAggregateFunction*
create_aggregate_function_min_max_by(const String& na
}
if (which.idx == TypeIndex::Decimal128) {
return
create_aggregate_function_min_max_by_impl>(
+
SingleValueDataDecimal>(
argument_types);
}
if (which.idx == TypeIndex::Decimal32) {
return
create_aggregate_function_min_max_by_impl>(
+
SingleValueDataDecimal>(
argument_types);
}
if (which.idx == TypeIndex::Decimal64) {
return
create_aggregate_function_min_max_by_impl>(
+
SingleValueDataDecimal>(
argument_types);
}
if (which.idx == TypeIndex::Decimal128I) {
return
create_aggregate_function_min_max_by_impl>(
+
SingleValueDataDecimal>(
argument_types);
}
return nullptr;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15144: [fix](auth) fix bug that user info may lost when upgrading to 1.2.0
hello-stephen commented on PR #15144: URL: https://github.com/apache/doris/pull/15144#issuecomment-1356802022 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.4 seconds load time: 687 seconds storage size: 17123386239 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221218134638_clickbench_pr_64758.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #15040: [Pipeline](runtime filter) Support runtime filters on pipeline engine
Gabriel39 merged PR #15040: URL: https://github.com/apache/doris/pull/15040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Pipeline](runtime filter) Support runtime filters on pipeline engine (#15040)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 13bc8c2ef8 [Pipeline](runtime filter) Support runtime filters on
pipeline engine (#15040)
13bc8c2ef8 is described below
commit 13bc8c2ef89fe598adf329a3c1a52c7569f81036
Author: Gabriel
AuthorDate: Sun Dec 18 21:48:00 2022 +0800
[Pipeline](runtime filter) Support runtime filters on pipeline engine
(#15040)
---
be/src/common/status.h| 4 +
be/src/exprs/runtime_filter.cpp | 101 ++
be/src/exprs/runtime_filter.h | 44 +--
be/src/exprs/runtime_filter_rpc.cpp | 1 +
be/src/pipeline/exec/operator.cpp | 15
be/src/pipeline/exec/operator.h | 6 +-
be/src/pipeline/exec/scan_operator.cpp| 18 +
be/src/pipeline/exec/scan_operator.h | 4 +-
be/src/pipeline/pipeline.cpp | 14 +---
be/src/pipeline/pipeline.h| 15 ++--
be/src/pipeline/pipeline_fragment_context.cpp | 4 -
be/src/pipeline/pipeline_fragment_context.h | 6 ++
be/src/pipeline/pipeline_task.cpp | 52 ++---
be/src/pipeline/pipeline_task.h | 20 ++---
be/src/pipeline/task_scheduler.cpp| 25 +--
be/src/runtime/fragment_mgr.cpp | 56 +++---
be/src/runtime/runtime_filter_mgr.cpp | 2 +
be/src/runtime/runtime_state.h| 5 +-
be/src/vec/exec/scan/vscan_node.cpp | 53 --
be/src/vec/exec/scan/vscan_node.h | 6 ++
be/src/vec/exec/vexchange_node.cpp| 16 +++-
be/src/vec/exec/vexchange_node.h | 1 +
be/src/vec/runtime/vdata_stream_recvr.h | 4 +-
gensrc/proto/internal_service.proto | 2 +
24 files changed, 323 insertions(+), 151 deletions(-)
diff --git a/be/src/common/status.h b/be/src/common/status.h
index 0cfc4517be..0970447a45 100644
--- a/be/src/common/status.h
+++ b/be/src/common/status.h
@@ -246,6 +246,7 @@ E(ROWSET_RENAME_FILE_FAILED, -3116);
E(SEGCOMPACTION_INIT_READER, -3117);
E(SEGCOMPACTION_INIT_WRITER, -3118);
E(SEGCOMPACTION_FAILED, -3119);
+E(PIP_WAIT_FOR_RF, -3120);
#undef E
}; // namespace ErrorCode
@@ -367,6 +368,7 @@ public:
ERROR_CTOR(NotSupported, NOT_IMPLEMENTED_ERROR)
ERROR_CTOR(EndOfFile, END_OF_FILE)
ERROR_CTOR(InternalError, INTERNAL_ERROR)
+ERROR_CTOR(WaitForRf, PIP_WAIT_FOR_RF)
ERROR_CTOR(RuntimeError, RUNTIME_ERROR)
ERROR_CTOR(Cancelled, CANCELLED)
ERROR_CTOR(MemoryLimitExceeded, MEM_LIMIT_EXCEEDED)
@@ -386,6 +388,8 @@ public:
bool ok() const { return _code == ErrorCode::OK; }
+bool is_blocked_by_rf() const { return _code ==
ErrorCode::PIP_WAIT_FOR_RF; }
+
bool is_io_error() const {
return ErrorCode::IO_ERROR == _code || ErrorCode::READ_UNENOUGH ==
_code ||
ErrorCode::CHECKSUM_ERROR == _code ||
ErrorCode::FILE_DATA_ERROR == _code ||
diff --git a/be/src/exprs/runtime_filter.cpp b/be/src/exprs/runtime_filter.cpp
index d584f06d17..a2d7c6118c 100644
--- a/be/src/exprs/runtime_filter.cpp
+++ b/be/src/exprs/runtime_filter.cpp
@@ -36,7 +36,6 @@
#include "runtime/large_int_value.h"
#include "runtime/primitive_type.h"
#include "runtime/runtime_filter_mgr.h"
-#include "runtime/runtime_state.h"
#include "util/runtime_profile.h"
#include "util/string_parser.hpp"
#include "vec/columns/column.h"
@@ -1205,7 +1204,7 @@ Status
IRuntimeFilter::get_prepared_context(std::vector* push_expr
if (_is_ignored) {
return Status::OK();
}
-DCHECK(_is_ready);
+DCHECK(!_state->enable_pipeline_exec() && _rf_state ==
RuntimeFilterState::READY);
DCHECK(is_consumer());
std::lock_guard guard(_inner_mutex);
@@ -1225,7 +1224,9 @@ Status
IRuntimeFilter::get_prepared_vexprs(std::vectorenable_pipeline_exec() && _rf_state ==
RuntimeFilterState::READY) ||
+ (_state->enable_pipeline_exec() &&
+_rf_state_atomic.load(std::memory_order_acquire) ==
RuntimeFilterState::READY));
DCHECK(is_consumer());
std::lock_guard guard(_inner_mutex);
@@ -1239,29 +1240,95 @@ Status
IRuntimeFilter::get_prepared_vexprs(std::vectorget_real_type() ==
RuntimeFilterType::BITMAP_FILTER
? _state->query_options().query_timeout
: _state->runtime_filter_wait_time_ms();
-std::unique_lock lock(_inner_mutex);
-if (!_is_ready) {
-int64_t ms_since_registration = MonotonicMillis() - registration_time_;
-int64_t ms_remaining = wait_times_ms - ms_since_registration;
-if (ms_remaining <= 0) {
-return _is_ready;
+if (_state->enable_pipeline_exec()) {
+auto expected = _rf_state_atomic.load
[GitHub] [doris] github-actions[bot] commented on pull request #14827: [Refactor](Nerieds) Refactor aggregate function/plan/rules and support related cbo rules
github-actions[bot] commented on PR #14827: URL: https://github.com/apache/doris/pull/14827#issuecomment-1356802536 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14827: [Refactor](Nerieds) Refactor aggregate function/plan/rules and support related cbo rules
github-actions[bot] commented on PR #14827: URL: https://github.com/apache/doris/pull/14827#issuecomment-1356802549 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #14827: [Refactor](Nerieds) Refactor aggregate function/plan/rules and support related cbo rules
morrySnow merged PR #14827: URL: https://github.com/apache/doris/pull/14827 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #14978: [refactor](resource) use resource to create external catalog
morningman commented on code in PR #14978:
URL: https://github.com/apache/doris/pull/14978#discussion_r1051608594
##
fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java:
##
@@ -17,88 +17,108 @@
package org.apache.doris.datasource;
-import org.apache.doris.catalog.HdfsResource;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.Resource;
import org.apache.doris.catalog.S3Resource;
+import org.apache.doris.common.UserException;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.persist.gson.GsonUtils;
-import com.google.common.collect.Maps;
import com.google.gson.annotations.SerializedName;
import lombok.Data;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.Optional;
/**
* CatalogProperty to store the properties for catalog.
*/
@Data
public class CatalogProperty implements Writable {
+private static final Logger LOG =
LogManager.getLogger(CatalogProperty.class);
+
+@SerializedName(value = "resource")
+private String resource;
@SerializedName(value = "properties")
-private Map properties = Maps.newHashMap();
+private Map properties;
-public String getOrDefault(String key, String defaultVal) {
-return properties.getOrDefault(key, defaultVal);
+private volatile Resource catalogResource = null;
+
+public CatalogProperty(String resource, Map properties) {
+this.resource = resource;
+this.properties = properties;
}
-// get all properties with dfs.* hadoop.* yarn.* hive.*
-// besides dfs.* and hadoop.username, we need other properties when enable
kerberos
-public Map getHdfsProperties() {
-Map dfsProperties = Maps.newHashMap();
-for (Map.Entry entry : properties.entrySet()) {
-if (entry.getKey().startsWith(HdfsResource.HADOOP_FS_PREFIX)
-|| entry.getKey().startsWith(HdfsResource.HADOOP_PREFIX)
-|| entry.getKey().startsWith(HdfsResource.HIVE_PREFIX)
-|| entry.getKey().startsWith(HdfsResource.YARN_PREFIX)) {
-dfsProperties.put(entry.getKey(), entry.getValue());
+private Resource catalogResource() throws UserException {
+if (catalogResource == null) {
+synchronized (this) {
+if (catalogResource == null) {
+catalogResource =
Optional.ofNullable(Env.getCurrentEnv().getResourceMgr().getResource(resource))
+.orElseThrow(() -> new UserException("Resource
doesn't exist: " + resource));
+}
}
}
-return dfsProperties;
+return catalogResource;
}
-// todo: remove and use S3Resource
-public Map getS3Properties() {
-Map s3Properties = Maps.newHashMap();
-if (properties.containsKey(S3Resource.S3_ACCESS_KEY)) {
-s3Properties.put("fs.s3a.access.key",
properties.get(S3Resource.S3_ACCESS_KEY));
-s3Properties.put(S3Resource.S3_ACCESS_KEY,
properties.get(S3Resource.S3_ACCESS_KEY));
-}
-if (properties.containsKey(S3Resource.S3_SECRET_KEY)) {
-s3Properties.put("fs.s3a.secret.key",
properties.get(S3Resource.S3_SECRET_KEY));
-s3Properties.put(S3Resource.S3_SECRET_KEY,
properties.get(S3Resource.S3_SECRET_KEY));
-}
-if (properties.containsKey(S3Resource.S3_ENDPOINT)) {
-s3Properties.put("fs.s3a.endpoint",
properties.get(S3Resource.S3_ENDPOINT));
-s3Properties.put(S3Resource.S3_ENDPOINT,
properties.get(S3Resource.S3_ENDPOINT));
-}
-if (properties.containsKey(S3Resource.S3_REGION)) {
-s3Properties.put("fs.s3a.endpoint.region",
properties.get(S3Resource.S3_REGION));
-s3Properties.put(S3Resource.S3_REGION,
properties.get(S3Resource.S3_REGION));
-}
-if (properties.containsKey(S3Resource.S3_MAX_CONNECTIONS)) {
-s3Properties.put("fs.s3a.connection.maximum",
properties.get(S3Resource.S3_MAX_CONNECTIONS));
-s3Properties.put(S3Resource.S3_MAX_CONNECTIONS,
properties.get(S3Resource.S3_MAX_CONNECTIONS));
+public String getOrDefault(String key, String defaultVal) {
+if (resource == null) {
+return properties.getOrDefault(key, defaultVal);
+} else {
+try {
+return
catalogResource().getCopiedProperties().getOrDefault(key, defaultVal);
+} catch (UserException e) {
+return defaultVal;
+}
}
-if (properties.containsKey(S3Resource.S3_REQUEST_TIMEOUT_MS)) {
-s3Properties.put("fs.s3a.connection.request.timeout",
properties.get(S3R
[GitHub] [doris] sohardforaname opened a new pull request, #15157: [Fix](Nereids)fix be core when select constant expression
sohardforaname opened a new pull request, #15157: URL: https://github.com/apache/doris/pull/15157 # Proposed changes Issue Number: close #xxx ## Problem summary fix be core when select !2 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [x] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15157: [Fix](Nereids)fix be core when select constant expression
hello-stephen commented on PR #15157: URL: https://github.com/apache/doris/pull/15157#issuecomment-1356810731 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.86 seconds load time: 656 seconds storage size: 17123062790 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221218142634_clickbench_pr_64762.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 0xflotus opened a new pull request, #15158: docs: fix small error
0xflotus opened a new pull request, #15158: URL: https://github.com/apache/doris/pull/15158 I only fixed a small error in docs. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10314: [Bug] Case clause is not supported for column in grouping set
github-actions[bot] commented on PR #10314: URL: https://github.com/apache/doris/pull/10314#issuecomment-1356910869 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] closed pull request #10266: [refactor] remove scheam hash
github-actions[bot] closed pull request #10266: [refactor] remove scheam hash URL: https://github.com/apache/doris/pull/10266 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15156: [Regression](multi catalog)P2 regression case for external hms catalog on emr.
yiguolei merged PR #15156: URL: https://github.com/apache/doris/pull/15156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (af4d9b636a -> 3506b568ff)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from af4d9b636a [refactor](Nerieds) Refactor aggregate function/plan/rules and support related cbo rules (#14827) add 3506b568ff [Regression](multi catalog)P2 regression case for external hms catalog on emr. #15156 No new revisions were added by this update. Summary of changes: .../doris/catalog/external/ExternalTable.java |2 +- .../hive/test_external_brown.out | 2347 ++ .../hive/test_external_github.out | 4805 .../hive/test_external_yandex.out | 913 .../hive/test_external_brown.groovy| 325 ++ .../hive/test_external_github.groovy | 586 +++ .../hive/test_external_yandex.groovy | 87 + 7 files changed, 9064 insertions(+), 1 deletion(-) create mode 100644 regression-test/data/external_table_emr_p2/hive/test_external_brown.out create mode 100644 regression-test/data/external_table_emr_p2/hive/test_external_github.out create mode 100644 regression-test/data/external_table_emr_p2/hive/test_external_yandex.out create mode 100644 regression-test/suites/external_table_emr_p2/hive/test_external_brown.groovy create mode 100644 regression-test/suites/external_table_emr_p2/hive/test_external_github.groovy create mode 100644 regression-test/suites/external_table_emr_p2/hive/test_external_yandex.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #14999: [Bug] Adding dynamic partitions failed when the create table statement with colocate_with has inconsistent buckets
yiguolei closed issue #14999: [Bug] Adding dynamic partitions failed when the create table statement with colocate_with has inconsistent buckets URL: https://github.com/apache/doris/issues/14999 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15071: [fix](schema) Fix create table error if Colocate tables not equal to bucket num
yiguolei merged PR #15071: URL: https://github.com/apache/doris/pull/15071 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](schema) Fix create table error if Colocate tables not equal to bucket num (#15071)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a75c302bdb [fix](schema) Fix create table error if Colocate tables not
equal to bucket num (#15071)
a75c302bdb is described below
commit a75c302bdb82719e0bd82354950310a29cd61b71
Author: yuxuan-luo <119841515+yuxuan-...@users.noreply.github.com>
AuthorDate: Mon Dec 19 09:24:14 2022 +0800
[fix](schema) Fix create table error if Colocate tables not equal to bucket
num (#15071)
Co-authored-by: hugoluo
---
.../java/org/apache/doris/catalog/ColocateGroupSchema.java | 12
.../src/main/java/org/apache/doris/common/ErrorCode.java | 2 ++
.../java/org/apache/doris/datasource/InternalCatalog.java| 1 +
3 files changed, 15 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/ColocateGroupSchema.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/ColocateGroupSchema.java
index 7d76a25cd0..ff1c957cd4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/ColocateGroupSchema.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/ColocateGroupSchema.java
@@ -30,6 +30,7 @@ import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.List;
+import java.util.Map;
import java.util.stream.Collectors;
/*
@@ -113,6 +114,17 @@ public class ColocateGroupSchema implements Writable {
}
}
+public void checkDynamicPartition(Map properties,
+ DistributionInfo distributionInfo)
throws DdlException {
+if (properties.get(DynamicPartitionProperty.BUCKETS) != null) {
+HashDistributionInfo info = (HashDistributionInfo)
distributionInfo;
+if (info.getBucketNum() !=
Integer.parseInt(properties.get(DynamicPartitionProperty.BUCKETS))) {
+ErrorReport.reportDdlException(
+
ErrorCode.ERR_DYNAMIC_PARTITION_MUST_HAS_SAME_BUCKET_NUM_WITH_COLOCATE_TABLE,
bucketsNum);
+}
+}
+}
+
public static ColocateGroupSchema read(DataInput in) throws IOException {
ColocateGroupSchema schema = new ColocateGroupSchema();
schema.readFields(in);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
index 4f21d77a0c..a6a78d054b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
@@ -1635,6 +1635,8 @@ public enum ErrorCode {
ERR_COLOCATE_NOT_COLOCATE_TABLE(5064, new byte[]{'4', '2', '0', '0', '0'},
"Table %s is not a colocated table"),
ERR_INVALID_OPERATION(5065, new byte[]{'4', '2', '0', '0', '0'},
"Operation %s is invalid"),
+ERR_DYNAMIC_PARTITION_MUST_HAS_SAME_BUCKET_NUM_WITH_COLOCATE_TABLE(5063,
new byte[]{'4', '2', '0', '0', '0'},
+"Dynamic partition buckets must equal the distribution buckets if
creating a colocate table: %s"),
ERROR_DYNAMIC_PARTITION_TIME_UNIT(5065, new byte[]{'4', '2', '0', '0',
'0'},
"Unsupported time unit %s. Expect HOUR/DAY/WEEK/MONTH."),
ERROR_DYNAMIC_PARTITION_START_ZERO(5066, new byte[]{'4', '2', '0', '0',
'0'},
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index d7c1a94405..85fd9d411c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -1970,6 +1970,7 @@ public class InternalCatalog implements
CatalogIf {
if (groupSchema != null) {
// group already exist, check if this table can be added
to this group
groupSchema.checkColocateSchema(olapTable);
+groupSchema.checkDynamicPartition(properties,
olapTable.getDefaultDistributionInfo());
}
// add table to this group, if group does not exist, create a
new one
Env.getCurrentColocateIndex()
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #15143: [fix](having-clause) having clause do not works correct with same alias name
morningman commented on PR #15143: URL: https://github.com/apache/doris/pull/15143#issuecomment-1356953235 please add regression test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15080: [enhancement](signal) output query_id when 'be' core dumped
yiguolei merged PR #15080: URL: https://github.com/apache/doris/pull/15080 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (a75c302bdb -> 8a08085356)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a75c302bdb [fix](schema) Fix create table error if Colocate tables not equal to bucket num (#15071) add 8a08085356 [enhancement](signal) output query_id when 'be' core dumped #15080 No new revisions were added by this update. Summary of changes: be/src/runtime/fragment_mgr.cpp | 5 - be/src/runtime/thread_context.cpp | 3 +++ 2 files changed, 3 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15151: [bug](udaf) fix java udaf incorrect get null value with row
yiguolei merged PR #15151: URL: https://github.com/apache/doris/pull/15151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [bug](udaf) fix java udaf incorrect get null value with row (#15151)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 17e14e9a63 [bug](udaf) fix java udaf incorrect get null value with row
(#15151)
17e14e9a63 is described below
commit 17e14e9a63b37cdd5a8327e057bea04cdc120908
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Mon Dec 19 10:07:12 2022 +0800
[bug](udaf) fix java udaf incorrect get null value with row (#15151)
---
fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java
b/fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java
index 598886a26b..f684363bf6 100644
--- a/fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java
+++ b/fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java
@@ -361,7 +361,8 @@ public class UdafExecutor {
for (int i = 0; i < argTypes.length; ++i) {
// skip the input column of current row is null
if (UdfUtils.UNSAFE.getLong(null,
UdfUtils.getAddressAtOffset(inputNullsPtrs, i)) != -1
-&& UdfUtils.UNSAFE.getByte(null,
UdfUtils.getAddressAtOffset(inputNullsPtrs, i) + row) == 1) {
+&& (UdfUtils.UNSAFE.getByte(null,
UdfUtils.UNSAFE.getLong(null,
+UdfUtils.getAddressAtOffset(inputNullsPtrs, i)) + row) ==
1)) {
inputObjects[i] = null;
continue;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #14994: [Bug] (merge-on-write) get many rows with same key from merge-on-write table
yiguolei closed issue #14994: [Bug] (merge-on-write) get many rows with same key from merge-on-write table URL: https://github.com/apache/doris/issues/14994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #14995: [fix](merge-on-write) delete all rows with same key in all pre segments
yiguolei merged PR #14995: URL: https://github.com/apache/doris/pull/14995 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (17e14e9a63 -> 1ed5ad3a16)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 17e14e9a63 [bug](udaf) fix java udaf incorrect get null value with row (#15151) add 1ed5ad3a16 [fix](merge-on-write) delete all rows with same key in all pre segments (#14995) No new revisions were added by this update. Summary of changes: be/src/olap/tablet.cpp | 51 ++ be/src/olap/tablet.h | 3 ++- 2 files changed, 29 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau commented on a diff in pull request #14978: [refactor](resource) use resource to create external catalog
AshinGau commented on code in PR #14978:
URL: https://github.com/apache/doris/pull/14978#discussion_r1051739236
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Resource.java:
##
@@ -78,6 +107,26 @@ public static Resource fromStmt(CreateResourceStmt stmt)
throws DdlException {
return resource;
}
+public synchronized boolean removeReference(String referenceName,
ReferenceType type) {
+String fullName = referenceName + REFERENCE_SPLIT + type.name();
+if (references.remove(fullName) != null) {
+LOG.info("Reference(type={}, name={}) is removed from resource {},
current set: {}",
+type, referenceName, name, references);
+return true;
+}
+return false;
+}
+
+public synchronized boolean addReference(String referenceName,
ReferenceType type) throws AnalysisException {
Review Comment:
Different statements may require different parameters even when using the
same resource, so some checks are added in `addReference`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15153: [Bug](decimalv3) Fix wrong argument for min_by/max_by
yiguolei merged PR #15153: URL: https://github.com/apache/doris/pull/15153 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (1ed5ad3a16 -> 0b6054a4ce)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1ed5ad3a16 [fix](merge-on-write) delete all rows with same key in all pre segments (#14995) add 0b6054a4ce [Bug](decimalv3) Fix wrong argument for min_by/max_by (#15153) No new revisions were added by this update. Summary of changes: be/src/vec/aggregate_functions/aggregate_function_min_max_by.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15101: [enhancement](load) verify the number of rows between different replicas when load data to avoid data inconsistency
github-actions[bot] commented on PR #15101: URL: https://github.com/apache/doris/pull/15101#issuecomment-1357003933 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on pull request #15038: [nereids](function)fix encryption function translation bug
924060929 commented on PR #15038: URL: https://github.com/apache/doris/pull/15038#issuecomment-1357005475 I think we should create new function like `checkArguments` in the new ScalarFunction, and do the check in the analysis stage -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg closed issue #15056: [Bug] broker load failed
yangzhg closed issue #15056: [Bug] broker load failed URL: https://github.com/apache/doris/issues/15056 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg merged pull request #15057: [fix](brokerload) fix broker load failed caused by the error path
yangzhg merged PR #15057: URL: https://github.com/apache/doris/pull/15057 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (0b6054a4ce -> 07f5d9562c)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 0b6054a4ce [Bug](decimalv3) Fix wrong argument for min_by/max_by (#15153) add 07f5d9562c [fix](brokerload) fix broker load failed aused by the error path (#15057) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/java/org/apache/doris/backup/BrokerStorage.java | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg opened a new issue, #15159: [Bug] AddressSanitizer: heap-use-after-free in DataStreamRecvr::SenderQueue::add_block
jacktengg opened a new issue, #15159:
URL: https://github.com/apache/doris/issues/15159
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no
similar issues.
### Version
master0b6054a4ce9dd5e2ecbf83e300f4e31a3bf502a1
### What's Wrong?
regression test failed, be.out:
```
==2611039==ERROR: AddressSanitizer: heap-use-after-free on address
0x603089623a50 at pc 0x5614a415dab0 bp 0x7fafac8e8670 sp 0x7fafac8e8660
READ of size 8 at 0x603089623a50 thread T56002 (FragmentMgrThre)
#0 0x5614a415daaf in
doris::vectorized::VDataStreamRecvr::SenderQueue::add_block(doris::vectorized::Block*,
bool)
/home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:192
#1 0x5614a4162666 in
doris::vectorized::VDataStreamRecvr::add_block(doris::vectorized::Block*, int,
bool)
/home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:366
#2 0x5614a407237c in
doris::vectorized::Channel::send_local_block(doris::vectorized::Block*)
/home/zcp/repo_center/doris_master/doris/be/src/vec/sink/vdata_stream_sender.cpp:132
#3 0x5614a407bc34 in
doris::vectorized::VDataStreamSender::send(doris::RuntimeState*,
doris::vectorized::Block*, bool)
/home/zcp/repo_center/doris_master/doris/be/src/vec/sink/vdata_stream_sender.cpp:479
#4 0x5614984e26f1 in
doris::PlanFragmentExecutor::open_vectorized_internal()
/home/zcp/repo_center/doris_master/doris/be/src/runtime/plan_fragment_executor.cpp:310
#5 0x5614984e123a in doris::PlanFragmentExecutor::open()
/home/zcp/repo_center/doris_master/doris/be/src/runtime/plan_fragment_executor.cpp:250
#6 0x561498468471 in doris::FragmentExecState::execute()
/home/zcp/repo_center/doris_master/doris/be/src/runtime/fragment_mgr.cpp:251
#7 0x561498470a73 in
doris::FragmentMgr::_exec_actual(std::shared_ptr,
std::function)
/home/zcp/repo_center/doris_master/doris/be/src/runtime/fragment_mgr.cpp:495
#8 0x561498472e00 in operator()
/home/zcp/repo_center/doris_master/doris/be/src/runtime/fragment_mgr.cpp:732
#9 0x561498481be5 in __invoke_impl&>
/var/local/ldb_toolchain/include/c++/11/bits/invoke.h:61
#10 0x5614984816c1 in __invoke_r&>
/var/local/ldb_toolchain/include/c++/11/bits/invoke.h:111
#11 0x561498480e0f in _M_invoke
/var/local/ldb_toolchain/include/c++/11/bits/std_function.h:291
#12 0x561498303a4d in std::function::operator()() const
/var/local/ldb_toolchain/include/c++/11/bits/std_function.h:560
#13 0x561498e4c8fd in doris::FunctionRunnable::run()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:46
#14 0x561498e479ef in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:535
#15 0x561498e699d5 in void std::_invoke_impl(std::_invoke_memfun_deref, void
(doris::ThreadPool::&)(), doris::ThreadPool&)
/var/local/ldb_toolchain/include/c++/11/bits/invoke.h:74
#16 0x561498e69274 in std::_invoke_result::type std::_invoke(void (doris::ThreadPool::&)(), doris::ThreadPool&)
/var/local/ldb_toolchain/include/c++/11/bits/invoke.h:96
#17 0x561498e68613 in void std::Bind::_call(std::tuple<>&&,
std::_Index_tuple<0ul>) /var/local/ldb_toolchain/include/c++/11/functional:420
#18 0x561498e67124 in void std::_Bind::operator()<, void>()
/var/local/ldb_toolchain/include/c++/11/functional:503
#19 0x561498e63d15 in void std::_invoke_impl&>(std::_invoke_other,
std::_Bind&)
/var/local/ldb_toolchain/include/c++/11/bits/invoke.h:61
#20 0x561498e611cd in std::enable_if&>, void>::type
std::_invoke_r&>(std::_Bind&)
/var/local/ldb_toolchain/include/c++/11/bits/invoke.h:111
#21 0x561498e5c4cc in std::_Function_handler >::_M_invoke(std::_Any_data const&)
/var/local/ldb_toolchain/include/c++/11/bits/std_function.h:291
#22 0x561498303a4d in std::function::operator()() const
/var/local/ldb_toolchain/include/c++/11/bits/std_function.h:560
#23 0x561498e277d9 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:454
#24 0x7fb7d9093608 in start_thread
/build/glibc-sMfBJT/glibc-2.31/nptl/pthread_create.c:477
#25 0x7fb7d91cd162 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f162)
0x603089623a50 is located 0 bytes inside of 32-byte region
[0x603089623a50,0x603089623a70)
freed by thread T56026 (FragmentMgrThre) here:
#0 0x561495f06767 in operator delete(void*, unsigned long)
(/mnt/ssd01/doris-master/VEC_ASAN/be/lib/doris_be+0x10e21767)
#1 0x561495f91568 in
doris::RuntimeProfile::HighWaterMarkCounter::~HighWaterMarkCounter()
/home/zcp/repo_center/doris_master/doris/be/src/util/runtime_profile.h:118
#2 0x561498caa530 in
doris::ObjectPool::add(doris::RuntimeProfile::HighWaterMarkCounter*)::{lambda(void*)#1}::operator()(void*
[GitHub] [doris] github-actions[bot] commented on pull request #15107: [refactor](fe) remove non vectorized code in fe
github-actions[bot] commented on PR #15107: URL: https://github.com/apache/doris/pull/15107#issuecomment-1357012735 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15107: [refactor](fe) remove non vectorized code in fe
github-actions[bot] commented on PR #15107: URL: https://github.com/apache/doris/pull/15107#issuecomment-1357012752 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg opened a new pull request, #15160: [fix](counter) fix coredump caused by updating destroyed counter
jacktengg opened a new pull request, #15160: URL: https://github.com/apache/doris/pull/15160 # Proposed changes Issue Number: close #15159 ## Problem summary VDataStreamRecvr may have been closed when it receive blocks, update counter need to be protected by _lock and check `_is_cancelled`. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15160: [fix](counter) fix coredump caused by updating destroyed counter
github-actions[bot] commented on PR #15160: URL: https://github.com/apache/doris/pull/15160#issuecomment-1357015045 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15154: [Feature] Support function roundBankers
github-actions[bot] commented on PR #15154: URL: https://github.com/apache/doris/pull/15154#issuecomment-1357017741 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15031: [vectorized](pipeline) support union node operator
github-actions[bot] commented on PR #15031: URL: https://github.com/apache/doris/pull/15031#issuecomment-1357023930 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #15161: [fix](pipeline) remove the redundant override of the close function in set operators
mrhhsg opened a new pull request, #15161: URL: https://github.com/apache/doris/pull/15161 # Proposed changes ```bash raise 0x7762337f abort 0x7760ddb5 0x6642d319 google::LogMessage::SendToLog() 0x6642392d google::LogMessage::Flush() 0x66423e2c google::LogMessageFatal::~LogMessageFatal() 0x66427349 doris::vectorized::VExprContext::~VExprContext vexpr_context.cpp:35 ::operator()(void *) const object_pool.h:40 ::__invoke(void *) object_pool.h:40 doris::ObjectPool::clear object_pool.h:53 doris::RuntimeState::~RuntimeState runtime_state.cpp:163 std::default_delete::operator() unique_ptr.h:85 std::unique_ptr::~unique_ptr unique_ptr.h:361 doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext pipeline_fragment_context.h:44 __gnu_cxx::new_allocator::destroy<…> new_allocator.h:162 std::allocator_traits::destroy<…> alloc_traits.h:531 std::_Sp_counted_ptr_inplace::_M_dispose shared_ptr_base.h:528 std::_Sp_counted_base::_M_release shared_ptr_base.h:168 std::__shared_count::~__shared_count shared_ptr_base.h:702 std::__shared_ptr::~__shared_ptr shared_ptr_base.h:1149 std::shared_ptr::~shared_ptr shared_ptr.h:122 doris::pipeline::PipelineFragmentContext::close_a_pipeline pipeline_fragment_context.cpp:555 doris::pipeline::TaskScheduler::_try_close_task task_scheduler.cpp:312 doris::pipeline::TaskScheduler::_do_work task_scheduler.cpp:231 std::__invoke_impl<…> invoke.h:74 std::__invoke<…> invoke.h:96 std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) functional:420 std::_Bind::operator()<, void>() functional:503 std::__invoke_impl<…>(std::__invoke_other, std::_Bind<…> &) invoke.h:61 std::__invoke_r<…>(std::_Bind<…> &) invoke.h:111 std::_Function_handler::_M_invoke(const std::_Any_data &) std_function.h:291 std::function::operator()() const std_function.h:560 doris::FunctionRunnable::run threadpool.cpp:46 doris::ThreadPool::dispatch_thread threadpool.cpp:535 std::__invoke_impl<…> invoke.h:74 std::__invoke<…> invoke.h:96 std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul>) functional:420 std::_Bind::operator()<, void>() functional:503 std::__invoke_impl<…>(std::__invoke_other, std::_Bind<…> &) invoke.h:61 std::__invoke_r<…>(std::_Bind<…> &) invoke.h:111 std::_Function_handler::_M_invoke(const std::_Any_data &) std_function.h:291 std::function::operator()() const std_function.h:560 doris::Thread::supervise_thread thread.cpp:454 start_thread 0x76e4e17a clone 0x776e8df3 ``` ## Problem summary The sink/source operators will share the same VSetOperationNode, so it is better to use the close of the base class to release resources. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on issue #11706: Good First Issue
Yukang-Lian commented on issue #11706: URL: https://github.com/apache/doris/issues/11706#issuecomment-1357032701 > > Good luck! If you have any questions, I am willing to help! > > Hi,I realized that `argmax` and `argmin` seem to act the same role of `max_by` and `min_by`.May I just treat `argmax` and `argmin` as aliases of the latter?Or plz point it out if I misunderstood.Thanks! Please add my wechat `abmdocrt_lyk`, let's talk about some details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15161: [fix](pipeline) remove the redundant override of the close function in set operators
github-actions[bot] commented on PR #15161: URL: https://github.com/apache/doris/pull/15161#issuecomment-1357033275 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15161: [fix](pipeline) remove the redundant override of the close function in set operators
github-actions[bot] commented on PR #15161: URL: https://github.com/apache/doris/pull/15161#issuecomment-1357035525 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15161: [fix](pipeline) remove the redundant override of the close function in set operators
github-actions[bot] commented on PR #15161: URL: https://github.com/apache/doris/pull/15161#issuecomment-1357035497 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15154: [Feature] Support function roundBankers
github-actions[bot] commented on PR #15154: URL: https://github.com/apache/doris/pull/15154#issuecomment-1357036008 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15154: [Feature] Support function roundBankers
github-actions[bot] commented on PR #15154: URL: https://github.com/apache/doris/pull/15154#issuecomment-1357036031 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] DongLiang-0 opened a new pull request, #94: [Improvement]Dynamically refresh the BE node from cache
DongLiang-0 opened a new pull request, #94: URL: https://github.com/apache/doris-flink-connector/pull/94 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #11579: [Feature](NGram BloomFilter Index) add new ngram bloom filter index to speed up like query
xiaokang commented on code in PR #11579:
URL: https://github.com/apache/doris/pull/11579#discussion_r1051746186
##
fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java:
##
@@ -177,6 +188,30 @@ public void checkColumn(Column column, KeysType keysType)
throws AnalysisExcepti
if (indexType == IndexType.INVERTED) {
InvertedIndexUtil.checkInvertedIndexParser(indexColName,
colType, properties);
+} else if (indexType == IndexType.NGRAM_BF) {
+if (colType != PrimitiveType.CHAR && colType !=
PrimitiveType.VARCHAR) {
Review Comment:
is STRING missed?
##
gensrc/proto/olap_file.proto:
##
@@ -199,12 +199,16 @@ message ColumnPB {
optional bool visible = 16 [default=true];
repeated ColumnPB children_columns = 17;
repeated string children_column_names = 18;
+optional bool ngram_bf_column = 19 [default=false];
Review Comment:
can we use properties in IndexPB
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java:
##
@@ -685,6 +685,17 @@ public void setIndexFlag(TColumn tColumn, OlapTable
olapTable) {
if (tColumn.getColumnName().equals(columns.get(0))) {
tColumn.setHasBitmapIndex(true);
}
+} else if (index.getIndexType() == IndexDef.IndexType.NGRAM_BF) {
+List columns = index.getColumns();
+if (tColumn.getColumnName().equals(columns.get(0))) {
+tColumn.setHasNgramBfIndex(true);
+String ngramSize =
index.getProperties().getOrDefault(IndexDef.NGRAM_SIZE_KEY,
+IndexDef.DEFAULT_NGRAM_SIZE);
+String bfSize =
index.getProperties().getOrDefault(IndexDef.NGRAM_BF_SIZE_KEY,
+IndexDef.DEFAULT_NGRAM_BF_SIZE);
+tColumn.setGramSize(Integer.parseInt(ngramSize));
Review Comment:
can we use properties instead of column field?
##
be/src/olap/rowset/segment_v2/bloom_filter_index_writer.cpp:
##
@@ -203,5 +260,22 @@ Status BloomFilterIndexWriter::create(const
BloomFilterOptions& bf_options,
return Status::OK();
}
+Status NGramBloomFilterIndexWriterImpl::create(const BloomFilterOptions&
bf_options,
+ const TypeInfo* typeinfo,
uint8_t gram_size,
+ uint16_t gram_bf_size,
+
std::unique_ptr* res) {
+FieldType type = typeinfo->type();
+switch (type) {
+case OLAP_FIELD_TYPE_CHAR:
+case OLAP_FIELD_TYPE_VARCHAR:
Review Comment:
OLAP_FIELD_TYPE_STRING
##
be/src/olap/tablet_meta.cpp:
##
@@ -284,6 +288,12 @@ void TabletMeta::init_column_from_tcolumn(uint32_t
unique_id, const TColumn& tco
if (tcolumn.__isset.is_bloom_filter_column) {
column->set_is_bf_column(tcolumn.is_bloom_filter_column);
}
+if (tcolumn.has_ngram_bf_index) {
Review Comment:
can we use index properties
##
fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java:
##
@@ -37,6 +37,11 @@ public class IndexDef {
private String comment;
private Map properties;
+public static final String NGRAM_SIZE_KEY = "gram_size";
+public static final String NGRAM_BF_SIZE_KEY = "bf_size";
Review Comment:
can we make it easier for users to set FPR of bf and calculate bf size
automatically, just like
https://www.postgresql.org/docs/14/brin-builtin-opclasses.html
##
be/src/olap/itoken_extractor.h:
##
@@ -0,0 +1,98 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_ITOKEN_EXTRACTOR_H
+#define DORIS_ITOKEN_EXTRACTOR_H
+
+#include
+
+#include
+
+#include "olap/rowset/segment_v2/bloom_filter.h"
+
+namespace doris {
+
+/// Interface for string parsers.
+struct ITokenExtractor {
+virtual ~ITokenExtractor() = default;
+
+/// Fast inplace implementation for regular use.
+/// Gets string (data ptr and len) and start position for extracting next
token (state of extractor).
+/// Returns false if parsing is finished, otherwise
[GitHub] [doris] hello-stephen commented on pull request #15160: [fix](counter) fix coredump caused by updating destroyed counter
hello-stephen commented on PR #15160: URL: https://github.com/apache/doris/pull/15160#issuecomment-1357043404 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.91 seconds load time: 628 seconds storage size: 17123621378 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221219033631_clickbench_pr_64843.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [fix](load) fix that flush memtable concurrently may cause data inconsistency (#15005)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new b73bb25dcc [fix](load) fix that flush memtable concurrently may cause
data inconsistency (#15005)
b73bb25dcc is described below
commit b73bb25dcc3c36fd06cea9e106e35c4e5cf0991f
Author: Xin Liao
AuthorDate: Tue Dec 13 09:27:35 2022 +0800
[fix](load) fix that flush memtable concurrently may cause data
inconsistency (#15005)
---
be/src/olap/memtable_flush_executor.cpp | 5 +++--
be/src/olap/task/engine_publish_version_task.cpp | 9 -
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/be/src/olap/memtable_flush_executor.cpp
b/be/src/olap/memtable_flush_executor.cpp
index 1dd4859e5a..c2fe380b1b 100644
--- a/be/src/olap/memtable_flush_executor.cpp
+++ b/be/src/olap/memtable_flush_executor.cpp
@@ -90,9 +90,10 @@ void FlushToken::_flush_memtable(MemTable* memtable, int64_t
submit_task_time) {
Status s = memtable->flush();
if (!s) {
LOG(WARNING) << "Flush memtable failed with res = " << s;
+// If s is not ok, ignore the code, just use other code is ok
+_flush_status.store(OLAP_ERR_OTHER_ERROR);
}
-// If s is not ok, ignore the code, just use other code is ok
-_flush_status.store(s.ok() ? OLAP_SUCCESS : OLAP_ERR_OTHER_ERROR);
+
if (_flush_status.load() != OLAP_SUCCESS) {
return;
}
diff --git a/be/src/olap/task/engine_publish_version_task.cpp
b/be/src/olap/task/engine_publish_version_task.cpp
index 25b55955c6..4e786a0455 100644
--- a/be/src/olap/task/engine_publish_version_task.cpp
+++ b/be/src/olap/task/engine_publish_version_task.cpp
@@ -227,11 +227,10 @@ void TabletPublishTxnTask::handle() {
return;
}
_engine_publish_version_task->add_succ_tablet_id(_tablet_info.tablet_id);
-VLOG_NOTICE << "publish version successfully on tablet. tablet=" <<
_tablet->full_name()
-<< ", transaction_id=" << _transaction_id << ", version=" <<
_version.first
-<< ", res=" << publish_status;
-
-return;
+LOG(INFO) << "publish version successfully on tablet"
+ << ", table_id=" << _tablet->table_id() << ", tablet=" <<
_tablet->full_name()
+ << ", transaction_id=" << _transaction_id << ", version=" <<
_version.first
+ << ", num_rows=" << _rowset->num_rows() << ", res=" <<
publish_status;
}
} // namespace doris
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly opened a new pull request, #15163: [feature](nereids)is_null_pred/is_not_null_pred
englefly opened a new pull request, #15163: URL: https://github.com/apache/doris/pull/15163 # Proposed changes make nereids support `is_null_pred` and `is_not_null_pred` Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated (b73bb25dcc -> 0d9d95a655)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git from b73bb25dcc [fix](load) fix that flush memtable concurrently may cause data inconsistency (#15005) new ed8d4d2df1 [refactor](resource) unified resource user interface (#14842) new d052138985 [feature](multi-catalog) support connecting to hive metastore with ke… (#15026) new 0d9d95a655 [cherry-pick](statistics) fix missing scanBytes and scanRows in query statistic #14828 The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/vec/exec/vexchange_node.cpp | 6 + be/src/vec/exec/vexchange_node.h | 1 + docs/en/docs/advanced/cold_hot_separation.md | 18 +- .../docs/ecosystem/external-table/multi-catalog.md | 28 ++- .../Alter/ALTER-RESOURCE.md| 22 +- .../Create/CREATE-RESOURCE.md | 57 +++-- .../sql-reference/Show-Statements/SHOW-POLICY.md | 18 +- docs/zh-CN/docs/advanced/cold_hot_separation.md| 18 +- .../docs/ecosystem/external-table/multi-catalog.md | 28 ++- .../Alter/ALTER-RESOURCE.md| 22 +- .../Create/CREATE-RESOURCE.md | 53 +++-- .../sql-reference/Show-Statements/SHOW-POLICY.md | 18 +- .../org/apache/doris/analysis/OutFileClause.java | 16 +- .../org/apache/doris/analysis/StorageDesc.java | 25 +-- .../java/org/apache/doris/backup/BlobStorage.java | 15 ++ .../org/apache/doris/backup/BrokerStorage.java | 7 +- .../java/org/apache/doris/backup/HdfsStorage.java | 36 +++- .../java/org/apache/doris/backup/S3Storage.java| 108 ++ .../java/org/apache/doris/catalog/HMSResource.java | 84 .../org/apache/doris/catalog/HdfsResource.java | 126 +++ .../doris/catalog/HiveMetaStoreClientHelper.java | 139 +++- .../java/org/apache/doris/catalog/HiveTable.java | 58 ++--- .../java/org/apache/doris/catalog/Resource.java| 14 +- .../java/org/apache/doris/catalog/ResourceMgr.java | 15 +- .../java/org/apache/doris/catalog/S3Resource.java | 213 +- .../doris/catalog/external/HMSExternalTable.java | 4 +- .../org/apache/doris/common/util/BrokerUtil.java | 161 ++ .../apache/doris/datasource/CatalogProperty.java | 59 ++--- .../doris/datasource/HMSExternalCatalog.java | 32 ++- .../apache/doris/datasource/InternalCatalog.java | 8 +- .../org/apache/doris/persist/gson/GsonUtils.java | 6 +- .../org/apache/doris/planner/BrokerScanNode.java | 3 +- .../org/apache/doris/planner/HiveScanNode.java | 3 +- .../org/apache/doris/planner/HudiScanNode.java | 3 +- .../doris/planner/external/HiveScanProvider.java | 2 +- .../doris/planner/external/LoadScanProvider.java | 4 +- .../doris/planner/external/QueryScanProvider.java | 3 +- .../org/apache/doris/policy/StoragePolicy.java | 9 +- .../apache/doris/service/FrontendServiceImpl.java | 8 +- .../ExternalFileTableValuedFunction.java | 3 +- .../tablefunction/HdfsTableValuedFunction.java | 2 +- .../doris/task/NotifyUpdateStoragePolicyTask.java | 6 +- .../java/org/apache/doris/alter/AlterTest.java | 20 +- .../org/apache/doris/backup/S3StorageTest.java | 4 +- .../org/apache/doris/catalog/ResourceMgrTest.java | 16 +- .../org/apache/doris/catalog/S3ResourceTest.java | 80 +++ .../cold_heat_separation/policy/alter.groovy | 88 .../cold_heat_separation/policy/create.groovy | 238 + .../suites/cold_heat_separation/policy/drop.groovy | 36 ++-- .../suites/cold_heat_separation/policy/show.groovy | 14 +- .../use_policy/alter_table_add_policy.groovy | 26 +-- .../create_table_use_partition_policy.groovy | 24 +-- .../use_policy/create_table_use_policy.groovy | 12 +- .../use_policy/modify_partition_add_policy.groovy | 12 +- .../use_policy/use_default_storage_policy.groovy | 16 +- 55 files changed, 1091 insertions(+), 956 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/catalog/HMSResource.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsResource.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 02/03: [feature](multi-catalog) support connecting to hive metastore with ke… (#15026)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git commit d052138985ed7127e1edeae398bb7bca7a8430c6 Author: wxy AuthorDate: Tue Dec 13 16:48:46 2022 +0800 [feature](multi-catalog) support connecting to hive metastore with ke… (#15026) Support kerberos authentication on hive external catalog --- .../docs/ecosystem/external-table/multi-catalog.md | 28 --- .../docs/ecosystem/external-table/multi-catalog.md | 28 --- .../org/apache/doris/catalog/HdfsResource.java | 3 ++ .../doris/catalog/external/HMSExternalTable.java | 4 +-- .../apache/doris/datasource/CatalogProperty.java | 10 --- .../doris/datasource/HMSExternalCatalog.java | 32 -- .../doris/planner/external/HiveScanProvider.java | 2 +- 7 files changed, 90 insertions(+), 17 deletions(-) diff --git a/docs/en/docs/ecosystem/external-table/multi-catalog.md b/docs/en/docs/ecosystem/external-table/multi-catalog.md index dabfcf4fb9..3be2f3bba0 100644 --- a/docs/en/docs/ecosystem/external-table/multi-catalog.md +++ b/docs/en/docs/ecosystem/external-table/multi-catalog.md @@ -85,11 +85,31 @@ CREATE CATALOG hive PROPERTIES ( "type"="hms", 'hive.metastore.uris' = 'thrift://172.21.0.1:7004', 'hadoop.username' = 'hive' -'dfs.nameservices'='service1', +'dfs.nameservices'='your-nameservice', +'dfs.ha.namenodes.your-nameservice'='nn1,nn2', +'dfs.namenode.rpc-address.your-nameservice.nn1'='172.21.0.2:4007', +'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:4007', + 'dfs.client.failover.proxy.provider.your-nameservice'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider' +); +``` + +If you want to connect to a Hive MetaStore with kerberos authentication, you can do like this: + +``` +CREATE CATALOG hive PROPERTIES ( +"type"="hms", +'hive.metastore.uris' = 'thrift://172.21.0.1:7004', +'hive.metastore.sasl.enabled' = 'true', +'dfs.nameservices'='your-nameservice', 'dfs.ha.namenodes. service1'='nn1,nn2', -'dfs.namenode.rpc-address.HDFS8000871.nn1'='172.21.0.2:4007', -'dfs.namenode.rpc-address.HDFS8000871.nn2'='172.21.0.3:4007', - 'dfs.client.failover.proxy.provider.HDFS8000871'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider' +'dfs.namenode.rpc-address.your-nameservice.nn1'='172.21.0.2:4007', +'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:4007', + 'dfs.client.failover.proxy.provider.your-nameservice'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider', +'hadoop.security.authentication' = 'kerberos', +'hadoop.kerberos.keytab' = '/your-keytab-filepath/your.keytab', +'hadoop.kerberos.principal' = 'your-princi...@your.com', +'yarn.resourcemanager.address' = 'your-rm-address:your-rm-port', +'yarn.resourcemanager.principal' = 'your-rm-principal/_h...@your.com' ); ``` diff --git a/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md b/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md index 841c6b6766..3627bb9221 100644 --- a/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md +++ b/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md @@ -85,14 +85,34 @@ CREATE CATALOG hive PROPERTIES ( "type"="hms", 'hive.metastore.uris' = 'thrift://172.21.0.1:7004', 'hadoop.username' = 'hive' -'dfs.nameservices'='service1', -'dfs.ha.namenodes. service1'='nn1,nn2', -'dfs.namenode.rpc-address.HDFS8000871.nn1'='172.21.0.2:4007', -'dfs.namenode.rpc-address.HDFS8000871.nn2'='172.21.0.3:4007', +'dfs.nameservices'='your-nameservice', +'dfs.ha.namenodes.service1'='nn1,nn2', +'dfs.namenode.rpc-address.your-nameservice.nn1'='172.21.0.2:4007', +'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:4007', 'dfs.client.failover.proxy.provider.HDFS8000871'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider' ); ``` +如果需要连接开启了 Kerberos 认证的 Hive MetaStore，示例如下： + +``` +CREATE CATALOG hive PROPERTIES ( +"type"="hms", +'hive.metastore.uris' = 'thrift://172.21.0.1:7004', +'hive.metastore.sasl.enabled' = 'true', +'dfs.nameservices'='your-nameservice', +'dfs.ha.namenodes.service1'='nn1,nn2', +'dfs.namenode.rpc-address.your-nameservice.nn1'='172.21.0.2:4007', +'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:4007', + 'dfs.client.failover.proxy.provider.your-nameservice'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider', +'hadoop.security.authentication' = 'kerberos', +'hadoop.kerberos.keytab' = '/your-keytab-filepath/your.keytab', +'hadoop.kerberos.principal' = 'your-princi...@your.com', +'yarn.resourcemanager.address' = 'your-rm-address:your-rm-port', +'yarn.resourcemanager.princip
[doris] 03/03: [cherry-pick](statistics) fix missing scanBytes and scanRows in query statistic #14828
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0d9d95a6555634406ac76df5172cfc7c99436ed2
Author: morningman
AuthorDate: Mon Dec 19 12:11:46 2022 +0800
[cherry-pick](statistics) fix missing scanBytes and scanRows in query
statistic #14828
---
be/src/vec/exec/vexchange_node.cpp | 6 ++
be/src/vec/exec/vexchange_node.h | 1 +
2 files changed, 7 insertions(+)
diff --git a/be/src/vec/exec/vexchange_node.cpp
b/be/src/vec/exec/vexchange_node.cpp
index c9cc87d6bd..8da3d2ead4 100644
--- a/be/src/vec/exec/vexchange_node.cpp
+++ b/be/src/vec/exec/vexchange_node.cpp
@@ -99,6 +99,12 @@ Status VExchangeNode::get_next(RuntimeState* state, Block*
block, bool* eos) {
return status;
}
+Status VExchangeNode::collect_query_statistics(QueryStatistics* statistics) {
+RETURN_IF_ERROR(ExecNode::collect_query_statistics(statistics));
+statistics->merge(_sub_plan_query_statistics_recvr.get());
+return Status::OK();
+}
+
Status VExchangeNode::close(RuntimeState* state) {
if (is_closed()) {
return Status::OK();
diff --git a/be/src/vec/exec/vexchange_node.h b/be/src/vec/exec/vexchange_node.h
index 5e2d4533af..b985097ca6 100644
--- a/be/src/vec/exec/vexchange_node.h
+++ b/be/src/vec/exec/vexchange_node.h
@@ -36,6 +36,7 @@ public:
virtual Status open(RuntimeState* state) override;
virtual Status get_next(RuntimeState* state, RowBatch* row_batch, bool*
eos) override;
virtual Status get_next(RuntimeState* state, Block* row_batch, bool* eos)
override;
+Status collect_query_statistics(QueryStatistics* statistics) override;
virtual Status close(RuntimeState* state) override;
// Status collect_query_statistics(QueryStatistics* statistics) override;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu commented on a diff in pull request #15114: [enhancement](gc) sub_file_cache checks the directory files when gc
pengxiangyu commented on code in PR #15114:
URL: https://github.com/apache/doris/pull/15114#discussion_r1051759865
##
be/src/io/cache/whole_file_cache.h:
##
@@ -48,15 +48,17 @@ class WholeFileCache final : public FileCache {
io::FileReaderSPtr remote_file_reader() const override { return
_remote_file_reader; }
-Status clean_timeout_cache() override;
+Status clean_timeout_cache();
+
+Status clean_cache_normal() override;
Status clean_all_cache() override;
Status clean_one_cache(size_t* cleaned_size) override;
-int64_t get_oldest_match_time() const override { return _last_match_time;
};
+int64_t get_oldest_match_time() const override { return _gc_match_time; };
Review Comment:
结尾多了一个分号
##
be/src/io/cache/file_cache.cpp:
##
@@ -87,32 +89,87 @@ Status FileCache::download_cache_to_local(const Path&
cache_file, const Path& ca
return Status::OK();
}
-Status FileCache::_remove_file(const Path& cache_file, const Path&
cache_done_file,
- size_t* cleaned_size) {
-bool done_file_exist = false;
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->exists(cache_done_file,
&done_file_exist),
-"Check local done file exist failed.");
-if (done_file_exist) {
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->delete_file(cache_done_file),
-fmt::format("Delete local done file failed: {}",
cache_done_file.native()));
-}
+Status FileCache::_remove_file(const Path& file, size_t* cleaned_size) {
bool cache_file_exist = false;
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->exists(cache_file,
&cache_file_exist),
-"Check local cache file exist failed.");
+RETURN_NOT_OK_STATUS_WITH_WARN(io::global_local_filesystem()->exists(file,
&cache_file_exist),
+ "Check local cache file exist failed.");
if (cache_file_exist) {
if (cleaned_size) {
RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->file_size(cache_file,
cleaned_size),
-fmt::format("get local cache file size failed: {}",
cache_file.native()));
+io::global_local_filesystem()->file_size(file,
cleaned_size),
+fmt::format("get local cache file size failed: {}",
file.native()));
}
RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->delete_file(cache_file),
-fmt::format("Delete local cache file failed: {}",
cache_file.native()));
+io::global_local_filesystem()->delete_file(file),
+fmt::format("Delete local cache file failed: {}",
file.native()));
+}
+LOG(INFO) << "Delete local cache file successfully: " << file.native();
+return Status::OK();
+}
+
+Status FileCache::_remove_cache_and_done(const Path& cache_file, const Path&
cache_done_file,
+ size_t* cleaned_size) {
+RETURN_IF_ERROR(_remove_file(cache_done_file, nullptr));
+RETURN_IF_ERROR(_remove_file(cache_file, cleaned_size));
+return Status::OK();
+}
+
+Status FileCache::_get_dir_file(const Path& cache_dir, std::vector&
cache_names,
Review Comment:
cache_names作为返回值，最好使用指针
##
be/src/io/cache/file_cache.cpp:
##
@@ -87,32 +89,87 @@ Status FileCache::download_cache_to_local(const Path&
cache_file, const Path& ca
return Status::OK();
}
-Status FileCache::_remove_file(const Path& cache_file, const Path&
cache_done_file,
- size_t* cleaned_size) {
-bool done_file_exist = false;
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->exists(cache_done_file,
&done_file_exist),
-"Check local done file exist failed.");
-if (done_file_exist) {
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->delete_file(cache_done_file),
-fmt::format("Delete local done file failed: {}",
cache_done_file.native()));
-}
+Status FileCache::_remove_file(const Path& file, size_t* cleaned_size) {
bool cache_file_exist = false;
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->exists(cache_file,
&cache_file_exist),
-"Check local cache file exist failed.");
+RETURN_NOT_OK_STATUS_WITH_WARN(io::global_local_filesystem()->exists(file,
&cache_file_exist),
+ "Check local cache file exist failed.");
if (cache_file_exist) {
if (cleaned_size) {
RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->file_size(cache_file,
cleaned_size),
-fmt::format("get local cache file size failed: {}",
cache_file.native()));
+io::global_local_f
[doris] 04/04: [Bug](datetimev2) Fix wrong result after insertion (#15052)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4eb5e14029edf89e8916c4dc7d7f791daaff3444
Author: Gabriel
AuthorDate: Thu Dec 15 09:54:18 2022 +0800
[Bug](datetimev2) Fix wrong result after insertion (#15052)
---
.../java/org/apache/doris/analysis/DateLiteral.java| 18 ++
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
index 8c2fc20303..b73cc7cd24 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
@@ -69,9 +69,9 @@ public class DateLiteral extends LiteralExpr {
private static final DateLiteral MAX_DATETIME = new DateLiteral(, 12,
31, 23, 59, 59);
private static final DateLiteral MIN_DATETIMEV2
-= new DateLiteral(, 1, 1, 0, 0, 0, 0);
+= new DateLiteral(, 1, 1, 0, 0, 0, 0, Type.DATETIMEV2);
private static final DateLiteral MAX_DATETIMEV2
-= new DateLiteral(, 12, 31, 23, 59, 59, 99L);
+= new DateLiteral(, 12, 31, 23, 59, 59, 99L,
Type.DATETIMEV2);
private static final int DATEKEY_LENGTH = 8;
private static final int DATETIMEKEY_LENGTH = 14;
private static final int MAX_MICROSECOND = 99;
@@ -284,7 +284,8 @@ public class DateLiteral extends LiteralExpr {
this.type = ScalarType.getDefaultDateType(Type.DATETIME);
}
-public DateLiteral(long year, long month, long day, long hour, long
minute, long second, long microsecond) {
+public DateLiteral(long year, long month, long day, long hour, long
minute, long second, long microsecond,
+Type type) {
this.hour = hour;
this.minute = minute;
this.second = second;
@@ -292,7 +293,8 @@ public class DateLiteral extends LiteralExpr {
this.month = month;
this.day = day;
this.microsecond = microsecond;
-this.type = Type.DATETIMEV2;
+Preconditions.checkArgument(type.isDatetimeV2());
+this.type = type;
}
public DateLiteral(long year, long month, long day, long hour, long
minute, long second, Type type) {
@@ -486,7 +488,7 @@ public class DateLiteral extends LiteralExpr {
return (year << 9) | (month << 5) | day;
} else if (type.equals(Type.DATETIMEV2)) {
return (year << 50) | (month << 46) | (day << 41) | (hour << 36)
-| (minute << 30) | (second << 24) | microsecond;
+| (minute << 30) | (second << 24) | microsecond;
} else {
Preconditions.checkState(false, "invalid date type: " + type);
return -1L;
@@ -638,8 +640,8 @@ public class DateLiteral extends LiteralExpr {
return new DateLiteral(this.year, this.month, this.day,
this.hour, this.minute, this.second,
targetType);
} else if (targetType.isDatetimeV2()) {
-return new DateLiteral(this.year, this.month, this.day,
this.hour, this.minute, this.microsecond,
-targetType);
+return new DateLiteral(this.year, this.month, this.day,
this.hour, this.minute, this.second,
+this.microsecond, targetType);
} else {
throw new AnalysisException("Error date literal type : " +
type);
}
@@ -672,7 +674,7 @@ public class DateLiteral extends LiteralExpr {
private long makePackedDatetimeV2() {
return (year << 50) | (month << 46) | (day << 41) | (hour << 36)
-| (minute << 30) | (second << 24) | microsecond;
+| (minute << 30) | (second << 24) | microsecond;
}
@Override
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 03/04: [improvement](config) opt the message when missing JAVA_HOME for BE (#15045)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bc4453f7f00d48846eeef872d9e7485256a71c91
Author: Mingyu Chen
AuthorDate: Wed Dec 14 23:17:46 2022 +0800
[improvement](config) opt the message when missing JAVA_HOME for BE (#15045)
Make the error message easy to understand
---
bin/start_be.sh | 8
conf/be.conf| 3 +++
2 files changed, 11 insertions(+)
diff --git a/bin/start_be.sh b/bin/start_be.sh
index b01cd98c7f..f73a6584f3 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -143,6 +143,14 @@ if [[ -e "${DORIS_HOME}/bin/palo_env.sh" ]]; then
source "${DORIS_HOME}/bin/palo_env.sh"
fi
+if [[ -z "${JAVA_HOME}" ]]; then
+echo "The JAVA_HOME environment variable is not defined correctly"
+echo "This environment variable is needed to run this program"
+echo "NB: JAVA_HOME should point to a JDK not a JRE"
+echo "You can set it in be.conf"
+exit 1
+fi
+
if [[ ! -d "${LOG_DIR}" ]]; then
mkdir -p "${LOG_DIR}"
fi
diff --git a/conf/be.conf b/conf/be.conf
index db67d7b1ff..7dd09d81f3 100644
--- a/conf/be.conf
+++ b/conf/be.conf
@@ -17,6 +17,9 @@
PPROF_TMPDIR="$DORIS_HOME/log/"
+# since 1.2, the JAVA_HOME need to be set to run BE process.
+# JAVA_HOME=/path/to/jdk/
+
# INFO, WARNING, ERROR, FATAL
sys_log_level = INFO
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated (0d9d95a655 -> 4eb5e14029)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git from 0d9d95a655 [cherry-pick](statistics) fix missing scanBytes and scanRows in query statistic #14828 new ca97e49b7d [improvement](query)optimize select stmt with limit 0 (#14956) new d18f33fcc7 [fix](string-type) rectify string type' len to MAX_STRING_LENGTH (#14985) new bc4453f7f0 [improvement](config) opt the message when missing JAVA_HOME for BE (#15045) new 4eb5e14029 [Bug](datetimev2) Fix wrong result after insertion (#15052) The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: bin/start_be.sh| 8 conf/be.conf | 3 +++ .../java/org/apache/doris/analysis/DateLiteral.java| 18 ++ .../src/main/java/org/apache/doris/catalog/Column.java | 15 ++- .../src/main/java/org/apache/doris/catalog/Type.java | 4 ++-- .../main/java/org/apache/doris/qe/StmtExecutor.java| 12 6 files changed, 49 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 01/04: [improvement](query)optimize select stmt with limit 0 (#14956)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ca97e49b7d3b785e290173ec59ebed43a57d870b
Author: chenlinzhong <490103...@qq.com>
AuthorDate: Wed Dec 14 13:48:09 2022 +0800
[improvement](query)optimize select stmt with limit 0 (#14956)
---
.../src/main/java/org/apache/doris/qe/StmtExecutor.java | 12
1 file changed, 12 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index 8feaaa3e6f..6b1041373d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -1105,6 +1105,18 @@ public class StmtExecutor implements ProfileWriter {
handleCacheStmt(cacheAnalyzer, channel, (SelectStmt) queryStmt);
return;
}
+
+// handle select .. from xx limit 0
+if (parsedStmt instanceof SelectStmt) {
+SelectStmt parsedSelectStmt = (SelectStmt) parsedStmt;
+if (parsedSelectStmt.getLimit() == 0) {
+LOG.info("ignore handle limit 0 ,sql:{}",
parsedSelectStmt.toSql());
+sendFields(queryStmt.getColLabels(),
exprToType(queryStmt.getResultExprs()));
+context.getState().setEof();
+return;
+}
+}
+
sendResult(isOutfileQuery, false, queryStmt, channel, null, null);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 02/04: [fix](string-type) rectify string type' len to MAX_STRING_LENGTH (#14985)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d18f33fcc75ec51f7bd9b536c629d3c53fa7f81a
Author: Mingyu Chen
AuthorDate: Wed Dec 14 15:41:08 2022 +0800
[fix](string-type) rectify string type' len to MAX_STRING_LENGTH (#14985)
cherry pick from #14587
---
.../src/main/java/org/apache/doris/catalog/Column.java| 15 ++-
.../src/main/java/org/apache/doris/catalog/Type.java | 4 ++--
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
index 532a49b0eb..4432db3335 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
@@ -29,6 +29,7 @@ import org.apache.doris.common.DdlException;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.common.util.SqlUtils;
+import org.apache.doris.persist.gson.GsonPostProcessable;
import org.apache.doris.persist.gson.GsonUtils;
import org.apache.doris.thrift.TColumn;
import org.apache.doris.thrift.TColumnType;
@@ -52,7 +53,7 @@ import java.util.Set;
/**
* This class represents the column-related metadata.
*/
-public class Column implements Writable {
+public class Column implements Writable, GsonPostProcessable {
private static final Logger LOG = LogManager.getLogger(Column.class);
public static final String DELETE_SIGN = "__DORIS_DELETE_SIGN__";
public static final String SEQUENCE_COL = "__DORIS_SEQUENCE_COL__";
@@ -699,4 +700,16 @@ public class Column implements Writable {
public void setCompoundKey(boolean compoundKey) {
isCompoundKey = compoundKey;
}
+
+@Override
+public void gsonPostProcess() throws IOException {
+// This just for bugfix. Because when user upgrade from 0.x to 1.1.x,
+// the length of String type become 1. The reason is not very clear
and maybe fixed by #14275.
+// Here we try to rectify the error string length, by setting all
String' length to MAX_STRING_LENGTH
+// when replaying edit log.
+if (type.isScalarType() && type.getPrimitiveType() ==
PrimitiveType.STRING
+&& type.getLength() != ScalarType.MAX_STRING_LENGTH) {
+((ScalarType) type).setLength(ScalarType.MAX_STRING_LENGTH);
+}
+}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
index 949c4a3e79..b8ee68aacd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
@@ -65,9 +65,9 @@ public abstract class Type {
public static final ScalarType DATEV2 = new
ScalarType(PrimitiveType.DATEV2);
public static final ScalarType TIMEV2 = new
ScalarType(PrimitiveType.TIMEV2);
public static final ScalarType TIME = new ScalarType(PrimitiveType.TIME);
-public static final ScalarType STRING = new
ScalarType(PrimitiveType.STRING);
+public static final ScalarType STRING = ScalarType.createStringType();
public static final ScalarType DEFAULT_DECIMALV2 =
ScalarType.createDecimalType(PrimitiveType.DECIMALV2,
-ScalarType.DEFAULT_PRECISION, ScalarType.DEFAULT_SCALE);
+ScalarType.DEFAULT_PRECISION, ScalarType.DEFAULT_SCALE);
public static final ScalarType MAX_DECIMALV2_TYPE =
ScalarType.createDecimalType(PrimitiveType.DECIMALV2,
ScalarType.MAX_DECIMALV2_PRECISION,
ScalarType.MAX_DECIMALV2_SCALE);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [fix](meta) fix bug that backend tag may change to default after upgrading to 1.2 (#15085)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 34ba9f59d2 [fix](meta) fix bug that backend tag may change to default
after upgrading to 1.2 (#15085)
34ba9f59d2 is described below
commit 34ba9f59d2abdf2c8ec208f795fb11251170819a
Author: Mingyu Chen
AuthorDate: Thu Dec 15 12:07:11 2022 +0800
[fix](meta) fix bug that backend tag may change to default after upgrading
to 1.2 (#15085)
---
fe/fe-core/src/main/java/org/apache/doris/system/Backend.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java
b/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java
index b102f3ed88..7868b50dab 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java
@@ -587,13 +587,12 @@ public class Backend implements Writable {
// When first upgrade from old version, tags may be null
tagMap = Maps.newHashMap();
}
-if (!tagMap.containsKey(Tag.TYPE_LOCATION)) {
+if (!locationTag.value.equals(tagMap.get(Tag.TYPE_LOCATION))) {
// ATTN: here we use Tag.TYPE_LOCATION directly, not
locationTag.type,
// because we need to make sure the previous tag must be a
location type tag,
// and if not, convert it to location type.
tagMap.put(Tag.TYPE_LOCATION, locationTag.value);
}
-locationTag = Tag.createNotCheck(Tag.TYPE_LOCATION,
tagMap.get(Tag.TYPE_LOCATION));
}
public static Backend read(DataInput in) throws IOException {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15161: [fix](pipeline) remove the redundant override of the close function in set operators
hello-stephen commented on PR #15161: URL: https://github.com/apache/doris/pull/15161#issuecomment-1357076656 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.56 seconds load time: 631 seconds storage size: 17123629853 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221219042259_clickbench_pr_64876.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated (34ba9f59d2 -> ae084be649)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git from 34ba9f59d2 [fix](meta) fix bug that backend tag may change to default after upgrading to 1.2 (#15085) new c7a97c012d [fix](resource) HdfsStorage can get default.Fs from path or configuration (#15079) new ae084be649 [Bug](function) fix overflow on concat_ws (#15043) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/vec/functions/function_timestamp.cpp| 13 +++--- be/src/vec/functions/function_utility.cpp | 4 +- .../org/apache/doris/analysis/OutFileClause.java | 5 ++- .../java/org/apache/doris/backup/HdfsStorage.java | 36 - .../org/apache/doris/catalog/HdfsResource.java | 8 .../external_catalog_p0/hive/test_hive_other.out | 8 .../hive/test_hive_other.groovy| 47 +- 7 files changed, 83 insertions(+), 38 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 02/02: [Bug](function) fix overflow on concat_ws (#15043)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ae084be6495969cc4fc092027befc1d1cb99270e
Author: Pxl
AuthorDate: Thu Dec 15 19:44:27 2022 +0800
[Bug](function) fix overflow on concat_ws (#15043)
fix overflow on concat_ws
---
be/src/vec/functions/function_timestamp.cpp | 13 +
be/src/vec/functions/function_utility.cpp | 4 +++-
2 files changed, 8 insertions(+), 9 deletions(-)
diff --git a/be/src/vec/functions/function_timestamp.cpp
b/be/src/vec/functions/function_timestamp.cpp
index f941408dc8..a467d7da8d 100644
--- a/be/src/vec/functions/function_timestamp.cpp
+++ b/be/src/vec/functions/function_timestamp.cpp
@@ -17,6 +17,7 @@
#include "runtime/runtime_state.h"
#include "udf/udf_internal.h"
+#include "vec/columns/column_const.h"
#include "vec/columns/column_nullable.h"
#include "vec/columns/column_string.h"
#include "vec/columns/column_vector.h"
@@ -397,14 +398,10 @@ struct UnixTimeStampImpl {
const ColumnNumbers& arguments, size_t result,
size_t input_rows_count) {
auto col_result = ColumnVector::create();
-col_result->resize(input_rows_count);
-// TODO: use a const column to store this value
-auto& col_result_data = col_result->get_data();
-auto res_value = context->impl()->state()->timestamp_ms() / 1000;
-for (int i = 0; i < input_rows_count; i++) {
-col_result_data[i] = res_value;
-}
-block.replace_by_position(result, std::move(col_result));
+col_result->resize(1);
+col_result->get_data()[0] = context->impl()->state()->timestamp_ms() /
1000;
+auto col_const = ColumnConst::create(std::move(col_result),
input_rows_count);
+block.replace_by_position(result, std::move(col_const));
return Status::OK();
}
};
diff --git a/be/src/vec/functions/function_utility.cpp
b/be/src/vec/functions/function_utility.cpp
index a42918ed08..9d529c9d01 100644
--- a/be/src/vec/functions/function_utility.cpp
+++ b/be/src/vec/functions/function_utility.cpp
@@ -16,6 +16,7 @@
// under the License.
#include
+#include "vec/columns/column_const.h"
#include "vec/data_types/data_type_number.h"
#include "vec/data_types/data_type_string.h"
#include "vec/functions/simple_function_factory.h"
@@ -103,7 +104,8 @@ public:
size_t result, size_t input_rows_count) override {
auto res_column = ColumnString::create();
res_column->insert_data(version.c_str(), version.length());
-block.replace_by_position(result, std::move(res_column));
+auto col_const = ColumnConst::create(std::move(res_column),
input_rows_count);
+block.replace_by_position(result, std::move(col_const));
return Status::OK();
}
};
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 01/02: [fix](resource) HdfsStorage can get default.Fs from path or configuration (#15079)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c7a97c012d5bd66f7f595fd618748fecb9396ba8
Author: Ashin Gau
AuthorDate: Thu Dec 15 16:56:32 2022 +0800
[fix](resource) HdfsStorage can get default.Fs from path or configuration
(#15079)
---
.../org/apache/doris/analysis/OutFileClause.java | 5 ++-
.../java/org/apache/doris/backup/HdfsStorage.java | 36 -
.../org/apache/doris/catalog/HdfsResource.java | 8
.../external_catalog_p0/hive/test_hive_other.out | 8
.../hive/test_hive_other.groovy| 47 +-
5 files changed, 75 insertions(+), 29 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
index 31bd346b06..3d3f82b74b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
@@ -624,7 +624,6 @@ public class OutFileClause {
} else if
(filePath.toUpperCase().startsWith(HDFS_FILE_PREFIX.toUpperCase())) {
brokerName = StorageBackend.StorageType.HDFS.name();
storageType = StorageBackend.StorageType.HDFS;
-filePath = filePath.substring(HDFS_FILE_PREFIX.length() - 1);
} else {
return;
}
@@ -651,7 +650,9 @@ public class OutFileClause {
if (storageType == StorageBackend.StorageType.S3) {
S3Storage.checkS3(brokerProps);
} else if (storageType == StorageBackend.StorageType.HDFS) {
-HdfsStorage.checkHDFS(brokerProps);
+if (!brokerProps.containsKey(HdfsResource.HADOOP_FS_NAME)) {
+brokerProps.put(HdfsResource.HADOOP_FS_NAME,
HdfsStorage.getFsName(filePath));
+}
}
brokerDesc = new BrokerDesc(brokerName, storageType, brokerProps);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/HdfsStorage.java
b/fe/fe-core/src/main/java/org/apache/doris/backup/HdfsStorage.java
index 6344a5ec75..e245ad6377 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/backup/HdfsStorage.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/backup/HdfsStorage.java
@@ -23,7 +23,6 @@ import org.apache.doris.catalog.HdfsResource;
import org.apache.doris.common.UserException;
import org.apache.doris.common.util.URI;
-import org.apache.commons.collections.map.CaseInsensitiveMap;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
@@ -49,6 +48,7 @@ import java.nio.file.FileVisitOption;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Comparator;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -58,7 +58,7 @@ import java.util.Map;
*/
public class HdfsStorage extends BlobStorage {
private static final Logger LOG = LogManager.getLogger(HdfsStorage.class);
-private final Map caseInsensitiveProperties;
+private final Map hdfsProperties;
private final int readBufferSize = 128 << 10; // 128k
private final int writeBufferSize = 128 << 10; // 128k
@@ -71,30 +71,26 @@ public class HdfsStorage extends BlobStorage {
* @param properties parameters to access HDFS.
*/
public HdfsStorage(Map properties) {
-caseInsensitiveProperties = new CaseInsensitiveMap();
+hdfsProperties = new HashMap<>();
setProperties(properties);
setType(StorageBackend.StorageType.HDFS);
setName(StorageBackend.StorageType.HDFS.name());
}
-public static void checkHDFS(Map properties) throws
UserException {
-for (String field : HdfsResource.REQUIRED_FIELDS) {
-if (!properties.containsKey(field)) {
-throw new UserException(
-String.format("The properties of hdfs is invalid. %s
are needed", field));
-}
-}
+public static String getFsName(String path) {
+Path hdfsPath = new Path(path);
+String fullPath = hdfsPath.toUri().toString();
+String filePath = hdfsPath.toUri().getPath();
+return fullPath.replace(filePath, "");
}
@Override
public FileSystem getFileSystem(String remotePath) throws UserException {
if (dfsFileSystem == null) {
-checkHDFS(caseInsensitiveProperties);
-String hdfsFsName =
caseInsensitiveProperties.get(HdfsResource.HADOOP_FS_NAME);
-String username =
caseInsensitiveProperties.get(HdfsResource.HADOOP_USER_NAME);
+String username =
hdfsProperties.get(HdfsResource.HADOOP_USER_NAME);
Configuration conf = new HdfsConfiguration();
boolean isSecurityEnabled = false;
-for (Map.Entry propEntry
[GitHub] [doris] luozenglin commented on a diff in pull request #15114: [enhancement](gc) sub_file_cache checks the directory files when gc
luozenglin commented on code in PR #15114:
URL: https://github.com/apache/doris/pull/15114#discussion_r1051788040
##
be/src/io/cache/file_cache.cpp:
##
@@ -87,32 +89,87 @@ Status FileCache::download_cache_to_local(const Path&
cache_file, const Path& ca
return Status::OK();
}
-Status FileCache::_remove_file(const Path& cache_file, const Path&
cache_done_file,
- size_t* cleaned_size) {
-bool done_file_exist = false;
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->exists(cache_done_file,
&done_file_exist),
-"Check local done file exist failed.");
-if (done_file_exist) {
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->delete_file(cache_done_file),
-fmt::format("Delete local done file failed: {}",
cache_done_file.native()));
-}
+Status FileCache::_remove_file(const Path& file, size_t* cleaned_size) {
bool cache_file_exist = false;
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->exists(cache_file,
&cache_file_exist),
-"Check local cache file exist failed.");
+RETURN_NOT_OK_STATUS_WITH_WARN(io::global_local_filesystem()->exists(file,
&cache_file_exist),
+ "Check local cache file exist failed.");
if (cache_file_exist) {
if (cleaned_size) {
RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->file_size(cache_file,
cleaned_size),
-fmt::format("get local cache file size failed: {}",
cache_file.native()));
+io::global_local_filesystem()->file_size(file,
cleaned_size),
+fmt::format("get local cache file size failed: {}",
file.native()));
}
RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->delete_file(cache_file),
-fmt::format("Delete local cache file failed: {}",
cache_file.native()));
+io::global_local_filesystem()->delete_file(file),
+fmt::format("Delete local cache file failed: {}",
file.native()));
+}
+LOG(INFO) << "Delete local cache file successfully: " << file.native();
+return Status::OK();
+}
+
+Status FileCache::_remove_cache_and_done(const Path& cache_file, const Path&
cache_done_file,
+ size_t* cleaned_size) {
+RETURN_IF_ERROR(_remove_file(cache_done_file, nullptr));
+RETURN_IF_ERROR(_remove_file(cache_file, cleaned_size));
+return Status::OK();
+}
+
+Status FileCache::_get_dir_file(const Path& cache_dir, std::vector&
cache_names,
+std::vector& unfinished_files) {
+// list all files
+std::vector cache_file_names;
+RETURN_NOT_OK_STATUS_WITH_WARN(
+io::global_local_filesystem()->list(cache_dir, &cache_file_names),
+fmt::format("List dir failed: {}", cache_dir.native()))
+
+// separate DATA file and DONE file
+std::set cache_names_temp;
+std::list done_names_temp;
+for (auto& cache_file_name : cache_file_names) {
+if (ends_with(cache_file_name.native(), CACHE_DONE_FILE_SUFFIX)) {
+done_names_temp.push_back(std::move(cache_file_name));
+} else {
+cache_names_temp.insert(std::move(cache_file_name));
+}
+}
+
+// match DONE file with DATA file
+for (auto done_file : done_names_temp) {
+Path cache_filename = StringReplace(done_file.native(),
CACHE_DONE_FILE_SUFFIX, "", true);
+if (auto cache_iter = cache_names_temp.find(cache_filename);
+cache_iter != cache_names_temp.end()) {
+cache_names_temp.erase(cache_iter);
+cache_names.push_back(std::move(cache_filename));
+} else {
+// not data file, but with DONE file
+unfinished_files.push_back(std::move(done_file));
+}
+}
+// data file without DONE file
+for (auto& file : cache_names_temp) {
Review Comment:
Downloading a file with a write lock, while `_get_dir_file` requires a read
lock, I don't think this will happen as described above.
##
be/src/io/cache/file_cache.cpp:
##
@@ -87,32 +89,87 @@ Status FileCache::download_cache_to_local(const Path&
cache_file, const Path& ca
return Status::OK();
}
-Status FileCache::_remove_file(const Path& cache_file, const Path&
cache_done_file,
- size_t* cleaned_size) {
-bool done_file_exist = false;
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->exists(cache_done_file,
&done_file_exist),
-"Check local done file exist failed.");
-if (done_file_exist) {
-RETURN_NOT_OK_STATUS_WITH_WARN(
-io::global_local_filesystem()->delete_file(cache_done_file),
-fmt::format("Delete local done file failed: {}",
[GitHub] [doris] Gabriel39 opened a new pull request, #15164: [Bug](pipeline) Fix bugs for scan node and join node
Gabriel39 opened a new pull request, #15164: URL: https://github.com/apache/doris/pull/15164 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15164: [Bug](pipeline) Fix bugs for scan node and join node
github-actions[bot] commented on PR #15164: URL: https://github.com/apache/doris/pull/15164#issuecomment-1357099888 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15164: [Bug](pipeline) Fix bugs for scan node and join node
github-actions[bot] commented on PR #15164: URL: https://github.com/apache/doris/pull/15164#issuecomment-1357100990 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 09/21: [vectorized](join) add try catch in create thread (#15065)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8853d515ae7ef22a19d1ee200aed8c54a5e95639
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Fri Dec 16 19:55:09 2022 +0800
[vectorized](join) add try catch in create thread (#15065)
---
be/src/vec/exec/join/vjoin_node_base.cpp | 15 ++-
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/be/src/vec/exec/join/vjoin_node_base.cpp
b/be/src/vec/exec/join/vjoin_node_base.cpp
index d5d5280f57..2ebedff219 100644
--- a/be/src/vec/exec/join/vjoin_node_base.cpp
+++ b/be/src/vec/exec/join/vjoin_node_base.cpp
@@ -159,11 +159,16 @@ Status VJoinNodeBase::open(RuntimeState* state) {
RETURN_IF_CANCELLED(state);
std::promise thread_status;
-std::thread([this, state, thread_status_p = &thread_status,
- parent_span = opentelemetry::trace::Tracer::GetCurrentSpan()]
{
-OpentelemetryScope scope {parent_span};
-this->_probe_side_open_thread(state, thread_status_p);
-}).detach();
+try {
+std::thread([this, state, thread_status_p = &thread_status,
+ parent_span =
opentelemetry::trace::Tracer::GetCurrentSpan()] {
+OpentelemetryScope scope {parent_span};
+this->_probe_side_open_thread(state, thread_status_p);
+}).detach();
+} catch (const std::system_error& e) {
+LOG(WARNING) << "In VJoinNodeBase::open create thread fail, " <<
e.what();
+return Status::InternalError(e.what());
+}
// Open the probe-side child so that it may perform any initialisation in
parallel.
// Don't exit even if we see an error, we still need to wait for the build
thread
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 03/21: [Bug](datev2) Fix wrong result when use datev2 as partition key (#15094)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a282d720c920610365393fb8edc10056b0199538
Author: Gabriel
AuthorDate: Thu Dec 15 21:27:05 2022 +0800
[Bug](datev2) Fix wrong result when use datev2 as partition key (#15094)
---
be/src/runtime/raw_value.h | 11 +---
.../org/apache/doris/analysis/DateLiteral.java | 52 ---
.../org/apache/doris/catalog/PartitionKey.java | 12 ++--
.../java/org/apache/doris/load/DeleteHandler.java | 6 +-
regression-test/data/delete_p0/test_delete.out | 16 ++---
.../data/partition_p0/test_datev2_partition.out| 39 +++
.../suites/delete_p0/test_delete.groovy| 22 +++
.../partition_p0/test_datev2_partition.groovy | 76 ++
8 files changed, 188 insertions(+), 46 deletions(-)
diff --git a/be/src/runtime/raw_value.h b/be/src/runtime/raw_value.h
index 70d12c60d5..ceb722e847 100644
--- a/be/src/runtime/raw_value.h
+++ b/be/src/runtime/raw_value.h
@@ -519,18 +519,11 @@ inline uint32_t RawValue::zlib_crc32(const void* v,
size_t len, const TypeDescri
}
case TYPE_DATEV2: {
-auto* date_v2_val = (const
vectorized::DateV2Value*)v;
-char buf[64];
-int date_v2_len = date_v2_val->to_buffer(buf);
-return HashUtil::zlib_crc_hash(buf, date_v2_len, seed);
+return HashUtil::zlib_crc_hash(v, 4, seed);
}
case TYPE_DATETIMEV2: {
-auto* date_v2_val =
-(const
vectorized::DateV2Value*)v;
-char buf[64];
-int date_v2_len = date_v2_val->to_buffer(buf);
-return HashUtil::zlib_crc_hash(buf, date_v2_len, seed);
+return HashUtil::zlib_crc_hash(v, 8, seed);
}
case TYPE_DECIMALV2: {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
index b73cc7cd24..2a6efebd40 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
@@ -486,9 +486,9 @@ public class DateLiteral extends LiteralExpr {
return (year * 1 + month * 100 + day) * 100L + hour *
1 + minute * 100 + second;
} else if (type.equals(Type.DATEV2)) {
return (year << 9) | (month << 5) | day;
-} else if (type.equals(Type.DATETIMEV2)) {
-return (year << 50) | (month << 46) | (day << 41) | (hour << 36)
-| (minute << 30) | (second << 24) | microsecond;
+} else if (type.isDatetimeV2()) {
+return (year << 46) | (month << 42) | (day << 37) | (hour << 32)
+| (minute << 26) | (second << 20) | (microsecond % (1 <<
20));
} else {
Preconditions.checkState(false, "invalid date type: " + type);
return -1L;
@@ -505,8 +505,8 @@ public class DateLiteral extends LiteralExpr {
buffer.order(ByteOrder.LITTLE_ENDIAN);
buffer.putInt(value);
} else if (type == PrimitiveType.DATETIMEV2) {
-long value = (year << 50) | (month << 46) | (day << 41) | (hour
<< 36)
-| (minute << 30) | (second << 24) | microsecond;
+long value = (year << 46) | (month << 42) | (day << 37) | (hour
<< 32)
+| (minute << 26) | (second << 20) | (microsecond % (1 <<
20));
buffer = ByteBuffer.allocate(8);
buffer.order(ByteOrder.LITTLE_ENDIAN);
buffer.putLong(value);
@@ -672,9 +672,40 @@ public class DateLiteral extends LiteralExpr {
return packedDatetime;
}
+private void fromPackedDatetimeV2(long packedTime) {
+microsecond = (packedTime % (1L << 20));
+long ymdhms = (packedTime >> 20);
+long ymd = ymdhms >> 17;
+day = ymd % (1 << 5);
+long ym = ymd >> 5;
+month = ym % (1 << 4);
+year = ym >> 4;
+
+long hms = ymdhms % (1 << 17);
+second = hms % (1 << 6);
+minute = (hms >> 6) % (1 << 6);
+hour = (hms >> 12);
+// set default date literal type to DATETIME
+// date literal read from meta will set type by flag bit;
+this.type = Type.DATETIMEV2;
+}
+
+private void fromPackedDateV2(long packedTime) {
+day = packedTime % (1 << 5);
+long ym = packedTime >> 5;
+month = ym % (1 << 4);
+year = ym >> 4;
+
+this.type = Type.DATEV2;
+}
+
private long makePackedDatetimeV2() {
-return (year << 50) | (month << 46) | (day << 41) | (hour << 36)
-| (minute << 30) | (second << 24) | microsecond;
+return (year << 46) | (month << 42) | (day << 37) | (hour << 32)
+| (minute << 26) | (second << 20) | (microsecond % (1 << 20));
+}
+
+pri
[doris] 08/21: [config](load) enable new load scan node by default (#14808)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3fe708cf56c3edcafa36f9e489e1d849e0fdf75a
Author: Mingyu Chen
AuthorDate: Fri Dec 16 09:41:43 2022 +0800
[config](load) enable new load scan node by default (#14808)
Set FE `enable_new_load_scan_node` to true by default.
So that all load tasks(broker load, stream load, routine load, insert into)
will use FileScanNode instead of BrokerScanNode
to read data
1. Support loading parquet file in stream load with new load scan node.
2. Fix bug that new parquet reader can not read column without logical or
converted type.
3. Change jsonb parser function to "jsonb_parse_error_to_null"
So that if the input string is not a valid json string, it will return
null for jsonb column in load task.
---
be/src/http/action/stream_load.cpp | 1 +
be/src/util/jsonb_parser.h | 25 +---
be/src/vec/exec/format/parquet/schema_desc.cpp | 14 +++--
be/src/vec/exec/scan/vfile_scanner.cpp | 6 +-
.../org/apache/doris/analysis/DataDescription.java | 27 ++--
.../main/java/org/apache/doris/common/Config.java | 2 +-
.../org/apache/doris/load/BrokerFileGroup.java | 2 +
.../org/apache/doris/planner/LoadScanNode.java | 5 +-
.../apache/doris/planner/StreamLoadPlanner.java| 14 -
.../planner/external/ExternalFileScanNode.java | 21 ++-
.../doris/planner/external/FileGroupInfo.java | 15 +
.../doris/planner/external/LoadScanProvider.java | 3 +-
.../apache/doris/service/FrontendServiceImpl.java | 3 +-
.../java/org/apache/doris/task/LoadTaskInfo.java | 5 ++
.../java/org/apache/doris/task/StreamLoadTask.java | 11
.../doris/planner/StreamLoadPlannerTest.java | 71 --
.../apache/doris/utframe/TestWithFeService.java| 8 ++-
gensrc/thrift/FrontendService.thrift | 1 +
.../table_valued_function/test_hdfs_tvf.groovy | 2 -
.../suites/export_p0/test_outfile_parquet.groovy | 2 +-
.../external_catalog_p0/hive/test_hive_orc.groovy | 27
.../hive/test_hive_other.groovy| 25
.../hive/test_hive_parquet.groovy | 28 -
.../jsonb_p0/test_jsonb_load_and_function.groovy | 26 +++-
.../test_jsonb_load_unique_key_and_function.groovy | 2 +
.../load_p0/broker_load/test_array_load.groovy | 8 ---
.../load_p0/broker_load/test_broker_load.groovy| 21 ---
...n_column_exclude_schema_without_jsonpath.groovy | 2 -
.../stream_load/load_json_null_to_nullable.groovy | 6 --
.../stream_load/load_json_with_jsonpath.groovy | 7 ---
.../load_p0/stream_load/test_hdfs_json_load.groovy | 6 --
.../load_p0/stream_load/test_json_load.groovy | 13
.../stream_load/test_txt_special_delimiter.groovy | 7 ---
.../test_streamload_perfomance.groovy | 2 +-
.../multi_catalog_query/hive_catalog_orc.groovy| 25
.../hive_catalog_parquet.groovy| 25
36 files changed, 164 insertions(+), 304 deletions(-)
diff --git a/be/src/http/action/stream_load.cpp
b/be/src/http/action/stream_load.cpp
index 69bfeabe48..b2acc0cedb 100644
--- a/be/src/http/action/stream_load.cpp
+++ b/be/src/http/action/stream_load.cpp
@@ -408,6 +408,7 @@ Status StreamLoadAction::_process_put(HttpRequest*
http_req, StreamLoadContext*
RETURN_IF_ERROR(file_sink->open());
request.__isset.path = true;
request.fileType = TFileType::FILE_LOCAL;
+request.__set_file_size(ctx->body_bytes);
ctx->body_sink = file_sink;
}
if (!http_req->header(HTTP_COLUMNS).empty()) {
diff --git a/be/src/util/jsonb_parser.h b/be/src/util/jsonb_parser.h
index c050fd305c..f4711f9a62 100644
--- a/be/src/util/jsonb_parser.h
+++ b/be/src/util/jsonb_parser.h
@@ -65,6 +65,7 @@
#include "jsonb_document.h"
#include "jsonb_error.h"
#include "jsonb_writer.h"
+#include "string_parser.hpp"
namespace doris {
@@ -894,8 +895,12 @@ private:
}
*pbuf = 0; // set null-terminator
-int64_t val = strtol(num_buf_, NULL, 10);
-if (errno == ERANGE) {
+StringParser::ParseResult parse_result = StringParser::PARSE_SUCCESS;
+int64_t val =
+StringParser::string_to_int(num_buf_, pbuf -
num_buf_, &parse_result);
+if (parse_result != StringParser::PARSE_SUCCESS) {
+VLOG_ROW << "debug string_to_int error for " << num_buf_ << "
val=" << val
+ << " parse_result=" << parse_result;
err_ = JsonbErrType::E_DECIMAL_OVERFLOW;
return false;
}
@@ -950,7 +955,7 @@ private:
}
*pbuf = 0; // set null-terminator
-return internConvertBufferToDouble();
+return internConvertBufferToDouble(num_buf_, pbuf
[doris] 02/21: [Enhancement](partition prune): calculate the column ranges of compound predicates (#14886)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5fce045f51a69d28e8824f3fc475b862f83821e4
Author: spaces-x
AuthorDate: Thu Dec 15 20:47:44 2022 +0800
[Enhancement](partition prune): calculate the column ranges of compound
predicates (#14886)
Doris does not support disjunctive predicates very well, which causes some
problems in partition prune.
For example, sqls like the followings will trigger a full table scan
without partition pruning
select * from test.t1
where (dt between 20211121 and 20211122) or (dt between 20211125 and
20211126)
---
.../java/org/apache/doris/planner/ScanNode.java| 83 ++
.../doris/analysis/RangePartitionPruneTest.java| 8 +++
2 files changed, 91 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
index 8523830c66..460ea01b7b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
@@ -43,7 +43,9 @@ import com.google.common.base.MoreObjects;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -260,6 +262,26 @@ public abstract class ScanNode extends PlanNode {
ColumnBound bound = ColumnBound.of((LiteralExpr)
inPredicate.getChild(i));
result.add(Range.closed(bound, bound));
}
+} else if (expr instanceof CompoundPredicate) {
+CompoundPredicate compoundPredicate = (CompoundPredicate) expr;
+ColumnRanges leftChildRange = null;
+ColumnRanges rightChildRange = null;
+switch (compoundPredicate.getOp()) {
+case AND:
+leftChildRange =
expressionToRanges(compoundPredicate.getChild(0), desc);
+rightChildRange =
expressionToRanges(compoundPredicate.getChild(1), desc);
+return leftChildRange.intersectRanges(rightChildRange);
+case OR:
+leftChildRange =
expressionToRanges(compoundPredicate.getChild(0), desc);
+rightChildRange =
expressionToRanges(compoundPredicate.getChild(1), desc);
+return leftChildRange.unionRanges(rightChildRange);
+case NOT:
+leftChildRange =
expressionToRanges(compoundPredicate.getChild(0), desc);
+return leftChildRange.complementOfRanges();
+default:
+throw new RuntimeException("unknown OP in compound
predicate: "
++ compoundPredicate.getOp().toString());
+}
}
if (result.isEmpty()) {
@@ -377,6 +399,67 @@ public abstract class ScanNode extends PlanNode {
return IS_NULL;
}
+public ColumnRanges complementOfRanges() {
+if (type == Type.CONVERT_SUCCESS) {
+RangeSet rangeSet = TreeRangeSet.create();
+rangeSet.addAll(ranges);
+return
create(Lists.newArrayList(rangeSet.complement().asRanges()));
+}
+return CONVERT_FAILURE;
+}
+
+public ColumnRanges intersectRanges(ColumnRanges other) {
+// intersect ranges can handle isnull
+switch (this.type) {
+case IS_NULL:
+return createIsNull();
+case CONVERT_FAILURE:
+return createFailure();
+case CONVERT_SUCCESS:
+switch (other.type) {
+case IS_NULL:
+return createIsNull();
+case CONVERT_FAILURE:
+return createFailure();
+case CONVERT_SUCCESS:
+RangeSet rangeSet =
TreeRangeSet.create();
+rangeSet.addAll(this.ranges);
+RangeSet intersectSet =
TreeRangeSet.create();
+
+other.ranges.forEach(range ->
intersectSet.addAll(rangeSet.subRangeSet(range)));
+return
create(Lists.newArrayList(intersectSet.asRanges()));
+default:
+return createFailure();
+}
+default:
+return createFailure();
+}
+}
+
+public ColumnRanges unionRanges(ColumnRanges other) {
+switch (this.type) {
+case IS
[doris] 01/21: [fix](merge-on-write) calc delete bitmap need all segments which _do_flush in one memtable (#15018)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b027ae3490627ed6bb780ee1a7d064eda23ec4b3
Author: xueweizhang
AuthorDate: Thu Dec 15 20:44:49 2022 +0800
[fix](merge-on-write) calc delete bitmap need all segments which _do_flush
in one memtable (#15018)
when some case(need modify be.conf), a memtable may flush many segments and
then calc delete bitmap with new data. but now, it just only load one segment
with max sgement id and this bug will not cala delte bitmap with all data of
all segment of one memtable, and will get many rows with same key from
merge-on-write table.
---
be/src/olap/memtable.cpp| 19 --
be/src/olap/memtable.h | 3 ++-
be/src/olap/rowset/beta_rowset.cpp | 35 -
be/src/olap/rowset/beta_rowset.h| 3 ++-
be/src/olap/rowset/beta_rowset_writer.h | 2 ++
be/src/olap/rowset/rowset_writer.h | 2 ++
6 files changed, 42 insertions(+), 22 deletions(-)
diff --git a/be/src/olap/memtable.cpp b/be/src/olap/memtable.cpp
index 892592b797..ab1a0f5209 100644
--- a/be/src/olap/memtable.cpp
+++ b/be/src/olap/memtable.cpp
@@ -415,7 +415,8 @@ bool MemTable::need_to_agg() {
: memory_usage() >=
config::memtable_max_buffer_size;
}
-Status MemTable::_generate_delete_bitmap() {
+Status MemTable::_generate_delete_bitmap(int64_t
atomic_num_segments_before_flush,
+ int64_t
atomic_num_segments_after_flush) {
// generate delete bitmap, build a tmp rowset and load recent segment
if (!_tablet->enable_unique_key_merge_on_write()) {
return Status::OK();
@@ -423,12 +424,11 @@ Status MemTable::_generate_delete_bitmap() {
auto rowset = _rowset_writer->build_tmp();
auto beta_rowset = reinterpret_cast(rowset.get());
std::vector segments;
-segment_v2::SegmentSharedPtr segment;
-if (beta_rowset->num_segments() == 0) {
+if (atomic_num_segments_before_flush >= atomic_num_segments_after_flush) {
return Status::OK();
}
-RETURN_IF_ERROR(beta_rowset->load_segment(beta_rowset->num_segments() - 1,
&segment));
-segments.push_back(segment);
+
RETURN_IF_ERROR(beta_rowset->load_segments(atomic_num_segments_before_flush,
+
atomic_num_segments_after_flush, &segments));
std::shared_lock meta_rlock(_tablet->get_header_lock());
// tablet is under alter process. The delete bitmap will be calculated
after conversion.
if (_tablet->tablet_state() == TABLET_NOTREADY &&
@@ -445,8 +445,15 @@ Status MemTable::flush() {
VLOG_CRITICAL << "begin to flush memtable for tablet: " << tablet_id()
<< ", memsize: " << memory_usage() << ", rows: " << _rows;
int64_t duration_ns = 0;
+// For merge_on_write table, it must get all segments in this flush.
+// The id of new segment is set by the _num_segment of beta_rowset_writer,
+// and new segment ids is between [atomic_num_segments_before_flush,
atomic_num_segments_after_flush),
+// and use the ids to load segment data file for calc delete bitmap.
+int64_t atomic_num_segments_before_flush =
_rowset_writer->get_atomic_num_segment();
RETURN_NOT_OK(_do_flush(duration_ns));
-RETURN_NOT_OK(_generate_delete_bitmap());
+int64_t atomic_num_segments_after_flush =
_rowset_writer->get_atomic_num_segment();
+RETURN_NOT_OK(_generate_delete_bitmap(atomic_num_segments_before_flush,
+ atomic_num_segments_after_flush));
DorisMetrics::instance()->memtable_flush_total->increment(1);
DorisMetrics::instance()->memtable_flush_duration_us->increment(duration_ns /
1000);
VLOG_CRITICAL << "after flush memtable for tablet: " << tablet_id()
diff --git a/be/src/olap/memtable.h b/be/src/olap/memtable.h
index 52589f35cc..e7f59ff151 100644
--- a/be/src/olap/memtable.h
+++ b/be/src/olap/memtable.h
@@ -146,7 +146,8 @@ private:
void _insert_one_row_from_block(RowInBlock* row_in_block);
void _aggregate_two_row_in_block(RowInBlock* new_row, RowInBlock*
row_in_skiplist);
-Status _generate_delete_bitmap();
+Status _generate_delete_bitmap(int64_t atomic_num_segments_before_flush,
+ int64_t atomic_num_segments_after_flush);
private:
TabletSharedPtr _tablet;
diff --git a/be/src/olap/rowset/beta_rowset.cpp
b/be/src/olap/rowset/beta_rowset.cpp
index 8ed48fdf70..f04fbf5922 100644
--- a/be/src/olap/rowset/beta_rowset.cpp
+++ b/be/src/olap/rowset/beta_rowset.cpp
@@ -130,20 +130,27 @@ Status
BetaRowset::load_segments(std::vector* segm
return Status::OK();
}
-Status BetaRowset::load_segment(int64_t seg_id, segment_v2::SegmentSharedPtr*
segment) {
-DCHECK(seg_id >= 0);
-auto fs
[doris] 05/21: [fix](explain) fix explain output format problem (#15019)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 315f0396857aaa5d93e062b27062417bb24a97e2
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Fri Dec 16 10:53:05 2022 +0800
[fix](explain) fix explain output format problem (#15019)
---
.../src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
index eb09d679e6..7a34084ec8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
@@ -244,7 +244,7 @@ public class NestedLoopJoinNode extends JoinNodeBase {
if (!runtimeFilters.isEmpty()) {
output.append(detailPrefix).append("runtime filters: ");
output.append(getRuntimeFilterExplainString(true));
-output.append("isOutputLeftSideOnly:
").append(isOutputLeftSideOnly).append("\n");
+output.append(detailPrefix).append("is output left side only:
").append(isOutputLeftSideOnly).append("\n");
}
output.append(detailPrefix).append(String.format("cardinality=%,d",
cardinality)).append("\n");
// todo unify in plan node
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 20/21: [bug](udaf) fix java udaf incorrect get null value with row (#15151)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6766c5135319b6902e2d94480de3ffc31ff4ff36
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Mon Dec 19 10:07:12 2022 +0800
[bug](udaf) fix java udaf incorrect get null value with row (#15151)
---
fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java
b/fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java
index 598886a26b..f684363bf6 100644
--- a/fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java
+++ b/fe/java-udf/src/main/java/org/apache/doris/udf/UdafExecutor.java
@@ -361,7 +361,8 @@ public class UdafExecutor {
for (int i = 0; i < argTypes.length; ++i) {
// skip the input column of current row is null
if (UdfUtils.UNSAFE.getLong(null,
UdfUtils.getAddressAtOffset(inputNullsPtrs, i)) != -1
-&& UdfUtils.UNSAFE.getByte(null,
UdfUtils.getAddressAtOffset(inputNullsPtrs, i) + row) == 1) {
+&& (UdfUtils.UNSAFE.getByte(null,
UdfUtils.UNSAFE.getLong(null,
+UdfUtils.getAddressAtOffset(inputNullsPtrs, i)) + row) ==
1)) {
inputObjects[i] = null;
continue;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 21/21: [fix](merge-on-write) delete all rows with same key in all pre segments (#14995)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 95aab248b03ed82b384cf0c2d3510f07b944dad3
Author: xueweizhang
AuthorDate: Mon Dec 19 10:08:38 2022 +0800
[fix](merge-on-write) delete all rows with same key in all pre segments
(#14995)
---
be/src/olap/tablet.cpp | 51 ++
be/src/olap/tablet.h | 3 ++-
2 files changed, 29 insertions(+), 25 deletions(-)
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index 11d125035e..e15bd61610 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -2032,10 +2032,10 @@ Status Tablet::calc_delete_bitmap(RowsetId rowset_id,
RowLocation loc;
// first check if exist in pre segment
if (check_pre_segments) {
-auto st = _check_pk_in_pre_segments(pre_segments, *key,
dummy_version,
-delete_bitmap, &loc);
+auto st = _check_pk_in_pre_segments(rowset_id,
pre_segments, *key,
+dummy_version,
delete_bitmap, &loc);
if (st.ok()) {
-delete_bitmap->add({loc.rowset_id, loc.segment_id,
dummy_version.first},
+delete_bitmap->add({rowset_id, loc.segment_id,
dummy_version.first},
loc.row_id);
++row_id;
continue;
@@ -2045,23 +2045,27 @@ Status Tablet::calc_delete_bitmap(RowsetId rowset_id,
continue;
}
}
-auto st = lookup_row_key(*key, specified_rowset_ids, &loc,
dummy_version.first - 1);
-CHECK(st.ok() || st.is_not_found() || st.is_already_exist());
-if (st.is_not_found()) {
-++row_id;
-continue;
-}
-// sequence id smaller than the previous one, so delete
current row
-if (st.is_already_exist()) {
-loc.rowset_id = rowset_id;
-loc.segment_id = seg->id();
-loc.row_id = row_id;
-}
+if (!specified_rowset_ids->empty()) {
+auto st = lookup_row_key(*key, specified_rowset_ids, &loc,
+ dummy_version.first - 1);
+CHECK(st.ok() || st.is_not_found() ||
st.is_already_exist());
+if (st.is_not_found()) {
+++row_id;
+continue;
+}
+// sequence id smaller than the previous one, so delete
current row
+if (st.is_already_exist()) {
+loc.rowset_id = rowset_id;
+loc.segment_id = seg->id();
+loc.row_id = row_id;
+}
+
+delete_bitmap->add({loc.rowset_id, loc.segment_id,
dummy_version.first},
+ loc.row_id);
+}
++row_id;
-delete_bitmap->add({loc.rowset_id, loc.segment_id,
dummy_version.first},
- loc.row_id);
}
remaining -= num_read;
}
@@ -2077,15 +2081,15 @@ Status Tablet::calc_delete_bitmap(RowsetId rowset_id,
}
Status Tablet::_check_pk_in_pre_segments(
-const std::vector& pre_segments, const
Slice& key,
-const Version& version, DeleteBitmapPtr delete_bitmap, RowLocation*
loc) {
+RowsetId rowset_id, const std::vector&
pre_segments,
+const Slice& key, const Version& version, DeleteBitmapPtr
delete_bitmap, RowLocation* loc) {
for (auto it = pre_segments.rbegin(); it != pre_segments.rend(); ++it) {
auto st = (*it)->lookup_row_key(key, loc);
CHECK(st.ok() || st.is_not_found() || st.is_already_exist());
if (st.is_not_found()) {
continue;
} else if (st.ok() && _schema->has_sequence_col() &&
- delete_bitmap->contains({loc->rowset_id, loc->segment_id,
version.first},
+ delete_bitmap->contains({rowset_id, loc->segment_id,
version.first},
loc->row_id)) {
// if has sequence col, we continue to compare the sequence_id of
// all segments, util we find an existing key.
@@ -2159,10 +2163,9 @@ Status Tablet::update_delete_bitmap(const
RowsetSharedPtr& rowset, DeleteBitmapP
for (const auto& to_del : rowset_ids_to_del) {
delete_bitmap->remove({to_del, 0, 0}, {to_del, UINT32_MAX, INT64_MAX});
}
-if (!rowset_ids_to_add.empty()) {
-RETURN_IF_ERROR(calc_delete_bitmap(row
[doris] branch branch-1.2-lts updated (ae084be649 -> 95aab248b0)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git from ae084be649 [Bug](function) fix overflow on concat_ws (#15043) new b027ae3490 [fix](merge-on-write) calc delete bitmap need all segments which _do_flush in one memtable (#15018) new 5fce045f51 [Enhancement](partition prune): calculate the column ranges of compound predicates (#14886) new a282d720c9 [Bug](datev2) Fix wrong result when use datev2 as partition key (#15094) new 12192e3f13 [fix](memtracker) Fix load channel memory tracker are not refreshed in time (#15048) new 315f039685 [fix](explain) fix explain output format problem (#15019) new e68a49f3fe [Bug](case function) do not crash if prepare failed (#15113) new d41a963c8f [fix](multi catalog)Return emtpy block while external table scanner couldn't find the file (#14997) new 3fe708cf56 [config](load) enable new load scan node by default (#14808) new 8853d515ae [vectorized](join) add try catch in create thread (#15065) new d23e7905b0 [fix](iceberg-v2) icebergv2 filter data path (#14470) new 2baa0e8891 [improvement](multi-catalog)Support invalid/not invalid option for refresh catalog and db. (#14922) new d6f4f08dcc [Bug](datev2) Fix compatible problems caused by datev2 (#15131) new f6936f256d [fix](type) forbid time type when creating table (#15093) new cc13983766 [vectorized](jdbc) fix external table of oracle with condition about … (#15092) new 4a1bcebf41 [fix](subquery)fix bug of using constexpr as subquery's output (#15119) new 22aacd849c [fix](load) delta writer init failed might cause data inconsistency between multiple replicas (#15058) new afbe5ba198 [fix](multi-catalog) hidden password for show create jdbc catalog (#15145) new 6234291506 [fix](schema) Fix create table error if Colocate tables not equal to bucket num (#15071) new d79b601796 [enhancement](signal) output query_id when 'be' core dumped #15080 new 6766c51353 [bug](udaf) fix java udaf incorrect get null value with row (#15151) new 95aab248b0 [fix](merge-on-write) delete all rows with same key in all pre segments (#14995) The 21 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/common/daemon.cpp | 9 +- be/src/http/action/stream_load.cpp | 1 + be/src/io/hdfs_file_reader.cpp | 3 + be/src/olap/delta_writer.cpp | 4 +- be/src/olap/memtable.cpp | 19 ++-- be/src/olap/memtable.h | 3 +- be/src/olap/rowset/beta_rowset.cpp | 35 --- be/src/olap/rowset/beta_rowset.h | 3 +- be/src/olap/rowset/beta_rowset_writer.h| 2 + be/src/olap/rowset/rowset_writer.h | 2 + be/src/olap/tablet.cpp | 51 +- be/src/olap/tablet.h | 3 +- be/src/runtime/fragment_mgr.cpp| 5 - be/src/runtime/raw_value.h | 11 +-- be/src/runtime/tablets_channel.cpp | 17 +++- be/src/runtime/thread_context.cpp | 3 + be/src/util/jsonb_parser.h | 25 +++-- be/src/vec/exec/format/parquet/parquet_common.h| 8 ++ be/src/vec/exec/format/parquet/schema_desc.cpp | 14 ++- be/src/vec/exec/format/parquet/schema_desc.h | 4 +- .../exec/format/parquet/vparquet_column_reader.cpp | 10 +- .../vec/exec/format/parquet/vparquet_page_index.h | 4 +- be/src/vec/exec/format/parquet/vparquet_reader.cpp | 58 +++- be/src/vec/exec/format/parquet/vparquet_reader.h | 3 +- be/src/vec/exec/format/table/iceberg_reader.cpp| 104 - be/src/vec/exec/format/table/iceberg_reader.h | 26 -- be/src/vec/exec/format/table/table_format_reader.h | 3 +- be/src/vec/exec/join/vjoin_node_base.cpp | 15 ++- be/src/vec/exec/scan/new_jdbc_scan_node.cpp| 2 +- be/src/vec/exec/scan/new_jdbc_scanner.cpp | 7 +- be/src/vec/exec/scan/new_jdbc_scanner.h| 2 +- be/src/vec/exec/scan/new_odbc_scan_node.cpp| 2 +- be/src/vec/exec/scan/new_odbc_scanner.cpp | 6 +- be/src/vec/exec/scan/new_odbc_scanner.h| 2 +- be/src/vec/exec/scan/scanner_scheduler.cpp | 12 ++- be/src/vec/exec/scan/vfile_scanner.cpp | 15 ++- be/src/vec/exprs/vcase_expr.cpp| 12 --- be/src/vec/functions/function_case.h | 9 +- fe/fe-core/src/main/cup/sql_parser.cup | 12 +-- .../org/apache/dori
[doris] 11/21: [improvement](multi-catalog)Support invalid/not invalid option for refresh catalog and db. (#14922)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2baa0e8891dfb9e1c90b6c546fe38191bbd71986
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Thu Dec 8 20:38:07 2022 +0800
[improvement](multi-catalog)Support invalid/not invalid option for refresh
catalog and db. (#14922)
Current refresh catalog/db operation always invalid all the related cache.
In some cases, it is not necessary,
for example, create a new db in external data source. This pr is to support
refresh without invalidate cache.
refresh catalog hive properties("invalid_cache" = "false");
refresh database hive.db1 properties("invalid_cache" = "false");
---
fe/fe-core/src/main/cup/sql_parser.cup | 12 ++--
.../org/apache/doris/analysis/RefreshCatalogStmt.java | 15 ++-
.../java/org/apache/doris/analysis/RefreshDbStmt.java | 18 --
.../java/org/apache/doris/catalog/RefreshManager.java | 7 ---
.../doris/catalog/external/ExternalDatabase.java | 8 ++--
.../org/apache/doris/datasource/CatalogFactory.java| 1 +
.../java/org/apache/doris/datasource/CatalogLog.java | 3 +++
.../java/org/apache/doris/datasource/CatalogMgr.java | 8
.../org/apache/doris/datasource/EsExternalCatalog.java | 2 +-
.../org/apache/doris/datasource/ExternalCatalog.java | 12 ++--
.../org/apache/doris/datasource/ExternalObjectLog.java | 3 +++
.../apache/doris/datasource/HMSExternalCatalog.java| 2 +-
.../apache/doris/datasource/JdbcExternalCatalog.java | 2 +-
13 files changed, 70 insertions(+), 23 deletions(-)
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup
b/fe/fe-core/src/main/cup/sql_parser.cup
index 6b3fff4bd5..5c209fe478 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -1137,13 +1137,13 @@ refresh_stmt ::=
{:
RESULT = new RefreshTableStmt(tbl);
:}
-| KW_REFRESH KW_DATABASE ident:db
+| KW_REFRESH KW_DATABASE ident:db opt_properties:properties
{:
-RESULT = new RefreshDbStmt(db);
+RESULT = new RefreshDbStmt(db, properties);
:}
-| KW_REFRESH KW_DATABASE ident:ctl DOT ident:db
+| KW_REFRESH KW_DATABASE ident:ctl DOT ident:db opt_properties:properties
{:
-RESULT = new RefreshDbStmt(ctl, db);
+RESULT = new RefreshDbStmt(ctl, db, properties);
:}
| KW_REFRESH KW_MATERIALIZED KW_VIEW table_name:mv
{:
@@ -1153,9 +1153,9 @@ refresh_stmt ::=
{:
RESULT = new RefreshMaterializedViewStmt(mv,
MVRefreshInfo.RefreshMethod.COMPLETE);
:}
-| KW_REFRESH KW_CATALOG ident:catalogName
+| KW_REFRESH KW_CATALOG ident:catalogName opt_properties:properties
{:
-RESULT = new RefreshCatalogStmt(catalogName);
+RESULT = new RefreshCatalogStmt(catalogName, properties);
:}
;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/RefreshCatalogStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/RefreshCatalogStmt.java
index 481ebf3b9f..e054dca050 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/RefreshCatalogStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/RefreshCatalogStmt.java
@@ -27,22 +27,32 @@ import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.qe.ConnectContext;
+import java.util.Map;
+
/**
* RefreshCatalogStmt
* Manually refresh the catalog metadata.
*/
public class RefreshCatalogStmt extends DdlStmt {
+private static final String INVALID_CACHE = "invalid_cache";
private final String catalogName;
+private Map properties;
+private boolean invalidCache = false;
-public RefreshCatalogStmt(String catalogName) {
+public RefreshCatalogStmt(String catalogName, Map
properties) {
this.catalogName = catalogName;
+this.properties = properties;
}
public String getCatalogName() {
return catalogName;
}
+public boolean isInvalidCache() {
+return invalidCache;
+}
+
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
@@ -56,6 +66,9 @@ public class RefreshCatalogStmt extends DdlStmt {
ErrorReport.reportAnalysisException(ErrorCode.ERR_CATALOG_ACCESS_DENIED,
analyzer.getQualifiedUser(), catalogName);
}
+String invalidConfig = properties == null ? null :
properties.get(INVALID_CACHE);
+// Default is to invalid cache.
+invalidCache = invalidConfig == null ? true :
invalidConfig.equalsIgnoreCase("true");
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/RefreshDbStmt.java
b/fe/fe-core/src/main/java/org/apach
[doris] 15/21: [fix](subquery)fix bug of using constexpr as subquery's output (#15119)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4a1bcebf411cba02228ea35347e5d39dd7569fba
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Fri Dec 16 21:58:58 2022 +0800
[fix](subquery)fix bug of using constexpr as subquery's output (#15119)
---
.../apache/doris/planner/SingleNodePlanner.java| 4 +++-
.../test_outer_join_with_subquery.out | 3 +++
.../test_outer_join_with_subquery.groovy | 25 ++
3 files changed, 31 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
index 5e6b779514..5e1973aaed 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
@@ -1608,7 +1608,9 @@ public class SingleNodePlanner {
}
// Set output smap of rootNode *before* creating a SelectNode for
proper resolution.
rootNode.setOutputSmap(outputSmap);
-
+if (rootNode instanceof UnionNode && ((UnionNode)
rootNode).isConstantUnion()) {
+rootNode.setWithoutTupleIsNullOutputSmap(outputSmap);
+}
//
rootNode.setOutputSmap(ExprSubstitutionMap.compose(inlineViewRef.getBaseTblSmap(),
// rootNode.getOutputSmap(), analyzer));
// Expr.substituteList(inlineViewRef.getViewStmt().getResultExprs(),
analyzer.getChangeResSmap());
diff --git
a/regression-test/data/correctness_p0/test_outer_join_with_subquery.out
b/regression-test/data/correctness_p0/test_outer_join_with_subquery.out
index 72d126351a..875d88ccc5 100644
--- a/regression-test/data/correctness_p0/test_outer_join_with_subquery.out
+++ b/regression-test/data/correctness_p0/test_outer_join_with_subquery.out
@@ -2,3 +2,6 @@
-- !select --
1
+-- !select --
+1
+
diff --git
a/regression-test/suites/correctness_p0/test_outer_join_with_subquery.groovy
b/regression-test/suites/correctness_p0/test_outer_join_with_subquery.groovy
index 59bb822e3b..97de116ca1 100644
--- a/regression-test/suites/correctness_p0/test_outer_join_with_subquery.groovy
+++ b/regression-test/suites/correctness_p0/test_outer_join_with_subquery.groovy
@@ -78,6 +78,31 @@ suite("test_outer_join_with_subquery") {
) as subq_2 on (subq_1.c1 = subq_2.c2);
"""
+qt_select"""
+with idm_org_table as (
+SELECT
+1 a
+)
+select
+a
+from
+test_outer_join_with_subquery_outerjoin_A
+where
+a in (
+SELECT
+a
+from
+test_outer_join_with_subquery_outerjoin_B
+where
+a = (
+select
+a
+from
+idm_org_table
+)
+);
+"""
+
sql """
drop table if exists test_outer_join_with_subquery_outerjoin_A;
"""
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 12/21: [Bug](datev2) Fix compatible problems caused by datev2 (#15131)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d6f4f08dcc14c435cb721e5009c162478a1ab25e
Author: Gabriel
AuthorDate: Fri Dec 16 21:52:39 2022 +0800
[Bug](datev2) Fix compatible problems caused by datev2 (#15131)
This bug is introduced by #15094
---
.../org/apache/doris/analysis/DateLiteral.java | 37 ++
.../org/apache/doris/catalog/PartitionKey.java | 10 ++
2 files changed, 47 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
index 2a6efebd40..2377351822 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
@@ -769,6 +769,33 @@ public class DateLiteral extends LiteralExpr {
}
}
+@Override
+public void checkValueValid() throws AnalysisException {
+if (year < 0 || year > ) {
+throw new AnalysisException("DateLiteral has invalid year value: "
+ year);
+}
+if (month < 1 || month > 12) {
+throw new AnalysisException("DateLiteral has invalid month value:
" + month);
+}
+if (day < 1 || day > 31) {
+throw new AnalysisException("DateLiteral has invalid day value: "
+ day);
+}
+if (type.isDatetimeV2() || type.isDatetime()) {
+if (hour < 0 || hour > 24) {
+throw new AnalysisException("DateLiteral has invalid hour
value: " + hour);
+}
+if (minute < 0 || minute > 60) {
+throw new AnalysisException("DateLiteral has invalid minute
value: " + minute);
+}
+if (second < 0 || second > 60) {
+throw new AnalysisException("DateLiteral has invalid second
value: " + second);
+}
+if (type.isDatetimeV2() && (microsecond < 0 || microsecond >
99)) {
+throw new AnalysisException("DateLiteral has invalid
microsecond value: " + microsecond);
+}
+}
+}
+
public static DateLiteral read(DataInput in) throws IOException {
DateLiteral literal = new DateLiteral();
literal.readFields(in);
@@ -1621,4 +1648,14 @@ public class DateLiteral extends LiteralExpr {
throw new AnalysisException("Datetime value is out of range: " +
dateStr);
}
}
+
+public void setMinValue() {
+year = 0;
+month = 1;
+day = 1;
+hour = 0;
+minute = 0;
+second = 0;
+microsecond = 0;
+}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionKey.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionKey.java
index 68b49c2974..6f26d4ff3a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionKey.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionKey.java
@@ -338,6 +338,16 @@ public class PartitionKey implements
Comparable, Writable {
if (type != PrimitiveType.DATETIMEV2) {
literal.setType(Type.fromPrimitiveType(type));
}
+if (type.isDateV2Type()) {
+try {
+literal.checkValueValid();
+} catch (AnalysisException e) {
+LOG.warn("Value {} for partition key [type = {}] is
invalid! This is a bug exists in Doris "
++ "1.2.0 and fixed since Doris 1.2.1. You should
create this table again using Doris "
++ "1.2.1+ .", literal.getStringValue(), type);
+((DateLiteral) literal).setMinValue();
+}
+}
keys.add(literal);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 14/21: [vectorized](jdbc) fix external table of oracle with condition about … (#15092)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit cc139837665d6f1fb3403642978f7ecd09ecca57
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Fri Dec 16 10:48:17 2022 +0800
[vectorized](jdbc) fix external table of oracle with condition about …
(#15092)
* [vectorized](jdbc) fix external table of oracle with condition about
datetime report error
* formatter
---
be/src/vec/exec/scan/new_jdbc_scan_node.cpp | 2 +-
be/src/vec/exec/scan/new_jdbc_scanner.cpp | 7 ++-
be/src/vec/exec/scan/new_jdbc_scanner.h | 2 +-
be/src/vec/exec/scan/new_odbc_scan_node.cpp | 2 +-
be/src/vec/exec/scan/new_odbc_scanner.cpp | 6 +-
be/src/vec/exec/scan/new_odbc_scanner.h | 2 +-
.../src/main/java/org/apache/doris/planner/OdbcScanNode.java | 8
7 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/be/src/vec/exec/scan/new_jdbc_scan_node.cpp
b/be/src/vec/exec/scan/new_jdbc_scan_node.cpp
index e86060ad41..15ddd607ba 100644
--- a/be/src/vec/exec/scan/new_jdbc_scan_node.cpp
+++ b/be/src/vec/exec/scan/new_jdbc_scan_node.cpp
@@ -52,7 +52,7 @@ Status NewJdbcScanNode::_init_scanners(std::list*
scanners) {
NewJdbcScanner* scanner =
new NewJdbcScanner(_state, this, _limit_per_scanner, _tuple_id,
_query_string);
_scanner_pool.add(scanner);
-RETURN_IF_ERROR(scanner->prepare(_state));
+RETURN_IF_ERROR(scanner->prepare(_state, _vconjunct_ctx_ptr.get()));
scanners->push_back(static_cast(scanner));
return Status::OK();
}
diff --git a/be/src/vec/exec/scan/new_jdbc_scanner.cpp
b/be/src/vec/exec/scan/new_jdbc_scanner.cpp
index b91300bf47..ef9004530d 100644
--- a/be/src/vec/exec/scan/new_jdbc_scanner.cpp
+++ b/be/src/vec/exec/scan/new_jdbc_scanner.cpp
@@ -27,8 +27,13 @@ NewJdbcScanner::NewJdbcScanner(RuntimeState* state,
NewJdbcScanNode* parent, int
_query_string(query_string),
_tuple_desc(nullptr) {}
-Status NewJdbcScanner::prepare(RuntimeState* state) {
+Status NewJdbcScanner::prepare(RuntimeState* state, VExprContext**
vconjunct_ctx_ptr) {
VLOG_CRITICAL << "NewJdbcScanner::Prepare";
+if (vconjunct_ctx_ptr != nullptr) {
+// Copy vconjunct_ctx_ptr from scan node to this scanner's
_vconjunct_ctx.
+RETURN_IF_ERROR((*vconjunct_ctx_ptr)->clone(state, &_vconjunct_ctx));
+}
+
if (_is_init) {
return Status::OK();
}
diff --git a/be/src/vec/exec/scan/new_jdbc_scanner.h
b/be/src/vec/exec/scan/new_jdbc_scanner.h
index 24c2649dfe..f5584cd5d0 100644
--- a/be/src/vec/exec/scan/new_jdbc_scanner.h
+++ b/be/src/vec/exec/scan/new_jdbc_scanner.h
@@ -32,7 +32,7 @@ public:
Status close(RuntimeState* state) override;
public:
-Status prepare(RuntimeState* state);
+Status prepare(RuntimeState* state, VExprContext** vconjunct_ctx_ptr);
protected:
Status _get_block_impl(RuntimeState* state, Block* block, bool* eos)
override;
diff --git a/be/src/vec/exec/scan/new_odbc_scan_node.cpp
b/be/src/vec/exec/scan/new_odbc_scan_node.cpp
index 571566b88a..db8f8bfc68 100644
--- a/be/src/vec/exec/scan/new_odbc_scan_node.cpp
+++ b/be/src/vec/exec/scan/new_odbc_scan_node.cpp
@@ -53,7 +53,7 @@ Status NewOdbcScanNode::_init_scanners(std::list*
scanners) {
}
NewOdbcScanner* scanner = new NewOdbcScanner(_state, this,
_limit_per_scanner, _odbc_scan_node);
_scanner_pool.add(scanner);
-RETURN_IF_ERROR(scanner->prepare(_state));
+RETURN_IF_ERROR(scanner->prepare(_state, _vconjunct_ctx_ptr.get()));
scanners->push_back(static_cast(scanner));
return Status::OK();
}
diff --git a/be/src/vec/exec/scan/new_odbc_scanner.cpp
b/be/src/vec/exec/scan/new_odbc_scanner.cpp
index 59c94cf887..71b1a8d525 100644
--- a/be/src/vec/exec/scan/new_odbc_scanner.cpp
+++ b/be/src/vec/exec/scan/new_odbc_scanner.cpp
@@ -36,8 +36,12 @@ NewOdbcScanner::NewOdbcScanner(RuntimeState* state,
NewOdbcScanNode* parent, int
_tuple_id(odbc_scan_node.tuple_id),
_tuple_desc(nullptr) {}
-Status NewOdbcScanner::prepare(RuntimeState* state) {
+Status NewOdbcScanner::prepare(RuntimeState* state, VExprContext**
vconjunct_ctx_ptr) {
VLOG_CRITICAL << NEW_SCANNER_TYPE << "::prepare";
+if (vconjunct_ctx_ptr != nullptr) {
+// Copy vconjunct_ctx_ptr from scan node to this scanner's
_vconjunct_ctx.
+RETURN_IF_ERROR((*vconjunct_ctx_ptr)->clone(state, &_vconjunct_ctx));
+}
if (_is_init) {
return Status::OK();
diff --git a/be/src/vec/exec/scan/new_odbc_scanner.h
b/be/src/vec/exec/scan/new_odbc_scanner.h
index 0b8d28ed99..012fc8b3c1 100644
--- a/be/src/vec/exec/scan/new_odbc_scanner.h
+++ b/be/src/vec/exec/scan/new_odbc_scanner.h
@@ -34,
