Re: [PR] [Chore](runtime-filter) adjust some check and error msg on runtime filter [doris]
BiteThet merged PR #35018: URL: https://github.com/apache/doris/pull/35018 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Chore](runtime-filter) adjust some check and error msg on runtime filter (#35018)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8f0ecf50d7b [Chore](runtime-filter) adjust some check and error msg on
runtime filter (#35018)
8f0ecf50d7b is described below
commit 8f0ecf50d7b8f8da0fca4f401e78965d75083068
Author: Pxl
AuthorDate: Sun May 19 15:48:11 2024 +0800
[Chore](runtime-filter) adjust some check and error msg on runtime filter
(#35018)
adjust some check and error msg on runtime filter
---
be/src/exprs/bloom_filter_func.h| 7 ---
be/src/exprs/runtime_filter.cpp | 24
be/src/exprs/runtime_filter.h | 9 -
be/src/exprs/runtime_filter_slots.h | 4
be/src/pipeline/dependency.cpp | 2 +-
be/src/runtime/runtime_filter_mgr.cpp | 8 +++-
be/src/vec/exec/runtime_filter_consumer.cpp | 2 +-
7 files changed, 33 insertions(+), 23 deletions(-)
diff --git a/be/src/exprs/bloom_filter_func.h b/be/src/exprs/bloom_filter_func.h
index bc56c7b505a..a831395a5ea 100644
--- a/be/src/exprs/bloom_filter_func.h
+++ b/be/src/exprs/bloom_filter_func.h
@@ -133,11 +133,6 @@ public:
}
Status init_with_fixed_length(int64_t bloom_filter_length) {
-if (_inited) {
-return Status::OK();
-}
-// TODO: really need the lock?
-std::lock_guard l(_lock);
if (_inited) {
return Status::OK();
}
@@ -154,7 +149,6 @@ public:
// If `_inited` is false, there is no memory allocated in bloom filter
and this is the first
// call for `merge` function. So we just reuse this bloom filter, and
we don't need to
// allocate memory again.
-std::lock_guard l(_lock);
if (!_inited) {
auto* other_func =
static_cast(bloomfilter_func);
DCHECK(_bloom_filter == nullptr);
@@ -228,7 +222,6 @@ protected:
int32_t _bloom_filter_alloced;
std::shared_ptr _bloom_filter;
bool _inited {};
-std::mutex _lock;
int64_t _bloom_filter_length;
bool _build_bf_exactly = false;
bool _bloom_filter_size_calculated_by_ndv = false;
diff --git a/be/src/exprs/runtime_filter.cpp b/be/src/exprs/runtime_filter.cpp
index 53b9953e346..fb81af5396b 100644
--- a/be/src/exprs/runtime_filter.cpp
+++ b/be/src/exprs/runtime_filter.cpp
@@ -364,9 +364,11 @@ public:
}
bool get_build_bf_cardinality() const {
-DCHECK(_filter_type == RuntimeFilterType::BLOOM_FILTER ||
- _filter_type == RuntimeFilterType::IN_OR_BLOOM_FILTER);
-return _context->bloom_filter_func->get_build_bf_cardinality();
+if (_filter_type == RuntimeFilterType::BLOOM_FILTER ||
+_filter_type == RuntimeFilterType::IN_OR_BLOOM_FILTER) {
+return _context->bloom_filter_func->get_build_bf_cardinality();
+}
+return false;
}
void insert_to_bloom_filter(BloomFilterFuncBase* bloom_filter) const {
@@ -1522,15 +1524,21 @@ void
IRuntimeFilter::update_runtime_filter_type_to_profile() {
_profile->add_info_string("RealRuntimeFilterType",
to_string(_wrapper->get_real_type()));
}
+std::string IRuntimeFilter::debug_string() const {
+return fmt::format(
+"RuntimeFilter: (id = {}, type = {}, need_local_merge: {},
is_broadcast: {}, "
+"build_bf_cardinality: {}",
+_filter_id, to_string(_runtime_filter_type), _need_local_merge,
_is_broadcast_join,
+_wrapper->get_build_bf_cardinality());
+}
+
Status IRuntimeFilter::merge_from(const RuntimePredicateWrapper* wrapper) {
auto status = _wrapper->merge(wrapper);
if (!status) {
-LOG(WARNING) << "runtime filter merge failed: " << _name
- << " ,need_local_merge: " << _need_local_merge
- << " ,is_broadcast: " << _is_broadcast_join;
-DCHECK(false); // rpc response is often ignored, so let it crash
directly here
+return Status::InternalError("runtime filter merge failed: {},
error_msg: {}",
+ debug_string(), status.msg());
}
-return status;
+return Status::OK();
}
template
diff --git a/be/src/exprs/runtime_filter.h b/be/src/exprs/runtime_filter.h
index 781f7ac34ff..1186c01db06 100644
--- a/be/src/exprs/runtime_filter.h
+++ b/be/src/exprs/runtime_filter.h
@@ -209,9 +209,9 @@ public:
_rf_wait_time_ms(_state->runtime_filter_wait_time_ms),
_enable_pipeline_exec(_state->enable_pipeline_exec),
_runtime_filter_type(get_runtime_filter_type(desc)),
- _name(fmt::format("RuntimeFilter: (id = {}, type = {})",
_filter_id,
-to_string(_runtime_filter_type))),
- _profile(new RuntimeProfile(_name)),
+
Re: [PR] [improve](http action) add http interface to calculate the crc of all files in tablet [doris]
csun5285 commented on PR #34915: URL: https://github.com/apache/doris/pull/34915#issuecomment-2119153407 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [chore](femetaversion) add a check in fe code to avoid fe meta version changed during pick PR [doris]
yiguolei opened a new pull request, #35039: URL: https://github.com/apache/doris/pull/35039 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](femetaversion) add a check in fe code to avoid fe meta version changed during pick PR [doris]
doris-robot commented on PR #35039: URL: https://github.com/apache/doris/pull/35039#issuecomment-2119171614 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](femetaversion) add a check in fe code to avoid fe meta version changed during pick PR [doris]
yiguolei commented on PR #35039: URL: https://github.com/apache/doris/pull/35039#issuecomment-2119171687 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](femetaversion) add a check in fe code to avoid fe meta version changed during pick PR [doris]
yiguolei commented on PR #35039: URL: https://github.com/apache/doris/pull/35039#issuecomment-2119172882 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) fix DCHECK failure of agg when failed to alloc memory [doris]
jacktengg commented on PR #35011: URL: https://github.com/apache/doris/pull/35011#issuecomment-2119173579 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)Misspelling of class names [doris]
wuwenchi commented on PR #34981: URL: https://github.com/apache/doris/pull/34981#issuecomment-2119174635 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) fix DCHECK failure of agg when failed to alloc memory [doris]
github-actions[bot] commented on PR #35011: URL: https://github.com/apache/doris/pull/35011#issuecomment-2119174921 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) fix DCHECK failure of agg when failed to alloc memory [doris]
github-actions[bot] commented on PR #35011: URL: https://github.com/apache/doris/pull/35011#issuecomment-2119174981 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](multi-table-load) fix be core when partial table load failed [doris]
HHoflittlefish777 commented on PR #34712: URL: https://github.com/apache/doris/pull/34712#issuecomment-2119183535 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](fuzzy) disable spill in fuzzy mode [doris]
mrhhsg commented on PR #35033: URL: https://github.com/apache/doris/pull/35033#issuecomment-2119183848 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](spill) handle oom exception in spill tasks [doris]
mrhhsg closed pull request #34960: [opt](spill) handle oom exception in spill tasks URL: https://github.com/apache/doris/pull/34960 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](multi-table-load) fix be core when partial table load failed [doris]
doris-robot commented on PR #34712: URL: https://github.com/apache/doris/pull/34712#issuecomment-2119191571 TeamCity be ut coverage result: Function Coverage: 35.70% (9007/25228) Line Coverage: 27.36% (74499/272290) Region Coverage: 26.60% (38529/144830) Branch Coverage: 23.43% (19660/83904) Coverage Report: http://coverage.selectdb-in.cc/coverage/bfcbfa0b901bb519510d8be5dbf0c0ba52cab2bf_bfcbfa0b901bb519510d8be5dbf0c0ba52cab2bf/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](femetaversion) add a check in fe code to avoid fe meta version changed during pick PR [doris]
yiguolei commented on PR #35039: URL: https://github.com/apache/doris/pull/35039#issuecomment-2119192514 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](status) refactor querycontext and runtime state status [doris]
github-actions[bot] commented on PR #35035: URL: https://github.com/apache/doris/pull/35035#issuecomment-2119193443 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](spill) handle oom exception in spill tasks [doris]
mrhhsg commented on PR #35025: URL: https://github.com/apache/doris/pull/35025#issuecomment-2119201085 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](status) refactor querycontext and runtime state status [doris]
yiguolei commented on PR #35035: URL: https://github.com/apache/doris/pull/35035#issuecomment-2119203723 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (22f85be7126 -> a6a398d7a4f)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from 22f85be7126 [fix](hive-ctas) support create hive table with full quolified name (#34984) add a6a398d7a4f [Fix](function) remove datev2 signature of microsecond #35017 No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/analysis/FunctionCallExpr.java | 16 .../trees/expressions/functions/scalar/Microsecond.java | 4 +--- .../datetime_functions/test_date_function.out| 3 +++ .../datetime_functions/test_date_function.groovy | 1 + 4 files changed, 13 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](status) refactor querycontext and runtime state status [doris]
doris-robot commented on PR #35035: URL: https://github.com/apache/doris/pull/35035#issuecomment-2119217347 TeamCity be ut coverage result: Function Coverage: 35.72% (9010/25224) Line Coverage: 27.37% (74506/272216) Region Coverage: 26.62% (38537/144793) Branch Coverage: 23.45% (19667/83884) Coverage Report: http://coverage.selectdb-in.cc/coverage/a5cfed1b1acd895899e3c2f1da424bf3638b5ff8_a5cfed1b1acd895899e3c2f1da424bf3638b5ff8/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](Variant) fix incorrect use of column index in TabletSchema [doris]
xiaokang merged PR #35019: URL: https://github.com/apache/doris/pull/35019 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Bug](Variant) fix incorrect use of column index in TabletSchema (#35019)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c0865cc7c0b [Bug](Variant) fix incorrect use of column index in
TabletSchema (#35019)
c0865cc7c0b is described below
commit c0865cc7c0b884e9a681c9620a77f6a9e9da9511
Author: lihangyu <15605149...@163.com>
AuthorDate: Sun May 19 20:47:10 2024 +0800
[Bug](Variant) fix incorrect use of column index in TabletSchema (#35019)
---
be/src/vec/common/schema_util.cpp | 4 +-
.../test_alter_add_drop_column.groovy | 53 ++
2 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/common/schema_util.cpp
b/be/src/vec/common/schema_util.cpp
index 2f9e5ded212..49e901e0846 100644
--- a/be/src/vec/common/schema_util.cpp
+++ b/be/src/vec/common/schema_util.cpp
@@ -395,9 +395,9 @@ void inherit_root_attributes(TabletSchemaSPtr& schema) {
col.type() != FieldType::OLAP_FIELD_TYPE_DOUBLE &&
col.type() != FieldType::OLAP_FIELD_TYPE_FLOAT) {
// above types are not supported in bf
-
col.set_is_bf_column(schema->column(col.parent_unique_id()).is_bf_column());
+
col.set_is_bf_column(schema->column_by_uid(col.parent_unique_id()).is_bf_column());
}
-
col.set_aggregation_method(schema->column(col.parent_unique_id()).aggregation());
+
col.set_aggregation_method(schema->column_by_uid(col.parent_unique_id()).aggregation());
auto it = variants_index_meta.find(col.parent_unique_id());
// variant has no index meta, ignore
if (it == variants_index_meta.end()) {
diff --git
a/regression-test/suites/variant_p0/schema_change/test_alter_add_drop_column.groovy
b/regression-test/suites/variant_p0/schema_change/test_alter_add_drop_column.groovy
new file mode 100644
index 000..6a38ec2804e
--- /dev/null
+++
b/regression-test/suites/variant_p0/schema_change/test_alter_add_drop_column.groovy
@@ -0,0 +1,53 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("regression_test_variant_add_drop_column", "variant_type"){
+def table_name = "variant_add_drop_column"
+sql "DROP TABLE IF EXISTS ${table_name}"
+sql """
+CREATE TABLE IF NOT EXISTS ${table_name} (
+k bigint,
+v variant
+)
+DUPLICATE KEY(`k`)
+DISTRIBUTED BY HASH(k) BUCKETS 1
+properties("replication_num" = "1", "bloom_filter_columns" = "v");
+"""
+sql """insert into variant_add_drop_column values (1, '{"a" : 12345,"b" :
2}')"""
+
+sql "alter table variant_add_drop_column add column v2 variant default
null"
+sql "alter table variant_add_drop_column add column t1 datetime default
null"
+sql "alter table variant_add_drop_column add column t2 datetime default
null"
+sql """insert into variant_add_drop_column values (1, '{"a" :
12345234567,"b" : 2}', '{"xxx" : 1}', "2021-01-01 01:01:01", "2021-01-01
01:01:01")"""
+sql "alter table variant_add_drop_column add column i1 int default null"
+sql """insert into variant_add_drop_column values (1, '{"a" : 12345,"b" :
2}', '{"xxx" : 1}', "2021-01-01 01:01:01", "2021-01-01 01:01:01", 12345)"""
+sql "alter table variant_add_drop_column drop column t1"
+sql """insert into variant_add_drop_column values (1, '{"a" : 12345,"b" :
2}', '{"xxx" : 1}', "2021-01-01 01:01:01", 12345)"""
+sql "alter table variant_add_drop_column drop column t2"
+sql """insert into variant_add_drop_column values (1, '{"a" : 12345,"b" :
2}', '{"xxx" : 1}', 12345)"""
+sql "alter table variant_add_drop_column drop column i1"
+sql """insert into variant_add_drop_column values (1, '{"a" : 12345,"b" :
2}', '{"xxx" : 1}')"""
+sql "alter table variant_add_drop_column drop column v"
+sql """insert into variant_add_drop_column values (1, '{"a" : 12345,"b" :
2}')"""
+sql "alter table variant_add_drop_column add column v variant default null"
+sql """insert into variant_add_drop_column values (1, '{"a" : 12345,"b" :
2}', '{"a" : 12345,"b" : 2}')
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
xinyiZzz commented on PR #34985: URL: https://github.com/apache/doris/pull/34985#issuecomment-2119252875 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
github-actions[bot] commented on PR #34985: URL: https://github.com/apache/doris/pull/34985#issuecomment-2119254918 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Paimon) support deletion vector for Paimon naive reader [doris]
morningman commented on PR #34743: URL: https://github.com/apache/doris/pull/34743#issuecomment-2119260211 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Paimon) support deletion vector for Paimon naive reader [doris]
github-actions[bot] commented on PR #34743: URL: https://github.com/apache/doris/pull/34743#issuecomment-2119261994 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
doris-robot commented on PR #34985: URL: https://github.com/apache/doris/pull/34985#issuecomment-2119265042 TeamCity be ut coverage result: Function Coverage: 35.70% (9011/25240) Line Coverage: 27.35% (74507/272402) Region Coverage: 26.61% (38538/144824) Branch Coverage: 23.44% (19664/83900) Coverage Report: http://coverage.selectdb-in.cc/coverage/778b81ef9a7f66b489fc8b31f590879289a78b3d_778b81ef9a7f66b489fc8b31f590879289a78b3d/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Paimon) support deletion vector for Paimon naive reader [doris]
doris-robot commented on PR #34743: URL: https://github.com/apache/doris/pull/34743#issuecomment-2119291152 TeamCity be ut coverage result: Function Coverage: 35.69% (9008/25242) Line Coverage: 27.35% (74502/272394) Region Coverage: 26.59% (38531/144885) Branch Coverage: 23.43% (19663/83920) Coverage Report: http://coverage.selectdb-in.cc/coverage/f39a510abda2fd1f788aeb0fa3ffe1a2e8ff627d_f39a510abda2fd1f788aeb0fa3ffe1a2e8ff627d/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](metrics)support be jvm metrics. [doris]
hubgeter commented on PR #35023: URL: https://github.com/apache/doris/pull/35023#issuecomment-2119295968 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](spill) handle oom exception in spill tasks [doris]
mrhhsg commented on PR #35025: URL: https://github.com/apache/doris/pull/35025#issuecomment-2119389289 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](fuzzy) disable spill in fuzzy mode [doris]
mrhhsg commented on PR #35033: URL: https://github.com/apache/doris/pull/35033#issuecomment-2119389349 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](spill) handle oom exception in spill tasks [doris]
doris-robot commented on PR #35025: URL: https://github.com/apache/doris/pull/35025#issuecomment-2119456565 TeamCity be ut coverage result: Function Coverage: 35.69% (9008/25241) Line Coverage: 27.36% (74501/272329) Region Coverage: 26.60% (38536/144896) Branch Coverage: 23.44% (19664/83906) Coverage Report: http://coverage.selectdb-in.cc/coverage/641fa29a75b1c5c83c378e5ff6e4525c52c58d81_641fa29a75b1c5c83c378e5ff6e4525c52c58d81/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](be) the mapped was not initialized when the agg state was destroyed [doris]
github-actions[bot] commented on PR #27313: URL: https://github.com/apache/doris/pull/27313#issuecomment-2119464879 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](multicatalog)support apche druid catalog [doris]
github-actions[bot] closed pull request #27270: [feature-wip](multicatalog)support apche druid catalog URL: https://github.com/apache/doris/pull/27270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](backup) add property for backup to ignore table error [doris]
github-actions[bot] closed pull request #27085: [enhancement](backup) add property for backup to ignore table error URL: https://github.com/apache/doris/pull/27085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] 是否支持血缘atlas插件 [doris]
heichong commented on issue #34577: URL: https://github.com/apache/doris/issues/34577#issuecomment-2119482052 这个确实有需要 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](thrift)Def topn filter in thirft [doris]
englefly commented on PR #35021: URL: https://github.com/apache/doris/pull/35021#issuecomment-2119495756 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [featrue](expr) check expr return type and block column type [doris]
Mryange commented on PR #35032: URL: https://github.com/apache/doris/pull/35032#issuecomment-2119497659 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids)choose agg mv in cbo [doris]
github-actions[bot] commented on PR #35020: URL: https://github.com/apache/doris/pull/35020#issuecomment-2119497876 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids)choose agg mv in cbo [doris]
github-actions[bot] commented on PR #35020: URL: https://github.com/apache/doris/pull/35020#issuecomment-2119497895 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](thrift)Def topn filter in thirft [doris]
github-actions[bot] commented on PR #35021: URL: https://github.com/apache/doris/pull/35021#issuecomment-2119498811 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](spill) handle oom exception in spill tasks [doris]
mrhhsg commented on PR #35025: URL: https://github.com/apache/doris/pull/35025#issuecomment-2119500313 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](datalake) return the error status instead of static_cast [doris]
github-actions[bot] commented on PR #34873: URL: https://github.com/apache/doris/pull/34873#issuecomment-2119503302 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Pick 2.1](inverted index) fix race condition for column reader load inverted index reader [doris]
airborne12 opened a new pull request, #35040: URL: https://github.com/apache/doris/pull/35040 ## Proposed changes Issue Number: close #xxx pick from (#34922) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Pick 2.1](inverted index) fix race condition for column reader load inverted index reader [doris]
doris-robot commented on PR #35040: URL: https://github.com/apache/doris/pull/35040#issuecomment-2119513141 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Pick 2.1](inverted index) fix race condition for column reader load inverted index reader [doris]
github-actions[bot] commented on PR #35040: URL: https://github.com/apache/doris/pull/35040#issuecomment-2119519155 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv) Fix table id overturn and optimize get table qualifier method [doris]
seawinde commented on PR #34768: URL: https://github.com/apache/doris/pull/34768#issuecomment-2119521539 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](test) inject fault during allocating memory [doris]
yiguolei commented on PR #34911: URL: https://github.com/apache/doris/pull/34911#issuecomment-2119529900 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Pick 2.1](inverted index) fix race condition for column reader load inverted index reader [doris]
yiguolei merged PR #35040: URL: https://github.com/apache/doris/pull/35040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [Fix](inverted index) fix race condition for column reader load inverted index reader (#34922) (#35040)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 7aff10b93ba [Fix](inverted index) fix race condition for column reader
load inverted index reader (#34922) (#35040)
7aff10b93ba is described below
commit 7aff10b93bad45b7a8e191be130a25c1d96f0b6e
Author: airborne12
AuthorDate: Mon May 20 09:52:22 2024 +0800
[Fix](inverted index) fix race condition for column reader load inverted
index reader (#34922) (#35040)
---
be/src/olap/rowset/segment_v2/column_reader.cpp | 11 +++
be/src/olap/rowset/segment_v2/column_reader.h | 2 +-
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/column_reader.cpp
b/be/src/olap/rowset/segment_v2/column_reader.cpp
index 9d19d1a64b4..6d5c28b8d6d 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/column_reader.cpp
@@ -262,9 +262,12 @@ Status ColumnReader::new_inverted_index_iterator(
std::shared_ptr index_file_reader, const
TabletIndex* index_meta,
const StorageReadOptions& read_options,
std::unique_ptr* iterator) {
RETURN_IF_ERROR(_ensure_inverted_index_loaded(index_file_reader,
index_meta));
-if (_inverted_index) {
-RETURN_IF_ERROR(_inverted_index->new_iterator(read_options.stats,
-
read_options.runtime_state, iterator));
+{
+std::shared_lock rlock(_load_index_lock);
+if (_inverted_index) {
+RETURN_IF_ERROR(_inverted_index->new_iterator(read_options.stats,
+
read_options.runtime_state, iterator));
+}
}
return Status::OK();
}
@@ -536,7 +539,7 @@ Status ColumnReader::_load_bitmap_index(bool
use_page_cache, bool kept_in_memory
Status ColumnReader::_load_inverted_index_index(
std::shared_ptr index_file_reader, const
TabletIndex* index_meta) {
-std::lock_guard wlock(_load_index_lock);
+std::unique_lock wlock(_load_index_lock);
if (_inverted_index && index_meta &&
_inverted_index->get_index_id() == index_meta->index_id()) {
diff --git a/be/src/olap/rowset/segment_v2/column_reader.h
b/be/src/olap/rowset/segment_v2/column_reader.h
index 9c889b848fd..30f916d00cd 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.h
+++ b/be/src/olap/rowset/segment_v2/column_reader.h
@@ -249,7 +249,7 @@ private:
// meta for various column indexes (null if the index is absent)
std::unique_ptr _segment_zone_map;
-mutable std::mutex _load_index_lock;
+mutable std::shared_mutex _load_index_lock;
std::unique_ptr _zone_map_index;
std::unique_ptr _ordinal_index;
std::unique_ptr _bitmap_index;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](parquet-reader) Fix INT96 timestamp min-max statistics is incorrect when was written by some old parquet writers by disable it. [doris]
kaka11chen opened a new pull request, #35041: URL: https://github.com/apache/doris/pull/35041 ## Proposed changes [Fix] (parquet-reader) Fix INT96 timestamp min-max statistics is incorrect when was written by some old parquet writers by disable it. Parquet INT96 timestamp values were compared incorrectly for the purposes of producing statistics by older parquet writers, so PARQUET-1065 deprecated them. The result is that any writer that produced stats was producing unusable incorrect values, except the special case where min == max and an incorrect ordering would not be material to the result. PARQUET-1026 made binary stats available and valid in that special case. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](parquet-reader) Fix INT96 timestamp min-max statistics is incorrect when was written by some old parquet writers by disable it. [doris]
doris-robot commented on PR #35041: URL: https://github.com/apache/doris/pull/35041#issuecomment-2119532880 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](test) inject fault during allocating memory [doris]
github-actions[bot] commented on PR #34911: URL: https://github.com/apache/doris/pull/34911#issuecomment-2119533949 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [exec](performance) opt the topn nullable column order performance in Heap Sort [doris]
HappenLee opened a new pull request, #35042: URL: https://github.com/apache/doris/pull/35042 ## Proposed changes Reduce the virtual Function call in nullable column order by Before： ``` exec 4.6s ``` After: ``` exec 2.7s ``` ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](performance) opt the topn nullable column order performance in Heap Sort [doris]
doris-robot commented on PR #35042: URL: https://github.com/apache/doris/pull/35042#issuecomment-2119534227 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](performance) opt the topn nullable column order performance in Heap Sort [doris]
HappenLee commented on PR #35042: URL: https://github.com/apache/doris/pull/35042#issuecomment-2119534284 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
morningman merged PR #34257: URL: https://github.com/apache/doris/pull/34257 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (c0865cc7c0b -> 09cb8b93845)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from c0865cc7c0b [Bug](Variant) fix incorrect use of column index in
TabletSchema (#35019)
add 09cb8b93845 [feature](iceberg)support iceberg insert grammar (#34257)
No new revisions were added by this update.
Summary of changes:
.../datasource/iceberg/IcebergExternalCatalog.java | 5 +-
.../datasource/iceberg/IcebergExternalTable.java | 16 +++
.../datasource/iceberg/IcebergTransaction.java | 4 +
.../doris/datasource/iceberg/IcebergUtils.java | 58 +++-
.../iceberg/source/IcebergApiSource.java | 11 +-
.../iceberg/source/IcebergHMSSource.java | 12 +-
.../datasource/iceberg/source/IcebergSource.java | 3 -
...Sink.java => UnboundBaseExternalTableSink.java} | 68 +++---
.../nereids/analyzer/UnboundHiveTableSink.java | 91 ++---
.../nereids/analyzer/UnboundIcebergTableSink.java | 84
.../nereids/analyzer/UnboundTableSinkCreator.java | 9 ++
.../glue/translator/PhysicalPlanTranslator.java| 25 +++-
.../pre/TurnOffPageCacheForInsertIntoSelect.java | 8 ++
.../nereids/properties/RequestPropertyDeriver.java | 12 ++
.../org/apache/doris/nereids/rules/RuleSet.java| 2 +
.../org/apache/doris/nereids/rules/RuleType.java | 2 +
.../doris/nereids/rules/analysis/BindSink.java | 68 +-
...IcebergTableSinkToPhysicalIcebergTableSink.java | 48 +++
.../apache/doris/nereids/trees/plans/PlanType.java | 2 +
... => BaseExternalTableInsertCommandContext.java} | 24 +---
...r.java => BaseExternalTableInsertExecutor.java} | 95 ++---
.../commands/insert/HiveInsertCommandContext.java | 11 +-
.../plans/commands/insert/HiveInsertExecutor.java | 110 ++-
.../commands/insert/IcebergInsertExecutor.java | 71 ++
.../commands/insert/InsertIntoTableCommand.java| 9 +-
.../trees/plans/commands/insert/InsertUtils.java | 6 +-
.../plans/logical/LogicalIcebergTableSink.java | 150 +
.../physical/PhysicalBaseExternalTableSink.java| 79 +++
.../plans/physical/PhysicalHiveTableSink.java | 50 ++-
...ableSink.java => PhysicalIcebergTableSink.java} | 93 +
.../nereids/trees/plans/visitor/SinkVisitor.java | 15 +++
.../doris/planner/BaseExternalTableDataSink.java | 97 +
.../java/org/apache/doris/planner/DataSink.java| 3 +
.../org/apache/doris/planner/HiveTableSink.java| 72 +++---
.../org/apache/doris/planner/IcebergTableSink.java | 146
.../main/java/org/apache/doris/qe/Coordinator.java | 17 +++
.../transaction/IcebergTransactionManager.java | 69 ++
.../transaction/TransactionManagerFactory.java | 5 +
.../datasource/hive/HiveDDLAndDMLPlanTest.java | 27 +++-
.../doris/datasource/hive/HmsCommitTest.java | 1 -
gensrc/thrift/DataSinks.thrift | 24
41 files changed, 1198 insertions(+), 504 deletions(-)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/{UnboundHiveTableSink.java
=> UnboundBaseExternalTableSink.java} (56%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundIcebergTableSink.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalIcebergTableSinkToPhysicalIcebergTableSink.java
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/{HiveInsertCommandContext.java
=> BaseExternalTableInsertCommandContext.java} (67%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/{HiveInsertExecutor.java
=> BaseExternalTableInsertExecutor.java} (66%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/IcebergInsertExecutor.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalIcebergTableSink.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalBaseExternalTableSink.java
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/{PhysicalHiveTableSink.java
=> PhysicalIcebergTableSink.java} (56%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/BaseExternalTableDataSink.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/IcebergTableSink.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/transaction/IcebergTransactionManager.java
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](parquet-reader) Fix INT96 timestamp min-max statistics is incorrect when was written by some old parquet writers by disable it. [doris]
github-actions[bot] commented on PR #35041: URL: https://github.com/apache/doris/pull/35041#issuecomment-2119534789 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](parquet-reader) Fix INT96 timestamp min-max statistics is incorrect when was written by some old parquet writers by disable it. [doris]
github-actions[bot] commented on PR #35041: URL: https://github.com/apache/doris/pull/35041#issuecomment-2119534816 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) fix DCHECK failure of agg when failed to alloc memory [doris]
github-actions[bot] commented on PR #35011: URL: https://github.com/apache/doris/pull/35011#issuecomment-2119537029 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) fix DCHECK failure of agg when failed to alloc memory [doris]
github-actions[bot] commented on PR #35011: URL: https://github.com/apache/doris/pull/35011#issuecomment-2119537053 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](status) refactor querycontext and runtime state status [doris]
github-actions[bot] commented on PR #35035: URL: https://github.com/apache/doris/pull/35035#issuecomment-2119537088 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](status) refactor querycontext and runtime state status [doris]
github-actions[bot] commented on PR #35035: URL: https://github.com/apache/doris/pull/35035#issuecomment-2119537112 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](parquet-reader) Fix INT96 timestamp min-max statistics is incorrect when was written by some old parquet writers by disable it. [doris]
github-actions[bot] commented on PR #35041: URL: https://github.com/apache/doris/pull/35041#issuecomment-2119537211 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](parquet-reader) Fix INT96 timestamp min-max statistics is incorrect when was written by some old parquet writers by disable it. [doris]
kaka11chen commented on PR #35041: URL: https://github.com/apache/doris/pull/35041#issuecomment-2119537612 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](performance) opt the topn nullable column order performance in Heap Sort [doris]
github-actions[bot] commented on PR #35042: URL: https://github.com/apache/doris/pull/35042#issuecomment-2119538085 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): Reject Commutativity Swap for Nested Loop Joins Affecting Parallelism #34639 [doris]
XieJiann commented on PR #34996: URL: https://github.com/apache/doris/pull/34996#issuecomment-2119539195 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv) Fix table id overturn and optimize get table qualifier method [doris]
seawinde commented on PR #34768: URL: https://github.com/apache/doris/pull/34768#issuecomment-2119540139 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids)choose agg mv in cbo [doris]
englefly merged PR #35020: URL: https://github.com/apache/doris/pull/35020 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (09cb8b93845 -> e34fb0f857d)
This is an automated email from the ASF dual-hosted git repository. englefly pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 09cb8b93845 [feature](iceberg)support iceberg insert grammar (#34257) add e34fb0f857d [feat](Nereids)choose agg mv in cbo #35020 No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/nereids/cost/CostModelV1.java | 13 - 1 file changed, 12 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [update](hudi) update hudi-spark bundle to 3.4.3 [doris]
AshinGau commented on PR #35013: URL: https://github.com/apache/doris/pull/35013#issuecomment-2119541459 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (e34fb0f857d -> fb89a46003e)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from e34fb0f857d [feat](Nereids)choose agg mv in cbo #35020 add fb89a46003e [fix](agg) fix DCHECK failure of agg when failed to alloc memory (#35011) No new revisions were added by this update. Summary of changes: be/src/vec/common/string_buffer.hpp | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) fix DCHECK failure of agg when failed to alloc memory [doris]
yiguolei merged PR #35011: URL: https://github.com/apache/doris/pull/35011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](agg) fix DCHECK failure of agg when failed to alloc memory (#35011)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 66565085796 [fix](agg) fix DCHECK failure of agg when failed to alloc
memory (#35011)
66565085796 is described below
commit 665650857961b335376d97e8611069da31e00050
Author: TengJianPing <18241664+jackte...@users.noreply.github.com>
AuthorDate: Mon May 20 10:10:11 2024 +0800
[fix](agg) fix DCHECK failure of agg when failed to alloc memory (#35011)
* [fix](agg) fix DCHECK failure of agg when failed to alloc memory
* add comment
---
be/src/vec/common/string_buffer.hpp | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/be/src/vec/common/string_buffer.hpp
b/be/src/vec/common/string_buffer.hpp
index 84f0dffbff7..3be04ad15e6 100644
--- a/be/src/vec/common/string_buffer.hpp
+++ b/be/src/vec/common/string_buffer.hpp
@@ -40,10 +40,12 @@ public:
_now_offset += 1;
}
-inline void commit() {
-ColumnString::check_chars_length(_offsets.back() + _now_offset, 0);
-_offsets.push_back(_offsets.back() + _now_offset);
+void commit() {
+auto now_offset = _now_offset;
_now_offset = 0;
+// the following code may throw exception, and DCHECK in destructor
will fail if _now_offset is not reset to 0
+ColumnString::check_chars_length(_offsets.back() + now_offset, 0);
+_offsets.push_back(_offsets.back() + now_offset);
}
~BufferWritable() { DCHECK(_now_offset == 0); }
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](binlog) Support gc binlogs by history nums and size [doris]
w41ter commented on PR #34888: URL: https://github.com/apache/doris/pull/34888#issuecomment-2119545007 run performance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [env](thrift) Generate move constructors and assignment operators. [doris]
Mryange closed pull request #34693: [env](thrift) Generate move constructors and assignment operators. URL: https://github.com/apache/doris/pull/34693 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [BUG] fix scan range boundary handling is incorrect [doris]
qzsee commented on PR #34832: URL: https://github.com/apache/doris/pull/34832#issuecomment-2119545574 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](Nereids) fix leading with multi level of brace pairs (#34169) [doris]
LiBinfeng-01 opened a new pull request, #35043:
URL: https://github.com/apache/doris/pull/35043
fix leading with multi level of brace pairs
example:
leading(t1 {{t2 t3} {t4 t5}} t6) can be reduced to leading(t1 {t2 t3 {t4
t5}} t6) also update cases which remove project node from explain shape plan
## Proposed changes
Issue Number: close #xxx
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Nereids) fix leading with multi level of brace pairs (#34169) [doris]
doris-robot commented on PR #35043: URL: https://github.com/apache/doris/pull/35043#issuecomment-2119549381 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [BUG] fix scan range boundary handling is incorrect [doris]
github-actions[bot] commented on PR #34832: URL: https://github.com/apache/doris/pull/34832#issuecomment-2119549696 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improve](txn insert) Set partition visible version once when finishTransaction [doris]
mymeiyi opened a new pull request, #35044: URL: https://github.com/apache/doris/pull/35044 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) Set partition visible version once when finishTransaction [doris]
doris-robot commented on PR #35044: URL: https://github.com/apache/doris/pull/35044#issuecomment-2119553594 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](dataque) use lock free queue to impl dataqueue [doris]
Mryange commented on PR #34468: URL: https://github.com/apache/doris/pull/34468#issuecomment-2119553930 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](multi-table-load) fix be core when partial table load failed [doris]
HHoflittlefish777 commented on PR #34712: URL: https://github.com/apache/doris/pull/34712#issuecomment-2119554490 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): Refactor `Eliminate_Group_By_Key` by functional dependencies [doris]
XieJiann commented on PR #34948: URL: https://github.com/apache/doris/pull/34948#issuecomment-2119555077 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Nereids) fix leading with multi level of brace pairs (#34169) [doris]
yiguolei merged PR #35043: URL: https://github.com/apache/doris/pull/35043 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [Fix](Nereids) fix leading with multi level of brace pairs (#34169) (#35043)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 7c29a964e51 [Fix](Nereids) fix leading with multi level of brace pairs
(#34169) (#35043)
7c29a964e51 is described below
commit 7c29a964e510d2c7ccaa31a0348ec6fff2310f55
Author: LiBinfeng <46676950+libinfeng...@users.noreply.github.com>
AuthorDate: Mon May 20 10:28:22 2024 +0800
[Fix](Nereids) fix leading with multi level of brace pairs (#34169) (#35043)
fix leading with multi level of brace pairs
example:
leading(t1 {{t2 t3} {t4 t5}} t6) can be reduced to leading(t1 {t2 t3 {t4
t5}} t6)
also update cases which remove project node from explain shape plan
---
.../org/apache/doris/nereids/hint/LeadingHint.java | 44 +-
.../data/nereids_hint_tpcds_p0/shape/query64.out | 18 +-
.../data/nereids_p0/hint/fix_leading.out | 59 +-
.../data/nereids_p0/hint/multi_leading.out | 575 +++
.../data/nereids_p0/hint/test_leading.out | 613 -
.../suites/nereids_p0/hint/fix_leading.groovy | 8 +
.../suites/nereids_p0/hint/multi_leading.groovy| 1 +
.../suites/nereids_p0/hint/test_leading.groovy | 1 +
8 files changed, 496 insertions(+), 823 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/hint/LeadingHint.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/hint/LeadingHint.java
index 3ef5217566a..4d58990fbc0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/hint/LeadingHint.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/hint/LeadingHint.java
@@ -55,7 +55,7 @@ public class LeadingHint extends Hint {
private List parameters;
private final List tablelist = new ArrayList<>();
-private final List levellist = new ArrayList<>();
+private final List levelList = new ArrayList<>();
private final Map distributeHints = new
HashMap<>();
@@ -120,18 +120,46 @@ public class LeadingHint extends Hint {
}
} else {
tablelist.add(parameter);
-levellist.add(level);
+levelList.add(level);
}
lastParameter = parameter;
}
+normalizeLevelList();
+}
+
+private void removeGap(int left, int right, int gap) {
+for (int i = left; i <= right; i++) {
+levelList.set(i, levelList.get(i) - (gap - 1));
+}
+}
+
+// when we write leading like: leading(t1 {{t2 t3} {t4 t5}} t6)
+// levelList would like 0 2 2 3 3 0, it could be reduced to 0 1 1 2 2 0
like leading(t1 {t2 t3 {t4 t5}} t6)
+// gap is like 0 to 2 or 3 to 0 in upper example, and this function is to
remove gap when we use a lot of braces
+private void normalizeLevelList() {
+int leftIndex = 0;
+// at lease two tables were needed
+for (int i = 1; i < levelList.size(); i++) {
+if ((levelList.get(i) - levelList.get(leftIndex)) > 1) {
+int rightIndex = i;
+for (int j = i; j < levelList.size(); j++) {
+if ((levelList.get(rightIndex) - levelList.get(j)) > 1) {
+removeGap(i, rightIndex, Math.min(levelList.get(i) -
levelList.get(leftIndex),
+levelList.get(rightIndex) - levelList.get(j)));
+}
+rightIndex = j;
+}
+}
+leftIndex = i;
+}
}
public List getTablelist() {
return tablelist;
}
-public List getLevellist() {
-return levellist;
+public List getLevelList() {
+return levelList;
}
public Map getRelationIdToScanMap() {
@@ -485,10 +513,10 @@ public class LeadingHint extends Hint {
}
logicalPlan = makeFilterPlanIfExist(getFilters(), logicalPlan);
assert (logicalPlan != null);
-stack.push(Pair.of(getLevellist().get(index), Pair.of(logicalPlan,
index)));
-int stackTopLevel = getLevellist().get(index++);
+stack.push(Pair.of(getLevelList().get(index), Pair.of(logicalPlan,
index)));
+int stackTopLevel = getLevelList().get(index++);
while (index < getTablelist().size()) {
-int currentLevel = getLevellist().get(index);
+int currentLevel = getLevelList().get(index);
if (currentLevel == stackTopLevel) {
// should return error if can not found table
logicalPlan =
getLogicalPlanByName(getTablelist().get(index++));
@@ -531,7 +559,7 @@ public class LeadingHint extends Hint {
logicalJoin.setBitmap(LongBitmap.or(getBitmap(newStackTop.second.first),
getBitmap(logicalPlan)));
if (stackTopLevel > 0) {
i
Re: [PR] [exec](performance) opt the topn nullable column order performance in Heap Sort [doris]
github-actions[bot] commented on PR #35042: URL: https://github.com/apache/doris/pull/35042#issuecomment-2119557259 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](performance) opt the topn nullable column order performance in Heap Sort [doris]
github-actions[bot] commented on PR #35042: URL: https://github.com/apache/doris/pull/35042#issuecomment-2119557278 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) Set partition visible version once when finishTransaction [doris]
mymeiyi commented on PR #35044: URL: https://github.com/apache/doris/pull/35044#issuecomment-2119558250 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) Txn load support cloud mode [doris]
mymeiyi commented on PR #34721: URL: https://github.com/apache/doris/pull/34721#issuecomment-2119560201 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](executor)Fix cloud p0 workload policy test failed [doris]
doris-robot commented on PR #35045: URL: https://github.com/apache/doris/pull/35045#issuecomment-2119560435 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](executor)Fix cloud p0 workload policy test failed [doris]
wangbo opened a new pull request, #35045:
URL: https://github.com/apache/doris/pull/35045
## Proposed changes
```
/ 3 disable test_set_var_policy2
sql "alter workload policy test_set_var_policy2
properties('enabled'='false');"
def result3 = connect(user = 'test_workload_sched_user', password = '12345',
url = context.config.jdbcUrl)
{ Thread.sleep(3000) sql "show variables like
'%parallel_pipeline_task_num%';" }
assertEquals("parallel_pipeline_task_num", result3[0][0])
assertEquals("33", result3[0][1])
^ERROR LINE^
```
This is because using daemon thread to publish policy, we can only reset the
interval in fe.conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](executor)Fix cloud p0 workload policy test failed [doris]
wangbo commented on PR #35045: URL: https://github.com/apache/doris/pull/35045#issuecomment-2119560493 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](pipeline) runtime filter wait time [doris]
github-actions[bot] commented on PR #34872: URL: https://github.com/apache/doris/pull/34872#issuecomment-2119560829 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](pipeline) runtime filter wait time [doris]
github-actions[bot] commented on PR #34872: URL: https://github.com/apache/doris/pull/34872#issuecomment-2119560844 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](Nereids) fix leading hint with update of alias name (#34434) [doris]
LiBinfeng-01 opened a new pull request, #35046: URL: https://github.com/apache/doris/pull/35046 pick: #34434 Problem: when using leading like leading(tbl1 tbl2) in "select * from (select tbl1.c1 from t1 as tbl1 join t2 as tbl2) join t3 as tbl2 on tbl2.c3 != 101;", in which tbl2.c3 means t3.c3 but not t2.c3 Causes and solved: when finding columns in condition, leading hint would find tbl2.c3's RelationId, and when we collect RelationId and aliasName we should update it if aliasName is repeat ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Nereids) fix leading hint with update of alias name (#34434) [doris]
doris-robot commented on PR #35046: URL: https://github.com/apache/doris/pull/35046#issuecomment-2119561864 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
