Re: [PR] [cherry-pick](branch-30)fix date_floor function return wrong result (#41948) [doris]
doris-robot commented on PR #43588: URL: https://github.com/apache/doris/pull/43588#issuecomment-2467358089 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](segcompaction) fix coredump for inverted index file writer close in segment compaction #43477 [doris]
airborne12 opened a new pull request, #43587: URL: https://github.com/apache/doris/pull/43587 cherry pick from #43477 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](time) use TimeValue to refine repetitive logic and remove unused code. [doris]
Gabriel39 merged PR #42978: URL: https://github.com/apache/doris/pull/42978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [misc](bvar) expose file uploading thread pool queue size [doris]
github-actions[bot] commented on code in PR #43590:
URL: https://github.com/apache/doris/pull/43590#discussion_r1836036269
##
be/src/io/fs/s3_file_bufferpool.cpp:
##
@@ -110,8 +112,14 @@ Status UploadFileBuffer::append_data(const Slice& data) {
*/
static Status submit_upload_buffer(std::shared_ptr buffer) {
TEST_SYNC_POINT_RETURN_WITH_VALUE("UploadFileBuffer::submit",
Status::OK(), buffer.get());
-return ExecEnv::GetInstance()->s3_file_upload_thread_pool()->submit_func(
-[buf = std::move(buffer)]() { buf->execute_async(); });
+auto pool = ExecEnv::GetInstance()->s3_file_upload_thread_pool();
Review Comment:
warning: 'auto pool' can be declared as 'auto *pool'
[readability-qualified-auto]
```suggestion
auto *pool = ExecEnv::GetInstance()->s3_file_upload_thread_pool();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Enhancement] replace table options to keep original dropped table in recycle bin [doris]
doris-robot commented on PR #43585: URL: https://github.com/apache/doris/pull/43585#issuecomment-2467375767 TPC-H: Total hot run time: 42351 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 4cabf34849dcda20786cd90b81f95427520efb55, data reload: false -- Round 1 -- q1 11487 742576257425 q2 1657140313081308 q3 5581159312241224 q4 1031766 717 717 q5 2837284628472846 q6 241 155 153 153 q7 981 636 611 611 q8 2017202920342029 q9 6516654664246424 q10 2566232323482323 q11 379 271 277 271 q12 431 222 224 222 q13 17522 299429952994 q14 251 215 231 215 q15 571 522 518 518 q16 653 603 603 603 q17 994 666 524 524 q18 7348662466556624 q19 11881184992 992 q20 393 201 202 201 q21 3982314834003148 q22 11021024979 979 Total cold run time: 69728 ms Total hot run time: 42351 ms - Round 2, with runtime_filter_mode=off - q1 7353734473027302 q2 418 307 303 303 q3 2927277227312731 q4 1939165517121655 q5 5404547054825470 q6 220 143 144 143 q7 2086163416871634 q8 3354362435503550 q9 8999898690468986 q10 3605355735043504 q11 593 503 498 498 q12 817 612 627 612 q13 8784319431933193 q14 321 274 269 269 q15 605 545 554 545 q16 693 667 678 667 q17 1879165116151615 q18 8117786777437743 q19 1728167915661566 q20 2084192718761876 q21 5455527554395275 q22 1123104710101010 Total cold run time: 68504 ms Total hot run time: 60147 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Nereids): Optimize Query Plan by Pulling Up Join with Common Child from Union (#42033) [doris]
feiniaofeiafei commented on PR #43591: URL: https://github.com/apache/doris/pull/43591#issuecomment-2467378959 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [refine](time) use TimeValue to refine repetitive logic and remove unused code. (#42978)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a8999f506ea [refine](time) use TimeValue to refine repetitive logic
and remove unused code. (#42978)
a8999f506ea is described below
commit a8999f506eae2a4245558f3e1fb57b64a8f14d94
Author: Mryange <59914473+mrya...@users.noreply.github.com>
AuthorDate: Mon Nov 11 14:49:14 2024 +0800
[refine](time) use TimeValue to refine repetitive logic and remove unused
code. (#42978)
---
.../vec/data_types/serde/data_type_time_serde.cpp | 27 --
be/src/vec/data_types/serde/data_type_time_serde.h | 18 ---
be/src/vec/functions/function_cast.h | 11 -
.../function_date_or_datetime_computation.h| 14 +--
be/src/vec/runtime/time_value.h| 11 +
5 files changed, 23 insertions(+), 58 deletions(-)
diff --git a/be/src/vec/data_types/serde/data_type_time_serde.cpp
b/be/src/vec/data_types/serde/data_type_time_serde.cpp
index 19b0b1cf6e5..01e9b5ecd5d 100644
--- a/be/src/vec/data_types/serde/data_type_time_serde.cpp
+++ b/be/src/vec/data_types/serde/data_type_time_serde.cpp
@@ -19,33 +19,6 @@
namespace doris {
namespace vectorized {
-template
-Status DataTypeTimeSerDe::_write_column_to_mysql(const IColumn& column,
-
MysqlRowBuffer& result,
- int row_idx, bool col_const,
- const FormatOptions& options)
const {
-auto& data = assert_cast(column).get_data();
-const auto col_index = index_check_const(row_idx, col_const);
-if (UNLIKELY(0 != result.push_time(data[col_index]))) {
-return Status::InternalError("pack mysql buffer failed.");
-}
-return Status::OK();
-}
-
-Status DataTypeTimeSerDe::write_column_to_mysql(const IColumn& column,
-MysqlRowBuffer&
row_buffer, int row_idx,
-bool col_const,
-const FormatOptions& options)
const {
-return _write_column_to_mysql(column, row_buffer, row_idx, col_const,
options);
-}
-
-Status DataTypeTimeSerDe::write_column_to_mysql(const IColumn& column,
-MysqlRowBuffer&
row_buffer, int row_idx,
-bool col_const,
-const FormatOptions& options)
const {
-return _write_column_to_mysql(column, row_buffer, row_idx, col_const,
options);
-}
-
Status DataTypeTimeV2SerDe::write_column_to_mysql(const IColumn& column,
MysqlRowBuffer&
row_buffer, int row_idx,
bool col_const,
diff --git a/be/src/vec/data_types/serde/data_type_time_serde.h
b/be/src/vec/data_types/serde/data_type_time_serde.h
index d9a373b2f5a..24e3da8ec21 100644
--- a/be/src/vec/data_types/serde/data_type_time_serde.h
+++ b/be/src/vec/data_types/serde/data_type_time_serde.h
@@ -28,24 +28,6 @@ namespace doris {
class JsonbOutStream;
namespace vectorized {
-class Arena;
-
-class DataTypeTimeSerDe : public DataTypeNumberSerDe {
-public:
-DataTypeTimeSerDe(int nesting_level = 1) :
DataTypeNumberSerDe(nesting_level) {};
-
-Status write_column_to_mysql(const IColumn& column, MysqlRowBuffer&
row_buffer,
- int row_idx, bool col_const,
- const FormatOptions& options) const override;
-Status write_column_to_mysql(const IColumn& column, MysqlRowBuffer&
row_buffer,
- int row_idx, bool col_const,
- const FormatOptions& options) const override;
-
-private:
-template
-Status _write_column_to_mysql(const IColumn& column,
MysqlRowBuffer& result,
- int row_idx, bool col_const, const
FormatOptions& options) const;
-};
class DataTypeTimeV2SerDe : public DataTypeNumberSerDe {
public:
DataTypeTimeV2SerDe(int scale = 0, int nesting_level = 1)
diff --git a/be/src/vec/functions/function_cast.h
b/be/src/vec/functions/function_cast.h
index 0e567a2b743..c2195fd7e69 100644
--- a/be/src/vec/functions/function_cast.h
+++ b/be/src/vec/functions/function_cast.h
@@ -94,6 +94,7 @@
#include "vec/functions/function_convert_tz.h"
#include "vec/functions/function_helpers.h"
#include "vec/io/reader_buffer.h"
+#include "vec/runtime/time_value.h"
#include "vec/runtime/vdatetime_value.h"
#include "vec/utils/util.hpp"
@@ -145,7 +146,7 @@ struct TimeCast {
} else {
if (VecDateTimeValue dv {}; dv.from_date_str(s, len,
local_time_zone)) {
Re: [PR] branch-3.0: [opt](log) change lzo decompress log to debug level [doris]
github-actions[bot] commented on PR #43582: URL: https://github.com/apache/doris/pull/43582#issuecomment-2467372429 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](build index)Optimize failed task check on same tablet (#42295) [doris]
github-actions[bot] commented on PR #43589: URL: https://github.com/apache/doris/pull/43589#issuecomment-2467373136 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [feature](Nereids): Optimize Query Plan by Pulling Up Join with Common Child from Union (#42033) [doris]
feiniaofeiafei opened a new pull request, #43591: URL: https://github.com/apache/doris/pull/43591 cherry-pick #42033 to branch-3.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Nereids): Optimize Query Plan by Pulling Up Join with Common Child from Union (#42033) [doris]
doris-robot commented on PR #43591: URL: https://github.com/apache/doris/pull/43591#issuecomment-2467378176 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [performance](move-memtable) async close tablet streams [doris]
github-actions[bot] commented on PR #41156: URL: https://github.com/apache/doris/pull/41156#issuecomment-2467380479 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [performance](move-memtable) async close tablet streams [doris]
github-actions[bot] commented on PR #41156: URL: https://github.com/apache/doris/pull/41156#issuecomment-2467380514 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) add performance profiling for remote io access in inverted index [doris]
zzzxl1993 commented on PR #43542: URL: https://github.com/apache/doris/pull/43542#issuecomment-2467384343 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](scan) remove useless enable_scan_node_run_serial #41559 [doris]
doris-robot commented on PR #42013: URL: https://github.com/apache/doris/pull/42013#issuecomment-2467387500 TPC-H: Total hot run time: 40397 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f1790a5bebfd1af35ea4cbc209ec9be680cd5ac7, data reload: false -- Round 1 -- q1 17981 980872917291 q2 2106191 171 171 q3 10588 1151 q4 10399 736 718 718 q5 7725277628112776 q6 240 150 142 142 q7 968 610 616 610 q8 9388190919591909 q9 6633641863926392 q10 7088228822962288 q11 455 255 257 255 q12 382 205 202 202 q13 17792 300429912991 q14 233 205 216 205 q15 576 525 515 515 q16 690 604 602 602 q17 992 505 562 505 q18 7379660864776477 q19 20361070991 991 q20 454 196 205 196 q21 3868315530873087 q22 1084963 982 963 Total cold run time: 109057 ms Total hot run time: 40397 ms - Round 2, with runtime_filter_mode=off - q1 7308723472717234 q2 390 314 318 314 q3 2999286628972866 q4 2074168917741689 q5 5662570357085703 q6 229 137 140 137 q7 2184173217461732 q8 3363348833653365 q9 8904884287538753 q10 3516350235203502 q11 592 497 493 493 q12 807 608 594 594 q13 16650 314931473147 q14 306 264 292 264 q15 605 555 553 553 q16 739 661 663 661 q17 1830160315721572 q18 8256772776747674 q19 1737150816691508 q20 2044184118491841 q21 5312534952515251 q22 1076102710421027 Total cold run time: 76583 ms Total hot run time: 59880 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud-mow) MS should create new Transaction to continue geting delete bitmap when encounter TXN_TOO_OLD [doris]
github-actions[bot] commented on PR #43509: URL: https://github.com/apache/doris/pull/43509#issuecomment-2467389389 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud-mow) MS should create new Transaction to continue geting delete bitmap when encounter TXN_TOO_OLD [doris]
github-actions[bot] commented on PR #43509: URL: https://github.com/apache/doris/pull/43509#issuecomment-2467389434 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Enhancement] replace table options to keep original dropped table in recycle bin [doris]
doris-robot commented on PR #43585: URL: https://github.com/apache/doris/pull/43585#issuecomment-2467391617 TPC-DS: Total hot run time: 191920 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 4cabf34849dcda20786cd90b81f95427520efb55, data reload: false query1 2422221922202219 query2 6379208120712071 query3 14325 9936236 236 query4 33178 23442 23650 23442 query5 3763460 434 434 query6 263 175 182 175 query7 3977313 320 313 query8 294 228 233 228 query9 9535270026882688 query10 469 265 257 257 query11 17983 15103 15163 15103 query12 138 100 99 99 query13 1539441 403 403 query14 9598641366426413 query15 232 187 172 172 query16 7744530 453 453 query17 1342600 622 600 query18 2300623 636 623 query19 254 193 203 193 query20 134 119 116 116 query21 209 107 105 105 query22 4623438743734373 query23 35031 34369 34079 34079 query24 7981343934353435 query25 585 392 390 390 query26 737 179 182 179 query27 1847294 295 294 query28 4067250924822482 query29 790 429 422 422 query30 397 303 309 303 query31 996 803 811 803 query32 89 54 53 53 query33 467 265 279 265 query34 893 511 509 509 query35 843 717 738 717 query36 1067921 924 921 query37 112 67 71 67 query38 4140385838883858 query39 1482144114821441 query40 194 96 96 96 query41 50 47 48 47 query42 111 94 100 94 query43 532 463 505 463 query44 1183791 781 781 query45 190 161 167 161 query46 1118753 753 753 query47 1874181917771777 query48 470 368 362 362 query49 840 374 374 374 query50 832 409 411 409 query51 7224697872676978 query52 97 91 88 88 query53 261 187 181 181 query54 566 460 446 446 query55 80 75 74 74 query56 242 232 232 232 query57 1200110610891089 query58 207 202 202 202 query59 3060286528652865 query60 277 254 259 254 query61 110 107 106 106 query62 807 656 655 655 query63 214 202 191 191 query64 3303644 615 615 query65 3301319831853185 query66 784 337 301 301 query67 16028 15459 15408 15408 query68 4442572 566 566 query69 417 260 257 257 query70 1135111610361036 query71 343 271 267 267 query72 6318402324792479 query73 766 345 343 343 query74 10197 890688818881 query75 3340265226022602 query76 23111022938 938 query77 417 277 269 269 query78 10513 979599029795 query79 3784596 595 595 query80 2394420 417 417 query81 585 242 240 240 query82 901 116 120 116 query83 276 165 161 161 query84 289 90 85 85 query85 1774294 290 290 query86 507 297 265 265 query87 4412423142244224 query88 4662236024102360 query89 551 297 293 293 query90 2037190 181 181 query91 180 144 157 144 query92 70 48 46 46 query93 6485524 531 524 query94 850 295 277 277 query95 349 254 255 254 query96 633 287 285 285 query97 3340310432083104 query98 226 210 198 198 query99 1742143413071307 Total cold run time: 303626 ms Total hot run time: 191920 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [feature](Nereids): Optimize Query Plan by Pulling Up Join with Common Child from Union (#42033) [doris]
feiniaofeiafei commented on PR #43591: URL: https://github.com/apache/doris/pull/43591#issuecomment-2467393157 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Enhancement] (nereids)implement KillQueryCommand in nereids [doris]
Yao-MR commented on issue #42533: URL: https://github.com/apache/doris/issues/42533#issuecomment-2467393897 @LiBinfeng-01 can you assign it me ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) add performance profiling for remote io access in inverted index [doris]
zzzxl1993 commented on PR #43542: URL: https://github.com/apache/doris/pull/43542#issuecomment-2467400346 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](store) allow to set storage_page_size for tables [doris]
kaijchen commented on code in PR #42680:
URL: https://github.com/apache/doris/pull/42680#discussion_r1836055939
##
be/src/olap/delta_writer_v2.cpp:
##
@@ -125,6 +125,9 @@ Status DeltaWriterV2::init() {
context.partial_update_info = _partial_update_info;
context.memtable_on_sink_support_index_v2 = true;
+auto tablet =
DORIS_TRY(ExecEnv::GetInstance()->storage_engine().get_tablet(_req.tablet_id));
Review Comment:
DeltaWriterV2 is on the upstream BE, the tablet requested here can be on
downstream BE.
This causes the following problem:
```
I20241110 23:46:32.273124 5731 pipeline_fragment_context.cpp:170]
PipelineFragmentContext::cancel|query_id=4334341c4795459e-ade78c01be07735d|fragment_id=0|reason=[INTERNAL_ERROR]failed
to get tablet: 14168, reason: tablet does not exist. 172.20.50.97
0# doris::StorageEngine::get_tablet(long) at
/home/zcp/repo_center/doris_master/doris/be/src/common/status.h:375
1# doris::DeltaWriterV2::init() at
/home/zcp/repo_center/doris_master/doris/be/src/util/expected.hpp:1986
2# doris::DeltaWriterV2::write(doris::vectorized::Block const*,
std::vector > const&) at
/home/zcp/repo_center/doris_master/doris/be/src/common/status.h:499
3#
doris::vectorized::VTabletWriterV2::_write_memtable(std::shared_ptr,
long, doris::vectorized::Rows const&) at
/home/zcp/repo_center/doris_master/doris/be/src/common/status.h:388
4# doris::vectorized::VTabletWriterV2::write(doris::RuntimeState*,
doris::vectorized::Block&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701
5#
doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*,
doris::RuntimeProfile*) at
/home/zcp/repo_center/doris_master/doris/be/src/common/status.h:499
6# std::_Function_handler::_M_invoke(std::_Any_data const&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/atomicity.h:98
7# doris::ThreadPool::dispatch_thread() at
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:0
8# doris::Thread::supervise_thread(void*) at
/var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
9# ?
10# ?
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Docs] Add doc for variant nested type [doris-website]
eldenmoon opened a new pull request, #1328: URL: https://github.com/apache/doris-website/pull/1328 # Versions - [ ] dev - [ ] 3.0 - [ ] 2.1 - [ ] 2.0 # Languages - [ ] Chinese - [ ] English -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](segcompaction) fix coredump for inverted index file writer close in segment compaction [doris]
github-actions[bot] commented on PR #43477: URL: https://github.com/apache/doris/pull/43477#issuecomment-2467329671 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Improvement](set) change set_operator's hash map to phmap (#43273)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dae8f607e17 [Improvement](set) change set_operator's hash map to phmap
(#43273)
dae8f607e17 is described below
commit dae8f607e176480306471754d0391df3daeee8a7
Author: Pxl
AuthorDate: Mon Nov 11 14:18:02 2024 +0800
[Improvement](set) change set_operator's hash map to phmap (#43273)
### What problem does this PR solve?
1. change set_operator's hash map to phmap.
2. optimize the logic of refresh hash table
```sql
+---+--+--+---+-+---+
| Field | Type | Null | Key | Default | Extra |
+---+--+--+---+-+---+
| k1| int | Yes | true | NULL| |
| k2| int | No | false | NULL| NONE |
| k3| bigint | Yes | false | NULL| NONE |
| k4| varchar(100) | Yes | false | NULL| NONE |
+---+--+--+---+-+---+
```
```sql
select count(*) from (select * from a_table intersect select * from
b_table)t;
before:Time(ms)=37768|CpuTimeMS=61754|PeakMemoryBytes=129921088
after: Time(ms)=25218|CpuTimeMS=44571|PeakMemoryBytes=129902624
```
```sql
select count(*) from (select k1 from a_table intersect select k1 from
b_table)t;
before: Time(ms)=7964|CpuTimeMS=13431|PeakMemoryBytes=130062400
after: Time(ms)=7459|CpuTimeMS=11976|PeakMemoryBytes=130091048
```
---
be/src/pipeline/common/set_utils.h | 74 +---
be/src/pipeline/exec/join/join_op.h | 18 ++
be/src/pipeline/exec/set_probe_sink_operator.cpp | 74 +---
be/src/pipeline/exec/set_source_operator.cpp | 17 +-
be/src/vec/common/hash_table/hash_map.h | 5 --
be/src/vec/common/hash_table/hash_map_context.h | 7 +--
be/src/vec/common/hash_table/ph_hash_map.h | 11 ++--
be/src/vec/common/hash_table/ph_hash_set.h | 3 +-
be/src/vec/common/hash_table/string_hash_map.h | 1 +
9 files changed, 96 insertions(+), 114 deletions(-)
diff --git a/be/src/pipeline/common/set_utils.h
b/be/src/pipeline/common/set_utils.h
index ed64035fb42..2caf5b7d0b8 100644
--- a/be/src/pipeline/common/set_utils.h
+++ b/be/src/pipeline/common/set_utils.h
@@ -25,21 +25,32 @@
namespace doris {
-template
-using SetFixedKeyHashTableContext =
-vectorized::MethodKeysFixed>>;
+template
+using SetData = PHHashMap>;
-template
-using SetPrimaryTypeHashTableContext =
-vectorized::MethodOneNumber>>;
+template
+using SetFixedKeyHashTableContext = vectorized::MethodKeysFixed>;
+
+template
+using SetPrimaryTypeHashTableContext = vectorized::MethodOneNumber>;
+
+template
+using SetPrimaryTypeHashTableContextNullable =
vectorized::MethodSingleNullableColumn<
+vectorized::MethodOneNumber>>>;
using SetSerializedHashTableContext =
-vectorized::MethodSerialized>;
+vectorized::MethodSerialized>;
using SetMethodOneString =
-vectorized::MethodStringNoCache>;
+vectorized::MethodStringNoCache>;
using SetHashTableVariants =
std::variant,
+
SetPrimaryTypeHashTableContextNullable,
+
SetPrimaryTypeHashTableContextNullable,
+
SetPrimaryTypeHashTableContextNullable,
+
SetPrimaryTypeHashTableContextNullable,
+
SetPrimaryTypeHashTableContextNullable,
SetPrimaryTypeHashTableContext,
SetPrimaryTypeHashTableContext,
SetPrimaryTypeHashTableContext,
@@ -51,9 +62,9 @@ using SetHashTableVariants =
SetFixedKeyHashTableContext,
SetFixedKeyHashTableContext>;
-struct SetDataVariants {
-SetHashTableVariants method_variant;
-
+struct SetDataVariants
+: public DataVariants {
void init(const std::vector& data_types,
HashKeyType type) {
bool nullable = data_types.size() == 1 && data_types[0]->is_nullable();
switch (type) {
@@ -61,51 +72,22 @@ struct SetDataVariants {
method_variant.emplace();
break;
case HashKeyType::int8_key:
-if (nullable) {
-
method_variant.emplace>(
-get_key_sizes(data_types));
-} else {
-
method_variant.emplace>();
-}
+emplace_single>(nullable);
break;
case HashKeyType::int16_key:
-if (nullable) {
-
method_variant.emplace>(
-get_key_sizes(data_types));
-} else {
-
method_variant.emplace>();
-}
+emplace_
Re: [PR] [fix](nereids)Solve the problem of pruning wrong partitions in multi-column partition pruning [doris]
github-actions[bot] commented on PR #43332: URL: https://github.com/apache/doris/pull/43332#issuecomment-2467329677 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](scan) remove useless enable_scan_node_run_serial #41559 [doris]
github-actions[bot] commented on PR #42013: URL: https://github.com/apache/doris/pull/42013#issuecomment-2467331018 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Optimize](Variant) move merge_rowsets_schema to sync_tablet_rowsets to make schema more accurate [doris]
github-actions[bot] commented on PR #43580: URL: https://github.com/apache/doris/pull/43580#issuecomment-2467331620 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](runtime-filter) add rpc error msg to RuntimeFilterContext [doris]
BiteThet merged PR #43517: URL: https://github.com/apache/doris/pull/43517 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (dae8f607e17 -> a71d4ecc113)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from dae8f607e17 [Improvement](set) change set_operator's hash map to phmap (#43273) add a71d4ecc113 [Chore](runtime-filter) add rpc error msg to RuntimeFilterContext (#43517) No new revisions were added by this update. Summary of changes: be/src/exprs/runtime_filter.cpp | 43 +++ be/src/vec/runtime/shared_hash_table_controller.h | 1 + 2 files changed, 22 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improvement](build index)Optimize failed task check on same tablet (#42295) [doris]
qidaye opened a new pull request, #43581: URL: https://github.com/apache/doris/pull/43581 bp #42295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](build index)Optimize failed task check on same tablet (#42295) [doris]
qidaye commented on PR #43581: URL: https://github.com/apache/doris/pull/43581#issuecomment-2467335970 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cluster key) fix some cluster key problems [doris]
dataroaring commented on code in PR #43574:
URL: https://github.com/apache/doris/pull/43574#discussion_r1836018689
##
be/src/olap/rowset/vertical_beta_rowset_writer.cpp:
##
@@ -50,7 +50,8 @@ template
requires std::is_base_of_v
Status VerticalBetaRowsetWriter::add_columns(const vectorized::Block* block,
const std::vector&
col_ids, bool is_key,
-uint32_t max_rows_per_segment)
{
+uint32_t max_rows_per_segment,
+bool has_cluster_key) {
Review Comment:
And it would generates too many segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](build index)Optimize failed task check on same tablet (#42295) [doris]
github-actions[bot] commented on PR #43581: URL: https://github.com/apache/doris/pull/43581#issuecomment-2467344458 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cluster key) add a config to random set cluster keys for mow [doris]
mymeiyi commented on PR #42565: URL: https://github.com/apache/doris/pull/42565#issuecomment-2467343266 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](set) change set_operator's hash map to phmap [doris]
BiteThet merged PR #43273: URL: https://github.com/apache/doris/pull/43273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [refactor](scanner) Delete meaning-less macro (#43555)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0cf7e74939b [refactor](scanner) Delete meaning-less macro (#43555)
0cf7e74939b is described below
commit 0cf7e74939bf430deb16521cc5b55e10a28e13fc
Author: Gabriel
AuthorDate: Mon Nov 11 14:21:42 2024 +0800
[refactor](scanner) Delete meaning-less macro (#43555)
Delete meaning-less macro
---
be/src/vec/exec/scan/new_olap_scanner.cpp | 179 ++
1 file changed, 86 insertions(+), 93 deletions(-)
diff --git a/be/src/vec/exec/scan/new_olap_scanner.cpp
b/be/src/vec/exec/scan/new_olap_scanner.cpp
index f40f30f5b16..4c0b30e440e 100644
--- a/be/src/vec/exec/scan/new_olap_scanner.cpp
+++ b/be/src/vec/exec/scan/new_olap_scanner.cpp
@@ -558,104 +558,97 @@ void NewOlapScanner::_collect_profile_before_close() {
VScanner::_collect_profile_before_close();
-#ifndef INCR_COUNTER
-#define INCR_COUNTER(Parent)
\
-COUNTER_UPDATE(Parent->_io_timer, stats.io_ns);
\
-COUNTER_UPDATE(Parent->_read_compressed_counter,
stats.compressed_bytes_read); \
-COUNTER_UPDATE(Parent->_scan_bytes, stats.compressed_bytes_read);
\
-COUNTER_UPDATE(Parent->_decompressor_timer, stats.decompress_ns);
\
-COUNTER_UPDATE(Parent->_read_uncompressed_counter,
stats.uncompressed_bytes_read); \
-COUNTER_UPDATE(Parent->_block_load_timer, stats.block_load_ns);
\
-COUNTER_UPDATE(Parent->_block_load_counter, stats.blocks_load);
\
-COUNTER_UPDATE(Parent->_block_fetch_timer, stats.block_fetch_ns);
\
-COUNTER_UPDATE(Parent->_delete_bitmap_get_agg_timer,
stats.delete_bitmap_get_agg_ns); \
-COUNTER_UPDATE(Parent->_scan_rows, stats.raw_rows_read);
\
-COUNTER_UPDATE(Parent->_vec_cond_timer, stats.vec_cond_ns);
\
-COUNTER_UPDATE(Parent->_short_cond_timer, stats.short_cond_ns);
\
-COUNTER_UPDATE(Parent->_expr_filter_timer, stats.expr_filter_ns);
\
-COUNTER_UPDATE(Parent->_block_init_timer, stats.block_init_ns);
\
-COUNTER_UPDATE(Parent->_block_init_seek_timer, stats.block_init_seek_ns);
\
-COUNTER_UPDATE(Parent->_block_init_seek_counter,
stats.block_init_seek_num);\
-COUNTER_UPDATE(Parent->_segment_generate_row_range_timer,
stats.generate_row_ranges_ns);\
-COUNTER_UPDATE(Parent->_segment_generate_row_range_by_bf_timer,
\
- stats.generate_row_ranges_by_bf_ns);
\
-COUNTER_UPDATE(Parent->_collect_iterator_merge_next_timer,
\
- stats.collect_iterator_merge_next_timer);
\
-COUNTER_UPDATE(Parent->_segment_generate_row_range_by_zonemap_timer,
\
- stats.generate_row_ranges_by_zonemap_ns);
\
-COUNTER_UPDATE(Parent->_segment_generate_row_range_by_dict_timer,
\
- stats.generate_row_ranges_by_dict_ns);
\
-COUNTER_UPDATE(Parent->_predicate_column_read_timer,
stats.predicate_column_read_ns); \
-COUNTER_UPDATE(Parent->_non_predicate_column_read_timer,
stats.non_predicate_read_ns); \
-COUNTER_UPDATE(Parent->_predicate_column_read_seek_timer,
\
- stats.predicate_column_read_seek_ns);
\
-COUNTER_UPDATE(Parent->_predicate_column_read_seek_counter,
\
- stats.predicate_column_read_seek_num);
\
-COUNTER_UPDATE(Parent->_lazy_read_timer, stats.lazy_read_ns);
\
-COUNTER_UPDATE(Parent->_lazy_read_seek_timer,
stats.block_lazy_read_seek_ns); \
-COUNTER_UPDATE(Parent->_lazy_read_seek_counter,
stats.block_lazy_read_seek_num);\
-COUNTER_UPDATE(Parent->_output_col_timer, stats.output_col_ns);
\
-COUNTER_UPDATE(Parent->_rows_vec_cond_filtered_counter,
stats.rows_vec_cond_filtered); \
-COUNTER_UPDATE(Parent->_rows_short_circuit_cond_filtered_counter,
\
- stats.rows_short_circuit_cond_filtered);
\
-COUNTER_UPDATE(P
Re: [PR] [refactor](scanner) Delete meaning-less macro [doris]
Gabriel39 merged PR #43555: URL: https://github.com/apache/doris/pull/43555 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](nereids)Compressed materialize for aggregate and sort [doris]
englefly commented on PR #42408: URL: https://github.com/apache/doris/pull/42408#issuecomment-2467337890 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](log) change lzo decompress log to debug level [doris]
morningman merged PR #43540: URL: https://github.com/apache/doris/pull/43540 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-3.0: [opt](log) change lzo decompress log to debug level [doris]
github-actions[bot] opened a new pull request, #43582: URL: https://github.com/apache/doris/pull/43582 Cherry-picked from #43540 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43540-branch-3.0 updated (91d95f63be7 -> 2741edad9f3)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43540-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 91d95f63be7 [fix](backup) Load backup meta and job info bytes from disk #43276 (#43520) add 2741edad9f3 [opt](log) change lzo decompress log to debug level (#43540) No new revisions were added by this update. Summary of changes: be/src/exec/lzo_decompressor.cpp | 19 ++- 1 file changed, 10 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Optimize](Variant) move merge_rowsets_schema to sync_tablet_rowsets to make schema more accurate [doris]
eldenmoon opened a new pull request, #43580: URL: https://github.com/apache/doris/pull/43580 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (0cf7e74939b -> 7b3e030a31f)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 0cf7e74939b [refactor](scanner) Delete meaning-less macro (#43555) add 7b3e030a31f [opt](log) change lzo decompress log to debug level (#43540) No new revisions were added by this update. Summary of changes: be/src/exec/lzo_decompressor.cpp | 19 ++- 1 file changed, 10 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43540-branch-3.0 created (now 91d95f63be7)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43540-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at 91d95f63be7 [fix](backup) Load backup meta and job info bytes from disk #43276 (#43520) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [opt](log) change lzo decompress log to debug level [doris]
doris-robot commented on PR #43583: URL: https://github.com/apache/doris/pull/43583#issuecomment-2467340783 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [opt](log) change lzo decompress log to debug level [doris]
doris-robot commented on PR #43582: URL: https://github.com/apache/doris/pull/43582#issuecomment-2467339873 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [opt](log) change lzo decompress log to debug level [doris]
doris-robot commented on PR #43582: URL: https://github.com/apache/doris/pull/43582#issuecomment-2467339900 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43540-branch-2.1 created (now 5dda61b410c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43540-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git at 5dda61b410c [fix](backup) Load backup meta and job info bytes from disk #43276 (#43519) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43540-branch-2.1 updated (5dda61b410c -> 011193711ec)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43540-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from 5dda61b410c [fix](backup) Load backup meta and job info bytes from disk #43276 (#43519) add 011193711ec [opt](log) change lzo decompress log to debug level (#43540) No new revisions were added by this update. Summary of changes: be/src/exec/lzo_decompressor.cpp | 19 ++- 1 file changed, 10 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [opt](log) change lzo decompress log to debug level [doris]
doris-robot commented on PR #43583: URL: https://github.com/apache/doris/pull/43583#issuecomment-2467340814 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-2.1: [opt](log) change lzo decompress log to debug level [doris]
github-actions[bot] opened a new pull request, #43583: URL: https://github.com/apache/doris/pull/43583 Cherry-picked from #43540 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cluster key) fix some cluster key problems [doris]
dataroaring commented on code in PR #43574:
URL: https://github.com/apache/doris/pull/43574#discussion_r1836018434
##
be/src/olap/rowset/vertical_beta_rowset_writer.cpp:
##
@@ -50,7 +50,8 @@ template
requires std::is_base_of_v
Status VerticalBetaRowsetWriter::add_columns(const vectorized::Block* block,
const std::vector&
col_ids, bool is_key,
-uint32_t max_rows_per_segment)
{
+uint32_t max_rows_per_segment,
+bool has_cluster_key) {
Review Comment:
has_cluster_key should be an attribute set when a rowset writer is created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](delete) Fix static type dispatch by mistake due to typo (#42260) [doris]
TangSiyang2001 commented on PR #43488: URL: https://github.com/apache/doris/pull/43488#issuecomment-2467346402 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) Fix aws sdk default retry policy not taking effect [doris]
dataroaring commented on PR #43575: URL: https://github.com/apache/doris/pull/43575#issuecomment-2467344958 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cherry-pick](branch-2.1)regression-test alter table partition [doris]
cjj2010 commented on PR #43561: URL: https://github.com/apache/doris/pull/43561#issuecomment-2467343358 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](build index)Optimize failed task check on same tablet (#42295) [doris]
doris-robot commented on PR #43581: URL: https://github.com/apache/doris/pull/43581#issuecomment-2467335871 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] replace table options to keep original dropped table in recycle bin [doris]
dataroaring merged PR #43538: URL: https://github.com/apache/doris/pull/43538 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Enhancement] replace table options to keep original dropped table in recycle bin [doris]
dataroaring closed issue #43537: [Enhancement] replace table options to keep original dropped table in recycle bin URL: https://github.com/apache/doris/issues/43537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (7b3e030a31f -> 37f489aab3b)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 7b3e030a31f [opt](log) change lzo decompress log to debug level
(#43540)
add 37f489aab3b [Enhancement] replace table options to keep original
dropped table in recycle bin (#43538)
No new revisions were added by this update.
Summary of changes:
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 2 +-
fe/fe-core/src/main/cup/sql_parser.cup | 4 +-
.../main/java/org/apache/doris/alter/Alter.java| 24 +---
.../apache/doris/analysis/ReplaceTableClause.java | 9 ++-
.../plans/commands/info/AlterMTMVReplaceInfo.java | 2 +-
.../doris/persist/ReplaceTableOperationLog.java| 9 ++-
.../java/org/apache/doris/qe/StmtExecutor.java | 3 +-
.../java/org/apache/doris/alter/AlterTest.java | 4 +-
.../persist/ReplaceTableOperationLogTest.java | 2 +-
...artition.out => test_replace_table_recycle.out} | 9 ++-
...on.groovy => test_replace_table_recycle.groovy} | 71 +-
11 files changed, 92 insertions(+), 47 deletions(-)
copy
regression-test/data/catalog_recycle_bin_p0/{test_insert_overwrite_recover_no_partition.out
=> test_replace_table_recycle.out} (68%)
copy
regression-test/suites/catalog_recycle_bin_p0/{test_insert_overwrite_recover_no_partition.groovy
=> test_replace_table_recycle.groovy} (55%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43538-branch-3.0 created (now 91d95f63be7)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43538-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at 91d95f63be7 [fix](backup) Load backup meta and job info bytes from disk #43276 (#43520) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mac-compile):Fix implicit type conversion error [doris]
doris-robot commented on PR #43584: URL: https://github.com/apache/doris/pull/43584#issuecomment-2467351747 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-3.0: [Enhancement] replace table options to keep original dropped table in recycle bin [doris]
github-actions[bot] opened a new pull request, #43585: URL: https://github.com/apache/doris/pull/43585 Cherry-picked from #43538 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43538-branch-3.0 updated (91d95f63be7 -> 4cabf34849d)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch auto-pick-43538-branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
from 91d95f63be7 [fix](backup) Load backup meta and job info bytes from
disk #43276 (#43520)
add 4cabf34849d [Enhancement] replace table options to keep original
dropped table in recycle bin (#43538)
No new revisions were added by this update.
Summary of changes:
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 2 +-
fe/fe-core/src/main/cup/sql_parser.cup | 4 +-
.../main/java/org/apache/doris/alter/Alter.java| 24 +---
.../apache/doris/analysis/ReplaceTableClause.java | 9 ++-
.../plans/commands/info/AlterMTMVReplaceInfo.java | 2 +-
.../doris/persist/ReplaceTableOperationLog.java| 9 ++-
.../java/org/apache/doris/qe/StmtExecutor.java | 3 +-
.../java/org/apache/doris/alter/AlterTest.java | 4 +-
.../persist/ReplaceTableOperationLogTest.java | 2 +-
...artition.out => test_replace_table_recycle.out} | 9 ++-
...on.groovy => test_replace_table_recycle.groovy} | 71 +-
11 files changed, 92 insertions(+), 47 deletions(-)
copy
regression-test/data/catalog_recycle_bin_p0/{test_insert_overwrite_recover_no_partition.out
=> test_replace_table_recycle.out} (68%)
copy
regression-test/suites/catalog_recycle_bin_p0/{test_insert_overwrite_recover_no_partition.groovy
=> test_replace_table_recycle.groovy} (55%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cluster key) fix some cluster key problems [doris]
mymeiyi commented on PR #43574: URL: https://github.com/apache/doris/pull/43574#issuecomment-2467351576 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](mac-compile):Fix implicit type conversion error [doris]
Tech-Circle-48 opened a new pull request, #43584: URL: https://github.com/apache/doris/pull/43584 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: addresses an implicit type conversion warning in the judge_selectivity function. The existing code divides two int64_t values, filter_rows and input_rows, resulting in a potential precision loss when compared to ignore_threshold, which is a double. This causes a compiler warning about implicit conversion from int64_t to long double. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](segcompaction) fix coredump for inverted index file writer close in segment compaction [doris]
airborne12 merged PR #43477: URL: https://github.com/apache/doris/pull/43477 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](segcompaction) fix coredump for inverted index file writer close in segment compaction (#43477)
This is an automated email from the ASF dual-hosted git repository.
airborne pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 14084eef148 [fix](segcompaction) fix coredump for inverted index file
writer close in segment compaction (#43477)
14084eef148 is described below
commit 14084eef1489b5829867786fc2baa943d9c3811c
Author: airborne12
AuthorDate: Mon Nov 11 14:37:23 2024 +0800
[fix](segcompaction) fix coredump for inverted index file writer close in
segment compaction (#43477)
Related PR: #41625
Problem Summary:
Fix coredump when doing segcompaction after refactor of inverted index
file writer.
Co-authored-by: airborne12
---
be/src/olap/rowset/segcompaction.cpp | 4 ++-
be/src/runtime/exec_env.cpp | 4 +++
be/src/runtime/exec_env.h| 2 ++
be/test/olap/segcompaction_test.cpp | 59
4 files changed, 68 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/rowset/segcompaction.cpp
b/be/src/olap/rowset/segcompaction.cpp
index 330174952b6..427236a6119 100644
--- a/be/src/olap/rowset/segcompaction.cpp
+++ b/be/src/olap/rowset/segcompaction.cpp
@@ -323,7 +323,9 @@ Status
SegcompactionWorker::_do_compact_segments(SegCompactionCandidatesSharedPt
_writer->_num_segcompacted);
}
RETURN_IF_ERROR(_writer->_rename_compacted_segments(begin, end));
-
+if (_inverted_index_file_writer != nullptr) {
+_inverted_index_file_writer.reset();
+}
if (VLOG_DEBUG_IS_ON) {
_writer->vlog_buffer.clear();
for (const auto& entry :
std::filesystem::directory_iterator(ctx.tablet_path)) {
diff --git a/be/src/runtime/exec_env.cpp b/be/src/runtime/exec_env.cpp
index e41cc982a74..872069ee70a 100644
--- a/be/src/runtime/exec_env.cpp
+++ b/be/src/runtime/exec_env.cpp
@@ -45,6 +45,10 @@ ExecEnv::~ExecEnv() {
}
#ifdef BE_TEST
+void ExecEnv::set_inverted_index_searcher_cache(
+segment_v2::InvertedIndexSearcherCache* inverted_index_searcher_cache)
{
+_inverted_index_searcher_cache = inverted_index_searcher_cache;
+}
void ExecEnv::set_storage_engine(std::unique_ptr&& engine) {
_storage_engine = std::move(engine);
}
diff --git a/be/src/runtime/exec_env.h b/be/src/runtime/exec_env.h
index b1617744eac..9321b17070e 100644
--- a/be/src/runtime/exec_env.h
+++ b/be/src/runtime/exec_env.h
@@ -273,6 +273,8 @@ public:
}
void set_storage_engine(std::unique_ptr&& engine);
+void set_inverted_index_searcher_cache(
+segment_v2::InvertedIndexSearcherCache*
inverted_index_searcher_cache);
void set_cache_manager(CacheManager* cm) { this->_cache_manager = cm; }
void set_process_profile(ProcessProfile* pp) { this->_process_profile =
pp; }
void set_tablet_schema_cache(TabletSchemaCache* c) {
this->_tablet_schema_cache = c; }
diff --git a/be/test/olap/segcompaction_test.cpp
b/be/test/olap/segcompaction_test.cpp
index b74bbb8f139..4029467dd42 100644
--- a/be/test/olap/segcompaction_test.cpp
+++ b/be/test/olap/segcompaction_test.cpp
@@ -49,6 +49,7 @@ using namespace ErrorCode;
static const uint32_t MAX_PATH_LEN = 1024;
static StorageEngine* l_engine = nullptr;
static const std::string lTestDir = "./data_test/data/segcompaction_test";
+constexpr static std::string_view tmp_dir = "./data_test/tmp";
class SegCompactionTest : public testing::Test {
public:
@@ -59,6 +60,7 @@ public:
config::tablet_map_shard_size = 1;
config::txn_map_shard_size = 1;
config::txn_shard_size = 1;
+config::inverted_index_fd_number_limit_percent = 0;
char buffer[MAX_PATH_LEN];
EXPECT_NE(getcwd(buffer, MAX_PATH_LEN), nullptr);
@@ -72,6 +74,23 @@ public:
std::vector paths;
paths.emplace_back(config::storage_root_path, -1);
+// tmp dir
+
EXPECT_TRUE(io::global_local_filesystem()->delete_directory(tmp_dir).ok());
+
EXPECT_TRUE(io::global_local_filesystem()->create_directory(tmp_dir).ok());
+paths.emplace_back(std::string(tmp_dir), 102400);
+auto tmp_file_dirs = std::make_unique(paths);
+EXPECT_TRUE(tmp_file_dirs->init().ok());
+ExecEnv::GetInstance()->set_tmp_file_dir(std::move(tmp_file_dirs));
+
+// use memory limit
+int64_t inverted_index_cache_limit = 0;
+_inverted_index_searcher_cache =
std::unique_ptr(
+
InvertedIndexSearcherCache::create_global_instance(inverted_index_cache_limit,
+ 256));
+
+ExecEnv::GetInstance()->set_inverted_index_searcher_cache(
+_inverted_index_searcher_cache.get());
+
doris::EngineOptions options;
options.store_paths = paths;
@@ -99,6 +118,7 @@ public:
ExecEnv* exec_env = doris::ExecEnv::GetI
Re: [PR] branch-3.0: [Enhancement] replace table options to keep original dropped table in recycle bin [doris]
doris-robot commented on PR #43585: URL: https://github.com/apache/doris/pull/43585#issuecomment-2467352837 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Enhancement] replace table options to keep original dropped table in recycle bin [doris]
doris-robot commented on PR #43585: URL: https://github.com/apache/doris/pull/43585#issuecomment-2467352816 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cherry-pick](branch-30)fix the exchange not skip offset rows (#42190) [doris]
zhangstar333 commented on PR #42490: URL: https://github.com/apache/doris/pull/42490#issuecomment-2467353828 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) Fix aws sdk default retry policy not taking effect [doris]
doris-robot commented on PR #43575: URL: https://github.com/apache/doris/pull/43575#issuecomment-2467405525 TPC-H: Total hot run time: 51720 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b24df3eaa9426633ff4e3f7d44602914f4db2677, data reload: false -- Round 1 -- q1 17569 756272907290 q2 2581132512861286 q3 9947117611511151 q4 10237 860 867 860 q5 7601295129452945 q6 235 144 144 144 q7 1030611 593 593 q8 9373237123662366 q9 12626 12662 12441 12441 q10 7157239324232393 q11 462 257 256 256 q12 430 211 217 211 q13 17793 301930233019 q14 251 210 212 210 q15 569 507 501 501 q16 648 602 609 602 q17 985 588 559 559 q18 7343674667766746 q19 13311011943 943 q20 3062284328102810 q21 4006312630303030 q22 1407136413841364 Total cold run time: 116643 ms Total hot run time: 51720 ms - Round 2, with runtime_filter_mode=off - q1 7478758774787478 q2 339 247 239 239 q3 3115301229822982 q4 2039183918111811 q5 56595636 q6 227 135 135 135 q7 2215176317981763 q8 3430359235343534 q9 14631 14791 14565 14565 q10 3647358635553555 q11 591 493 493 493 q12 815 614 603 603 q13 13210 320132663201 q14 319 276 252 252 q15 596 535 554 535 q16 677 661 626 626 q17 1842161416021602 q18 8299773275747574 q19 1693154915941549 q20 2154193419191919 q21 5405507451435074 q22 633 547 536 536 Total cold run time: 79014 ms Total hot run time: 65581 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] (bloom filter) Fix the bloom filter calculation for date and datetime [doris]
github-actions[bot] commented on PR #43351: URL: https://github.com/apache/doris/pull/43351#issuecomment-246745 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] (bloom filter) Fix the bloom filter calculation for date and datetime [doris]
github-actions[bot] commented on PR #43351: URL: https://github.com/apache/doris/pull/43351#issuecomment-2467411067 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
doris-robot commented on PR #43592: URL: https://github.com/apache/doris/pull/43592#issuecomment-2467411813 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
Mryange commented on PR #43592: URL: https://github.com/apache/doris/pull/43592#issuecomment-2467412492 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [only test] [doris]
Mryange opened a new pull request, #43592: URL: https://github.com/apache/doris/pull/43592 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.1](log) Pick all BE execution log reduction [doris]
github-actions[bot] commented on PR #43267: URL: https://github.com/apache/doris/pull/43267#issuecomment-2467305665 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] debug perfmance [doris]
hello-stephen commented on PR #43578: URL: https://github.com/apache/doris/pull/43578#issuecomment-2467311132 run performance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-3.0](log) Pick all BE execution log reduction [doris]
github-actions[bot] commented on PR #43579: URL: https://github.com/apache/doris/pull/43579#issuecomment-2467312128 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](build index)Optimize failed task check on same tablet [doris]
github-actions[bot] commented on PR #42295: URL: https://github.com/apache/doris/pull/42295#issuecomment-2467312539 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](backup) Load backup meta and job info bytes from disk #43276 (#43519)
This is an automated email from the ASF dual-hosted git repository.
w41ter pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 5dda61b410c [fix](backup) Load backup meta and job info bytes from
disk #43276 (#43519)
5dda61b410c is described below
commit 5dda61b410cdde1dc7495dee9a95eeafe6f6fb25
Author: walter
AuthorDate: Mon Nov 11 14:08:35 2024 +0800
[fix](backup) Load backup meta and job info bytes from disk #43276 (#43519)
cherry pick from #43276
---
.../java/org/apache/doris/backup/AbstractJob.java | 2 +
.../org/apache/doris/backup/BackupHandler.java | 50 +++---
.../java/org/apache/doris/backup/BackupJob.java| 45 ---
.../java/org/apache/doris/backup/RestoreJob.java | 5 +++
.../apache/doris/service/FrontendServiceImpl.java | 7 ++-
5 files changed, 85 insertions(+), 24 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/AbstractJob.java
b/fe/fe-core/src/main/java/org/apache/doris/backup/AbstractJob.java
index 4e2c3fd1990..f22598dd86b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/backup/AbstractJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/backup/AbstractJob.java
@@ -155,6 +155,8 @@ public abstract class AbstractJob implements Writable {
public abstract boolean isCancelled();
+public abstract boolean isFinished();
+
public abstract Status updateRepo(Repository repo);
public static AbstractJob read(DataInput in) throws IOException {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java
b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java
index d9b7659cfc1..49190acce1a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java
@@ -109,10 +109,10 @@ public class BackupHandler extends MasterDaemon
implements Writable {
private Env env;
-// map to store backup info, key is label name, value is Pair,
meta && info is bytes
-// this map not present in persist && only in fe master memory
+// map to store backup info, key is label name, value is the BackupJob
+// this map not present in persist && only in fe memory
// one table only keep one snapshot info, only keep last
-private final Map localSnapshots = new HashMap<>();
+private final Map localSnapshots = new HashMap<>();
private ReadWriteLock localSnapshotsLock = new ReentrantReadWriteLock();
public BackupHandler() {
@@ -167,6 +167,7 @@ public class BackupHandler extends MasterDaemon implements
Writable {
return false;
}
}
+
isInit = true;
return true;
}
@@ -544,11 +545,15 @@ public class BackupHandler extends MasterDaemon
implements Writable {
return;
}
+List removedLabels = Lists.newArrayList();
jobLock.lock();
try {
Deque jobs =
dbIdToBackupOrRestoreJobs.computeIfAbsent(dbId, k -> Lists.newLinkedList());
while (jobs.size() >= Config.max_backup_restore_job_num_per_db) {
-jobs.removeFirst();
+AbstractJob removedJob = jobs.removeFirst();
+if (removedJob instanceof BackupJob && ((BackupJob)
removedJob).isLocalSnapshot()) {
+removedLabels.add(removedJob.getLabel());
+}
}
AbstractJob lastJob = jobs.peekLast();
@@ -561,6 +566,17 @@ public class BackupHandler extends MasterDaemon implements
Writable {
} finally {
jobLock.unlock();
}
+
+if (job.isFinished() && job instanceof BackupJob) {
+// Save snapshot to local repo, when reload backupHandler from
image.
+BackupJob backupJob = (BackupJob) job;
+if (backupJob.isLocalSnapshot()) {
+addSnapshot(backupJob.getLabel(), backupJob);
+}
+}
+for (String label : removedLabels) {
+removeSnapshot(label);
+}
}
private List getAllCurrentJobs() {
@@ -799,22 +815,42 @@ public class BackupHandler extends MasterDaemon
implements Writable {
return false;
}
-public void addSnapshot(String labelName, Snapshot snapshot) {
+public void addSnapshot(String labelName, BackupJob backupJob) {
+assert backupJob.isFinished();
+
+LOG.info("add snapshot {} to local repo", labelName);
localSnapshotsLock.writeLock().lock();
try {
-localSnapshots.put(labelName, snapshot);
+localSnapshots.put(labelName, backupJob);
+} finally {
+localSnapshotsLock.writeLock().unlock();
+}
+}
+
+public void removeSnapshot(String labelName) {
+LOG.info("remove snapshot {} from local repo", l
Re: [PR] [Fix](Nereids) fix floor/round/ceil/truncate functions type compute precision problem [doris]
LiBinfeng-01 commented on PR #43422: URL: https://github.com/apache/doris/pull/43422#issuecomment-2467317889 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](backup) Load backup meta and job info bytes from disk #43276 [doris]
w41ter merged PR #43519: URL: https://github.com/apache/doris/pull/43519 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](function) change hash set usage of all functions [doris]
BiteThet merged PR #43289: URL: https://github.com/apache/doris/pull/43289 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Chore](function) change hash set usage of all functions (#43289)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1b49f847e2d [Chore](function) change hash set usage of all functions
(#43289)
1b49f847e2d is described below
commit 1b49f847e2df8e5288006e8438b747f378913c37
Author: Pxl
AuthorDate: Mon Nov 11 14:08:53 2024 +0800
[Chore](function) change hash set usage of all functions (#43289)
remove all ck hash_set usage and use phmap_set
some test results have changed because the order in which data is stored
within the hash table has changed.
---
.../aggregate_function_collect.h | 38 ++-
.../aggregate_function_distinct.h | 44 +++---
.../vec/functions/array/function_array_distinct.h | 10 ++---
.../vec/functions/array/function_array_except.cpp | 4 +-
be/src/vec/functions/array/function_array_map.h| 1 -
be/src/vec/functions/array/function_array_set.h| 5 +--
.../vec/functions/array/function_arrays_overlap.h | 9 ++---
.../test_vertical_compaction_agg_state.out | 4 +-
.../data/function_p0/test_agg_foreach.out | 2 +-
.../data/function_p0/test_agg_foreach_notnull.out | 2 +-
.../agg_function/group_unique_array.out| 16
.../test_aggregate_window_functions.out| 16
.../test_vertical_compaction_agg_state.groovy | 4 +-
.../agg_function/group_unique_array.groovy | 4 +-
14 files changed, 80 insertions(+), 79 deletions(-)
diff --git a/be/src/vec/aggregate_functions/aggregate_function_collect.h
b/be/src/vec/aggregate_functions/aggregate_function_collect.h
index 68de426ea1f..02490be56a0 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_collect.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_collect.h
@@ -35,7 +35,6 @@
#include "vec/columns/column_string.h"
#include "vec/columns/columns_number.h"
#include "vec/common/assert_cast.h"
-#include "vec/common/hash_table/hash_set.h"
#include "vec/common/pod_array_fwd.h"
#include "vec/common/string_buffer.hpp"
#include "vec/common/string_ref.h"
@@ -62,7 +61,7 @@ struct AggregateFunctionCollectSetData {
using ColVecType = ColumnVectorOrDecimal;
using ElementNativeType = typename NativeType::Type;
using SelfType = AggregateFunctionCollectSetData;
-using Set = HashSetWithStackMemory, 4>;
+using Set = phmap::flat_hash_set;
Set data_set;
Int64 max_size = -1;
@@ -83,20 +82,29 @@ struct AggregateFunctionCollectSetData {
if (size() >= max_size) {
return;
}
-data_set.insert(rhs_elem.get_value());
+data_set.insert(rhs_elem);
}
} else {
-data_set.merge(rhs.data_set);
+data_set.merge(Set(rhs.data_set));
}
}
void write(BufferWritable& buf) const {
-data_set.write(buf);
+write_var_uint(data_set.size(), buf);
+for (const auto& value : data_set) {
+write_binary(value, buf);
+}
write_var_int(max_size, buf);
}
void read(BufferReadable& buf) {
-data_set.read(buf);
+size_t new_size = 0;
+read_var_uint(new_size, buf);
+ElementNativeType x;
+for (size_t i = 0; i < new_size; ++i) {
+read_binary(x, buf);
+data_set.insert(x);
+}
read_var_int(max_size, buf);
}
@@ -104,7 +112,7 @@ struct AggregateFunctionCollectSetData {
auto& vec = assert_cast(to).get_data();
vec.reserve(size());
for (const auto& item : data_set) {
-vec.push_back(item.key);
+vec.push_back(item);
}
}
@@ -116,23 +124,19 @@ struct AggregateFunctionCollectSetData {
using ElementType = StringRef;
using ColVecType = ColumnString;
using SelfType = AggregateFunctionCollectSetData;
-using Set = HashSetWithStackMemory,
4>;
+using Set = phmap::flat_hash_set;
Set data_set;
Int64 max_size = -1;
size_t size() const { return data_set.size(); }
void add(const IColumn& column, size_t row_num, Arena* arena) {
-Set::LookupResult it;
-bool inserted;
auto key = column.get_data_at(row_num);
key.data = arena->insert(key.data, key.size);
-data_set.emplace(key, it, inserted);
+data_set.insert(key);
}
void merge(const SelfType& rhs, Arena* arena) {
-bool inserted;
-Set::LookupResult it;
if (max_size == -1) {
max_size = rhs.max_size;
}
@@ -145,16 +149,16 @@ struct AggregateFunctionCollectSetData {
}
}
assert(arena != nullptr);
-StringRef key = rhs_elem.get_value();
+StringRef key = rhs_e
Re: [PR] [fix](variant) fix index in variant [doris]
github-actions[bot] commented on PR #43375: URL: https://github.com/apache/doris/pull/43375#issuecomment-2467321175 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](variant) fix index in variant [doris]
github-actions[bot] commented on PR #43375: URL: https://github.com/apache/doris/pull/43375#issuecomment-2467321132 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](build index)Optimize failed task check on same tablet [doris]
qidaye merged PR #42295: URL: https://github.com/apache/doris/pull/42295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
doris-robot commented on PR #41701: URL: https://github.com/apache/doris/pull/41701#issuecomment-2467295779 TeamCity be ut coverage result: Function Coverage: 37.88% (9867/26047) Line Coverage: 29.05% (82244/283147) Region Coverage: 28.20% (42342/150156) Branch Coverage: 24.74% (21437/86638) Coverage Report: http://coverage.selectdb-in.cc/coverage/a35d1360e38a6d95dbf89a478f03033af139bd79_a35d1360e38a6d95dbf89a478f03033af139bd79/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) optimize small sql [doris]
github-actions[bot] commented on PR #43546: URL: https://github.com/apache/doris/pull/43546#issuecomment-2467301295 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [BUG] Doris Stream Load Memory Overcommit Issue Causing Cancellation (Data Does Not Exceed GC Settings) [doris]
Han-lai commented on issue #43550: URL: https://github.com/apache/doris/issues/43550#issuecomment-2467285370 > How much data was loaded in on stream load?流載入時載入了多少數據？ In this stream load process, a total of 100 rows of data were loaded, as indicated by the LoadedRows column. The LoadBytes field shows that the data size loaded is 618,362,793 bytes (approximately 618 MB).In this stream load process, a total of 100 rows of data were loaded, as indicated by the LoadedRows column. The LoadBytes field shows that the data size loaded is 618,362,793 bytes (approximately 618 MB). However, one of my fields is in JSON format, and its length is very large, with the total length exceeding 5,000,000. `AND length(Message) >= 500` | Label | Db | Table | ClientIp| Status | Message | Url | TotalRows | LoadedRows | FilteredRows | UnselectedRows | LoadBytes | StartTime | FinishTime | User | Comment | |---|-|---|-||-|-|---||--||---|-|-|---|-| | pkmsg_autobkt_LIMIT_905708683449597953_0_1730704481157 | test_variant | pkmsg_autobkt_LIMIT | 1X.XXX.XXX.XXX | Fail | [CANCELLED]GC wg for hard limit, wg id:1, name:normal, used:2.99 GB, limit:2.70 GB, backend:1X.XXX.XXX.XX. cancel top memory used tracker consumption 2.66 GB. details:process memory used 2.63 GB exceed soft limit 9.72 GB or sys available memory 10.23 GB less than warning water mark 1.20 GB., Execute again after enough memory, details see be.INFO. | N/A | 100 | 100| 0 | 0 | 618362793 | 2024-11-04 07:14:41.232 | 2024-11-04 07:16:07.364 | root | | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.1](log) Pick all BE execution log reduction [doris]
zclllyybb commented on PR #43267: URL: https://github.com/apache/doris/pull/43267#issuecomment-2467295949 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) optimize small sql [doris]
924060929 commented on PR #43546: URL: https://github.com/apache/doris/pull/43546#issuecomment-2467296843 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.1](log) Pick all BE execution log reduction [doris]
zclllyybb commented on PR #43267: URL: https://github.com/apache/doris/pull/43267#issuecomment-2467297988 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] debug perfmance [doris]
github-actions[bot] commented on PR #43578: URL: https://github.com/apache/doris/pull/43578#issuecomment-2467270787 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-3.0](log) Pick all BE execution log reduction [doris]
doris-robot commented on PR #43579: URL: https://github.com/apache/doris/pull/43579#issuecomment-2467302383 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [branch-3.0](log) Pick all BE execution log reduction [doris]
zclllyybb opened a new pull request, #43579: URL: https://github.com/apache/doris/pull/43579 pick https://github.com/apache/doris/pull/42383 pick https://github.com/apache/doris/pull/42900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-3.0](log) Pick all BE execution log reduction [doris]
zclllyybb commented on PR #43579: URL: https://github.com/apache/doris/pull/43579#issuecomment-2467302390 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.1](log) Pick all BE execution log reduction [doris]
github-actions[bot] commented on PR #43267: URL: https://github.com/apache/doris/pull/43267#issuecomment-2467303546 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
