Re: [PR] [improve](explode-json-array)change explode-json-array-xx func signature from string to json type [doris]
doris-robot commented on PR #37278: URL: https://github.com/apache/doris/pull/37278#issuecomment-2208258105 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner) fix bug of select stmt toSql [doris]
feiniaofeiafei commented on PR #37274: URL: https://github.com/apache/doris/pull/37274#issuecomment-2208260828 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [improve](serde) support json string format with escaped charactors (#37176)
This is an automated email from the ASF dual-hosted git repository.
eldenmoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 195fb730d40 [improve](serde) support json string format with escaped
charactors (#37176)
195fb730d40 is described below
commit 195fb730d409c522da7bf03705cf9adb633fb679
Author: amory
AuthorDate: Thu Jul 4 15:06:06 2024 +0800
[improve](serde) support json string format with escaped charactors (#37176)
before this pr
if we use some escape from nested types like array, we can not make this
string cast to json
---
.../vec/data_types/serde/data_type_string_serde.h | 46 -
be/src/vec/functions/function_cast.h | 1 +
regression-test/data/jsonb_p0/test_jsonb_cast.csv | 4 ++
regression-test/data/jsonb_p0/test_jsonb_cast.out | 27
.../suites/jsonb_p0/test_jsonb_cast.groovy | 79 ++
5 files changed, 154 insertions(+), 3 deletions(-)
diff --git a/be/src/vec/data_types/serde/data_type_string_serde.h
b/be/src/vec/data_types/serde/data_type_string_serde.h
index b74b5857086..24f99a12e67 100644
--- a/be/src/vec/data_types/serde/data_type_string_serde.h
+++ b/be/src/vec/data_types/serde/data_type_string_serde.h
@@ -73,19 +73,59 @@ public:
auto result = check_column_const_set_readability(column, row_num);
ColumnPtr ptr = result.first;
row_num = result.second;
+const auto& value = assert_cast(*ptr).get_data_at(row_num);
if (_nesting_level > 1) {
bw.write('"');
}
-
-const auto& value = assert_cast(*ptr).get_data_at(row_num);
-bw.write(value.data, value.size);
+if constexpr (std::is_same_v) {
+if (options.escape_char != 0) {
+// we should make deal with some special characters in json
str if we have escape_char
+StringRef str_ref = value;
+write_with_escaped_char_to_json(str_ref, bw);
+} else {
+bw.write(value.data, value.size);
+}
+} else {
+bw.write(value.data, value.size);
+}
if (_nesting_level > 1) {
bw.write('"');
}
+
return Status::OK();
}
+inline void write_with_escaped_char_to_json(StringRef value,
BufferWritable& bw) const {
+for (char it : value) {
+switch (it) {
+case '\b':
+bw.write("\\b", 2);
+break;
+case '\f':
+bw.write("\\f", 2);
+break;
+case '\n':
+bw.write("\\n", 2);
+break;
+case '\r':
+bw.write("\\r", 2);
+break;
+case '\t':
+bw.write("\\t", 2);
+break;
+case '\\':
+bw.write("", 2);
+break;
+case '"':
+bw.write("\\\"", 2);
+break;
+default:
+bw.write(it);
+}
+}
+}
+
Status serialize_column_to_json(const IColumn& column, int start_idx, int
end_idx,
BufferWritable& bw, FormatOptions&
options) const override {
SERIALIZE_COLUMN_TO_JSON();
diff --git a/be/src/vec/functions/function_cast.h
b/be/src/vec/functions/function_cast.h
index 17250e10fd7..d4b21aacc5c 100644
--- a/be/src/vec/functions/function_cast.h
+++ b/be/src/vec/functions/function_cast.h
@@ -766,6 +766,7 @@ struct ConvertImplGenericToJsonb {
auto tmp_col = ColumnString::create();
vectorized::DataTypeSerDe::FormatOptions options;
+options.escape_char = '\\';
for (size_t i = 0; i < input_rows_count; i++) {
// convert to string
tmp_col->clear();
diff --git a/regression-test/data/jsonb_p0/test_jsonb_cast.csv
b/regression-test/data/jsonb_p0/test_jsonb_cast.csv
new file mode 100644
index 000..08b694ddea8
--- /dev/null
+++ b/regression-test/data/jsonb_p0/test_jsonb_cast.csv
@@ -0,0 +1,4 @@
+1 \N
+2 ['{\'x\' : \'{"y" : 1}\', \'t\' : \'{"y" : 2}\'}', '{"x" : 1}']
+3 ['foo\'bar', 'foo"bar', 'foo\\'bar', 'foo\'\'bar']
+4 ['\/some\/cool\/url', '/some/cool/url',
'a\\_\\c\\l\\i\\c\\k\\h\\o\\u\\s\\e']
\ No newline at end of file
diff --git a/regression-test/data/jsonb_p0/test_jsonb_cast.out
b/regression-test/data/jsonb_p0/test_jsonb_cast.out
new file mode 100644
index 000..2ab4174c746
--- /dev/null
+++ b/regression-test/data/jsonb_p0/test_jsonb_cast.out
@@ -0,0 +1,27 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select_1 --
+1 \N
+2 ["{\\'x\\' : \\'{"y" : 1}\\', \\'t\\' : \\'{"y" : 2}\\'}", "{"x" : 1}"]
+3 ["foo\\'bar', 'foo"bar', 'foo'bar', 'foo\\'\
Re: [PR] [improve](serde) support json string format with escaped charactors [doris]
eldenmoon merged PR #37176: URL: https://github.com/apache/doris/pull/37176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](Cloud) Avoid unnecessary RPC to meta service for watershed txn id. [doris]
doris-robot commented on PR #37204: URL: https://github.com/apache/doris/pull/37204#issuecomment-2208267419 TPC-H: Total hot run time: 40089 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 1c48102ef642c9f17ec33014f9280c84fe82b157, data reload: false -- Round 1 -- q1 18072 539944304430 q2 2567198 194 194 q3 12113 117910571057 q4 10849 808 812 808 q5 7882270426612661 q6 225 140 142 140 q7 961 600 606 600 q8 9221204920792049 q9 8916647564376437 q10 8956372037343720 q11 467 241 242 241 q12 423 242 228 228 q13 17886 299629842984 q14 257 240 219 219 q15 525 484 483 483 q16 509 377 390 377 q17 963 671 690 671 q18 8082741774687417 q19 2228157515781575 q20 672 340 330 330 q21 4997312738933127 q22 406 343 341 341 Total cold run time: 117177 ms Total hot run time: 40089 ms - Round 2, with runtime_filter_mode=off - q1 4422430942474247 q2 368 269 260 260 q3 3023272327772723 q4 1867164716351635 q5 5271526552885265 q6 221 129 132 129 q7 2109175516861686 q8 3182332333103310 q9 8348832683378326 q10 3871370236053605 q11 604 495 508 495 q12 788 624 607 607 q13 16996 299130242991 q14 305 265 254 254 q15 521 487 482 482 q16 457 403 417 403 q17 1755158714581458 q18 7708744173027302 q19 4747161815611561 q20 1997178617561756 q21 4827479046804680 q22 638 518 543 518 Total cold run time: 74025 ms Total hot run time: 53693 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](regression) Fix regression run in cloud use non-root user #34340 [doris]
deardeng opened a new pull request, #37279: URL: https://github.com/apache/doris/pull/37279 cherry pick from #34340 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression) Fix regression run in cloud use non-root user #34340 [doris]
deardeng commented on PR #37279: URL: https://github.com/apache/doris/pull/37279#issuecomment-2208270592 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression) Fix regression run in cloud use non-root user #34340 [doris]
doris-robot commented on PR #37279: URL: https://github.com/apache/doris/pull/37279#issuecomment-2208270636 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix]check nullptr when log thread num [doris]
github-actions[bot] commented on PR #37263: URL: https://github.com/apache/doris/pull/37263#issuecomment-2208273539 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression) Fix regression run in cloud use non-root user #34340 [doris]
github-actions[bot] commented on PR #37279: URL: https://github.com/apache/doris/pull/37279#issuecomment-2208274312 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](HadoopLz4BlockCompression)Fixed the bug that HadoopLz4BlockCompression creates _decompressor every time it decompresses. [doris]
morningman merged PR #37187: URL: https://github.com/apache/doris/pull/37187 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (195fb730d40 -> ac17cfa14f5)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 195fb730d40 [improve](serde) support json string format with escaped charactors (#37176) add ac17cfa14f5 [fix](HadoopLz4BlockCompression)Fixed the bug that HadoopLz4BlockCompression creates _decompressor every time it decompresses. (#37187) No new revisions were added by this update. Summary of changes: be/src/util/block_compression.cpp | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix]check nullptr when log thread num [doris]
github-actions[bot] commented on PR #37263: URL: https://github.com/apache/doris/pull/37263#issuecomment-2208273603 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression) Fix regression run in cloud use non-root user #34340 [doris]
github-actions[bot] commented on PR #37279: URL: https://github.com/apache/doris/pull/37279#issuecomment-2208274253 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipeline](datagen) Improve datagen operator parallism [doris]
github-actions[bot] commented on PR #37195: URL: https://github.com/apache/doris/pull/37195#issuecomment-2208279968 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [case](paimon/iceberg)move cases from p2 to p0 [doris]
doris-robot commented on PR #37276: URL: https://github.com/apache/doris/pull/37276#issuecomment-2208283208 TPC-H: Total hot run time: 39645 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 431c2cbaf4d489b8f0b9521f68716f6a00a20821, data reload: false -- Round 1 -- q1 17630 431942934293 q2 2013190 189 189 q3 10442 113711471137 q4 10180 790 800 790 q5 7504266625472547 q6 216 140 134 134 q7 954 593 614 593 q8 9217206820532053 q9 8883650564376437 q10 9009373637083708 q11 446 241 239 239 q12 477 236 227 227 q13 17768 299129932991 q14 271 224 220 220 q15 516 478 481 478 q16 517 370 368 368 q17 962 627 642 627 q18 7983744173637363 q19 5129141614241416 q20 661 320 345 320 q21 4904318037693180 q22 387 335 341 335 Total cold run time: 116069 ms Total hot run time: 39645 ms - Round 2, with runtime_filter_mode=off - q1 4364424742794247 q2 368 267 261 261 q3 2984273429352734 q4 2027172617081708 q5 5632551654695469 q6 222 134 130 130 q7 2143183519271835 q8 3251343234493432 q9 8709862788258627 q10 4100390937523752 q11 582 498 497 497 q12 828 642 619 619 q13 17273 314632033146 q14 307 279 281 279 q15 534 479 475 475 q16 490 432 438 432 q17 1826152215331522 q18 8112787977437743 q19 2057153517221535 q20 2544188818711871 q21 5215502448944894 q22 624 535 556 535 Total cold run time: 74192 ms Total hot run time: 55743 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](Cloud) Avoid unnecessary RPC to meta service for watershed txn id. [doris]
doris-robot commented on PR #37204: URL: https://github.com/apache/doris/pull/37204#issuecomment-2208284182 TPC-DS: Total hot run time: 172744 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 1c48102ef642c9f17ec33014f9280c84fe82b157, data reload: false query1 938 377 367 367 query2 6472253423132313 query3 6669206 208 206 query4 20578 17464 17492 17464 query5 4184504 499 499 query6 286 170 176 170 query7 4603300 300 300 query8 322 305 295 295 query9 8562245824342434 query10 602 302 299 299 query11 10413 10005 10047 10005 query12 137 85 86 85 query13 1654375 410 375 query14 10226 759872127212 query15 236 192 194 192 query16 7914324 314 314 query17 1803584 515 515 query18 1997276 267 267 query19 197 145 150 145 query20 88 81 82 81 query21 215 145 126 126 query22 4374399640013996 query23 33705 33073 33081 33073 query24 11870 283528382835 query25 658 360 361 360 query26 1755156 151 151 query27 2963311 316 311 query28 7267210020852085 query29 1040653 607 607 query30 267 149 155 149 query31 936 724 725 724 query32 93 52 54 52 query33 768 293 282 282 query34 955 455 476 455 query35 748 639 604 604 query36 1092975 916 916 query37 194 82 76 76 query38 2879273927942739 query39 840 828 801 801 query40 281 131 127 127 query41 57 52 51 51 query42 123 102 104 102 query43 595 542 568 542 query44 1171738 740 738 query45 205 164 163 163 query46 1069753 741 741 query47 1879178518001785 query48 366 293 301 293 query49 1243426 411 411 query50 763 380 393 380 query51 6837684468596844 query52 108 95 91 91 query53 363 291 292 291 query54 939 458 453 453 query55 79 75 75 75 query56 293 264 273 264 query57 1210105210821052 query58 265 259 252 252 query59 3397316533563165 query60 298 275 275 275 query61 95 92 97 92 query62 655 450 446 446 query63 332 291 290 290 query64 9846225917571757 query65 3150315831043104 query66 1386333 329 329 query67 15567 14922 14924 14922 query68 4615570 558 558 query69 466 310 311 310 query70 1162115011681150 query71 402 276 287 276 query72 7287535355795353 query73 736 328 326 326 query74 5897546555525465 query75 3423268526512651 query76 2528942 945 942 query77 454 308 307 307 query78 9426912688138813 query79 2276526 527 526 query80 1340478 463 463 query81 579 219 223 219 query82 756 104 106 104 query83 244 167 165 165 query84 241 86 85 85 query85 2042287 273 273 query86 497 310 333 310 query87 3264316231113111 query88 3673237223582358 query89 485 396 386 386 query90 1915192 193 192 query91 129 99 103 99 query92 61 50 48 48 query93 2459519 515 515 query94 1284213 209 209 query95 401 312 315 312 query96 595 268 269 268 query97 3198301829952995 query98 214 207 199 199 query99 1347846 841 841 Total cold run time: 276468 ms Total hot run time: 172744 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [Improvement](Cloud) Avoid unnecessary RPC to meta service for watershed txn id. [doris]
doris-robot commented on PR #37204: URL: https://github.com/apache/doris/pull/37204#issuecomment-2208293305 ClickBench: Total hot run time: 30.97 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 1c48102ef642c9f17ec33014f9280c84fe82b157, data reload: false query1 0.040.030.03 query2 0.080.040.04 query3 0.220.050.05 query4 1.660.090.09 query5 0.500.480.48 query6 1.150.730.73 query7 0.020.020.01 query8 0.050.040.04 query9 0.550.500.49 query10 0.550.550.55 query11 0.150.110.12 query12 0.160.120.13 query13 0.610.590.59 query14 0.800.790.80 query15 0.820.830.82 query16 0.370.370.36 query17 1.020.991.00 query18 0.220.240.25 query19 1.871.821.69 query20 0.010.020.01 query21 15.41 0.760.66 query22 4.066.932.16 query23 18.31 1.371.26 query24 2.180.240.23 query25 0.170.090.08 query26 0.270.180.18 query27 0.090.090.08 query28 13.18 1.011.01 query29 12.60 3.303.29 query30 0.250.060.05 query31 2.870.390.39 query32 3.270.490.48 query33 2.902.942.86 query34 17.23 4.434.41 query35 4.514.544.51 query36 0.660.470.47 query37 0.190.160.16 query38 0.170.160.15 query39 0.040.030.04 query40 0.160.150.15 query41 0.100.050.05 query42 0.060.040.04 query43 0.040.040.04 Total cold run time: 109.57 s Total hot run time: 30.97 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix]check nullptr when log thread num [doris]
wangbo merged PR #37263: URL: https://github.com/apache/doris/pull/37263 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick]log thread num [doris]
wangbo merged PR #37258: URL: https://github.com/apache/doris/pull/37258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (ac17cfa14f5 -> 8eb9a1a0661)
This is an automated email from the ASF dual-hosted git repository. wangbo pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ac17cfa14f5 [fix](HadoopLz4BlockCompression)Fixed the bug that HadoopLz4BlockCompression creates _decompressor every time it decompresses. (#37187) add 8eb9a1a0661 [Fix]check nullptr when log thread num (#37263) No new revisions were added by this update. Summary of changes: be/src/runtime/workload_group/workload_group.cpp | 30 +--- 1 file changed, 21 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [pick]log thread num (#37258)
This is an automated email from the ASF dual-hosted git repository.
wangbo pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new b272247a578 [pick]log thread num (#37258)
b272247a578 is described below
commit b272247a578a49a6f48292c2596baf35da034984
Author: wangbo
AuthorDate: Thu Jul 4 15:27:52 2024 +0800
[pick]log thread num (#37258)
## Proposed changes
pick #37159
---
be/src/agent/cgroup_cpu_ctl.cpp | 7 +++--
be/src/agent/cgroup_cpu_ctl.h| 4 +--
be/src/agent/workload_group_listener.cpp | 14 -
be/src/common/config.cpp | 1 -
be/src/common/config.h | 1 -
be/src/pipeline/task_scheduler.h | 2 ++
be/src/runtime/fragment_mgr.cpp | 4 +--
be/src/runtime/workload_group/workload_group.cpp | 40 +++-
be/src/runtime/workload_group/workload_group.h | 6 ++--
be/src/util/threadpool.h | 7 +
be/src/vec/exec/scan/scanner_scheduler.h | 2 ++
11 files changed, 61 insertions(+), 27 deletions(-)
diff --git a/be/src/agent/cgroup_cpu_ctl.cpp b/be/src/agent/cgroup_cpu_ctl.cpp
index 3fe0778ecba..b141676545e 100644
--- a/be/src/agent/cgroup_cpu_ctl.cpp
+++ b/be/src/agent/cgroup_cpu_ctl.cpp
@@ -130,7 +130,7 @@ Status CgroupV1CpuCtl::init() {
return Status::InternalError("invalid cgroup path, not find cpu
quota file");
}
-if (_tg_id == -1) {
+if (_wg_id == -1) {
// means current cgroup cpu ctl is just used to clear dir,
// it does not contains workload group.
// todo(wb) rethinking whether need to refactor cgroup_cpu_ctl
@@ -140,7 +140,7 @@ Status CgroupV1CpuCtl::init() {
}
// workload group path
-_cgroup_v1_cpu_tg_path = _cgroup_v1_cpu_query_path + "/" +
std::to_string(_tg_id);
+_cgroup_v1_cpu_tg_path = _cgroup_v1_cpu_query_path + "/" +
std::to_string(_wg_id);
if (access(_cgroup_v1_cpu_tg_path.c_str(), F_OK) != 0) {
int ret = mkdir(_cgroup_v1_cpu_tg_path.c_str(), S_IRWXU);
if (ret != 0) {
@@ -186,7 +186,8 @@ Status CgroupV1CpuCtl::add_thread_to_cgroup() {
return Status::OK();
#else
int tid = static_cast(syscall(SYS_gettid));
-std::string msg = "add thread " + std::to_string(tid) + " to group";
+std::string msg =
+"add thread " + std::to_string(tid) + " to group" + " " +
std::to_string(_wg_id);
std::lock_guard w_lock(_lock_mutex);
return CgroupCpuCtl::write_cg_sys_file(_cgroup_v1_cpu_tg_task_file, tid,
msg, true);
#endif
diff --git a/be/src/agent/cgroup_cpu_ctl.h b/be/src/agent/cgroup_cpu_ctl.h
index 1289f26307b..b5f8d2d5d80 100644
--- a/be/src/agent/cgroup_cpu_ctl.h
+++ b/be/src/agent/cgroup_cpu_ctl.h
@@ -35,7 +35,7 @@ class CgroupCpuCtl {
public:
virtual ~CgroupCpuCtl() = default;
CgroupCpuCtl() = default;
-CgroupCpuCtl(uint64_t tg_id) { _tg_id = tg_id; }
+CgroupCpuCtl(uint64_t wg_id) { _wg_id = wg_id; }
virtual Status init();
@@ -63,7 +63,7 @@ protected:
int _cpu_hard_limit = 0;
std::shared_mutex _lock_mutex;
bool _init_succ = false;
-uint64_t _tg_id = -1; // workload group id
+uint64_t _wg_id = -1; // workload group id
uint64_t _cpu_shares = 0;
};
diff --git a/be/src/agent/workload_group_listener.cpp
b/be/src/agent/workload_group_listener.cpp
index 822e3c692f7..aca3ea597ce 100644
--- a/be/src/agent/workload_group_listener.cpp
+++ b/be/src/agent/workload_group_listener.cpp
@@ -43,13 +43,13 @@ void WorkloadGroupListener::handle_topic_info(const
std::vector& topi
current_wg_ids.insert(workload_group_info.id);
}
if (!ret.ok()) {
-LOG(INFO) << "[topic_publish_wg]parse topic info failed, tg_id="
+LOG(INFO) << "[topic_publish_wg]parse topic info failed, wg_id="
<< workload_group_info.id << ", reason:" <<
ret.to_string();
continue;
}
// 2 update workload group
-auto tg =
+auto wg =
_exec_env->workload_group_mgr()->get_or_create_workload_group(workload_group_info);
// 3 set cpu soft hard limit switch
@@ -57,17 +57,15 @@ void WorkloadGroupListener::handle_topic_info(const
std::vector& topi
workload_group_info.enable_cpu_hard_limit);
// 4 create and update task scheduler
-tg->upsert_task_scheduler(&workload_group_info, _exec_env);
+wg->upsert_task_scheduler(&workload_group_info, _exec_env);
-LOG(INFO) << "[topic_publish_wg]update workload group finish, tg info="
- << tg->debug_string() << ", enable_cpu_hard_limit="
+LOG(INFO) << "[topic_publish_wg]update workload group finish, wg info="
+ << wg->debug_string() << ", enable_cpu_hard_
[PR] [chore](Nereids) opt part not exists error msg in bind relation (#36792)(#37160) [doris]
morrySnow opened a new pull request, #37280: URL: https://github.com/apache/doris/pull/37280 pick from master #36792 #37160 print table name when partition not exists in bind relation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](Nereids) opt part not exists error msg in bind relation (#36792)(#37160) [doris]
doris-robot commented on PR #37280: URL: https://github.com/apache/doris/pull/37280#issuecomment-2208299298 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](Nereids) opt part not exists error msg in bind relation (#36792)(#37160) [doris]
morrySnow commented on PR #37280: URL: https://github.com/apache/doris/pull/37280#issuecomment-2208299303 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [case](paimon/iceberg)move cases from p2 to p0 [doris]
doris-robot commented on PR #37276: URL: https://github.com/apache/doris/pull/37276#issuecomment-2208301341 TPC-DS: Total hot run time: 173708 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 431c2cbaf4d489b8f0b9521f68716f6a00a20821, data reload: false query1 922 374 372 372 query2 6460252323232323 query3 6649204 214 204 query4 18890 17639 17285 17285 query5 3676497 488 488 query6 269 159 159 159 query7 4584299 288 288 query8 310 284 294 284 query9 8319246524422442 query10 578 308 280 280 query11 10446 10093 10113 10093 query12 120 92 83 83 query13 1647378 375 375 query14 10438 746179977461 query15 237 187 193 187 query16 7739324 314 314 query17 1360586 550 550 query18 1923282 280 280 query19 202 158 161 158 query20 97 83 89 83 query21 213 138 132 132 query22 4215408540134013 query23 33787 33732 33579 33579 query24 10993 286729162867 query25 601 407 399 399 query26 712 159 157 157 query27 2197326 330 326 query28 6228218621682168 query29 981 647 642 642 query30 239 176 169 169 query31 984 773 752 752 query32 100 57 60 57 query33 672 308 372 308 query34 897 476 489 476 query35 801 654 641 641 query36 1144988 978 978 query37 142 84 80 80 query38 2935288527862786 query39 892 837 818 818 query40 216 136 129 129 query41 52 50 50 50 query42 119 107 103 103 query43 583 547 559 547 query44 1070750 746 746 query45 198 167 169 167 query46 1075716 714 714 query47 1815175117741751 query48 366 297 304 297 query49 853 407 419 407 query50 755 377 381 377 query51 6890678567086708 query52 115 94 97 94 query53 382 297 304 297 query54 888 447 443 443 query55 75 77 73 73 query56 304 268 270 268 query57 1148104210791042 query58 265 248 250 248 query59 3296313930333033 query60 307 283 274 274 query61 97 95 92 92 query62 608 447 454 447 query63 331 300 314 300 query64 8560232917511751 query65 3180333831493149 query66 748 325 329 325 query67 15586 14832 14848 14832 query68 6394545 537 537 query69 748 452 316 316 query70 1231116411621162 query71 500 273 319 273 query72 8949559555045504 query73 823 322 314 314 query74 5882543455345434 query75 5004268326752675 query76 4406985 980 980 query77 781 313 304 304 query78 10761 883788268826 query79 8130519 531 519 query80 1065489 474 474 query81 538 217 225 217 query82 714 108 107 107 query83 332 172 173 172 query84 265 83 88 83 query85 1348273 279 273 query86 436 310 323 310 query87 3319305931093059 query88 4840236723702367 query89 543 401 399 399 query90 1970184 188 184 query91 136 100 103 100 query92 62 52 48 48 query93 6232515 528 515 query94 1281215 218 215 query95 409 328 325 325 query96 610 264 267 264 query97 3169300330253003 query98 225 210 199 199 query99 1132858 829 829 Total cold run time: 285716 ms Total hot run time: 173708 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [pipeline](datagen) Improve datagen operator parallism [doris]
Gabriel39 commented on PR #37195: URL: https://github.com/apache/doris/pull/37195#issuecomment-2208302574 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](fe) Remove cloudWarmUpJob from DEPRECATED_MODULES_NAMES [doris]
w41ter commented on PR #37203: URL: https://github.com/apache/doris/pull/37203#issuecomment-2208303628 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [case](paimon/iceberg)move cases from p2 to p0 [doris]
doris-robot commented on PR #37276: URL: https://github.com/apache/doris/pull/37276#issuecomment-2208310260 ClickBench: Total hot run time: 30.36 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 431c2cbaf4d489b8f0b9521f68716f6a00a20821, data reload: false query1 0.040.030.03 query2 0.070.040.04 query3 0.220.040.05 query4 1.680.080.08 query5 0.500.500.51 query6 1.140.730.73 query7 0.020.020.01 query8 0.050.040.05 query9 0.550.490.49 query10 0.530.540.56 query11 0.150.110.12 query12 0.160.130.13 query13 0.590.600.59 query14 0.760.780.77 query15 0.830.800.81 query16 0.370.350.36 query17 0.980.940.94 query18 0.220.250.24 query19 1.771.691.67 query20 0.020.010.01 query21 15.39 0.740.66 query22 3.837.421.71 query23 18.27 1.361.24 query24 2.050.220.21 query25 0.150.090.09 query26 0.250.180.18 query27 0.080.080.08 query28 13.29 1.011.00 query29 12.60 3.303.27 query30 0.250.060.06 query31 2.940.400.38 query32 3.220.470.47 query33 2.882.912.91 query34 17.26 4.454.46 query35 4.464.484.52 query36 0.650.470.47 query37 0.180.160.15 query38 0.150.150.16 query39 0.040.030.04 query40 0.170.140.15 query41 0.080.040.05 query42 0.050.040.04 query43 0.040.040.04 Total cold run time: 108.93 s Total hot run time: 30.36 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipeline](datagen) Improve datagen operator parallism [doris]
github-actions[bot] commented on PR #37195: URL: https://github.com/apache/doris/pull/37195#issuecomment-2208311755 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](Nereids) null type in result set will be cast to tinyint (#37019) [doris]
morrySnow opened a new pull request, #37281: URL: https://github.com/apache/doris/pull/37281 pick from master #37019 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) null type in result set will be cast to tinyint (#37019) [doris]
doris-robot commented on PR #37281: URL: https://github.com/apache/doris/pull/37281#issuecomment-2208312469 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) null type in result set will be cast to tinyint (#37019) [doris]
morrySnow commented on PR #37281: URL: https://github.com/apache/doris/pull/37281#issuecomment-2208315088 run buildal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) null type in result set will be cast to tinyint (#37019) [doris]
morrySnow commented on PR #37281: URL: https://github.com/apache/doris/pull/37281#issuecomment-2208315452 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Improvement](scan) Refine ignoring data distribution without join [doris]
Gabriel39 opened a new pull request, #37282: URL: https://github.com/apache/doris/pull/37282 ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](scan) Refine ignoring data distribution without join [doris]
doris-robot commented on PR #37282: URL: https://github.com/apache/doris/pull/37282#issuecomment-2208320958 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) simplify window expression should inherit data type (#37061) [doris]
doris-robot commented on PR #37283: URL: https://github.com/apache/doris/pull/37283#issuecomment-2208321679 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](scan) Refine ignoring data distribution without join [doris]
Gabriel39 commented on PR #37282: URL: https://github.com/apache/doris/pull/37282#issuecomment-2208321051 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](Nereids) simplify window expression should inherit data type (#37061) [doris]
morrySnow opened a new pull request, #37283: URL: https://github.com/apache/doris/pull/37283 pick from master #37061 after window expression rewritten by literal. literal's data type should same with original window expression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) simplify window expression should inherit data type (#37061) [doris]
morrySnow commented on PR #37283: URL: https://github.com/apache/doris/pull/37283#issuecomment-2208322321 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](multi-catalog)fix paimon meta properties convert [doris]
wsjz commented on PR #37249: URL: https://github.com/apache/doris/pull/37249#issuecomment-2208326742 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore][fe] Get meta returns dropped partitions in binlog manager [doris]
github-actions[bot] commented on PR #37196: URL: https://github.com/apache/doris/pull/37196#issuecomment-2208327126 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](multi-catalog)fix paimon meta properties convert [doris]
wsjz commented on PR #37249: URL: https://github.com/apache/doris/pull/37249#issuecomment-2208327577 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](ctas) add a variable to control varchar length in ctas (#37069) [doris]
doris-robot commented on PR #37284: URL: https://github.com/apache/doris/pull/37284#issuecomment-2208327597 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](ctas) add a variable to control varchar length in ctas (#37069) [doris]
morrySnow commented on PR #37284: URL: https://github.com/apache/doris/pull/37284#issuecomment-2208327722 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](OSS) Retry when encountering TooManyRequest response [doris]
gavinchou commented on code in PR #37199:
URL: https://github.com/apache/doris/pull/37199#discussion_r1665282400
##
be/src/io/fs/s3_file_reader.cpp:
##
@@ -187,16 +139,10 @@ void S3FileReader::_collect_profile_before_close() {
ADD_TIMER(_profile, s3_profile_name);
RuntimeProfile::Counter* total_get_request_counter =
ADD_CHILD_COUNTER(_profile, "TotalGetRequest", TUnit::UNIT,
s3_profile_name);
-RuntimeProfile::Counter* too_many_request_err_counter =
Review Comment:
do not remove these profile items
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](multi-catalog)fix paimon meta properties convert [doris]
wsjz commented on PR #37249: URL: https://github.com/apache/doris/pull/37249#issuecomment-2208330328 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](scan) Refine ignoring data distribution without join [doris]
Gabriel39 commented on PR #37282: URL: https://github.com/apache/doris/pull/37282#issuecomment-2208340765 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](paritial update) Fix the case of partial update failing in cloud mode [doris]
github-actions[bot] commented on PR #37151: URL: https://github.com/apache/doris/pull/37151#issuecomment-2208342054 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](memory) ScopedAllocatorTagger [doris]
xinyiZzz commented on code in PR #37066:
URL: https://github.com/apache/doris/pull/37066#discussion_r1665289269
##
be/src/runtime/thread_context.h:
##
@@ -460,17 +478,22 @@ class ScopeSkipMemoryCheck {
// Mem Hook to consume thread mem tracker
#define CONSUME_THREAD_MEM_TRACKER_BY_HOOK(size) \
do { \
+allocator_detect(size);\
if (doris::use_mem_hook) { \
doris::thread_context()->consume_memory(size); \
} \
} while (0)
+
#define RELEASE_THREAD_MEM_TRACKER_BY_HOOK(size)
CONSUME_THREAD_MEM_TRACKER_BY_HOOK(-size)
+
#define CONSUME_THREAD_MEM_TRACKER_BY_HOOK_WITH_FN(size_fn, ...) \
do { \
+allocator_detect(size_fn(__VA_ARGS__));\
Review Comment:
```
#ifndef NDEBUG
allocator_detect(size_fn(__VA_ARGS__));
#endif
```
avoid unnecessary function calls `size_fn(__VA_ARGS__)` and
`allocator_detect`
##
be/src/vec/common/allocator.cpp:
##
@@ -244,13 +244,15 @@ void Allocator::remove_address_sanitizer
template
void* Allocator::alloc(size_t size,
size_t alignment) {
+doris::ScopedAllocatorTagger tagger;
Review Comment:
```
doris::allocator_working = true;
Defer defer {[] { doris::allocator_working = false; }};
```
seems simpler
##
be/src/vec/common/allocator.cpp:
##
@@ -244,13 +244,15 @@ void Allocator::remove_address_sanitizer
template
void* Allocator::alloc(size_t size,
size_t alignment) {
+doris::ScopedAllocatorTagger tagger;
return alloc_impl(size, alignment);
}
template
void* Allocator::realloc(void* buf,
size_t old_size,
size_t
new_size,
size_t
alignment) {
+doris::ScopedAllocatorTagger tagger;
Review Comment:
```
doris::allocator_working = true;
Defer defer {[] { doris::allocator_working = false; }};
```
seems simpler
##
be/src/common/config.cpp:
##
@@ -1318,6 +1318,9 @@ DEFINE_Int64(min_row_group_size, "134217728");
// filter wrong data.
DEFINE_mBool(enable_parquet_page_index, "true");
+// If native memory allocator allocates more memory than this value, an
exception will be thrown.
+DEFINE_Int64(debug_max_memory_size_by_native_allocator, "1048576")
Review Comment:
`debug_max_memory_size_by_native_allocator_bytes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](Nereids) normalize aggregate should not push down lambda's param (#37109) [doris]
morrySnow opened a new pull request, #37285: URL: https://github.com/apache/doris/pull/37285 pick from master #37109 ArrayItemSlot should not be inputSlot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](regression-test) dup modify value case [doris]
doris-robot commented on PR #37286: URL: https://github.com/apache/doris/pull/37286#issuecomment-2208345609 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](recyclebin) db can not be serialized with gson directly. [doris]
github-actions[bot] commented on PR #37261: URL: https://github.com/apache/doris/pull/37261#issuecomment-2208345839 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](case) add log for flink connector case download file [doris]
github-actions[bot] commented on PR #37202: URL: https://github.com/apache/doris/pull/37202#issuecomment-2208346430 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) normalize aggregate should not push down lambda's param (#37109) [doris]
doris-robot commented on PR #37285: URL: https://github.com/apache/doris/pull/37285#issuecomment-2208344497 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [enhancement](regression-test) dup modify value case [doris]
cjj2010 opened a new pull request, #37286: URL: https://github.com/apache/doris/pull/37286 ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](recyclebin) db can not be serialized with gson directly. [doris]
github-actions[bot] commented on PR #37261: URL: https://github.com/apache/doris/pull/37261#issuecomment-2208345901 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](regression-test) dup modify value case [doris]
cjj2010 commented on PR #37286: URL: https://github.com/apache/doris/pull/37286#issuecomment-2208345973 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](case) add log for flink connector case download file [doris]
github-actions[bot] commented on PR #37202: URL: https://github.com/apache/doris/pull/37202#issuecomment-2208346374 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](memory) ScopedAllocatorTagger [doris]
zhiqiang- commented on code in PR #37066:
URL: https://github.com/apache/doris/pull/37066#discussion_r1665299420
##
be/src/runtime/thread_context.h:
##
@@ -460,17 +478,22 @@ class ScopeSkipMemoryCheck {
// Mem Hook to consume thread mem tracker
#define CONSUME_THREAD_MEM_TRACKER_BY_HOOK(size) \
do { \
+allocator_detect(size);\
if (doris::use_mem_hook) { \
doris::thread_context()->consume_memory(size); \
} \
} while (0)
+
#define RELEASE_THREAD_MEM_TRACKER_BY_HOOK(size)
CONSUME_THREAD_MEM_TRACKER_BY_HOOK(-size)
+
#define CONSUME_THREAD_MEM_TRACKER_BY_HOOK_WITH_FN(size_fn, ...) \
do { \
+allocator_detect(size_fn(__VA_ARGS__));\
Review Comment:
function all will be erased with -O3
##
be/src/runtime/thread_context.h:
##
@@ -460,17 +478,22 @@ class ScopeSkipMemoryCheck {
// Mem Hook to consume thread mem tracker
#define CONSUME_THREAD_MEM_TRACKER_BY_HOOK(size) \
do { \
+allocator_detect(size);\
if (doris::use_mem_hook) { \
doris::thread_context()->consume_memory(size); \
} \
} while (0)
+
#define RELEASE_THREAD_MEM_TRACKER_BY_HOOK(size)
CONSUME_THREAD_MEM_TRACKER_BY_HOOK(-size)
+
#define CONSUME_THREAD_MEM_TRACKER_BY_HOOK_WITH_FN(size_fn, ...) \
do { \
+allocator_detect(size_fn(__VA_ARGS__));\
Review Comment:
function call will be erased with -O3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [chore](cloud) Define std::stringstream in local [doris]
w41ter opened a new pull request, #37287: URL: https://github.com/apache/doris/pull/37287 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](cloud) Define std::stringstream in local [doris]
doris-robot commented on PR #37287: URL: https://github.com/apache/doris/pull/37287#issuecomment-2208358577 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) normalize aggregate should not push down lambda's param (#37109) [doris]
morrySnow commented on PR #37285: URL: https://github.com/apache/doris/pull/37285#issuecomment-2208357384 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](cloud) Define std::stringstream in local [doris]
w41ter commented on PR #37287: URL: https://github.com/apache/doris/pull/37287#issuecomment-2208358671 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](routine-load) fix routine load pause when Kafka data deleted after TTL [doris]
sollhui opened a new pull request, #37288:
URL: https://github.com/apache/doris/pull/37288
## Proposed changes
When use routine load, After the data load is completed, the lag is still a
positive number:
```
Lag:
{"0":16,"1":15,"2":16,"3":16,"4":16,"5":16,"6":15,"7":16,"8":16,"9":16,"10":15,"11":16,"12":15,"13":15,"14":16,"15":16,"16":17,"17":15,"18":16,"19":15,"20":16,"21":16,"22":16,"23":16,"24":15,"25":17,"26":17,"27":16,"28":16,"29":16,"30":16,"31":17,"32":14,"33":16,"34":17,"35":16,"36":15,"37":15,"38":15,"39":16,"40":16,"41":16,"42":15,"43":15,"44":17,"45":16,"46":15,"47":15,"48":16,"49":17,"50":16,"51":15,"52":16,"53":15,"54":15,"55":17,"56":16,"57":17,"58":16,"59":16,"60":15,"61":15,"62":16,"63":16,"64":17,"65":16,"66":15,"67":16,"68":17,"69":16,"70":15,"71":17}
```
and the routing load is paused when the Kafka data reaches TTL and is
deleted, the error is `out of range`.
The reason why this happened is EOF is has it is offset which need statistic.
**note(important):**
After bug fixed, if you set
```
"property.enable.partition.eof" = "false"
```
in your routine load job, it will meet the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](cloud) Define std::stringstream in local [doris]
gavinchou commented on code in PR #37287: URL: https://github.com/apache/doris/pull/37287#discussion_r1665310018 ## cloud/src/meta-service/meta_service.cpp: ## @@ -1591,13 +1594,13 @@ static bool check_delete_bitmap_lock(MetaServiceCode& code, std::string& msg, st return true; } -static bool process_pending_delete_bitmap(MetaServiceCode& code, std::string& msg, - std::stringstream& ss, std::unique_ptr& txn, Review Comment: We have to check if `ss` already has something in it, say, ``` ss << "test"; // called somewhere ... process_pending_delete_bitmap(..., ss, ...); ... ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](routine-load) fix routine load pause when Kafka data deleted after TTL [doris]
doris-robot commented on PR #37288: URL: https://github.com/apache/doris/pull/37288#issuecomment-2208366255 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](memory) ScopedAllocatorTagger [doris]
zhiqiang- commented on code in PR #37066: URL: https://github.com/apache/doris/pull/37066#discussion_r1665309882 ## be/src/common/config.cpp: ## @@ -1318,6 +1318,9 @@ DEFINE_Int64(min_row_group_size, "134217728"); // filter wrong data. DEFINE_mBool(enable_parquet_page_index, "true"); +// If native memory allocator allocates more memory than this value, an exception will be thrown. +DEFINE_Int64(debug_max_memory_size_by_native_allocator, "1048576") Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](memory) ScopedAllocatorTagger [doris]
zhiqiang- commented on PR #37066: URL: https://github.com/apache/doris/pull/37066#issuecomment-2208367290 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](routine-load) fix routine load pause when Kafka data deleted after TTL [doris]
sollhui commented on PR #37288: URL: https://github.com/apache/doris/pull/37288#issuecomment-2208369055 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Improvement]Log be thread num [doris]
wangbo opened a new pull request, #37289: URL: https://github.com/apache/doris/pull/37289 ## Proposed changes Log doris_be process's thread num at a fixed interval, when a BE cores because of thread exhaustion, we can use this log to find which logic uses the most threads. ``` log be thread num: pid=3545627 thread_name_count=106 total_thread_num=7103 details: RScan_auth_grou=960 RScan_normal [w=960 RScan_test_wg [=960 RemoteScanThrea=960 brpc_heavy=384 brpc_light=384 doris_be=294 Scan_normal [wo=192 Scan_auth_group=192 LimitedScanThre=192 Scan_test_wg [w=192 SpillIOThreadPo=192 local_scan=192 Pipe_test_wg [w=96 Pipe_normal [wo=96 Pipe_auth_group=96 PipeNoGSchePool=96 DownloadThreadP=64 SendReportThrea=64 SendBatchThread=64 FragmentMgrThre=64 JoinNodeThreadP=64 EvHttpServer [w=48 FragmentInstanc=48 TabletPublishTx=32 S3FileUploadThr=16 BufferedReaderP=16 NonBlockCloseTh=12 MultiGetTaskThr=10 TaskWP_PUBLISH_=8 SendTableStatsT=8 MemTableHighPri=6 MemTableFlushTh=6 wg_flush_test_w=6 wg_flush_auth_g=6 SegmentFileWrit=6 wg_flush_normal=6 WriteCooldownMe=5 TaskWP_MAKE_SNA=5 CooldownTaskThr=5 SegCompactionTa=5 TaskWP_RELEASE_=5 BaseCompactionT=4 cancel_timeout_=3 TaskWP_DELETE [=3 TaskWP_DROP_TAB=3 TaskWP_CREATE_T=3 TaskWP_CLONE [w=3 TaskWP_ALTER_TA=3 TaskWP_ALTER_IN=3 TaskWP_.PUSH [w=3 HighPriorPool.P=3 ColdDataCompact=2 memtable_memory=1 query_runtime_s=1 memory_maintena=1 memory_gc_threa=1 GroupCommitThre=1 je_purge_dirty_=1 gc_expired_cont=1 garbage_sweeper=1 evict_querying_=1 disk_stat_monit=1 cooldown_tasks_=1 compaction_task=1 cold_data_compa=1 path_gc_thread-=1 ping_worker-354=1 clean_expired_t=1 remove_unused_r=1 replay_wal-3547=1 spill_gc_thread=1 tablet_checkpoi=1 unused_rowset_m=1 update_wal_dir_=1 GroupCommitRepl=1 CumuCompactionT=1 wg_mem_refresh_=1 workload_schedu=1 TaskWP_UPDATE_T=1 TaskWP_CLEAN_UD=1 TaskWP_CLEAR_TR=1 TaskWP_CHECK_CO=1 TaskWP_DOWNLOAD=1 TaskWP_GC_BINLO=1 TabletCalcDelet=1 TaskWP_MOVE [wo=1 SingleReplicaCo=1 TaskWP_PUSH_COO=1 TaskWP_PUSH_STO=1 REPORT_TASK-354=1 TaskWP_STORAGE_=1 TaskWP_SUBMIT_T=1 clean_idle_cons=1 TaskWP_UPDATE_V=1 TaskWP_UPLOAD [=1 REPORT_OLAP_TAB=1 _update_replica=1 async_publish_v=1 be_proc_monitor=1 REPORT_DISK_STA=1 MigrationTaskTh=1 cache_clean_thr=1 calculate_metri=1 LazyReleaseMemo=1 TaskWP_CLEAN_TR=1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
doris-robot commented on PR #37289: URL: https://github.com/apache/doris/pull/37289#issuecomment-2208369172 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](cloud) Define std::stringstream in local [doris]
github-actions[bot] commented on code in PR #37287:
URL: https://github.com/apache/doris/pull/37287#discussion_r1665312253
##
cloud/src/meta-service/meta_service_job.cpp:
##
@@ -499,14 +500,15 @@
txn->put(lock_key, lock_val);
}
-void process_compaction_job(MetaServiceCode& code, std::string& msg,
std::stringstream& ss,
-std::unique_ptr& txn,
+void process_compaction_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
Review Comment:
warning: function 'process_compaction_job' exceeds recommended
size/complexity thresholds [readability-function-size]
```cpp
void process_compaction_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
^
```
Additional context
**cloud/src/meta-service/meta_service_job.cpp:502:** 349 lines including
whitespace and comments (threshold 80)
```cpp
void process_compaction_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
^
```
##
cloud/src/meta-service/meta_service_job.cpp:
##
@@ -49,10 +47,10 @@ namespace doris::cloud {
static constexpr int COMPACTION_DELETE_BITMAP_LOCK_ID = -1;
static constexpr int SCHEMA_CHANGE_DELETE_BITMAP_LOCK_ID = -2;
-void start_compaction_job(MetaServiceCode& code, std::string& msg,
std::stringstream& ss,
- std::unique_ptr& txn, const
StartTabletJobRequest* request,
- StartTabletJobResponse* response, std::string&
instance_id,
- bool& need_commit) {
+void start_compaction_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
Review Comment:
warning: function 'start_compaction_job' exceeds recommended size/complexity
thresholds [readability-function-size]
```cpp
void start_compaction_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
^
```
Additional context
**cloud/src/meta-service/meta_service_job.cpp:49:** 159 lines including
whitespace and comments (threshold 80)
```cpp
void start_compaction_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
^
```
##
cloud/src/meta-service/meta_service_job.cpp:
##
@@ -853,14 +855,15 @@
need_commit = true;
}
-void process_schema_change_job(MetaServiceCode& code, std::string& msg,
std::stringstream& ss,
- std::unique_ptr& txn,
+void process_schema_change_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
Review Comment:
warning: function 'process_schema_change_job' exceeds recommended
size/complexity thresholds [readability-function-size]
```cpp
void process_schema_change_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
^
```
Additional context
**cloud/src/meta-service/meta_service_job.cpp:857:** 269 lines including
whitespace and comments (threshold 80)
```cpp
void process_schema_change_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
^
```
##
cloud/src/meta-service/meta_service_job.cpp:
##
@@ -499,14 +500,15 @@
txn->put(lock_key, lock_val);
}
-void process_compaction_job(MetaServiceCode& code, std::string& msg,
std::stringstream& ss,
-std::unique_ptr& txn,
+void process_compaction_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
Review Comment:
warning: function 'process_compaction_job' has cognitive complexity of 52
(threshold 50) [readability-function-cognitive-complexity]
```cpp
void process_compaction_job(MetaServiceCode& code, std::string& msg,
Transaction* txn,
^
```
Additional context
**cloud/src/meta-service/meta_service_job.cpp:515:** +1, including nesting
penalty of 0, nesting level increased to 1
```cpp
if (recorded_job.compaction().empty()) {
^
```
**cloud/src/meta-service/meta_service_job.cpp:525:** +1, including nesting
penalty of 0, nesting level increased to 1
```cpp
for (; recorded_compaction != recorded_job.mutable_compaction()->end();
++recorded_compaction) {
^
```
**cloud/src/meta-service/meta_service_job.cpp:526:** +2, including nesting
penalty of 1, nesting level increased to 2
```cpp
if (recorded_compaction->id() == compaction.id()) break;
^
```
**cloud/src/meta-service/meta_service_job.cpp:528:** +1, including nesting
penalty of 0, nesting level increased to 1
```cpp
if (recorded_compaction == recorded_job.mutable_compaction()->end()) {
^
```
**cloud/src/meta-service/meta_service_job.cpp:538:** +1, including nesting
penalty of 0, nesting level increased to 1
```cpp
if (recorded_compaction->expiration() > 0 &&
recorded_compaction->expiration() < now) {
^
```
**cloud/src/meta-service/meta_service_job.cpp:538:** +1
```cpp
if (recorded_co
Re: [PR] [improvement](statistics)Enable estimate hive table row count using file size. [doris]
github-actions[bot] commented on PR #37218: URL: https://github.com/apache/doris/pull/37218#issuecomment-2208372461 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](cloud) add selected partition ids and visible version in log [doris]
liaoxin01 commented on code in PR #37267:
URL: https://github.com/apache/doris/pull/37267#discussion_r1665314968
##
fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java:
##
@@ -731,6 +731,11 @@ private Collection distributionPrune(
// Update the visible version of the scan range locations.
public void updateScanRangeVersions(Map visibleVersionMap) {
+if (ConnectContext.get() != null) {
+LOG.info("query id: {}, selectedPartitionIds: {},
visibleVersionMap: {}",
Review Comment:
Many logs may be printed, using debug level maybe better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
wangbo commented on PR #37289: URL: https://github.com/apache/doris/pull/37289#issuecomment-2208373011 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](memory) ScopedAllocatorTagger [doris]
github-actions[bot] commented on PR #37066: URL: https://github.com/apache/doris/pull/37066#issuecomment-2208373749 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](case) add log for flink connector case download file [doris]
JNSimba merged PR #37202: URL: https://github.com/apache/doris/pull/37202 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ES Catalog)Support control scroll level by config #37180 [doris]
qidaye commented on PR #37271: URL: https://github.com/apache/doris/pull/37271#issuecomment-2208376062 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [dependencies](fe)upgrade paimon to 0.8.1 [doris]
CalvinKirs merged PR #37205: URL: https://github.com/apache/doris/pull/37205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (8eb9a1a0661 -> 110329c548e)
This is an automated email from the ASF dual-hosted git repository. diwu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8eb9a1a0661 [Fix]check nullptr when log thread num (#37263) add 110329c548e [Fix](case) add log for flink connector case download file (#37202) No new revisions were added by this update. Summary of changes: .../flink_connector_syncdb.groovy | 21 - 1 file changed, 16 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](routine-load) fix routine load pause when Kafka data deleted after TTL [doris]
github-actions[bot] commented on PR #37288: URL: https://github.com/apache/doris/pull/37288#issuecomment-2208376593 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (110329c548e -> a8aa0384568)
This is an automated email from the ASF dual-hosted git repository. kirs pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 110329c548e [Fix](case) add log for flink connector case download file (#37202) add a8aa0384568 [dependencies](fe)upgrade paimon to 0.8.1 (#37205) No new revisions were added by this update. Summary of changes: fe/pom.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](memory) ScopedAllocatorTagger [doris]
github-actions[bot] commented on PR #37066: URL: https://github.com/apache/doris/pull/37066#issuecomment-2208377746 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](ip)improve cidr query for looking up ips [doris]
github-actions[bot] commented on PR #37185: URL: https://github.com/apache/doris/pull/37185#issuecomment-2208379730 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression) Fix p0 case `test_modify_reorder_column` [doris]
zhannngchen merged PR #37256: URL: https://github.com/apache/doris/pull/37256 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](regression) Fix p0 case `test_modify_reorder_column` (#37256)
This is an automated email from the ASF dual-hosted git repository.
zhangchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9ce57071d8e [fix](regression) Fix p0 case `test_modify_reorder_column`
(#37256)
9ce57071d8e is described below
commit 9ce57071d8e0b67afa7ac9e95d016f134c8bf048
Author: bobhan1
AuthorDate: Thu Jul 4 16:16:43 2024 +0800
[fix](regression) Fix p0 case `test_modify_reorder_column` (#37256)
https://github.com/apache/doris/pull/37067 passed all pipeline cases
based on the branch that doesn't contain
https://github.com/apache/doris/pull/37039 which changed the output
format merged later, causing this case's failure.
---
.../schema_change_p0/test_modify_reorder_column.out| 18 +-
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git
a/regression-test/data/schema_change_p0/test_modify_reorder_column.out
b/regression-test/data/schema_change_p0/test_modify_reorder_column.out
index 64e030f740e..ce2b54972c4 100644
--- a/regression-test/data/schema_change_p0/test_modify_reorder_column.out
+++ b/regression-test/data/schema_change_p0/test_modify_reorder_column.out
@@ -1,22 +1,22 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !dup --
-1 {"f1": "A", "f2": "B", "f3": 10, "f4": 3.14}{"f1": "C", "f2": "D",
"f3": 20, "f4": 8.343} {"a":1,"b":[1],"c":1.0}
+1 {"f1":"A", "f2":"B", "f3":10, "f4":3.14}{"f1":"C", "f2":"D",
"f3":20, "f4":8.343} {"a":1,"b":[1],"c":1.0}
-- !dup --
-1 {"f1": "A", "f2": "B", "f3": 10, "f4": 3.14}{"a":1,"b":[1],"c":1.0}
{"f1": "C", "f2": "D", "f3": 20, "f4": 8.343}
-2 {"f1": "E", "f2": "F", "f3": 30, "f4": 484.3234}
{"a":1,"b":[1],"c":1.0} \N
+1 {"f1":"A", "f2":"B", "f3":10, "f4":3.14}{"a":1,"b":[1],"c":1.0}
{"f1":"C", "f2":"D", "f3":20, "f4":8.343}
+2 {"f1":"E", "f2":"F", "f3":30, "f4":484.3234}{"a":1,"b":[1],"c":1.0}
\N
-- !mor --
-1 {"f1": "A", "f2": "B", "f3": 10, "f4": 3.14}{"f1": "C", "f2": "D",
"f3": 20, "f4": 8.343} {"a":1,"b":[1],"c":1.0}
+1 {"f1":"A", "f2":"B", "f3":10, "f4":3.14}{"f1":"C", "f2":"D",
"f3":20, "f4":8.343} {"a":1,"b":[1],"c":1.0}
-- !mor --
-1 {"f1": "A", "f2": "B", "f3": 10, "f4": 3.14}{"a":1,"b":[1],"c":1.0}
{"f1": "C", "f2": "D", "f3": 20, "f4": 8.343}
-2 {"f1": "E", "f2": "F", "f3": 30, "f4": 484.3234}
{"a":1,"b":[1],"c":1.0} \N
+1 {"f1":"A", "f2":"B", "f3":10, "f4":3.14}{"a":1,"b":[1],"c":1.0}
{"f1":"C", "f2":"D", "f3":20, "f4":8.343}
+2 {"f1":"E", "f2":"F", "f3":30, "f4":484.3234}{"a":1,"b":[1],"c":1.0}
\N
-- !mow --
-1 {"f1": "A", "f2": "B", "f3": 10, "f4": 3.14}{"f1": "C", "f2": "D",
"f3": 20, "f4": 8.343} {"a":1,"b":[1],"c":1.0}
+1 {"f1":"A", "f2":"B", "f3":10, "f4":3.14}{"f1":"C", "f2":"D",
"f3":20, "f4":8.343} {"a":1,"b":[1],"c":1.0}
-- !mow --
-1 {"f1": "A", "f2": "B", "f3": 10, "f4": 3.14}{"a":1,"b":[1],"c":1.0}
{"f1": "C", "f2": "D", "f3": 20, "f4": 8.343}
-2 {"f1": "E", "f2": "F", "f3": 30, "f4": 484.3234}
{"a":1,"b":[1],"c":1.0} \N
+1 {"f1":"A", "f2":"B", "f3":10, "f4":3.14}{"a":1,"b":[1],"c":1.0}
{"f1":"C", "f2":"D", "f3":20, "f4":8.343}
+2 {"f1":"E", "f2":"F", "f3":30, "f4":484.3234}{"a":1,"b":[1],"c":1.0}
\N
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](partial-update) make partial-update on agg table use less memory [doris]
github-actions[bot] commented on PR #32200: URL: https://github.com/apache/doris/pull/32200#issuecomment-2208380494 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
github-actions[bot] commented on code in PR #37289: URL: https://github.com/apache/doris/pull/37289#discussion_r1665321664 ## be/src/runtime/be_proc_monitor.cpp: ## @@ -0,0 +1,81 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "runtime/be_proc_monitor.h" + +#include Review Comment: warning: 'fmt/format.h' file not found [clang-diagnostic-error] ```cpp #include ^ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
wangbo commented on PR #37289: URL: https://github.com/apache/doris/pull/37289#issuecomment-2208381316 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [feature](ES Catalog)Support control scroll level by config #37180 [doris]
qidaye opened a new pull request, #37290: URL: https://github.com/apache/doris/pull/37290 ## Proposed changes backport #37180 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ES Catalog)Support control scroll level by config #37180 [doris]
doris-robot commented on PR #37290: URL: https://github.com/apache/doris/pull/37290#issuecomment-2208384896 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](partial-update) make partial-update on agg table use less memory [doris]
hust-hhb commented on PR #32200: URL: https://github.com/apache/doris/pull/32200#issuecomment-2208386442 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipeline](datagen) Improve datagen operator parallelism [doris]
doris-robot commented on PR #37195: URL: https://github.com/apache/doris/pull/37195#issuecomment-2208388298 TPC-H: Total hot run time: 40169 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 524adcd61ad12787ed0638f89cc4428d354aa8ce, data reload: false -- Round 1 -- q1 17832 513843654365 q2 2031197 193 193 q3 10471 116711661166 q4 10196 829 723 723 q5 7496267326662666 q6 225 143 140 140 q7 986 627 602 602 q8 9255211420912091 q9 9133653665636536 q10 8957374737703747 q11 463 240 242 240 q12 417 242 229 229 q13 18940 298829732973 q14 272 224 224 224 q15 529 494 490 490 q16 527 383 370 370 q17 986 682 745 682 q18 8057744574237423 q19 7289144115141441 q20 690 315 327 315 q21 4936325532153215 q22 401 338 339 338 Total cold run time: 120089 ms Total hot run time: 40169 ms - Round 2, with runtime_filter_mode=off - q1 4378429743034297 q2 361 279 269 269 q3 2982276727762767 q4 1847165315551555 q5 5257528753035287 q6 219 129 131 129 q7 2124180917721772 q8 3218336533673365 q9 8384831783478317 q10 3893367636793676 q11 587 496 512 496 q12 842 599 615 599 q13 17527 297330212973 q14 292 259 256 256 q15 519 490 468 468 q16 476 414 432 414 q17 1768147514781475 q18 7773740773847384 q19 1712158615871586 q20 1982179217741774 q21 4895486147574757 q22 631 547 558 547 Total cold run time: 71667 ms Total hot run time: 54163 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](fe) Remove cloudWarmUpJob from DEPRECATED_MODULES_NAMES [doris]
doris-robot commented on PR #37203: URL: https://github.com/apache/doris/pull/37203#issuecomment-2208389650 TPC-H: Total hot run time: 40223 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit c46e72cfd21631551309d0398bbe64378ea40d98, data reload: false -- Round 1 -- q1 18662 530345584558 q2 2626199 205 199 q3 10979 123510691069 q4 10438 764 765 764 q5 7971268727082687 q6 237 143 143 143 q7 990 640 608 608 q8 9576213020832083 q9 9046653964946494 q10 8885373037683730 q11 461 248 238 238 q12 437 227 223 223 q13 17769 298529792979 q14 274 222 226 222 q15 504 476 460 460 q16 498 370 375 370 q17 957 574 685 574 q18 7975742674267426 q19 6495143914161416 q20 665 322 347 322 q21 5047332640173326 q22 410 333 332 332 Total cold run time: 120902 ms Total hot run time: 40223 ms - Round 2, with runtime_filter_mode=off - q1 4352430242014201 q2 371 258 273 258 q3 2969274027422740 q4 1868161916161616 q5 5248530552635263 q6 222 128 131 128 q7 2118175416931693 q8 3175334032873287 q9 8363838283928382 q10 3924365937163659 q11 571 486 484 484 q12 806 606 608 606 q13 16359 299630382996 q14 305 279 260 260 q15 510 492 485 485 q16 460 423 426 423 q17 1785146014641460 q18 7649752374007400 q19 1741156116841561 q20 1946178717761776 q21 5009484046364636 q22 602 548 544 544 Total cold run time: 70353 ms Total hot run time: 53858 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ES Catalog)Support control scroll level by config #37180 [doris]
qidaye commented on PR #37290: URL: https://github.com/apache/doris/pull/37290#issuecomment-2208390026 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](partial-update) make partial-update on agg table use less memory [doris]
github-actions[bot] commented on PR #32200: URL: https://github.com/apache/doris/pull/32200#issuecomment-2208395037 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ES Catalog)Support control scroll level by config #37180 [doris]
github-actions[bot] commented on PR #37290: URL: https://github.com/apache/doris/pull/37290#issuecomment-2208395347 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](Cloud) Support session variable disable_file_cache in query. [doris]
doris-robot commented on PR #37141: URL: https://github.com/apache/doris/pull/37141#issuecomment-2208399294 TPC-H: Total hot run time: 39896 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit d31abdf8a659db45d77d1cf9433b31562542ba13, data reload: false -- Round 1 -- q1 18023 455743394339 q2 2040196 192 192 q3 10553 116211201120 q4 10211 860 749 749 q5 7470269826602660 q6 227 143 141 141 q7 995 607 612 607 q8 9230210821012101 q9 9052652265146514 q10 8963370037033700 q11 475 239 238 238 q12 438 234 232 232 q13 17779 297430052974 q14 267 234 226 226 q15 520 494 491 491 q16 509 385 377 377 q17 980 702 639 639 q18 8070759573277327 q19 6405152214901490 q20 690 336 340 336 q21 4870310638823106 q22 389 347 337 337 Total cold run time: 118156 ms Total hot run time: 39896 ms - Round 2, with runtime_filter_mode=off - q1 4430430042134213 q2 381 257 273 257 q3 3039288429862884 q4 1965166917491669 q5 5570553754595459 q6 227 141 140 140 q7 2247191718511851 q8 3296348234393439 q9 8752877087568756 q10 4179374337673743 q11 604 504 503 503 q12 861 633 647 633 q13 15960 317432113174 q14 311 291 281 281 q15 540 484 498 484 q16 498 443 444 443 q17 1846156515411541 q18 8098799778287828 q19 1753166517721665 q20 2154188118701870 q21 5195495746134613 q22 638 568 566 566 Total cold run time: 72544 ms Total hot run time: 56012 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner) fix bug of select stmt toSql [doris]
doris-robot commented on PR #37274: URL: https://github.com/apache/doris/pull/37274#issuecomment-2208402269 TPC-H: Total hot run time: 40236 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9c122dcc0a5b812b7a3a26ec3988ba6ea0a61fe1, data reload: false -- Round 1 -- q1 17609 504042994299 q2 2014189 190 189 q3 10447 120911791179 q4 10181 767 747 747 q5 7484261826502618 q6 221 139 136 136 q7 962 595 599 595 q8 9224210320972097 q9 9057651164736473 q10 8984379237603760 q11 455 242 249 242 q12 536 253 248 248 q13 19004 298229632963 q14 281 228 233 228 q15 536 486 486 486 q16 514 388 385 385 q17 973 699 668 668 q18 8177749774467446 q19 8009157416201574 q20 665 332 328 328 q21 4944323432513234 q22 399 341 343 341 Total cold run time: 120676 ms Total hot run time: 40236 ms - Round 2, with runtime_filter_mode=off - q1 4438427242274227 q2 384 277 288 277 q3 3235296729552955 q4 1984166317551663 q5 5525544054575440 q6 221 145 139 139 q7 2258182918171817 q8 3355340134393401 q9 8666885388118811 q10 4087371539133715 q11 581 494 494 494 q12 820 604 642 604 q13 16383 316632243166 q14 335 285 289 285 q15 538 480 479 479 q16 507 436 415 415 q17 1819154315361536 q18 8124804077737773 q19 1801173917181718 q20 2211186618661866 q21 5165468248864682 q22 648 589 539 539 Total cold run time: 73085 ms Total hot run time: 56002 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
