Re: [PR] [fix](funcs) map struct construct funcs #39699 [doris]
xiaokang commented on PR #40502: URL: https://github.com/apache/doris/pull/40502#issuecomment-2336605013 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow) merge and remove old version of delete bitmap when cumulative compaction is done [doris]
hust-hhb commented on PR #40204: URL: https://github.com/apache/doris/pull/40204#issuecomment-2336605493 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](funcs) map struct construct funcs #39699 [doris]
doris-robot commented on PR #40502: URL: https://github.com/apache/doris/pull/40502#issuecomment-2336614724 TPC-H: Total hot run time: 48858 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 19b2e187188d030716e84dc171a67b93d6ef6124, data reload: false -- Round 1 -- q1 17788 440143344334 q2 2065150 141 141 q3 10452 190519281905 q4 10286 127112991271 q5 8390388438483848 q6 230 126 122 122 q7 2039158716131587 q8 9282271426912691 q9 10086 973298159732 q10 8679356735533553 q11 420 240 233 233 q12 474 294 303 294 q13 18326 392440183924 q14 364 331 329 329 q15 506 451 450 450 q16 542 457 451 451 q17 1131964 932 932 q18 7230699568066806 q19 1681156214761476 q20 561 324 304 304 q21 4363414740904090 q22 481 392 385 385 Total cold run time: 115376 ms Total hot run time: 48858 ms - Round 2, with runtime_filter_mode=off - q1 4361435542714271 q2 328 224 222 222 q3 4131410641104106 q4 2735272927382729 q5 7148704671037046 q6 234 118 118 118 q7 3228282428602824 q8 4308441745294417 q9 13658 13735 13424 13424 q10 4224425442764254 q11 776 663 679 663 q12 1011851 841 841 q13 7091373937523739 q14 452 425 429 425 q15 513 465 455 455 q16 638 613 577 577 q17 3810383838363836 q18 8643874286678667 q19 1723171516391639 q20 2397210621082106 q21 8400847783968396 q22 1059953 949 949 Total cold run time: 80868 ms Total hot run time: 75704 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](funcs) map struct construct funcs #39699 [doris]
doris-robot commented on PR #40502: URL: https://github.com/apache/doris/pull/40502#issuecomment-2336616279 TeamCity be ut coverage result: Function Coverage: 37.85% (8137/21498) Line Coverage: 29.59% (66984/226336) Region Coverage: 29.08% (34561/118835) Branch Coverage: 24.99% (17809/71266) Coverage Report: http://coverage.selectdb-in.cc/coverage/19b2e187188d030716e84dc171a67b93d6ef6124_19b2e187188d030716e84dc171a67b93d6ef6124/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](funcs) map struct construct funcs #39699 [doris]
doris-robot commented on PR #40502: URL: https://github.com/apache/doris/pull/40502#issuecomment-2336620768 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 19b2e187188d030716e84dc171a67b93d6ef6124 with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.2 seconds inserted 1000 Rows, about 471K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression) mem gc when load regression failure due to error msg update [doris]
dataroaring merged PR #40483: URL: https://github.com/apache/doris/pull/40483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](regression) mem gc when load regression failure due to error msg update (#40483)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 93e19b06566 [fix](regression) mem gc when load regression failure due
to error msg update (#40483)
93e19b06566 is described below
commit 93e19b065664d97ed0d2dc581a463371335def06
Author: zhengyu
AuthorDate: Sun Sep 8 17:54:27 2024 +0800
[fix](regression) mem gc when load regression failure due to error msg
update (#40483)
The error msg for memory shortage is changed from "Process has no memory
available" to "less than low water mark".
Signed-off-by: freemandealer
---
.../fault_injection_p0/test_mem_gc_when_load_fault_injection.groovy | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/regression-test/suites/fault_injection_p0/test_mem_gc_when_load_fault_injection.groovy
b/regression-test/suites/fault_injection_p0/test_mem_gc_when_load_fault_injection.groovy
index 852566c0e05..2f72c635df3 100644
---
a/regression-test/suites/fault_injection_p0/test_mem_gc_when_load_fault_injection.groovy
+++
b/regression-test/suites/fault_injection_p0/test_mem_gc_when_load_fault_injection.groovy
@@ -72,7 +72,7 @@ suite("mem_gc_when_load", "nonConcurrent") {
GetDebugPoint().enableDebugPointForAllBEs("VNodeChannel.try_send_and_fetch_status_full_gc")
sql "insert into test select * from baseall where k1 <= 3"
} catch(Exception e) {
-assertTrue(e.getMessage().contains("Process has no memory available"))
// the msg should contain the root cause
+assertTrue(e.getMessage().contains("less than low water mark")) //
the msg should contain the root cause
} finally {
GetDebugPoint().disableDebugPointForAllBEs("VNodeChannel.try_send_and_fetch_status_full_gc")
}
@@ -82,7 +82,7 @@ suite("mem_gc_when_load", "nonConcurrent") {
GetDebugPoint().enableDebugPointForAllBEs("VNodeChannel.close_wait_full_gc")
sql "insert into test select * from baseall where k1 <= 3"
} catch(Exception e) {
-assertTrue(e.getMessage().contains("Process has no memory available"))
// the msg should contain the root cause
+assertTrue(e.getMessage().contains("less than low water mark")) //
the msg should contain the root cause
} finally {
GetDebugPoint().disableDebugPointForAllBEs("VNodeChannel.close_wait_full_gc")
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](compatibility) fix agg functions coredump when upgrade [doris]
dataroaring merged PR #40472: URL: https://github.com/apache/doris/pull/40472 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Bug](compatibility) fix agg functions coredump when upgrade (#40472)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ca38271482c [Bug](compatibility) fix agg functions coredump when
upgrade (#40472)
ca38271482c is described below
commit ca38271482c045bf56757c32b507e3b0d1e325be
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Sun Sep 8 18:18:14 2024 +0800
[Bug](compatibility) fix agg functions coredump when upgrade (#40472)
## Proposed changes
before use check (IAggregateFunction::version < AGG_FUNCTION_NULLABLE)
and then choose the return type and insert method.
but the version maybe update in the same branch-version，and get wrong
type
so put those function to all of object class alone.
---
.../aggregate_functions/aggregate_function_covar.h | 19 +--
.../aggregate_function_percentile.h| 151 -
2 files changed, 126 insertions(+), 44 deletions(-)
diff --git a/be/src/vec/aggregate_functions/aggregate_function_covar.h
b/be/src/vec/aggregate_functions/aggregate_function_covar.h
index 179e723285e..9b4b1b70c1f 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_covar.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_covar.h
@@ -120,8 +120,6 @@ struct BaseData {
count += 1;
}
-static DataTypePtr get_return_type() { return
std::make_shared>(); }
-
double sum_x;
double sum_y;
double sum_xy;
@@ -134,6 +132,7 @@ struct PopData : Data {
auto& col = assert_cast(to);
col.get_data().push_back(this->get_pop_result());
}
+static DataTypePtr get_return_type() { return
std::make_shared>(); }
};
template
@@ -148,6 +147,9 @@ struct SampData_OLDER : Data {
nullable_column.get_null_map_data().push_back(0);
}
}
+static DataTypePtr get_return_type() {
+return make_nullable(std::make_shared>());
+}
};
template
@@ -160,6 +162,7 @@ struct SampData : Data {
col.get_data().push_back(this->get_samp_result());
}
}
+static DataTypePtr get_return_type() { return
std::make_shared>(); }
};
template
@@ -184,17 +187,7 @@ public:
String get_name() const override { return Data::name(); }
-DataTypePtr get_return_type() const override {
-if constexpr (is_pop) {
-return Data::get_return_type();
-} else {
-if (IAggregateFunction::version < AGG_FUNCTION_NULLABLE) {
-return make_nullable(Data::get_return_type());
-} else {
-return Data::get_return_type();
-}
-}
-}
+DataTypePtr get_return_type() const override { return
Data::get_return_type(); }
void add(AggregateDataPtr __restrict place, const IColumn** columns,
ssize_t row_num,
Arena*) const override {
diff --git a/be/src/vec/aggregate_functions/aggregate_function_percentile.h
b/be/src/vec/aggregate_functions/aggregate_function_percentile.h
index aae66eb4f78..cd328a85f34 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_percentile.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_percentile.h
@@ -162,13 +162,6 @@ public:
String get_name() const override { return "percentile_approx"; }
-DataTypePtr get_return_type() const override {
-if (IAggregateFunction::version < AGG_FUNCTION_NULLABLE) {
-return make_nullable(std::make_shared());
-}
-return std::make_shared();
-}
-
void reset(AggregateDataPtr __restrict place) const override {
AggregateFunctionPercentileApprox::data(place).reset();
}
@@ -187,30 +180,6 @@ public:
Arena*) const override {
AggregateFunctionPercentileApprox::data(place).read(buf);
}
-
-void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn&
to) const override {
-if (IAggregateFunction::version < AGG_FUNCTION_NULLABLE) {
-ColumnNullable& nullable_column = assert_cast(to);
-double result =
AggregateFunctionPercentileApprox::data(place).get();
-
-if (std::isnan(result)) {
-nullable_column.insert_default();
-} else {
-auto& col =
assert_cast(nullable_column.get_nested_column());
-col.get_data().push_back(result);
-nullable_column.get_null_map_data().push_back(0);
-}
-} else {
-auto& col = assert_cast(to);
-double result =
AggregateFunctionPercentileApprox::data(place).get();
-
-if (std::isnan(result)) {
-col.insert_default();
-} else {
-col.get_data().push_back(result);
-}
-}
-}
};
template
@@ -256,6 +225,23 @@ public:
Re: [PR] [opt](scan) unify the local and remote scan bytes stats for all scanners [doris]
doris-robot commented on PR #40493: URL: https://github.com/apache/doris/pull/40493#issuecomment-2336639159 TPC-DS: Total hot run time: 193085 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit a6d8c99fde14ebaea4f113f07f8c4c80196348f8, data reload: false query1 1275893 870 870 query2 6366191719021902 query3 10628 395140313951 query4 59323 24251 23260 23260 query5 5356504 514 504 query6 402 167 169 167 query7 5761302 285 285 query8 303 213 206 206 query9 8684244924692449 query10 485 286 263 263 query11 18024 15055 15042 15042 query12 150 102 105 102 query13 1602397 383 383 query14 11073 740972837283 query15 253 182 182 182 query16 7243499 506 499 query17 1087568 557 557 query18 2000305 296 296 query19 287 209 144 144 query20 116 112 109 109 query21 204 109 102 102 query22 4960466445864586 query23 34156 33388 33383 33383 query24 5995295928102810 query25 527 396 381 381 query26 687 162 151 151 query27 1788274 279 274 query28 3768203220192019 query29 662 401 409 401 query30 222 157 155 155 query31 961 717 772 717 query32 86 51 55 51 query33 449 278 288 278 query34 869 480 468 468 query35 856 700 722 700 query36 1062965 916 916 query37 143 87 84 84 query38 4048389338853885 query39 1441139014041390 query40 202 117 116 116 query41 49 47 47 47 query42 120 97 99 97 query43 499 474 441 441 query44 1093768 750 750 query45 201 169 169 169 query46 1102745 742 742 query47 1919178918211789 query48 392 305 328 305 query49 805 458 450 450 query50 826 411 420 411 query51 7011681868476818 query52 100 88 88 88 query53 250 178 181 178 query54 574 474 465 465 query55 80 78 85 78 query56 280 264 275 264 query57 1184106811031068 query58 234 240 232 232 query59 2915296527712771 query60 306 280 288 280 query61 125 123 122 122 query62 764 666 650 650 query63 215 190 186 186 query64 2896761 764 761 query65 3179317731253125 query66 684 350 367 350 query67 15458 15363 15486 15363 query68 3403588 578 578 query69 406 285 289 285 query70 1192112610871087 query71 354 283 279 279 query72 6341419141834183 query73 748 324 328 324 query74 9218883188958831 query75 3406263726862637 query76 15311021962 962 query77 522 370 316 316 query78 9931908790219021 query79 2013526 535 526 query80 986 502 510 502 query81 573 231 229 229 query82 233 132 143 132 query83 179 151 151 151 query84 267 75 80 75 query85 1031313 278 278 query86 411 294 303 294 query87 4541425142544251 query88 3946239223752375 query89 396 277 273 273 query90 1879192 190 190 query91 129 99 105 99 query92 64 49 51 49 query93 2054538 541 538 query94 802 296 285 285 query95 359 252 257 252 query96 604 266 267 266 query97 3179304530853045 query98 220 198 195 195 query99 1591127112561256 Total cold run time: 309274 ms Total hot run time: 193085 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [opt](scan) unify the local and remote scan bytes stats for all scanners [doris]
doris-robot commented on PR #40493: URL: https://github.com/apache/doris/pull/40493#issuecomment-2336640979 ClickBench: Total hot run time: 31.51 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit a6d8c99fde14ebaea4f113f07f8c4c80196348f8, data reload: false query1 0.050.050.04 query2 0.090.050.04 query3 0.220.050.06 query4 1.690.080.08 query5 0.490.500.49 query6 1.130.740.72 query7 0.020.010.01 query8 0.060.050.04 query9 0.550.500.49 query10 0.550.560.53 query11 0.160.110.12 query12 0.150.120.12 query13 0.600.590.59 query14 1.381.431.47 query15 0.830.810.80 query16 0.370.370.35 query17 0.971.040.99 query18 0.210.190.21 query19 1.911.791.73 query20 0.020.010.01 query21 15.39 0.650.64 query22 4.057.422.20 query23 18.25 1.291.29 query24 2.060.210.21 query25 0.150.090.08 query26 0.250.170.18 query27 0.070.070.08 query28 13.36 1.010.99 query29 12.58 3.343.34 query30 0.240.060.05 query31 2.880.390.38 query32 3.280.480.47 query33 2.983.013.01 query34 16.94 4.374.38 query35 4.484.394.53 query36 0.660.470.50 query37 0.180.160.15 query38 0.150.150.14 query39 0.040.040.03 query40 0.150.140.12 query41 0.100.040.05 query42 0.070.050.05 query43 0.040.040.04 Total cold run time: 109.8 s Total hot run time: 31.51 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Enhancement] Insert overwrite case need to keep old partition data in recycle-bin [doris]
Vallishp opened a new pull request, #40512: URL: https://github.com/apache/doris/pull/40512 ## Proposed changes Issue Number: close #xxx Insert overwrite command will trigger drop of partition and create partition and add the rows to table. In this case need to move the old data to catalog recycle bin so that user get chance to recover the data if needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] Insert overwrite case need to keep old partition data in recycle-bin [doris]
doris-robot commented on PR #40512: URL: https://github.com/apache/doris/pull/40512#issuecomment-2336648899 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] Insert overwrite case need to keep old partition data in recycle-bin [doris]
Vallishp commented on PR #40512: URL: https://github.com/apache/doris/pull/40512#issuecomment-2336648999 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] Insert overwrite case need to keep old partition data in recycle-bin [doris]
doris-robot commented on PR #40512: URL: https://github.com/apache/doris/pull/40512#issuecomment-2336659770 TPC-H: Total hot run time: 37904 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit def98dd182fcfc33a6db82e06cf14d023c7bf524, data reload: false -- Round 1 -- q1 17607 439642994299 q2 2020193 188 188 q3 11780 976 1144976 q4 10519 721 668 668 q5 7751284127562756 q6 221 139 140 139 q7 952 607 591 591 q8 9321208920782078 q9 7032649965546499 q10 7016225622312231 q11 481 239 240 239 q12 388 214 224 214 q13 17849 314830783078 q14 287 244 239 239 q15 539 489 492 489 q16 519 427 429 427 q17 969 711 708 708 q18 7396696169866961 q19 1390103210201020 q20 711 336 340 336 q21 4355305527522752 q22 1141101610221016 Total cold run time: 110244 ms Total hot run time: 37904 ms - Round 2, with runtime_filter_mode=off - q1 4380428945014289 q2 388 283 276 276 q3 2921271826522652 q4 1997171316581658 q5 5655569457855694 q6 232 140 139 139 q7 2242186318491849 q8 3324342534323425 q9 8862906890569056 q10 3667355135143514 q11 616 536 531 531 q12 862 681 743 681 q13 16401 336933433343 q14 325 280 282 280 q15 547 485 489 485 q16 535 483 497 483 q17 1842156015481548 q18 8326790078877887 q19 1814153315351533 q20 2156190218991899 q21 5737529154785291 q22 1115104710581047 Total cold run time: 73944 ms Total hot run time: 57560 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/01: [fix](log) fix fe doesn't print location in log when log mode is async
This is an automated email from the ASF dual-hosted git repository.
caiconghui pushed a commit to branch log4j-fix
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 25c15687f41be16704ae2b6fce79af7cbacd3b8a
Author: caiconghui1
AuthorDate: Sun Sep 8 20:07:52 2024 +0800
[fix](log) fix fe doesn't print location in log when log mode is async
---
.../main/java/org/apache/doris/common/Log4jConfig.java | 18 +-
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
b/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
index 206d1cb2089..39d13b0e989 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
@@ -252,13 +252,18 @@ public class Log4jConfig extends XmlConfiguration {
}
newXmlConfTemplate =
newXmlConfTemplate.replaceAll(VERBOSE_MODULE_PLACEHOLDER, sb.toString());
-if (sysLogMode.equalsIgnoreCase("NORMAL")) {
+// BRIEF: async, no location
+// ASYNC: async, with location
+// NORMAL: sync, with location
+boolean includeLocation = !sysLogMode.equalsIgnoreCase("BRIEF");
+boolean immediateFlush = sysLogMode.equalsIgnoreCase("NORMAL");
+if (includeLocation) {
newXmlConfTemplate =
newXmlConfTemplate.replaceAll(RUNTIME_LOG_FORMAT_PLACEHOLDER, " [%C{1}.%M():%L]
");
} else {
newXmlConfTemplate =
newXmlConfTemplate.replaceAll(RUNTIME_LOG_FORMAT_PLACEHOLDER, " ");
-if (sysLogMode.equalsIgnoreCase("ASYNC")) {
-newXmlConfTemplate = newXmlConfTemplate.replaceAll("Root",
"AsyncRoot");
-}
+}
+if (!immediateFlush) {
+newXmlConfTemplate = newXmlConfTemplate.replaceAll("Root",
"AsyncRoot");
}
if (Config.enable_file_logger) {
@@ -298,11 +303,6 @@ public class Log4jConfig extends XmlConfiguration {
properties.put("warn_sys_accumulated_file_size",
String.valueOf(Config.warn_sys_accumulated_file_size));
properties.put("audit_sys_accumulated_file_size",
String.valueOf(Config.audit_sys_accumulated_file_size));
-// BRIEF: async, no location
-// ASYNC: async, with location
-// NORMAL: sync, with location
-boolean includeLocation = !sysLogMode.equalsIgnoreCase("BRIEF");
-boolean immediateFlush = sysLogMode.equalsIgnoreCase("NORMAL");
properties.put("include_location_flag",
Boolean.toString(includeLocation));
properties.put("immediate_flush_flag",
Boolean.toString(immediateFlush));
properties.put("audit_file_postfix", compressAuditLog ? ".gz" : "");
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch log4j-fix created (now 25c15687f41)
This is an automated email from the ASF dual-hosted git repository. caiconghui pushed a change to branch log4j-fix in repository https://gitbox.apache.org/repos/asf/doris.git at 25c15687f41 [fix](log) fix fe doesn't print location in log when log mode is async This branch includes the following new commits: new 25c15687f41 [fix](log) fix fe doesn't print location in log when log mode is async The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](log) fix fe doesn't print location in log when log mode is async [doris]
caiconghui opened a new pull request, #40513: URL: https://github.com/apache/doris/pull/40513 ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](log) fix fe doesn't print location in log when log mode is async [doris]
caiconghui commented on PR #40513: URL: https://github.com/apache/doris/pull/40513#issuecomment-2336664522 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] Insert overwrite case need to keep old partition data in recycle-bin [doris]
doris-robot commented on PR #40512: URL: https://github.com/apache/doris/pull/40512#issuecomment-2336665750 ClickBench: Total hot run time: 31.21 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit def98dd182fcfc33a6db82e06cf14d023c7bf524, data reload: false query1 0.040.050.04 query2 0.080.040.04 query3 0.240.060.05 query4 1.670.100.09 query5 0.500.490.49 query6 1.120.740.73 query7 0.020.020.02 query8 0.060.040.04 query9 0.540.500.49 query10 0.540.560.53 query11 0.150.110.11 query12 0.150.130.12 query13 0.600.580.58 query14 1.381.391.40 query15 0.840.820.80 query16 0.360.370.39 query17 0.990.950.98 query18 0.200.190.19 query19 1.871.821.70 query20 0.020.010.00 query21 15.38 0.710.71 query22 4.587.391.71 query23 18.32 1.441.35 query24 2.050.230.23 query25 0.140.080.08 query26 0.280.190.18 query27 0.070.070.07 query28 13.27 1.020.99 query29 12.60 3.313.31 query30 0.240.060.05 query31 2.870.400.39 query32 3.270.470.48 query33 2.933.012.98 query34 17.14 4.394.45 query35 4.544.494.49 query36 0.660.510.49 query37 0.180.150.15 query38 0.160.140.14 query39 0.040.040.04 query40 0.160.120.12 query41 0.090.050.05 query42 0.050.050.05 query43 0.040.040.05 Total cold run time: 110.43 s Total hot run time: 31.21 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](log) fix fe doesn't print location in log when log mode is async [doris]
doris-robot commented on PR #40513: URL: https://github.com/apache/doris/pull/40513#issuecomment-2336672466 TPC-H: Total hot run time: 38092 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 25c15687f41be16704ae2b6fce79af7cbacd3b8a, data reload: false -- Round 1 -- q1 18156 445943894389 q2 2799206 186 186 q3 11147 119111841184 q4 10287 792 743 743 q5 7833289328032803 q6 231 140 143 140 q7 967 640 616 616 q8 9662204620422042 q9 7249653365266526 q10 7006218321362136 q11 457 239 251 239 q12 400 223 220 220 q13 17969 310730753075 q14 299 228 239 228 q15 539 482 476 476 q16 520 434 428 428 q17 994 675 730 675 q18 7351685967936793 q19 13861056970 970 q20 668 331 334 331 q21 3876287431012874 q22 1104101810361018 Total cold run time: 110900 ms Total hot run time: 38092 ms - Round 2, with runtime_filter_mode=off - q1 4341431842644264 q2 377 267 278 267 q3 2883268225902590 q4 1936159916571599 q5 5421540053935393 q6 224 131 131 131 q7 2098173617281728 q8 3190334433483344 q9 8490846484798464 q10 3457320131823182 q11 596 491 500 491 q12 786 611 591 591 q13 12567 309930953095 q14 310 277 277 277 q15 521 475 479 475 q16 528 477 482 477 q17 1793149714921492 q18 7719740576087405 q19 1663157914031403 q20 2027181018161810 q21 5369519652555196 q22 1094103110371031 Total cold run time: 67390 ms Total hot run time: 54705 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0-fix](merge-on-write) Fix some cases [doris]
doris-robot commented on PR #40514: URL: https://github.com/apache/doris/pull/40514#issuecomment-2336708855 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Cherry-pick](branch-2.0) Pick "[Enhancement](delete) Modify some delete cases to ajust enable_mow_light_delete config #40387" [doris]
Yukang-Lian opened a new pull request, #40515: URL: https://github.com/apache/doris/pull/40515 ## Proposed changes Pick 40387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](vault) Do not use latest_fs() in vault mode [doris]
gavinchou commented on PR #40516: URL: https://github.com/apache/doris/pull/40516#issuecomment-2336735378 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Feature] least and greatest function enhanced [doris]
HappenLee commented on issue #40275: URL: https://github.com/apache/doris/issues/40275#issuecomment-2336735831 `least(ifnull(col1, -1), ifnull(col2, -1))` you can write sql like this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](vault) Do not use latest_fs() in vault mode [doris]
doris-robot commented on PR #40516: URL: https://github.com/apache/doris/pull/40516#issuecomment-2336755577 TeamCity be ut coverage result: Function Coverage: 36.83% (9382/25474) Line Coverage: 28.23% (77383/274105) Region Coverage: 27.63% (39960/144609) Branch Coverage: 24.27% (20336/83802) Coverage Report: http://coverage.selectdb-in.cc/coverage/64d7fc4ed979eda13ea87e609f407a375dc89ad4_64d7fc4ed979eda13ea87e609f407a375dc89ad4/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] 同样的数据，排序键顺序不一样，数据量差30% [doris]
HappenLee closed issue #39608: [Bug] 同样的数据，排序键顺序不一样，数据量差30% URL: https://github.com/apache/doris/issues/39608 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](Short Circuit) short circuit query supports `IN` [doris]
doris-robot commented on PR #39468: URL: https://github.com/apache/doris/pull/39468#issuecomment-2336764129 TPC-H: Total hot run time: 38307 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 08a0bb26792bfb31d614511f91602fefcf39626f, data reload: false -- Round 1 -- q1 18267 475643904390 q2 2040198 188 188 q3 11754 995 1098995 q4 10225 762 787 762 q5 7795289228182818 q6 224 141 139 139 q7 973 625 609 609 q8 9330206721032067 q9 7065654065766540 q10 7004218321942183 q11 454 241 243 241 q12 394 224 232 224 q13 18936 308430873084 q14 289 229 230 229 q15 537 476 481 476 q16 518 436 426 426 q17 997 711 703 703 q18 7524694768356835 q19 1394108410941084 q20 665 352 336 336 q21 4011294031712940 q22 1132104310381038 Total cold run time: 111528 ms Total hot run time: 38307 ms - Round 2, with runtime_filter_mode=off - q1 4406436143034303 q2 385 272 274 272 q3 2954270427072704 q4 1983168116821681 q5 5676569158595691 q6 235 139 130 130 q7 2213186718391839 q8 3297342335243423 q9 8877881888138813 q10 3715337133543354 q11 596 515 521 515 q12 837 669 686 669 q13 13853 314332923143 q14 317 289 284 284 q15 562 478 502 478 q16 541 490 502 490 q17 1845155615311531 q18 8294794778527852 q19 1746165316671653 q20 2165190719161907 q21 5810544056605440 q22 1168106010611060 Total cold run time: 71475 ms Total hot run time: 57232 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](Short Circuit) short circuit query supports `IN` [doris]
doris-robot commented on PR #39468: URL: https://github.com/apache/doris/pull/39468#issuecomment-2336770762 TeamCity be ut coverage result: Function Coverage: 36.82% (9379/25476) Line Coverage: 28.21% (77329/274130) Region Coverage: 27.62% (39950/144616) Branch Coverage: 24.25% (20327/83806) Coverage Report: http://coverage.selectdb-in.cc/coverage/08a0bb26792bfb31d614511f91602fefcf39626f_08a0bb26792bfb31d614511f91602fefcf39626f/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] 开启strict模式后不符合要求的数据成功导入 [doris]
HappenLee commented on issue #40128: URL: https://github.com/apache/doris/issues/40128#issuecomment-2336783781 after you commit will get error msg. the invalid data not really insert success in doris -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-WIP](query cache) cache tablets aggregate result, BE part [doris]
HappenLee commented on PR #40171: URL: https://github.com/apache/doris/pull/40171#issuecomment-2336784541 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](delete_predicate) fix wrong data after upgrade from v2.0 version [doris]
github-actions[bot] commented on PR #40401: URL: https://github.com/apache/doris/pull/40401#issuecomment-2336925062 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](delete_predicate) fix wrong data after upgrade from v2.0 version (#40401)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dc706ed49b5 [fix](delete_predicate) fix wrong data after upgrade from
v2.0 version (#40401)
dc706ed49b5 is described below
commit dc706ed49b5d1b45d45ea58e78d39de011c18bc5
Author: camby
AuthorDate: Mon Sep 9 10:20:55 2024 +0900
[fix](delete_predicate) fix wrong data after upgrade from v2.0 version
(#40401)
issue: https://github.com/apache/doris/issues/40390
related pr: https://github.com/apache/doris/pull/22442
---
be/src/olap/delete_handler.cpp | 15 ++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/delete_handler.cpp b/be/src/olap/delete_handler.cpp
index 10c6f50b300..2f4e77d036b 100644
--- a/be/src/olap/delete_handler.cpp
+++ b/be/src/olap/delete_handler.cpp
@@ -411,7 +411,20 @@ Status DeleteHandler::init(TabletSchemaSPtr tablet_schema,
for (const auto& in_predicate : delete_condition.in_predicates()) {
TCondition condition;
condition.__set_column_name(in_predicate.column_name());
-auto col_unique_id = in_predicate.column_unique_id();
+
+int32_t col_unique_id = -1;
+if (in_predicate.has_column_unique_id()) {
+col_unique_id = in_predicate.column_unique_id();
+} else {
+// if upgrade from version 2.0.x, column_unique_id maybe not
set
+const auto& pre_column =
+
*DORIS_TRY(delete_pred_related_schema->column(condition.column_name));
+col_unique_id = pre_column.unique_id();
+}
+if (col_unique_id == -1) {
+return Status::Error(
+"cannot get column_unique_id for column {}",
condition.column_name);
+}
condition.__set_column_unique_id(col_unique_id);
if (in_predicate.is_not_in()) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Fix typos in documents metacache.md [doris-website]
RengarLee commented on PR #1089: URL: https://github.com/apache/doris-website/pull/1089#issuecomment-2336942218 @KassieZ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](backup) limit the involved tablets in a backup job #39987 [doris]
w41ter commented on PR #40518: URL: https://github.com/apache/doris/pull/40518#issuecomment-2336958465 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](backup) limit the involved tablets in a backup job #39987 [doris]
doris-robot commented on PR #40518: URL: https://github.com/apache/doris/pull/40518#issuecomment-2336958505 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] explode_map() funcation ERROR:Unknown table [doris]
HappenLee commented on issue #39614: URL: https://github.com/apache/doris/issues/39614#issuecomment-2336962662 https://github.com/apache/doris/pull/39972#event-14026257428 fix buy here，please update to 2.1.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Opt](ShortCircuit) optimize [doris]
eldenmoon opened a new pull request, #40519: URL: https://github.com/apache/doris/pull/40519 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](scanner) Fix deadlock when scanner submit failed [doris]
yiguolei commented on code in PR #40495:
URL: https://github.com/apache/doris/pull/40495#discussion_r1749465993
##
be/src/vec/exec/scan/scanner_context.cpp:
##
@@ -284,11 +299,14 @@ Status ScannerContext::get_block_from_queue(RuntimeState*
state, vectorized::Blo
}
} else {
// resubmit current running scanner to read the next block
-submit_scan_task(scan_task);
+Status submit_status = submit_scan_task(scan_task);
+if (!submit_status.ok()) {
+append_block_to_queue_locked(scan_task, l);
Review Comment:
not need
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](scanner) Fix deadlock when scanner submit failed [doris]
yiguolei commented on code in PR #40495:
URL: https://github.com/apache/doris/pull/40495#discussion_r1749466401
##
be/src/vec/exec/scan/scanner_context.cpp:
##
@@ -263,13 +272,19 @@ Status ScannerContext::get_block_from_queue(RuntimeState*
state, vectorized::Blo
block->swap(*current_block);
return_free_block(std::move(current_block));
} else {
+// This scan task do not have any cached blocks.
_blocks_queue.pop_front();
-if (scan_task->is_eos()) { // current scanner is finished, and no
more data to read
+// current scanner is finished, and no more data to read
+if (scan_task->is_eos()) {
_num_finished_scanners++;
std::weak_ptr next_scanner;
// submit one of the remaining scanners
if (_scanners.try_dequeue(next_scanner)) {
-submit_scan_task(std::make_shared(next_scanner));
+auto scan_task_ptr =
std::make_shared(next_scanner);
+Status submit_status = submit_scan_task(scan_task_ptr);
+if (!submit_status.ok()) {
+append_block_to_queue_locked(scan_task_ptr, l);
Review Comment:
also should update the scanner context's status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage API. [doris]
hubgeter commented on PR #40225: URL: https://github.com/apache/doris/pull/40225#issuecomment-2336976642 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](scanner) Fix deadlock when scanner submit failed [doris]
yiguolei commented on code in PR #40495: URL: https://github.com/apache/doris/pull/40495#discussion_r1749466893 ## be/src/vec/exec/scan/scanner_context.cpp: ## @@ -214,7 +214,12 @@ void ScannerContext::append_block_to_queue(std::shared_ptr scan_task) } } } -std::lock_guard l(_transfer_lock); +std::unique_lock l(_transfer_lock); +append_block_to_queue_locked(scan_task, l); Review Comment: 如果我们不处理submit 之后的重试，那么这个方法的拆分就意义不大了 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values [doris]
github-actions[bot] commented on PR #40385: URL: https://github.com/apache/doris/pull/40385#issuecomment-2336981754 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values [doris]
github-actions[bot] commented on PR #40385: URL: https://github.com/apache/doris/pull/40385#issuecomment-2336981783 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](scanner) Fix deadlock when scanner submit failed [doris]
yiguolei commented on code in PR #40495:
URL: https://github.com/apache/doris/pull/40495#discussion_r1749466401
##
be/src/vec/exec/scan/scanner_context.cpp:
##
@@ -263,13 +272,19 @@ Status ScannerContext::get_block_from_queue(RuntimeState*
state, vectorized::Blo
block->swap(*current_block);
return_free_block(std::move(current_block));
} else {
+// This scan task do not have any cached blocks.
_blocks_queue.pop_front();
-if (scan_task->is_eos()) { // current scanner is finished, and no
more data to read
+// current scanner is finished, and no more data to read
+if (scan_task->is_eos()) {
_num_finished_scanners++;
std::weak_ptr next_scanner;
// submit one of the remaining scanners
if (_scanners.try_dequeue(next_scanner)) {
-submit_scan_task(std::make_shared(next_scanner));
+auto scan_task_ptr =
std::make_shared(next_scanner);
+Status submit_status = submit_scan_task(scan_task_ptr);
+if (!submit_status.ok()) {
+append_block_to_queue_locked(scan_task_ptr, l);
Review Comment:
also should update the scanner context's status
if (!_process_status.ok()) {
_set_scanner_done();
return _process_status;
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](funcs) map struct construct funcs #39699 [doris]
xiaokang commented on PR #40502: URL: https://github.com/apache/doris/pull/40502#issuecomment-2336983112 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](scanner) Fix deadlock when scanner submit failed [doris]
yiguolei commented on code in PR #40495:
URL: https://github.com/apache/doris/pull/40495#discussion_r1749471749
##
be/src/vec/exec/scan/scanner_context.cpp:
##
@@ -284,11 +299,14 @@ Status ScannerContext::get_block_from_queue(RuntimeState*
state, vectorized::Blo
}
} else {
// resubmit current running scanner to read the next block
-submit_scan_task(scan_task);
+Status submit_status = submit_scan_task(scan_task);
+if (!submit_status.ok()) {
+append_block_to_queue_locked(scan_task, l);
Review Comment:
set process status to abnormal and return the error status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feature](ES Catalog)Add FE open API for ES Catalog (#40444)
This is an automated email from the ASF dual-hosted git repository.
jianliangqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a06625ee6b6 [feature](ES Catalog)Add FE open API for ES Catalog
(#40444)
a06625ee6b6 is described below
commit a06625ee6b626ce95d9e665f8cdd279ea153096b
Author: qiye
AuthorDate: Mon Sep 9 10:20:38 2024 +0800
[feature](ES Catalog)Add FE open API for ES Catalog (#40444)
Add api to get info from Elasticsearch, it's very useful for debugging.
1. get index mapping api
```
GET /rest/v2/api/es_catalog/get_mapping?catalog=xxx&table=xxx
```
2. search index api
```
POST /rest/v2/api/es_catalog/search?catalog=xxx&table=xxx
{
request_body
...
}
```
---
.../apache/doris/datasource/es/EsRestClient.java | 36 ++-
.../doris/httpv2/restv2/ESCatalogAction.java | 108 +
.../plugins/plugin_curl_requester.groovy | 16 +--
.../es/test_es_catalog_http_open_api.groovy| 84
4 files changed, 233 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/es/EsRestClient.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/es/EsRestClient.java
index 21a85126c9e..e2c7fa0b688 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/es/EsRestClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/es/EsRestClient.java
@@ -24,8 +24,10 @@ import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.google.common.collect.ImmutableList;
import okhttp3.Credentials;
+import okhttp3.MediaType;
import okhttp3.OkHttpClient;
import okhttp3.Request;
+import okhttp3.RequestBody;
import okhttp3.Response;
import org.apache.http.HttpHeaders;
import org.apache.logging.log4j.LogManager;
@@ -126,12 +128,27 @@ public class EsRestClient {
return indexMapping;
}
+/**
+ * Search specific index
+ */
+public String searchIndex(String indexName, String body) throws
DorisEsException {
+String path = indexName + "/_search";
+RequestBody requestBody = null;
+if (Strings.isNotEmpty(body)) {
+requestBody = RequestBody.create(
+body,
+MediaType.get("application/json")
+);
+}
+return executeWithRequestBody(path, requestBody);
+}
+
/**
* Check whether index exist.
**/
public boolean existIndex(OkHttpClient httpClient, String indexName) {
String path = indexName + "/_mapping";
-try (Response response = executeResponse(httpClient, path)) {
+try (Response response = executeResponse(httpClient, path, null)) {
if (response.isSuccessful()) {
return true;
}
@@ -228,7 +245,7 @@ public class EsRestClient {
return sslNetworkClient;
}
-private Response executeResponse(OkHttpClient httpClient, String path)
throws IOException {
+private Response executeResponse(OkHttpClient httpClient, String path,
RequestBody requestBody) throws IOException {
currentNode = currentNode.trim();
if (!(currentNode.startsWith("http://";) ||
currentNode.startsWith("https://";))) {
currentNode = "http://"; + currentNode;
@@ -239,7 +256,12 @@ public class EsRestClient {
String url = currentNode + path;
try {
SecurityChecker.getInstance().startSSRFChecking(url);
-Request request = builder.get().url(currentNode + path).build();
+Request request;
+if (requestBody != null) {
+request = builder.post(requestBody).url(currentNode +
path).build();
+} else {
+request = builder.get().url(currentNode + path).build();
+}
if (LOG.isInfoEnabled()) {
LOG.info("es rest client request URL: {}",
request.url().toString());
}
@@ -251,13 +273,17 @@ public class EsRestClient {
}
}
+private String execute(String path) throws DorisEsException {
+return executeWithRequestBody(path, null);
+}
+
/**
* execute request for specific path，it will try again nodes.length times
if it fails
*
* @param path the path must not leading with '/'
* @return response
*/
-private String execute(String path) throws DorisEsException {
+private String executeWithRequestBody(String path, RequestBody
requestBody) throws DorisEsException {
// try 3 times for every node
int retrySize = nodes.length * 3;
DorisEsException scratchExceptionForThrow = null;
@@ -277,7 +303,7 @@ public class EsRestClient {
if (LOG.isTraceEnabled()) {
LOG.trace("es rest client request
Re: [PR] [regression-test](point query) add `enable_mow_light_delete=false` to related cases [doris]
doris-robot commented on PR #40520: URL: https://github.com/apache/doris/pull/40520#issuecomment-2336987049 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-WIP](query cache) cache tablets aggregate result, BE part [doris]
HappenLee commented on PR #40171: URL: https://github.com/apache/doris/pull/40171#issuecomment-2336987395 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) [doris]
doris-robot commented on PR #40521: URL: https://github.com/apache/doris/pull/40521#issuecomment-2336988554 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) [doris]
qidaye commented on PR #40521: URL: https://github.com/apache/doris/pull/40521#issuecomment-2336988656 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) [doris]
qidaye opened a new pull request, #40522: URL: https://github.com/apache/doris/pull/40522 bp #40385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) [doris]
qidaye commented on PR #40522: URL: https://github.com/apache/doris/pull/40522#issuecomment-2336990552 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression-test](point query) add `enable_mow_light_delete=false` to related cases [doris]
github-actions[bot] commented on PR #40520: URL: https://github.com/apache/doris/pull/40520#issuecomment-2336990788 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression-test](point query) add `enable_mow_light_delete=false` to related cases [doris]
eldenmoon commented on PR #40523: URL: https://github.com/apache/doris/pull/40523#issuecomment-2336994042 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](multi table) do not use strlen to calculate the length of msg (#40367) (#40511)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 1c91fbc167e [fix](multi table) do not use strlen to calculate the
length of msg (#40367) (#40511)
1c91fbc167e is described below
commit 1c91fbc167ee25df26d829175bd1d3b1e747c66a
Author: hui lai <1353307...@qq.com>
AuthorDate: Mon Sep 9 10:35:59 2024 +0800
[fix](multi table) do not use strlen to calculate the length of msg
(#40367) (#40511)
pick #40367
Meet code dump when using single stream multi table load:
```
SUMMARY: AddressSanitizer: heap-buffer-overflow
/root/doris/be/src/io/fs/multi_table_pipe.cpp:99:22 in
doris::io::MultiTablePipe::dispatch(std::__cxx11::basic_string, std::allocator> const&, char const*, unsigned
long, doris::Status (doris::io::KafkaConsumerPipe::*)(char const*, unsigned
long))
```
1. It is hard to guaranteed that msg is a C-style string ending in '\0'
character. If not, it may cause the core dump to access memory out of
bounds.
2. It is not need to calculate the length of msg twice.
Therefore, deleting the logic that using strlen to calculate the length
of msg.
---
be/src/io/fs/multi_table_pipe.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/io/fs/multi_table_pipe.cpp
b/be/src/io/fs/multi_table_pipe.cpp
index 75fd05f2ef8..be77b3685fc 100644
--- a/be/src/io/fs/multi_table_pipe.cpp
+++ b/be/src/io/fs/multi_table_pipe.cpp
@@ -96,7 +96,7 @@ std::string MultiTablePipe::parse_dst_table(const char* data,
size_t size) {
Status MultiTablePipe::dispatch(const std::string& table, const char* data,
size_t size,
AppendFunc cb) {
-if (size == 0 || strlen(data) == 0) {
+if (size == 0) {
LOG(WARNING) << "empty data for table: " << table << ", ctx: " <<
_ctx->brief();
return Status::InternalError("empty data");
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) [doris]
github-actions[bot] commented on PR #40522: URL: https://github.com/apache/doris/pull/40522#issuecomment-2336996313 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [optimize](short circuit) avoid set cacheId when non prepared execute [doris]
doris-robot commented on PR #40525: URL: https://github.com/apache/doris/pull/40525#issuecomment-2336998458 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement]Add more tip when drop workload group failed [doris]
github-actions[bot] commented on PR #40468: URL: https://github.com/apache/doris/pull/40468#issuecomment-2337000362 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](funcs) map struct construct funcs #39699 [doris]
doris-robot commented on PR #40502: URL: https://github.com/apache/doris/pull/40502#issuecomment-2337001829 TPC-H: Total hot run time: 49279 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit bfa788f3188ac13edb8e02bf9c082391d801815b, data reload: false -- Round 1 -- q1 18113 448043334333 q2 2068153 150 150 q3 10432 187718761876 q4 10336 123713051237 q5 8548390939083908 q6 233 123 123 123 q7 2011163216051605 q8 9318271526992699 q9 10277 985699279856 q10 8651354534913491 q11 422 237 250 237 q12 480 300 295 295 q13 18363 403240134013 q14 356 318 320 318 q15 507 464 454 454 q16 546 453 458 453 q17 1137952 986 952 q18 7327697169746971 q19 1696154415461544 q20 549 324 303 303 q21 4414413740674067 q22 505 394 395 394 Total cold run time: 116289 ms Total hot run time: 49279 ms - Round 2, with runtime_filter_mode=off - q1 4333428042734273 q2 314 228 223 223 q3 4192413441634134 q4 2751275827622758 q5 7131710070727072 q6 235 118 117 117 q7 3288285727772777 q8 4322446044834460 q9 13648 13664 13789 13664 q10 4250426442684264 q11 748 680 685 680 q12 1032847 862 847 q13 6500371437443714 q14 453 428 416 416 q15 502 472 463 463 q16 630 597 610 597 q17 3860387038303830 q18 8723864987278649 q19 1715165116641651 q20 2370214220942094 q21 8596840484958404 q22 1001918 919 918 Total cold run time: 80594 ms Total hot run time: 76005 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [feature](ES Catalog)Add FE open API for ES Catalog (#40444) [doris]
qidaye opened a new pull request, #40526: URL: https://github.com/apache/doris/pull/40526 bp #40444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ES Catalog)Add FE open API for ES Catalog (#40444) [doris]
doris-robot commented on PR #40526: URL: https://github.com/apache/doris/pull/40526#issuecomment-2337002003 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ES Catalog)Add FE open API for ES Catalog (#40444) [doris]
qidaye commented on PR #40526: URL: https://github.com/apache/doris/pull/40526#issuecomment-2337002364 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (a963709fed0 -> e0b22b5104b)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from a963709fed0 [opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927 (#40357) add e0b22b5104b [enhancement](schema-change) Log out end version before truncating new tablet data (#39924) (#40239) No new revisions were added by this update. Summary of changes: be/src/olap/schema_change.cpp | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](nereids)set actual row count in physical plan according to merged profile [doris]
doris-robot commented on PR #40361: URL: https://github.com/apache/doris/pull/40361#issuecomment-2337004872 TPC-H: Total hot run time: 37522 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b1d5172903add964ebe275e9200598045f5cda42, data reload: false -- Round 1 -- q1 17604 435742994299 q2 2033183 196 183 q3 11644 967 1090967 q4 10515 679 711 679 q5 7756282828092809 q6 230 136 134 134 q7 952 615 598 598 q8 9313204420782044 q9 7006648364696469 q10 7002221622342216 q11 472 241 239 239 q12 393 220 215 215 q13 19035 309531023095 q14 279 243 239 239 q15 534 493 494 493 q16 518 444 426 426 q17 967 620 632 620 q18 7232664568786645 q19 1384109910061006 q20 697 337 321 321 q21 4328286528282828 q22 1108997 1004997 Total cold run time: 111002 ms Total hot run time: 37522 ms - Round 2, with runtime_filter_mode=off - q1 4376428243194282 q2 376 272 273 272 q3 2838266226192619 q4 1907164416701644 q5 5599568156615661 q6 229 134 144 134 q7 2221178717831783 q8 3268338933923389 q9 8790882287968796 q10 3543337333903373 q11 614 508 519 508 q12 807 647 652 647 q13 14622 320533073205 q14 321 291 285 285 q15 535 475 479 475 q16 561 515 506 506 q17 1835154615231523 q18 8177791780557917 q19 1691151615351516 q20 2168190319331903 q21 5715539153525352 q22 1113102210021002 Total cold run time: 71306 ms Total hot run time: 56792 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [feature](ES Catalog)Add FE open API for ES Catalog (#40444) [doris]
qidaye opened a new pull request, #40527: URL: https://github.com/apache/doris/pull/40527 bp #40444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Improvement](sort) do not sort partitial when spill disabled [doris]
BiteThet opened a new pull request, #40528: URL: https://github.com/apache/doris/pull/40528 ## Proposed changes do not sort partitial when spill disabled -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ES Catalog)Add FE open API for ES Catalog (#40444) [doris]
qidaye commented on PR #40527: URL: https://github.com/apache/doris/pull/40527#issuecomment-2337007768 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](routine load) make error msg clear if routine load name illegal (#40037) [doris]
sollhui commented on PR #40510: URL: https://github.com/apache/doris/pull/40510#issuecomment-2337009229 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](Short Circuit) short circuit query supports `IN` [doris]
lxr599 commented on PR #39468: URL: https://github.com/apache/doris/pull/39468#issuecomment-2337010439 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement]Add more tip when drop workload group failed [doris]
wangbo merged PR #40468: URL: https://github.com/apache/doris/pull/40468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Improvement]Add more tip when drop workload group failed (#40468)
This is an automated email from the ASF dual-hosted git repository.
wangbo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 903c05b3af3 [Improvement]Add more tip when drop workload group failed
(#40468)
903c05b3af3 is described below
commit 903c05b3af3b99a125ba44693ff453d8a995e7ed
Author: wangbo
AuthorDate: Mon Sep 9 10:55:02 2024 +0800
[Improvement]Add more tip when drop workload group failed (#40468)
## Proposed changes
Add more tip to tell user what to do when drop wg failed.
---
.../org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java| 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
index 5ddc5fb68f8..8ba9b1a7609 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
@@ -429,7 +429,10 @@ public class WorkloadGroupMgr extends MasterDaemon
implements Writable, GsonPost
// user need to reset user property first
Pair ret =
Env.getCurrentEnv().getAuth().isWorkloadGroupInUse(workloadGroupName);
if (ret.first) {
-throw new DdlException("workload group " + workloadGroupName + "
is set for user " + ret.second);
+throw new DdlException("workload group " + workloadGroupName + "
is set for user " + ret.second
++ ", you can reset the user's property(eg, "
++ "set property for " + ret.second + "
'default_workload_group'='xxx'; ), "
++ "then you can drop the group.");
}
// A group with related policies should not be deleted.
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](schema-change) Support light schema change on hash columns and agg key columns with varchar type to change length (#39319) [doris]
yiguolei merged PR #40236: URL: https://github.com/apache/doris/pull/40236 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv) Generate mtmv cache should use ADMIN user, and rewritten plan should not check privilege [doris]
seawinde commented on PR #40374: URL: https://github.com/apache/doris/pull/40374#issuecomment-2337012200 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) [doris]
doris-robot commented on PR #40522: URL: https://github.com/apache/doris/pull/40522#issuecomment-2337017773 TPC-H: Total hot run time: 49284 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0867cd57e5a52b20c6e8b49428653beaf18be029, data reload: false -- Round 1 -- q1 18142 433343144314 q2 2035184 142 142 q3 10410 186019041860 q4 10337 129413251294 q5 8398393738743874 q6 228 124 122 122 q7 2020162316131613 q8 9304273027042704 q9 10591 10133 10084 10084 q10 8606353134903490 q11 425 244 245 244 q12 468 295 302 295 q13 18336 397340423973 q14 362 333 331 331 q15 517 458 461 458 q16 544 471 451 451 q17 1123927 913 913 q18 7184693368356835 q19 1691157415171517 q20 548 303 297 297 q21 4385414240804080 q22 500 393 397 393 Total cold run time: 116154 ms Total hot run time: 49284 ms - Round 2, with runtime_filter_mode=off - q1 4321432443324324 q2 321 227 223 223 q3 4197414541354135 q4 2756273827462738 q5 7140715271127112 q6 237 120 121 120 q7 3220279928352799 q8 4404447144744471 q9 14213 14015 13910 13910 q10 4225428342724272 q11 762 697 666 666 q12 1036839 840 839 q13 6926393537413741 q14 441 416 416 416 q15 497 462 445 445 q16 630 586 599 586 q17 3777391439173914 q18 8674876287528752 q19 1711167516501650 q20 2393210321142103 q21 8457846884628462 q22 1031956 968 956 Total cold run time: 81369 ms Total hot run time: 76634 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feat](Nereids) support string functions fold constant by fe [doris]
doris-robot commented on PR #40441: URL: https://github.com/apache/doris/pull/40441#issuecomment-2337018943 TPC-H: Total hot run time: 38006 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 54275d49f7d10ca215ab7a109cdd22246ffd24af, data reload: false -- Round 1 -- q1 17625 447243094309 q2 2024207 184 184 q3 11844 952 1162952 q4 10525 763 751 751 q5 7772284328112811 q6 225 136 135 135 q7 961 621 605 605 q8 9316208620652065 q9 7201650665846506 q10 7003222622422226 q11 478 234 240 234 q12 392 219 221 219 q13 18715 308330523052 q14 287 231 248 231 q15 540 492 487 487 q16 531 420 437 420 q17 975 708 681 681 q18 7344694967626762 q19 1389112210271027 q20 691 331 331 331 q21 3929312630003000 q22 1156104310181018 Total cold run time: 110923 ms Total hot run time: 38006 ms - Round 2, with runtime_filter_mode=off - q1 4346433142584258 q2 391 281 267 267 q3 2915273426502650 q4 1976168616471647 q5 5672564057705640 q6 233 140 142 140 q7 2177187618091809 q8 3233347334583458 q9 8775878688078786 q10 3550338333453345 q11 597 499 511 499 q12 833 698 653 653 q13 15157 330632533253 q14 324 293 297 293 q15 517 484 497 484 q16 545 516 486 486 q17 1837152515161516 q18 800576647664 q19 1700153015391530 q20 2160190219261902 q21 5779555455915554 q22 1169107010541054 Total cold run time: 71891 ms Total hot run time: 56888 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](mtmv) Support to add use_for_rewrite property when create materialized view [doris]
zddr commented on code in PR #40332:
URL: https://github.com/apache/doris/pull/40332#discussion_r1749490925
##
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java:
##
@@ -66,6 +67,8 @@ public static void analyzeProperty(String key, String value) {
break;
case PropertyAnalyzer.PROPERTIES_ENABLE_NONDETERMINISTIC_FUNCTION:
break;
+case PropertyAnalyzer.PROPERTIES_USE_FOR_REWRITE:
Review Comment:
we need check type is `boolean`?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java:
##
@@ -129,15 +131,24 @@ private List
createAsyncMaterializationContext(CascadesC
Set usedTables) {
Set availableMTMVs = getAvailableMTMVs(usedTables,
cascadesContext);
if (availableMTMVs.isEmpty()) {
-LOG.debug(String.format("Enable materialized view rewrite but
availableMTMVs is empty, current queryId "
-+ "is %s",
cascadesContext.getConnectContext().getQueryIdentifier()));
+LOG.debug("Enable materialized view rewrite but availableMTMVs is
empty, current queryId "
++ "is {}",
cascadesContext.getConnectContext().getQueryIdentifier());
return ImmutableList.of();
}
List asyncMaterializationContext = new
ArrayList<>();
for (MTMV materializedView : availableMTMVs) {
MTMVCache mtmvCache = null;
try {
mtmvCache =
materializedView.getOrGenerateCache(cascadesContext.getConnectContext());
+// If mv property use_for_rewrite is set false, should not
partition in
+// query rewrite by materialized view
+String usedForRewrite = materializedView.getMvProperties().get(
+PropertyAnalyzer.PROPERTIES_USE_FOR_REWRITE);
+if (!StringUtils.isEmpty(usedForRewrite) &&
!Boolean.parseBoolean(usedForRewrite)) {
Review Comment:
Suggest extracting a method from the logic for obtaining usedForRewrite and
placing it in MTMV, similar to getWorkload Group,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) tabe row count priority: user injected > BE report > analyzed [doris]
englefly commented on PR #40529: URL: https://github.com/apache/doris/pull/40529#issuecomment-2337019959 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [optimize](short circuit) avoid set cacheId when non prepared execute [doris]
doris-robot commented on PR #40525: URL: https://github.com/apache/doris/pull/40525#issuecomment-2337020066 TPC-H: Total hot run time: 38019 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 332273fbb0fc4ef557b73d24828545d17c810c20, data reload: false -- Round 1 -- q1 17620 436842964296 q2 2019195 183 183 q3 11990 936 1135936 q4 10510 808 691 691 q5 7760287828322832 q6 230 138 137 137 q7 957 621 611 611 q8 9317209020832083 q9 7233653665536536 q10 7006223522262226 q11 458 250 246 246 q12 407 222 229 222 q13 18881 308430783078 q14 290 239 251 239 q15 534 490 488 488 q16 520 443 414 414 q17 975 746 653 653 q18 7369686468416841 q19 1393102610651026 q20 677 342 334 334 q21 3954293530152935 q22 1135101210251012 Total cold run time: 111235 ms Total hot run time: 38019 ms - Round 2, with runtime_filter_mode=off - q1 4309428042324232 q2 384 279 274 274 q3 2920267326692669 q4 1975160816271608 q5 5698570557085705 q6 228 136 137 136 q7 2257187218091809 q8 3295341634603416 q9 8923879488028794 q10 3528344234093409 q11 607 512 509 509 q12 835 671 647 647 q13 14539 317532873175 q14 332 289 296 289 q15 546 493 491 491 q16 527 492 509 492 q17 1817155215421542 q18 8155789977827782 q19 1767166317521663 q20 2175190519171905 q21 5832571056045604 q22 1133102610501026 Total cold run time: 71782 ms Total hot run time: 57177 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feat](Nereids) support string functions fold constant by fe [doris]
LiBinfeng-01 commented on PR #40441: URL: https://github.com/apache/doris/pull/40441#issuecomment-2337020495 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](nereids)set actual row count in physical plan according to merged profile [doris]
doris-robot commented on PR #40361: URL: https://github.com/apache/doris/pull/40361#issuecomment-2337020996 ClickBench: Total hot run time: 31.49 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b1d5172903add964ebe275e9200598045f5cda42, data reload: false query1 0.040.050.04 query2 0.080.040.04 query3 0.220.050.05 query4 1.670.070.08 query5 0.490.490.50 query6 1.130.730.72 query7 0.020.010.02 query8 0.060.040.04 query9 0.550.490.49 query10 0.550.580.54 query11 0.150.120.11 query12 0.140.120.11 query13 0.600.580.58 query14 1.411.431.43 query15 0.830.830.84 query16 0.370.350.39 query17 1.021.021.04 query18 0.210.200.19 query19 1.841.811.83 query20 0.010.010.01 query21 15.41 0.680.66 query22 4.016.802.05 query23 18.32 1.381.27 query24 2.100.240.22 query25 0.140.080.08 query26 0.260.190.18 query27 0.080.070.08 query28 13.19 1.010.98 query29 12.59 3.313.32 query30 0.260.060.06 query31 2.850.410.38 query32 3.270.480.48 query33 2.962.993.01 query34 17.17 4.374.44 query35 4.434.384.44 query36 0.650.500.49 query37 0.190.150.15 query38 0.160.140.14 query39 0.050.040.05 query40 0.150.130.13 query41 0.090.050.05 query42 0.050.040.04 query43 0.050.040.04 Total cold run time: 109.82 s Total hot run time: 31.49 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auth)Fix some issues with incorrect permission verification (#3… [doris]
yiguolei merged PR #40410: URL: https://github.com/apache/doris/pull/40410 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](funcs) map struct construct funcs #39699 [doris]
doris-robot commented on PR #40502: URL: https://github.com/apache/doris/pull/40502#issuecomment-2337021021 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit bfa788f3188ac13edb8e02bf9c082391d801815b with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.4 seconds inserted 1000 Rows, about 467K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) [doris]
doris-robot commented on PR #40522: URL: https://github.com/apache/doris/pull/40522#issuecomment-2337021084 TeamCity be ut coverage result: Function Coverage: 37.84% (8135/21499) Line Coverage: 29.59% (66970/226357) Region Coverage: 29.08% (34556/118851) Branch Coverage: 24.99% (17811/71272) Coverage Report: http://coverage.selectdb-in.cc/coverage/0867cd57e5a52b20c6e8b49428653beaf18be029_0867cd57e5a52b20c6e8b49428653beaf18be029/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](auth)Fix some issues with incorrect permission verification (#3… (#40410)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 93a2518d12e [fix](auth)Fix some issues with incorrect permission
verification (#3… (#40410)
93a2518d12e is described below
commit 93a2518d12efe15312825002ac144840be951317
Author: zhangdong <493738...@qq.com>
AuthorDate: Mon Sep 9 11:04:49 2024 +0800
[fix](auth)Fix some issues with incorrect permission verification (#3…
(#40410)
…9726)
pick: https://github.com/apache/doris/pull/39726
---
.../main/java/org/apache/doris/analysis/ShowColumnStmt.java | 11 +++
.../src/main/java/org/apache/doris/analysis/ShowDataStmt.java | 2 +-
.../main/java/org/apache/doris/analysis/ShowSyncJobStmt.java | 9 +
.../apache/doris/analysis/ShowTabletStorageFormatStmt.java| 6 ++
.../src/main/java/org/apache/doris/qe/ConnectScheduler.java | 2 +-
5 files changed, 24 insertions(+), 6 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStmt.java
index eb7fcaf0285..9af269104cc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStmt.java
@@ -18,9 +18,14 @@
package org.apache.doris.analysis;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.InfoSchemaDb;
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.ShowResultSetMetaData;
import com.google.common.base.Strings;
@@ -103,6 +108,12 @@ public class ShowColumnStmt extends ShowStmt {
} else {
metaData = META_DATA;
}
+if (!Env.getCurrentEnv().getAccessManager()
+.checkTblPriv(ConnectContext.get(), tableName.getCtl(),
tableName.getDb(),
+tableName.getTbl(), PrivPredicate.SHOW)) {
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLE_ACCESS_DENIED_ERROR,
+PrivPredicate.SHOW.getPrivs().toString(), tableName);
+}
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataStmt.java
index dd2053750ba..799fa68bcf7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataStmt.java
@@ -115,7 +115,7 @@ public class ShowDataStmt extends ShowStmt {
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
dbName = analyzer.getDefaultDb();
-if (Strings.isNullOrEmpty(dbName)) {
+if (Strings.isNullOrEmpty(dbName) && tableName == null) {
getAllDbStats();
return;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowSyncJobStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowSyncJobStmt.java
index 25980ea16a8..f0671f8afe0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowSyncJobStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowSyncJobStmt.java
@@ -18,10 +18,14 @@
package org.apache.doris.analysis;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.InternalCatalog;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.ShowResultSetMetaData;
import com.google.common.base.Strings;
@@ -60,6 +64,11 @@ public class ShowSyncJobStmt extends ShowStmt {
ErrorReport.reportAnalysisException(ErrorCode.ERR_NO_DB_ERROR);
}
}
+if (!Env.getCurrentEnv().getAccessManager()
+.checkDbPriv(ConnectContext.get(),
InternalCatalog.INTERNAL_CATALOG_NAME, dbName, PrivPredicate.SHOW)) {
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_DB_ACCESS_DENIED_ERROR,
+PrivPredicate.SHOW.getPrivs().toString(), dbName);
+}
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTabletStorageFormatStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTabletStorageFormatStmt.java
index 441f0f1d7d5..9d0f3b88e6c 100644
---
a/fe/fe-core/src/ma
Re: [PR] [chore](mtmv)add plan log for UpdateMvByPartitionCommand (#40072) [doris]
yiguolei merged PR #40413: URL: https://github.com/apache/doris/pull/40413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auth) ordinary users can see the processes of other users (#39747) [doris]
yiguolei merged PR #40415: URL: https://github.com/apache/doris/pull/40415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (a3eba2aad5e -> ecb75c2e7da)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
from a3eba2aad5e [fix](auth) ordinary users can see the processes of other
users (#39747) (#40415)
add ecb75c2e7da [fix](mtmv) Mtmv support set both immediate and starttime
(#39573) (#40418)
No new revisions were added by this update.
Summary of changes:
.../java/org/apache/doris/mtmv/MTMVJobManager.java | 8 ++--
...ate_table_mtmv.out => test_start_time_mtmv.out} | 11 --
...iew_mtmv.groovy => test_start_time_mtmv.groovy} | 44 --
3 files changed, 36 insertions(+), 27 deletions(-)
copy regression-test/data/mtmv_p0/{test_truncate_table_mtmv.out =>
test_start_time_mtmv.out} (50%)
copy regression-test/suites/mtmv_p0/{test_create_view_mtmv.groovy =>
test_start_time_mtmv.groovy} (70%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) [doris]
doris-robot commented on PR #40522: URL: https://github.com/apache/doris/pull/40522#issuecomment-2337028615 TPC-DS: Total hot run time: 213134 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0867cd57e5a52b20c6e8b49428653beaf18be029, data reload: false query1 923 391 420 391 query2 6533225321572157 query3 6942208 210 208 query4 23486 21754 22266 21754 query5 23084 659966486599 query6 1130216 230 216 query7 6027310 317 310 query8 253 288 241 241 query9 3060258725422542 query10 499 307 305 305 query11 16038 15492 15224 15224 query12 135 76 76 76 query13 1051444 448 444 query14 19188 13617 13482 13482 query15 358 224 218 218 query16 6431281 259 259 query17 1572930 886 886 query18 893 323 314 314 query19 210 145 144 144 query20 82 77 83 77 query21 195 100 93 93 query22 5327504250845042 query23 34112 33352 33503 33352 query24 6505626962846269 query25 519 431 413 413 query26 800 167 165 165 query27 2279300 299 299 query28 6083229822402240 query29 2918271526442644 query30 243 169 170 169 query31 972 752 793 752 query32 69 63 57 57 query33 438 253 270 253 query34 887 470 468 468 query35 1142956 909 909 query36 1437133812131213 query37 90 60 57 57 query38 3082296529062906 query39 1407133113141314 query40 203 96 89 89 query41 40 38 38 38 query42 86 88 82 82 query43 656 604 580 580 query44 1159721 729 721 query45 245 236 226 226 query46 1239967 958 958 query47 1870174819631748 query48 526 409 410 409 query49 611 380 376 376 query50 859 601 621 601 query51 4745464046384638 query52 88 87 83 83 query53 235 191 178 178 query54 2722247325122473 query55 97 84 82 82 query56 229 192 211 192 query57 1178115310801080 query58 223 222 227 222 query59 3516348732283228 query60 213 206 211 206 query61 99 97 99 97 query62 788 466 435 435 query63 203 175 178 175 query64 3253151713421342 query65 3632355235863552 query66 735 403 403 403 query67 16630 15284 15372 15284 query68 9793638 648 638 query69 503 284 280 280 query70 1521131313661313 query71 426 310 307 307 query72 7022492248714871 query73 758 320 311 311 query74 6462596258965896 query75 5491369736213621 query76 5567115712301157 query77 942 256 251 251 query78 12746 11882 11803 11803 query79 6943642 644 642 query80 1534382 378 378 query81 488 238 239 238 query82 1679100 95 95 query83 173 137 137 137 query84 256 70 69 69 query85 895 319 323 319 query86 326 292 304 292 query87 3203303230023002 query88 4673232323172317 query89 389 283 296 283 query90 2002205 216 205 query91 160 142 126 126 query92 57 50 51 50 query93 5867549 559 549 query94 721 211 209 209 query95 2085194220341942 query96 648 325 326 325 query97 6578644265736442 query98 219 211 204 204 query99 3016902 973 902 Total cold run time: 325623 ms Total hot run time: 213134 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [optimize](short circuit) avoid set cacheId when non prepared execute [doris]
doris-robot commented on PR #40525: URL: https://github.com/apache/doris/pull/40525#issuecomment-2337029991 TPC-DS: Total hot run time: 192430 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 332273fbb0fc4ef557b73d24828545d17c810c20, data reload: false query1 1272903 864 864 query2 6358195518951895 query3 10732 437140984098 query4 59588 24465 23262 23262 query5 5430497 510 497 query6 419 158 153 153 query7 5758295 292 292 query8 331 220 224 220 query9 8944252024962496 query10 493 287 258 258 query11 18018 15027 15488 15027 query12 164 101 108 101 query13 1576403 408 403 query14 11205 701374447013 query15 255 169 180 169 query16 7587496 519 496 query17 1171575 574 574 query18 1938287 294 287 query19 300 148 146 146 query20 132 110 114 110 query21 212 106 104 104 query22 4612444547064445 query23 34353 33463 33365 33365 query24 5966285027742774 query25 528 376 387 376 query26 681 156 156 156 query27 1793272 275 272 query28 3850202320112011 query29 691 397 416 397 query30 234 155 151 151 query31 933 776 807 776 query32 88 51 53 51 query33 445 290 274 274 query34 862 480 468 468 query35 811 721 724 721 query36 1070930 947 930 query37 147 81 85 81 query38 4043384438723844 query39 1441141813861386 query40 196 114 107 107 query41 47 43 45 43 query42 113 92 93 92 query43 531 473 477 473 query44 1060761 731 731 query45 196 164 165 164 query46 1087716 720 716 query47 1868178417851784 query48 359 284 285 284 query49 764 432 442 432 query50 817 405 417 405 query51 7046686468426842 query52 101 86 85 85 query53 247 175 177 175 query54 561 454 489 454 query55 73 73 74 73 query56 274 260 248 248 query57 1165108810941088 query58 216 229 227 227 query59 3098288829072888 query60 298 270 256 256 query61 101 99 105 99 query62 750 637 671 637 query63 216 182 183 182 query64 2839736 673 673 query65 3207311631933116 query66 683 343 333 333 query67 15408 15278 15309 15278 query68 2971583 561 561 query69 389 274 270 270 query70 1207113911421139 query71 345 273 275 273 query72 4975405039953995 query73 743 320 322 320 query74 9265891087908790 query75 3356271526962696 query76 1462932 967 932 query77 525 320 318 318 query78 9647912690209020 query79 1005532 523 523 query80 677 543 501 501 query81 464 228 230 228 query82 232 140 140 140 query83 186 150 152 150 query84 257 80 84 80 query85 696 292 287 287 query86 292 287 286 286 query87 4362432843044304 query88 2920229222852285 query89 385 292 286 286 query90 1903189 190 189 query91 127 97 100 97 query92 58 47 49 47 query93 1020530 525 525 query94 702 307 313 307 query95 311 248 256 248 query96 582 263 265 263 query97 3203306830783068 query98 272 197 200 197 query99 1501127712841277 Total cold run time: 303792 ms Total hot run time: 192430 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [enhance](mtmv) Prohibit creating materialized views in external cata… [doris]
yiguolei merged PR #40420: URL: https://github.com/apache/doris/pull/40420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [enhance](mtmv) Prohibit creating materialized views in external cata… (#40420)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new fc4c2aed649 [enhance](mtmv) Prohibit creating materialized views in
external cata… (#40420)
fc4c2aed649 is described below
commit fc4c2aed6496162a31e4e2f191153283a80329b9
Author: zhangdong <493738...@qq.com>
AuthorDate: Mon Sep 9 11:16:40 2024 +0800
[enhance](mtmv) Prohibit creating materialized views in external cata…
(#40420)
…logs (#39557)
pick: https://github.com/apache/doris/pull/39557
---
.../nereids/trees/plans/commands/info/CreateMTMVInfo.java | 4
regression-test/suites/mtmv_p0/test_hive_mtmv.groovy | 10 ++
2 files changed, 14 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
index d4dc0b95a17..adf0eee8ebe 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
@@ -38,6 +38,7 @@ import org.apache.doris.common.FeConstants;
import org.apache.doris.common.FeNameFormat;
import org.apache.doris.common.util.DynamicPartitionUtil;
import org.apache.doris.common.util.PropertyAnalyzer;
+import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.mtmv.EnvInfo;
import org.apache.doris.mtmv.MTMVPartitionInfo;
import org.apache.doris.mtmv.MTMVPartitionInfo.MTMVPartitionType;
@@ -151,6 +152,9 @@ public class CreateMTMVInfo {
public void analyze(ConnectContext ctx) throws Exception {
// analyze table name
mvName.analyze(ctx);
+if (!InternalCatalog.INTERNAL_CATALOG_NAME.equals(mvName.getCtl())) {
+throw new AnalysisException("Only support creating asynchronous
materialized views in internal catalog");
+}
try {
FeNameFormat.checkTableName(mvName.getTbl());
} catch (org.apache.doris.common.AnalysisException e) {
diff --git a/regression-test/suites/mtmv_p0/test_hive_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_hive_mtmv.groovy
index 4ac5ad9e890..7b3abc2300f 100644
--- a/regression-test/suites/mtmv_p0/test_hive_mtmv.groovy
+++ b/regression-test/suites/mtmv_p0/test_hive_mtmv.groovy
@@ -38,6 +38,16 @@ suite("test_hive_mtmv",
"p0,external,hive,external_docker,external_docker_hive")
def dbName = "regression_test_mtmv_p0"
sql """drop materialized view if exists ${mvName};"""
+ test {
+sql """CREATE MATERIALIZED VIEW
${catalog_name}.`default`.${mvName}
+ BUILD DEFERRED REFRESH AUTO ON MANUAL
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES ('replication_num' = '1')
+ AS
+ SELECT * FROM
${catalog_name}.`default`.mtmv_base1;"""
+exception "internal"
+}
+
sql """
CREATE MATERIALIZED VIEW ${mvName}
BUILD DEFERRED REFRESH AUTO ON MANUAL
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) [doris]
doris-robot commented on PR #40522: URL: https://github.com/apache/doris/pull/40522#issuecomment-2337032962 ClickBench: Total hot run time: 31.23 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 0867cd57e5a52b20c6e8b49428653beaf18be029, data reload: false query1 0.030.030.02 query2 0.070.030.03 query3 0.250.050.04 query4 1.790.080.08 query5 0.550.530.52 query6 1.250.630.63 query7 0.020.010.01 query8 0.030.030.03 query9 0.520.490.48 query10 0.530.540.54 query11 0.130.090.09 query12 0.110.080.08 query13 0.620.610.62 query14 0.790.780.79 query15 0.800.760.75 query16 0.360.370.37 query17 1.011.021.00 query18 0.190.270.26 query19 1.921.821.86 query20 0.010.010.01 query21 15.45 0.570.57 query22 2.142.581.65 query23 17.47 0.930.92 query24 6.331.251.62 query25 0.360.070.06 query26 0.830.170.15 query27 0.040.030.03 query28 5.980.740.75 query29 12.68 2.322.34 query30 0.630.530.54 query31 2.800.390.37 query32 3.370.500.48 query33 3.113.063.06 query34 15.25 4.834.84 query35 4.874.864.86 query36 1.061.011.02 query37 0.060.050.04 query38 0.040.020.02 query39 0.020.010.02 query40 0.160.140.14 query41 0.070.020.02 query42 0.020.010.02 query43 0.020.010.02 Total cold run time: 103.74 s Total hot run time: 31.23 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)Optimize the speed of obtaining the last update time o… [doris]
yiguolei merged PR #40421: URL: https://github.com/apache/doris/pull/40421 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [enhance](mtmv)Optimize the speed of obtaining the last update time o… (#40421)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 4084ff2c4a1 [enhance](mtmv)Optimize the speed of obtaining the last
update time o… (#40421)
4084ff2c4a1 is described below
commit 4084ff2c4a13459ce5aacafc46ad204f548b5597
Author: zhangdong <493738...@qq.com>
AuthorDate: Mon Sep 9 11:20:40 2024 +0800
[enhance](mtmv)Optimize the speed of obtaining the last update time o…
(#40421)
…f Hive (#40169)
pick: https://github.com/apache/doris/pull/40169
---
.../doris/datasource/hive/HMSExternalTable.java| 79 --
.../doris/datasource/hive/HiveMetaStoreCache.java | 4 ++
2 files changed, 46 insertions(+), 37 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
index 6bb2be58743..4b87f938dac 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
@@ -53,6 +53,7 @@ import com.google.common.collect.BiMap;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
+import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.collections.MapUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.hive.metastore.api.ColumnStatisticsData;
@@ -742,18 +743,17 @@ public class HMSExternalTable extends ExternalTable
implements MTMVRelatedTableI
return res;
}
-private HiveMetaStoreCache.HivePartitionValues getHivePartitionValues() {
-HiveMetaStoreCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
-.getMetaStoreCache((HMSExternalCatalog) getCatalog());
-return cache.getPartitionValues(
-getDbName(), getName(), getPartitionColumnTypes());
-}
-
@Override
public MTMVSnapshotIf getPartitionSnapshot(String partitionName,
MTMVRefreshContext context)
throws AnalysisException {
-long partitionLastModifyTime =
getPartitionLastModifyTime(partitionName);
-return new MTMVTimestampSnapshot(partitionLastModifyTime);
+HiveMetaStoreCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
+.getMetaStoreCache((HMSExternalCatalog) getCatalog());
+HiveMetaStoreCache.HivePartitionValues hivePartitionValues =
cache.getPartitionValues(
+getDbName(), getName(), getPartitionColumnTypes());
+Long partitionId =
getPartitionIdByNameOrAnalysisException(partitionName, hivePartitionValues);
+HivePartition hivePartition =
getHivePartitionByIdOrAnalysisException(partitionId,
+hivePartitionValues, cache);
+return new MTMVTimestampSnapshot(hivePartition.getLastModifiedTime());
}
@Override
@@ -761,45 +761,50 @@ public class HMSExternalTable extends ExternalTable
implements MTMVRelatedTableI
if (getPartitionType() == PartitionType.UNPARTITIONED) {
return new MTMVMaxTimestampSnapshot(getName(), getLastDdlTime());
}
-String partitionName = "";
+Long maxPartitionId = 0L;
long maxVersionTime = 0L;
long visibleVersionTime;
-for (Entry entry :
getAndCopyPartitionItems().entrySet()) {
-visibleVersionTime = getPartitionLastModifyTime(entry.getKey());
+HiveMetaStoreCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
+.getMetaStoreCache((HMSExternalCatalog) getCatalog());
+HiveMetaStoreCache.HivePartitionValues hivePartitionValues =
cache.getPartitionValues(
+getDbName(), getName(), getPartitionColumnTypes());
+BiMap idToName =
hivePartitionValues.getPartitionNameToIdMap().inverse();
+if (MapUtils.isEmpty(idToName)) {
+throw new AnalysisException("partitions is empty for : " +
getName());
+}
+for (Long partitionId : idToName.keySet()) {
+visibleVersionTime =
getHivePartitionByIdOrAnalysisException(partitionId, hivePartitionValues,
+cache).getLastModifiedTime();
if (visibleVersionTime > maxVersionTime) {
maxVersionTime = visibleVersionTime;
-partitionName = entry.getKey();
+maxPartitionId = partitionId;
}
}
-return new MTMVMaxTimestampSnapshot(partitionName, maxVersionTime);
+return new MTMVMaxTimestampSnapshot(idToName.get(maxPartitionId),
maxVersionTime);
}
-private long getPartitionLastModifyTime(String partitionName) throws
AnalysisException {
-return getPartitionByName(partitionName).getLastModifiedTime();
-
Re: [PR] [optimize](short circuit) avoid set cacheId when non prepared execute [doris]
doris-robot commented on PR #40525: URL: https://github.com/apache/doris/pull/40525#issuecomment-2337034277 ClickBench: Total hot run time: 31.14 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 332273fbb0fc4ef557b73d24828545d17c810c20, data reload: false query1 0.040.050.04 query2 0.080.040.04 query3 0.220.040.05 query4 1.670.080.07 query5 0.500.490.49 query6 1.140.740.73 query7 0.010.010.01 query8 0.050.040.05 query9 0.540.500.50 query10 0.540.580.54 query11 0.160.120.12 query12 0.150.120.12 query13 0.600.580.59 query14 1.411.451.43 query15 0.840.820.82 query16 0.370.370.38 query17 1.071.031.04 query18 0.210.210.20 query19 1.841.701.81 query20 0.010.020.01 query21 15.38 0.670.67 query22 4.126.451.74 query23 18.32 1.381.27 query24 2.140.220.22 query25 0.140.080.07 query26 0.260.180.19 query27 0.080.080.07 query28 13.18 1.020.99 query29 12.58 3.323.26 query30 0.240.050.05 query31 2.880.400.38 query32 3.260.480.48 query33 3.002.973.03 query34 17.08 4.444.42 query35 4.544.444.49 query36 0.660.490.46 query37 0.190.160.15 query38 0.160.140.15 query39 0.060.040.04 query40 0.150.130.14 query41 0.090.060.05 query42 0.060.050.04 query43 0.050.040.04 Total cold run time: 110.07 s Total hot run time: 31.14 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auth)fix case should grant after create view (#40108) [doris]
yiguolei merged PR #40488: URL: https://github.com/apache/doris/pull/40488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](auth)fix case should grant after create view (#40108) (#40488)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 2c588e2738b [fix](auth)fix case should grant after create view
(#40108) (#40488)
2c588e2738b is described below
commit 2c588e2738b6a842ca63a4792924d27a31082721
Author: zhangdong <493738...@qq.com>
AuthorDate: Mon Sep 9 11:21:52 2024 +0800
[fix](auth)fix case should grant after create view (#40108) (#40488)
pick: https://github.com/apache/doris/pull/40108
---
.../suites/external_table_p0/tvf/test_s3_tvf_with_resource.groovy | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/regression-test/suites/external_table_p0/tvf/test_s3_tvf_with_resource.groovy
b/regression-test/suites/external_table_p0/tvf/test_s3_tvf_with_resource.groovy
index 8fc03a42312..45796cad882 100644
---
a/regression-test/suites/external_table_p0/tvf/test_s3_tvf_with_resource.groovy
+++
b/regression-test/suites/external_table_p0/tvf/test_s3_tvf_with_resource.groovy
@@ -187,7 +187,6 @@ suite("test_s3_tvf_with_resource", "p0") {
String viewName = "test_s3_tvf_with_resource_view"
try_sql("DROP USER ${user}")
sql """CREATE USER '${user}' IDENTIFIED BY '${pwd}'"""
-sql """grant select_priv on ${db}.${viewName} to ${user}"""
sql "drop view if exists ${viewName}"
sql """
create view ${viewName} as
@@ -198,7 +197,7 @@ suite("test_s3_tvf_with_resource", "p0") {
"resource" = "${resource_name}"
) where k1 > 100 order by k3,k2,k1;
"""
-
+sql """grant select_priv on ${db}.${viewName} to ${user}"""
// not have usage priv, can not select tvf with resource
connect(user=user, password="${pwd}", url=url) {
sql """set enable_fallback_to_original_planner=false;"""
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv) Generate mtmv cache should use ADMIN user, and rewritten plan should not check privilege [doris]
doris-robot commented on PR #40374: URL: https://github.com/apache/doris/pull/40374#issuecomment-2337035589 TPC-H: Total hot run time: 38190 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit e30f2e9b9f0908a1db926b781881104753c353eb, data reload: false -- Round 1 -- q1 17627 450542784278 q2 2012186 196 186 q3 10454 113410791079 q4 10140 674 734 674 q5 7733289928302830 q6 226 140 141 140 q7 969 631 622 622 q8 9363207120552055 q9 7141654865466546 q10 7008215422302154 q11 449 250 243 243 q12 389 224 230 224 q13 17771 310230793079 q14 294 241 244 241 q15 534 513 480 480 q16 520 434 441 434 q17 982 695 703 695 q18 7392679568666795 q19 1393100310691003 q20 697 345 340 340 q21 4216309830903090 q22 1129104010021002 Total cold run time: 108439 ms Total hot run time: 38190 ms - Round 2, with runtime_filter_mode=off - q1 4309431742834283 q2 389 272 281 272 q3 2915267426642664 q4 1950165016731650 q5 5398539754085397 q6 224 130 134 130 q7 2095175517921755 q8 3184335333513351 q9 8487838384098383 q10 3461321432293214 q11 617 508 507 507 q12 777 617 587 587 q13 16678 308731033087 q14 301 276 267 267 q15 511 485 488 485 q16 521 475 502 475 q17 1802150814861486 q18 7714762674617461 q19 1688154715361536 q20 2086184818431843 q21 5392533353465333 q22 1162105610241024 Total cold run time: 71661 ms Total hot run time: 55190 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
