Re: [PR] [improvement](statistics)Support get index row count and table delta rows. [doris]
doris-robot commented on PR #38492: URL: https://github.com/apache/doris/pull/38492#issuecomment-2257617612 TPC-H: Total hot run time: 41331 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 795e48a427820ff0c47c804af20c86b2ab9cd935, data reload: false -- Round 1 -- q1 17687 407240464046 q2 2012214 198 198 q3 10447 123813091238 q4 10160 837 910 837 q5 7661297729472947 q6 218 137 135 135 q7 1022608 602 602 q8 9436188819211888 q9 8483659266046592 q10 8731382938353829 q11 437 249 248 248 q12 437 229 228 228 q13 17756 297429592959 q14 270 242 243 242 q15 522 485 499 485 q16 521 386 384 384 q17 942 936 890 890 q18 8028722871717171 q19 3025114712011147 q20 557 320 335 320 q21 5309471546664666 q22 345 279 293 279 Total cold run time: 114006 ms Total hot run time: 41331 ms - Round 2, with runtime_filter_mode=off - q1 4138399939933993 q2 327 230 219 219 q3 2989301131153011 q4 1987203119761976 q5 5537545554315431 q6 221 131 130 130 q7 2117176718181767 q8 3305339533563356 q9 8717864287508642 q10 3973400239473947 q11 539 448 449 448 q12 749 600 620 600 q13 16357 313531183118 q14 309 288 282 282 q15 543 495 516 495 q16 481 411 414 411 q17 1753170817131708 q18 8110772776907690 q19 3082175817231723 q20 2041184118521841 q21 5711551253625362 q22 501 446 471 446 Total cold run time: 73487 ms Total hot run time: 56596 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip the alignment process of some rowsets in partial update [doris]
doris-robot commented on PR #38487: URL: https://github.com/apache/doris/pull/38487#issuecomment-2257617814 ClickBench: Total hot run time: 29.81 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 6e4f5ddc602c4fc2cb208d9ecbfa47958c9b081f, data reload: false query1 0.040.030.03 query2 0.080.040.04 query3 0.220.050.05 query4 1.680.080.08 query5 0.500.480.47 query6 1.150.710.71 query7 0.020.010.01 query8 0.050.040.04 query9 0.560.500.50 query10 0.550.570.57 query11 0.150.110.11 query12 0.140.120.12 query13 0.610.610.60 query14 0.760.790.80 query15 0.890.870.88 query16 0.360.360.35 query17 1.001.011.01 query18 0.220.220.22 query19 1.901.811.77 query20 0.040.010.01 query21 15.39 0.770.65 query22 3.797.991.07 query23 17.70 1.301.25 query24 2.260.230.22 query25 0.170.090.07 query26 0.320.200.21 query27 0.460.230.22 query28 13.16 1.000.97 query29 12.51 3.283.28 query30 0.270.060.05 query31 3.060.420.41 query32 3.680.490.50 query33 2.952.973.00 query34 15.40 4.284.29 query35 4.324.294.30 query36 0.670.470.52 query37 0.200.160.17 query38 0.160.160.14 query39 0.040.040.03 query40 0.170.130.12 query41 0.090.040.04 query42 0.050.040.05 query43 0.040.040.04 Total cold run time: 107.78 s Total hot run time: 29.81 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](S3) Use allocator for s3 buffer's allocation [doris]
ByteYue commented on PR #38519: URL: https://github.com/apache/doris/pull/38519#issuecomment-2257620922 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [enhance](S3) Use allocator for s3 buffer's allocation [doris]
ByteYue opened a new pull request, #38519: URL: https://github.com/apache/doris/pull/38519 ## Proposed changes Remove the static buffer pool for S3 upload. Use the dynamic allocator to control the memory usage for S3 upload which is more adaptive and easy to use. Remove all the useless config plus. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](S3) Use allocator for s3 buffer's allocation [doris]
doris-robot commented on PR #38519: URL: https://github.com/apache/doris/pull/38519#issuecomment-2257619537 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Support get index row count and table delta rows. [doris]
doris-robot commented on PR #38520: URL: https://github.com/apache/doris/pull/38520#issuecomment-2257625879 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] Support get index row count and table delta rows. [doris]
Jibing-Li opened a new pull request, #38520: URL: https://github.com/apache/doris/pull/38520 backport: https://github.com/apache/doris/pull/38492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](schema scan) Use async scanner for schema scanners [doris]
yiguolei commented on code in PR #38403:
URL: https://github.com/apache/doris/pull/38403#discussion_r1696428144
##
be/src/exec/schema_scanner.h:
##
@@ -101,8 +106,23 @@ class SchemaScanner {
// factory function
static std::unique_ptr create(TSchemaTableType::type type);
TSchemaTableType::type type() const { return _schema_table_type; }
+void set_dependency(std::shared_ptr dep,
+std::shared_ptr fin_dep) {
+_dependency = dep;
+_finish_dependency = fin_dep;
+}
+
+void init_block(vectorized::Block* src_block);
protected:
Review Comment:
schema scanner
FeBasedSchemaScanner
vector _block;
bool has_started = false
dependency dep;
get_block_async() {
thread_pool.submit() {
if (!has_started) {
start();
}
get_block_internal();
queue.push_block();
}
}
get_block(Block* block) {
block = _block;
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (cloud-merge) Fix the coredump because of change_cache_type to origin_type [doris]
doris-robot commented on PR #38518: URL: https://github.com/apache/doris/pull/38518#issuecomment-2257627595 TPC-DS: Total hot run time: 169618 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 95046e10ae7aa315cb2beef2f9db8b6b49d10694, data reload: false query1 908 368 365 365 query2 6469167916381638 query3 6686218 226 218 query4 19049 17341 17303 17303 query5 4322521 531 521 query6 292 188 170 170 query7 4620311 305 305 query8 260 208 212 208 query9 8526234523072307 query10 434 297 308 297 query11 10627 996599599959 query12 142 90 88 88 query13 1630389 396 389 query14 9782777376817681 query15 207 174 166 166 query16 7198470 444 444 query17 968 575 577 575 query18 1896336 277 277 query19 192 146 150 146 query20 93 88 86 86 query21 202 106 98 98 query22 4197410438873887 query23 33901 32919 32955 32919 query24 10353 308330133013 query25 672 382 396 382 query26 1805150 147 147 query27 2953280 284 280 query28 7019195419641954 query29 1303419 409 409 query30 280 146 148 146 query31 945 768 758 758 query32 103 56 56 56 query33 712 310 314 310 query34 899 478 488 478 query35 863 737 721 721 query36 967 862 842 842 query37 293 80 82 80 query38 2880279727642764 query39 879 821 816 816 query40 288 118 113 113 query41 47 45 44 44 query42 116 99 99 99 query43 474 426 416 416 query44 1230747 736 736 query45 205 178 176 176 query46 1078810 788 788 query47 1792167817051678 query48 403 297 293 293 query49 1193432 428 428 query50 892 439 449 439 query51 6735662266136613 query52 107 89 93 89 query53 265 182 188 182 query54 635 452 468 452 query55 78 76 77 76 query56 289 250 263 250 query57 1118105310681053 query58 279 261 266 261 query59 2534239124802391 query60 320 268 276 268 query61 104 131 95 95 query62 906 653 670 653 query63 220 193 197 193 query64 5923194319051905 query65 3147313031133113 query66 1427351 333 333 query67 15240 14825 14790 14790 query68 4340581 630 581 query69 437 312 303 303 query70 1134108510711071 query71 411 285 280 280 query72 7173268125452545 query73 754 333 334 333 query74 6060565756775657 query75 3358275127232723 query76 2497135213941352 query77 466 312 321 312 query78 9468882788548827 query79 2324547 543 543 query80 1124520 507 507 query81 581 230 230 230 query82 1037133 134 133 query83 246 174 172 172 query84 267 81 81 81 query85 1365324 310 310 query86 512 300 299 299 query87 3282315430943094 query88 3105248824922488 query89 383 286 295 286 query90 1842257 215 215 query91 132 103 104 103 query92 65 52 51 51 query93 1983633 639 633 query94 952 300 303 300 query95 386 273 272 272 query96 599 284 288 284 query97 3192302930993029 query98 215 197 193 193 query99 1644134012811281 Total cold run time: 263906 ms Total hot run time: 169618 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [enhance](S3) Use allocator for s3 buffer's allocation [doris]
github-actions[bot] commented on code in PR #38519:
URL: https://github.com/apache/doris/pull/38519#discussion_r1696432938
##
be/src/io/fs/s3_file_write_bufferpool.cpp:
##
@@ -29,13 +30,43 @@
namespace doris {
namespace io {
+
+bvar::Adder s3_file_buffer_allocated("s3_file_buffer_allocated");
+
+template >
+struct Memory : boost::noncopyable, Allocator {
+Memory() = default;
+explicit Memory(size_t size) : _size(size) {
+alloc(size);
+s3_file_buffer_allocated << 1;
+}
+~Memory() {
+dealloc();
+s3_file_buffer_allocated << -1;
+}
+void alloc(size_t size) { _data =
static_cast(Allocator::alloc(size, 0)); }
+void dealloc() {
+if (_data == nullptr) {
+return;
+}
+Allocator::free(_data, _size);
+_data = nullptr;
+}
+size_t _size;
+char* _data;
+};
+
+struct S3FileBuffer::PartData {
+Memory<> _memory;
+PartData() : _memory(config::s3_write_buffer_size) {}
Review Comment:
warning: use '= default' to define a trivial default constructor
[modernize-use-equals-default]
```suggestion
PartData() : _memory(config::s3_write_buffer_size) = default;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)Support get index row count and table delta rows. [doris]
doris-robot commented on PR #38516: URL: https://github.com/apache/doris/pull/38516#issuecomment-2257635459 TPC-DS: Total hot run time: 203860 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit cf9647b946a198c057aa1f15bb7c60386a6209d5, data reload: false query1 951 430 386 386 query2 6528273826742674 query3 6920207 203 203 query4 20026 18297 18201 18201 query5 19722 653563706370 query6 287 231 243 231 query7 4163306 297 297 query8 424 403 416 403 query9 3149272826582658 query10 407 314 308 308 query11 11343 10833 10722 10722 query12 122 76 78 76 query13 5603719 702 702 query14 17315 13282 13777 13282 query15 359 251 260 251 query16 6469289 274 274 query17 16931464875 875 query18 2323426 435 426 query19 214 155 147 147 query20 80 81 80 80 query21 190 101 94 94 query22 5169508249954995 query23 32466 32011 31969 31969 query24 7098662565086508 query25 534 430 423 423 query26 528 168 162 162 query27 1876292 293 292 query28 6131241623782378 query29 2910272228132722 query30 243 172 168 168 query31 945 729 741 729 query32 67 68 59 59 query33 400 260 274 260 query34 845 483 486 483 query35 1140912 924 912 query36 1403117115441171 query37 90 63 62 62 query38 3131295829702958 query39 1378132213201320 query40 211 99 101 99 query41 47 45 46 45 query42 83 83 86 83 query43 742 769 622 622 query44 1115722 733 722 query45 251 238 237 237 query46 1221984 966 966 query47 1860177117161716 query48 1039734 722 722 query49 631 374 360 360 query50 866 630 627 627 query51 4756465846934658 query52 85 97 82 82 query53 453 320 328 320 query54 2661249024662466 query55 86 93 86 86 query56 241 219 220 219 query57 1274108910691069 query58 218 199 187 187 query59 3966409342724093 query60 225 185 222 185 query61 97 98 96 96 query62 837 535 526 526 query63 486 348 340 340 query64 2589155413731373 query65 3652358135343534 query66 797 380 382 380 query67 15300 15774 15083 15083 query68 9026662 663 662 query69 569 370 355 355 query70 1636153714181418 query71 414 314 313 313 query72 6563351034893489 query73 745 323 329 323 query74 6415581058785810 query75 5400378137103710 query76 5450113011771130 query77 940 259 258 258 query78 12575 11668 11561 11561 query79 7927640 636 636 query80 1266407 420 407 query81 499 239 235 235 query82 1584110 100 100 query83 190 131 129 129 query84 261 71 73 71 query85 901 335 347 335 query86 331 299 296 296 query87 3221300530273005 query88 4659232323182318 query89 404 283 290 283 query90 1954222 218 218 query91 195 138 144 138 query92 62 52 54 52 query93 6548569 572 569 query94 714 219 214 214 query95 1107106610541054 query96 655 326 321 321 query97 6516635264256352 query98 185 178 182 178 query99 3063793 972 793 Total cold run time: 312406 ms Total hot run time: 203860 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] (cloud-merge) Fix the coredump because of change_cache_type to origin_type [doris]
doris-robot commented on PR #38518: URL: https://github.com/apache/doris/pull/38518#issuecomment-2257636277 ClickBench: Total hot run time: 30.7 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 95046e10ae7aa315cb2beef2f9db8b6b49d10694, data reload: false query1 0.040.040.03 query2 0.070.030.04 query3 0.230.040.04 query4 1.680.070.08 query5 0.500.480.48 query6 1.130.720.70 query7 0.010.010.01 query8 0.050.040.04 query9 0.580.500.51 query10 0.560.570.56 query11 0.160.110.11 query12 0.140.120.12 query13 0.620.600.61 query14 0.780.790.80 query15 0.880.850.86 query16 0.360.350.35 query17 0.991.020.96 query18 0.250.210.22 query19 1.901.741.79 query20 0.010.010.01 query21 15.40 0.790.68 query22 3.616.322.05 query23 18.03 1.301.25 query24 2.250.220.22 query25 0.190.080.08 query26 0.310.210.22 query27 0.460.230.23 query28 13.18 1.000.97 query29 12.52 3.373.31 query30 0.260.060.06 query31 2.860.400.40 query32 3.250.500.48 query33 3.012.942.95 query34 15.45 4.284.25 query35 4.314.294.30 query36 0.680.480.47 query37 0.190.160.15 query38 0.170.160.15 query39 0.040.040.04 query40 0.160.130.14 query41 0.100.050.05 query42 0.050.050.05 query43 0.040.040.04 Total cold run time: 107.46 s Total hot run time: 30.7 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)Support get index row count and table delta rows. [doris]
doris-robot commented on PR #38492: URL: https://github.com/apache/doris/pull/38492#issuecomment-2257636282 TPC-DS: Total hot run time: 169931 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 795e48a427820ff0c47c804af20c86b2ab9cd935, data reload: false query1 909 375 364 364 query2 6481172116571657 query3 6662209 223 209 query4 20172 17402 17308 17308 query5 3674501 506 501 query6 256 162 170 162 query7 4600293 295 293 query8 254 190 202 190 query9 8520236323402340 query10 428 281 268 268 query11 10421 10052 99859985 query12 123 89 89 89 query13 1647380 370 370 query14 8798792577847784 query15 202 156 163 156 query16 6816428 449 428 query17 946 541 549 541 query18 1899270 274 270 query19 195 146 140 140 query20 86 86 83 83 query21 202 99 96 96 query22 4347402140794021 query23 33580 33657 33491 33491 query24 10023 313731093109 query25 649 427 408 408 query26 1576144 151 144 query27 3006278 279 278 query28 7606199419871987 query29 1174432 442 432 query30 243 159 167 159 query31 952 775 755 755 query32 100 55 58 55 query33 667 322 330 322 query34 920 479 495 479 query35 918 745 745 745 query36 1034894 904 894 query37 169 89 82 82 query38 3044281427662766 query39 864 819 812 812 query40 255 112 111 111 query41 45 43 42 42 query42 123 95 98 95 query43 461 414 415 414 query44 1186714 738 714 query45 213 173 181 173 query46 1094810 771 771 query47 1808168717251687 query48 359 291 284 284 query49 871 423 411 411 query50 909 425 423 423 query51 6771669166636663 query52 96 85 88 85 query53 249 179 180 179 query54 613 442 476 442 query55 78 73 75 73 query56 273 246 253 246 query57 1134103510461035 query58 265 286 276 276 query59 2719238424582384 query60 301 271 266 266 query61 98 104 108 104 query62 888 649 651 649 query63 211 183 182 182 query64 5684191019401910 query65 3190308330473047 query66 1320329 336 329 query67 15156 14732 14814 14732 query68 4314557 561 557 query69 563 328 288 288 query70 1122106110771061 query71 374 283 267 267 query72 7122272425042504 query73 803 333 325 325 query74 5938563755695569 query75 3345271627672716 query76 2160136614071366 query77 412 310 297 297 query78 9437891688598859 query79 1491523 532 523 query80 1100517 508 508 query81 550 217 223 217 query82 1076129 126 126 query83 228 173 164 164 query84 265 78 81 78 query85 1272312 303 303 query86 390 300 300 300 query87 3274309231083092 query88 2896242923922392 query89 388 292 289 289 query90 1673184 187 184 query91 124 100 101 100 query92 55 51 49 49 query93 1478596 596 596 query94 765 294 298 294 query95 376 266 256 256 query96 592 279 277 277 query97 3192300130243001 query98 214 193 198 193 query99 1591128013041280 Total cold run time: 259113 ms Total hot run time: 169931 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [improve](columns)replace logFatal with exception and del some useless code in columns [doris]
amorynan commented on PR #38035: URL: https://github.com/apache/doris/pull/38035#issuecomment-2257637220 run p1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](Backup) Do connectivity check when creating repository [doris]
ByteYue commented on PR #38350: URL: https://github.com/apache/doris/pull/38350#issuecomment-2257637231 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](Backup) Do connectivity check when creating repository [doris]
github-actions[bot] commented on PR #38350: URL: https://github.com/apache/doris/pull/38350#issuecomment-2257643528 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)Support get index row count and table delta rows. [doris]
doris-robot commented on PR #38516: URL: https://github.com/apache/doris/pull/38516#issuecomment-2257644424 ClickBench: Total hot run time: 30.2 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit cf9647b946a198c057aa1f15bb7c60386a6209d5, data reload: false query1 0.020.020.02 query2 0.080.030.02 query3 0.260.050.04 query4 1.790.070.06 query5 0.540.520.52 query6 1.240.620.62 query7 0.020.010.02 query8 0.030.020.02 query9 0.520.470.47 query10 0.540.550.53 query11 0.120.080.09 query12 0.110.090.09 query13 0.610.610.61 query14 0.790.780.79 query15 0.780.750.76 query16 0.390.350.38 query17 1.011.001.04 query18 0.240.250.23 query19 1.911.861.82 query20 0.010.010.01 query21 15.48 0.600.55 query22 2.022.231.47 query23 16.87 1.120.94 query24 7.110.611.09 query25 0.400.080.05 query26 0.740.170.15 query27 0.040.040.05 query28 6.300.770.75 query29 12.67 2.282.32 query30 0.510.520.54 query31 2.800.390.38 query32 3.380.500.49 query33 3.053.133.04 query34 15.26 4.784.79 query35 4.864.864.85 query36 1.051.021.02 query37 0.060.040.04 query38 0.040.020.02 query39 0.020.020.01 query40 0.160.140.14 query41 0.070.020.02 query42 0.020.010.02 query43 0.020.010.01 Total cold run time: 103.94 s Total hot run time: 30.2 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)Support get index row count and table delta rows. [doris]
doris-robot commented on PR #38492: URL: https://github.com/apache/doris/pull/38492#issuecomment-2257645462 ClickBench: Total hot run time: 29.81 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 795e48a427820ff0c47c804af20c86b2ab9cd935, data reload: false query1 0.040.040.03 query2 0.070.040.04 query3 0.220.050.05 query4 1.680.060.06 query5 0.490.480.49 query6 1.140.710.72 query7 0.020.010.02 query8 0.060.040.05 query9 0.560.500.51 query10 0.560.540.55 query11 0.160.120.12 query12 0.140.120.14 query13 0.620.600.60 query14 0.760.810.80 query15 0.890.860.85 query16 0.350.350.36 query17 1.030.990.98 query18 0.210.210.21 query19 1.791.731.75 query20 0.010.010.01 query21 15.39 0.780.67 query22 3.868.431.24 query23 17.82 1.331.28 query24 2.260.230.21 query25 0.170.080.08 query26 0.330.210.20 query27 0.460.230.23 query28 13.17 1.000.97 query29 12.55 3.313.27 query30 0.250.060.05 query31 2.880.410.41 query32 3.220.480.48 query33 2.872.942.90 query34 15.46 4.264.23 query35 4.274.294.30 query36 0.670.470.49 query37 0.190.160.17 query38 0.160.150.15 query39 0.040.030.04 query40 0.160.120.13 query41 0.090.050.05 query42 0.060.050.05 query43 0.050.040.04 Total cold run time: 107.18 s Total hot run time: 29.81 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improment]Add schema table workload_group_privileges [doris]
zddr commented on code in PR #38436:
URL: https://github.com/apache/doris/pull/38436#discussion_r1696425649
##
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Auth.java:
##
@@ -1407,6 +1408,58 @@ private void getUserAuthInfo(List>
userAuthInfos, UserIdentity user
userAuthInfos.add(userAuthInfo);
}
+public static boolean isAdminPriv(GlobalPrivTable userGlobalPrivTable) {
+PrivBitSet globalPrivs = new PrivBitSet();
+for (PrivEntry pe : userGlobalPrivTable.getEntries()) {
+globalPrivs.or(pe.privSet);
+}
+if (globalPrivs.satisfy(PrivPredicate.ADMIN)) {
+return true;
+}
+return false;
+}
+
+public void getUserRoleWorkloadGroupPrivs(List> result,
UserIdentity currentUserIdentity) {
+readLock();
+try {
+boolean isCurrentUserAdmin =
isAdminPriv(getUserGlobalPrivTable(currentUserIdentity));
Review Comment:
Can `checkGlobalPriv (current UserIdentity, PrivPredicte. ADMIN)` be used
instead?
##
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/RoleManager.java:
##
@@ -211,6 +212,31 @@ public void getRoleInfo(List> results) {
}
}
+public void getRoleWorkloadGroupPrivs(List> result,
Set limitedRole) {
+for (Role role : roles.values()) {
+if
(ClusterNamespace.getNameFromFullName(role.getRoleName()).startsWith(DEFAULT_ROLE_PREFIX))
{
+continue;
+}
+
+if (limitedRole != null &&
!limitedRole.contains(role.getRoleName())) {
+continue;
+}
+String isGrantable = Auth.isAdminPriv(role.getGlobalPrivTable()) ?
"YES" : "NO";
Review Comment:
role.checkGlobalPriv(wanted) instaed？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)Support get index row count and table delta rows. [doris]
doris-robot commented on PR #38516: URL: https://github.com/apache/doris/pull/38516#issuecomment-2257649715 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit cf9647b946a198c057aa1f15bb7c60386a6209d5 with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 22.1 seconds inserted 1000 Rows, about 452K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](compatibility) type toSql should return lowercase string (#38012) [doris]
morrySnow opened a new pull request, #38521: URL: https://github.com/apache/doris/pull/38521 pick from master #38012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compatibility) type toSql should return lowercase string (#38012) [doris]
doris-robot commented on PR #38521: URL: https://github.com/apache/doris/pull/38521#issuecomment-2257657990 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compatibility) type toSql should return lowercase string (#38012) [doris]
morrySnow commented on PR #38521: URL: https://github.com/apache/doris/pull/38521#issuecomment-2257658104 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](catalog)fix db name may be null in NotificationEvent [doris]
zddr commented on PR #38421: URL: https://github.com/apache/doris/pull/38421#issuecomment-2257658749 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](Backup) Do connectivity check when creating repository [doris]
doris-robot commented on PR #38350: URL: https://github.com/apache/doris/pull/38350#issuecomment-2257666702 TPC-H: Total hot run time: 41740 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit d37d2c48977b9c5d26a4faf58c5cc111268dd2af, data reload: false -- Round 1 -- q1 17855 421541914191 q2 3061218 221 218 q3 11264 142713801380 q4 10584 801 944 801 q5 8754292029982920 q6 225 142 140 140 q7 1037613 625 613 q8 9441179519611795 q9 8559661866626618 q10 8735382738523827 q11 426 259 261 259 q12 410 231 223 223 q13 17764 297329582958 q14 273 243 249 243 q15 541 491 496 491 q16 491 422 398 398 q17 964 943 898 898 q18 8010732271877187 q19 1398124812321232 q20 571 347 324 324 q21 5214473747794737 q22 354 287 289 287 Total cold run time: 115931 ms Total hot run time: 41740 ms - Round 2, with runtime_filter_mode=off - q1 4087407240634063 q2 332 219 220 219 q3 3011299230112992 q4 1878189718701870 q5 5260525352455245 q6 218 136 137 136 q7 2096169316751675 q8 3248335133233323 q9 8338832883088308 q10 3778384738243824 q11 546 470 448 448 q12 718 542 551 542 q13 14619 300429692969 q14 285 257 258 257 q15 521 482 477 477 q16 442 394 403 394 q17 1726169317101693 q18 7753729371767176 q19 1693169416981694 q20 1989178417741774 q21 5472520553975205 q22 518 463 468 463 Total cold run time: 68528 ms Total hot run time: 54747 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](Backup) Do connectivity check when creating repository [doris]
hello-stephen commented on PR #38350: URL: https://github.com/apache/doris/pull/38350#issuecomment-2257668974 run compile -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](S3) Use allocator for s3 buffer's allocation [doris]
ByteYue commented on PR #38519: URL: https://github.com/apache/doris/pull/38519#issuecomment-2257669367 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cast) differences between FE and BE in parsing strings to integers. [doris]
Mryange commented on PR #38511: URL: https://github.com/apache/doris/pull/38511#issuecomment-2257679459 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](schema scan) Use async scanner for schema scanners [doris]
github-actions[bot] commented on code in PR #38403:
URL: https://github.com/apache/doris/pull/38403#discussion_r1696478776
##
be/src/exec/schema_scanner.cpp:
##
@@ -85,7 +89,54 @@ Status SchemaScanner::start(RuntimeState* state) {
return Status::OK();
}
-Status SchemaScanner::get_next_block(vectorized::Block* block, bool* eos) {
+Status SchemaScanner::get_next_block(RuntimeState* state, vectorized::Block*
block, bool* eos) {
Review Comment:
warning: pointer parameter 'eos' can be pointer to const
[readability-non-const-parameter]
be/src/exec/schema_scanner.h:101:
```diff
- Status get_next_block(RuntimeState* state, vectorized::Block* block,
bool* eos);
+ Status get_next_block(RuntimeState* state, vectorized::Block* block,
const bool* eos);
```
```suggestion
Status SchemaScanner::get_next_block(RuntimeState* state, vectorized::Block*
block, const bool* eos) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](Backup) Do connectivity check when creating repository [doris]
doris-robot commented on PR #38350: URL: https://github.com/apache/doris/pull/38350#issuecomment-2257685799 TPC-DS: Total hot run time: 168749 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit d37d2c48977b9c5d26a4faf58c5cc111268dd2af, data reload: false query1 906 382 368 368 query2 6473174516751675 query3 6692214 236 214 query4 20081 17585 17269 17269 query5 4336509 513 509 query6 286 170 157 157 query7 4614314 300 300 query8 270 184 191 184 query9 8481236323672363 query10 436 275 269 269 query11 10377 10006 10006 10006 query12 133 93 90 90 query13 1638385 394 385 query14 9630749770947094 query15 202 163 162 162 query16 7082432 464 432 query17 950 547 543 543 query18 1927280 277 277 query19 195 145 145 145 query20 91 88 88 88 query21 207 104 95 95 query22 4020393939743939 query23 33887 32841 33080 32841 query24 10320 314030313031 query25 709 387 387 387 query26 1840154 149 149 query27 3019267 272 267 query28 7039194419401940 query29 1349417 435 417 query30 282 149 152 149 query31 917 751 766 751 query32 98 54 58 54 query33 710 308 322 308 query34 908 480 481 480 query35 877 732 739 732 query36 1028895 885 885 query37 299 79 88 79 query38 2888278027682768 query39 873 827 811 811 query40 279 120 112 112 query41 47 46 48 46 query42 123 98 103 98 query43 486 416 447 416 query44 1196732 732 732 query45 204 174 175 174 query46 1099800 772 772 query47 1767168616941686 query48 363 285 292 285 query49 1200451 422 422 query50 910 441 445 441 query51 6829673867356735 query52 109 89 91 89 query53 254 185 184 184 query54 643 457 452 452 query55 77 79 80 79 query56 289 284 257 257 query57 1122103810051005 query58 290 273 273 273 query59 2589261223412341 query60 286 274 273 273 query61 101 103 99 99 query62 918 682 661 661 query63 209 194 186 186 query64 5907195318991899 query65 3175312231193119 query66 1447337 341 337 query67 15228 14595 15193 14595 query68 4379556 573 556 query69 435 310 304 304 query70 107210941094 query71 384 277 280 277 query72 7215267724862486 query73 778 334 330 330 query74 6087570457175704 query75 3399281027582758 query76 2245137914321379 query77 467 317 304 304 query78 9371894188668866 query79 1400556 543 543 query80 1037507 513 507 query81 552 221 230 221 query82 1069133 127 127 query83 241 174 170 170 query84 271 85 81 81 query85 1278330 304 304 query86 396 288 296 288 query87 3261308731233087 query88 2970240223872387 query89 388 292 286 286 query90 1809202 196 196 query91 128 100 103 100 query92 64 117 50 50 query93 1393613 598 598 query94 887 314 292 292 query95 390 270 267 267 query96 603 280 280 280 query97 3187301630143014 query98 207 197 197 197 query99 1641131513171315 Total cold run time: 262221 ms Total hot run time: 168749 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [Improvement](schema scan) Use async scanner for schema scanners [doris]
Gabriel39 commented on PR #38403: URL: https://github.com/apache/doris/pull/38403#issuecomment-2257686115 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] (compaction) fix time series compaction policy [doris]
github-actions[bot] commented on PR #38220: URL: https://github.com/apache/doris/pull/38220#issuecomment-2257687844 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](FE) add rest api to get last profile #38268 [doris]
englefly commented on PR #38450: URL: https://github.com/apache/doris/pull/38450#issuecomment-2257690752 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Optimize `ClearCacheAction`implementation [doris]
xinyiZzz commented on PR #38438: URL: https://github.com/apache/doris/pull/38438#issuecomment-2257692061 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) optimize cost model to make join order good for runtime filter when stats is not available [doris]
englefly commented on PR #38444: URL: https://github.com/apache/doris/pull/38444#issuecomment-2257693588 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feat](nereids)Support date_trunc function in partition prune [doris]
feiniaofeiafei commented on PR #38025: URL: https://github.com/apache/doris/pull/38025#issuecomment-2257693014 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improment]publish workload to BE by tag [doris]
github-actions[bot] commented on PR #38486: URL: https://github.com/apache/doris/pull/38486#issuecomment-2257694474 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](Backup) Do connectivity check when creating repository [doris]
doris-robot commented on PR #38350: URL: https://github.com/apache/doris/pull/38350#issuecomment-2257695175 ClickBench: Total hot run time: 29.8 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit d37d2c48977b9c5d26a4faf58c5cc111268dd2af, data reload: false query1 0.040.030.03 query2 0.080.040.03 query3 0.220.050.05 query4 1.690.060.08 query5 0.490.470.49 query6 1.120.720.72 query7 0.020.010.02 query8 0.060.040.04 query9 0.560.510.51 query10 0.560.580.57 query11 0.160.120.11 query12 0.160.130.12 query13 0.620.610.60 query14 0.770.780.79 query15 0.910.860.86 query16 0.350.340.34 query17 1.000.970.99 query18 0.220.210.24 query19 1.841.681.70 query20 0.010.000.00 query21 15.39 0.760.65 query22 4.148.121.18 query23 18.07 1.451.34 query24 2.290.220.22 query25 0.180.080.08 query26 0.330.210.21 query27 0.470.230.23 query28 13.15 1.000.97 query29 12.52 3.293.30 query30 0.260.050.05 query31 2.880.410.40 query32 3.260.490.49 query33 2.942.942.93 query34 15.42 4.224.26 query35 4.314.304.29 query36 0.680.480.47 query37 0.200.160.16 query38 0.170.150.15 query39 0.040.040.03 query40 0.160.120.14 query41 0.100.050.05 query42 0.060.060.06 query43 0.050.050.05 Total cold run time: 107.95 s Total hot run time: 29.8 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch 2.1_38126 created (now 0d314af2d77)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch 2.1_38126 in repository https://gitbox.apache.org/repos/asf/doris.git at 0d314af2d77 [opt](compatibility) fill schema and table name in protocol column def (#38126) This branch includes the following new commits: new 0d314af2d77 [opt](compatibility) fill schema and table name in protocol column def (#38126) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/01: [opt](compatibility) fill schema and table name in protocol column def (#38126)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch 2.1_38126
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0d314af2d77f2b1c599e16e1d5055d93c90e5d20
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Tue Jul 23 14:26:16 2024 +0800
[opt](compatibility) fill schema and table name in protocol column def
(#38126)
pick from master #38126
according to mysql text protocol Protocol::ColumnDefinition41, should
fill schema name, table name into it if column from table.
---
.../java/org/apache/doris/analysis/Queriable.java | 6 +++
.../doris/common/NereidsSqlCacheManager.java | 4 +-
.../java/org/apache/doris/mysql/FieldInfo.java | 58 +
.../org/apache/doris/mysql/MysqlSerializer.java| 29 +++
.../org/apache/doris/nereids/NereidsPlanner.java | 37 ++---
.../org/apache/doris/nereids/SqlCacheContext.java | 10
.../doris/nereids/glue/LogicalPlanAdapter.java | 11
.../trees/plans/logical/LogicalSqlCache.java | 9 +++-
.../trees/plans/physical/PhysicalSqlCache.java | 9 +++-
.../java/org/apache/doris/qe/ConnectProcessor.java | 1 +
.../java/org/apache/doris/qe/StmtExecutor.java | 54 ++-
.../suites/mysql_compatibility_p0/metadata.groovy | 60 ++
12 files changed, 266 insertions(+), 22 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/Queriable.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/Queriable.java
index 035ea70f882..7303190f4fe 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/Queriable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/Queriable.java
@@ -17,6 +17,8 @@
package org.apache.doris.analysis;
+import org.apache.doris.mysql.FieldInfo;
+
import java.util.ArrayList;
import java.util.List;
@@ -36,5 +38,9 @@ public interface Queriable {
ArrayList getColLabels();
+default List getFieldInfos() {
+return null;
+}
+
String toDigest();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/NereidsSqlCacheManager.java
b/fe/fe-core/src/main/java/org/apache/doris/common/NereidsSqlCacheManager.java
index 6c4d5901709..e6a72d069b4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/NereidsSqlCacheManager.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/NereidsSqlCacheManager.java
@@ -240,7 +240,7 @@ public class NereidsSqlCacheManager {
String cachedPlan = sqlCacheContext.getPhysicalPlan();
LogicalSqlCache logicalSqlCache = new LogicalSqlCache(
-sqlCacheContext.getQueryId(),
sqlCacheContext.getColLabels(),
+sqlCacheContext.getQueryId(),
sqlCacheContext.getColLabels(), sqlCacheContext.getFieldInfos(),
sqlCacheContext.getResultExprs(), resultSetInFe,
ImmutableList.of(),
"none", cachedPlan
);
@@ -265,7 +265,7 @@ public class NereidsSqlCacheManager {
MetricRepo.COUNTER_CACHE_HIT_SQL.increase(1L);
LogicalSqlCache logicalSqlCache = new LogicalSqlCache(
-sqlCacheContext.getQueryId(),
sqlCacheContext.getColLabels(),
+sqlCacheContext.getQueryId(),
sqlCacheContext.getColLabels(), sqlCacheContext.getFieldInfos(),
sqlCacheContext.getResultExprs(), Optional.empty(),
cacheValues, backendAddress, cachedPlan
);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/FieldInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/FieldInfo.java
new file mode 100644
index 000..2ebf64fc127
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/FieldInfo.java
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mysql;
+
+/**
+ * according to mysql text protocol ColumnDefinition41. Field should be filled
by these attribute.
+ */
+public class FieldInfo {
+
+private final String schema;
+private final String table;
+priva
Re: [PR] [fix](simdjson) fix simdjson reader will parse given root twice when data is object array [doris]
github-actions[bot] commented on PR #38490: URL: https://github.com/apache/doris/pull/38490#issuecomment-2257702230 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](simdjson) fix simdjson reader will parse given root twice when data is object array [doris]
github-actions[bot] commented on PR #38490: URL: https://github.com/apache/doris/pull/38490#issuecomment-2257702331 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](compatibility) fill schema and table name in protocol column def (#38126) [doris]
morrySnow opened a new pull request, #38522: URL: https://github.com/apache/doris/pull/38522 pick from master #38126 according to mysql text protocol Protocol::ColumnDefinition41, should fill schema name, table name into it if column from table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](simdjson) fix simdjson reader for read json object array when jsonroot set [doris]
xiaokang merged PR #38500: URL: https://github.com/apache/doris/pull/38500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](simdjson) fix simdjson reader for read json object array when jsonroot set #38490 (#38500)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 973f2878ba4 [fix](simdjson) fix simdjson reader for read json object
array when jsonroot set #38490 (#38500)
973f2878ba4 is described below
commit 973f2878ba4ae53dedcbcc2e4dee2852581af72c
Author: amory
AuthorDate: Tue Jul 30 15:51:15 2024 +0800
[fix](simdjson) fix simdjson reader for read json object array when
jsonroot set #38490 (#38500)
---
be/src/vec/exec/format/json/new_json_reader.cpp| 3 +-
be/src/vec/exec/format/json/new_json_reader.h | 1 +
.../load_p0/stream_load/load_object_array_json.out | 4 +
.../load_p0/stream_load/test_json_object_array.csv | 1 +
.../stream_load/load_object_array_json.groovy | 100 +
5 files changed, 108 insertions(+), 1 deletion(-)
diff --git a/be/src/vec/exec/format/json/new_json_reader.cpp
b/be/src/vec/exec/format/json/new_json_reader.cpp
index 0ef75a8ebfe..e3ed69b150e 100644
--- a/be/src/vec/exec/format/json/new_json_reader.cpp
+++ b/be/src/vec/exec/format/json/new_json_reader.cpp
@@ -1251,7 +1251,7 @@ Status
NewJsonReader::_simdjson_handle_flat_array_complex_json(
bool valid = true;
cur = (*_array_iter).get_object();
// extract root
-if (_parsed_json_root.size() != 0) {
+if (!_parsed_from_json_root && _parsed_json_root.size() != 0) {
simdjson::ondemand::value val;
Status st = JsonFunctions::extract_from_object(cur,
_parsed_json_root, &val);
if (UNLIKELY(!st.ok())) {
@@ -1668,6 +1668,7 @@ Status NewJsonReader::_simdjson_parse_json_doc(size_t*
size, bool* eof) {
fmt::format_to(error_msg, "{}", st.to_string());
return return_quality_error(error_msg,
std::string((char*)json_str, *size));
}
+_parsed_from_json_root = true;
} catch (simdjson::simdjson_error& e) {
fmt::memory_buffer error_msg;
fmt::format_to(error_msg, "Encounter error while
extract_from_object, error: {}",
diff --git a/be/src/vec/exec/format/json/new_json_reader.h
b/be/src/vec/exec/format/json/new_json_reader.h
index 9d4696a6422..6ad33fb4ffb 100644
--- a/be/src/vec/exec/format/json/new_json_reader.h
+++ b/be/src/vec/exec/format/json/new_json_reader.h
@@ -223,6 +223,7 @@ private:
std::vector> _parsed_jsonpaths;
std::vector _parsed_json_root;
+bool _parsed_from_json_root = false; // to avoid parsing json root
multiple times
char _value_buffer[4 * 1024 * 1024]; // 4MB
char _parse_buffer[512 * 1024]; // 512KB
diff --git
a/regression-test/data/load_p0/stream_load/load_object_array_json.out
b/regression-test/data/load_p0/stream_load/load_object_array_json.out
new file mode 100644
index 000..53598f37e7d
--- /dev/null
+++ b/regression-test/data/load_p0/stream_load/load_object_array_json.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select --
+10210213387802624011021021338700570624 0 239.
87621955C31942 0 0. 128.1
2024-07-19T11:34:17 239.0 0.
+
diff --git
a/regression-test/data/load_p0/stream_load/test_json_object_array.csv
b/regression-test/data/load_p0/stream_load/test_json_object_array.csv
new file mode 100644
index 000..63fc1b5d460
--- /dev/null
+++ b/regression-test/data/load_p0/stream_load/test_json_object_array.csv
@@ -0,0 +1 @@
+{"data":[{"id":"1021021338780262401","type":"INSERT","owner_id":"0","amount_tag":"239.0","barcode":"87621955C31942","retail_order_bill_id":"1021021338700570624","status":"0","amount_retail":"0.0","amount":"128.0","qty":"1","timestamp":"2024-07-19
11:34:17","price_cost":"239.0","is_gift":"0","amount_discount":"0.0"}],"type":"INSERT"}
diff --git
a/regression-test/suites/load_p0/stream_load/load_object_array_json.groovy
b/regression-test/suites/load_p0/stream_load/load_object_array_json.groovy
new file mode 100644
index 000..aeb7f9297cc
--- /dev/null
+++ b/regression-test/suites/load_p0/stream_load/load_object_array_json.groovy
@@ -0,0 +1,100 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed
Re: [PR] [opt](compatibility) fill schema and table name in protocol column def (#38126) [doris]
morrySnow commented on PR #38522: URL: https://github.com/apache/doris/pull/38522#issuecomment-2257704689 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Optimize `ClearCacheAction`implementation [doris]
github-actions[bot] commented on PR #38438: URL: https://github.com/apache/doris/pull/38438#issuecomment-2257704729 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](compatibility) fill schema and table name in protocol column def (#38126) [doris]
doris-robot commented on PR #38522: URL: https://github.com/apache/doris/pull/38522#issuecomment-2257704665 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](simdjson) fix simdjson reader will parse given root twice when data is object array (#38490)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 15eac463d18 [fix](simdjson) fix simdjson reader will parse given root
twice when data is object array (#38490)
15eac463d18 is described below
commit 15eac463d18d3bab869bfc108a2d69a28a97adc2
Author: amory
AuthorDate: Tue Jul 30 15:53:41 2024 +0800
[fix](simdjson) fix simdjson reader will parse given root twice when data
is object array (#38490)
---
be/src/vec/exec/format/json/new_json_reader.cpp| 3 +-
be/src/vec/exec/format/json/new_json_reader.h | 1 +
.../load_p0/stream_load/load_object_array_json.out | 4 +
.../load_p0/stream_load/test_json_object_array.csv | 1 +
.../stream_load/load_object_array_json.groovy | 100 +
5 files changed, 108 insertions(+), 1 deletion(-)
diff --git a/be/src/vec/exec/format/json/new_json_reader.cpp
b/be/src/vec/exec/format/json/new_json_reader.cpp
index f0ea0f4c7f0..2aff2cb4e7e 100644
--- a/be/src/vec/exec/format/json/new_json_reader.cpp
+++ b/be/src/vec/exec/format/json/new_json_reader.cpp
@@ -1215,7 +1215,7 @@ Status
NewJsonReader::_simdjson_handle_flat_array_complex_json_write_columns(
while (true) {
cur = (*_array_iter).get_object();
// extract root
-if (!_parsed_json_root.empty()) {
+if (!_parsed_from_json_root && !_parsed_json_root.empty()) {
simdjson::ondemand::value val;
Status st = JsonFunctions::extract_from_object(cur,
_parsed_json_root, &val);
if (UNLIKELY(!st.ok())) {
@@ -1611,6 +1611,7 @@ Status NewJsonReader::_get_json_value(size_t* size, bool*
eof, simdjson::error_c
fmt::format_to(error_msg, "{}", st.to_string());
return return_quality_error(error_msg,
std::string((char*)_json_str, *size));
}
+_parsed_from_json_root = true;
} catch (simdjson::simdjson_error& e) {
fmt::memory_buffer error_msg;
fmt::format_to(error_msg, "Encounter error while
extract_from_object, error: {}",
diff --git a/be/src/vec/exec/format/json/new_json_reader.h
b/be/src/vec/exec/format/json/new_json_reader.h
index f044e06e62e..0df3747b8c2 100644
--- a/be/src/vec/exec/format/json/new_json_reader.h
+++ b/be/src/vec/exec/format/json/new_json_reader.h
@@ -233,6 +233,7 @@ private:
std::vector> _parsed_jsonpaths;
std::vector _parsed_json_root;
+bool _parsed_from_json_root = false; // to avoid parsing json root
multiple times
char _value_buffer[4 * 1024 * 1024]; // 4MB
char _parse_buffer[512 * 1024]; // 512KB
diff --git
a/regression-test/data/load_p0/stream_load/load_object_array_json.out
b/regression-test/data/load_p0/stream_load/load_object_array_json.out
new file mode 100644
index 000..53598f37e7d
--- /dev/null
+++ b/regression-test/data/load_p0/stream_load/load_object_array_json.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select --
+10210213387802624011021021338700570624 0 239.
87621955C31942 0 0. 128.1
2024-07-19T11:34:17 239.0 0.
+
diff --git
a/regression-test/data/load_p0/stream_load/test_json_object_array.csv
b/regression-test/data/load_p0/stream_load/test_json_object_array.csv
new file mode 100644
index 000..63fc1b5d460
--- /dev/null
+++ b/regression-test/data/load_p0/stream_load/test_json_object_array.csv
@@ -0,0 +1 @@
+{"data":[{"id":"1021021338780262401","type":"INSERT","owner_id":"0","amount_tag":"239.0","barcode":"87621955C31942","retail_order_bill_id":"1021021338700570624","status":"0","amount_retail":"0.0","amount":"128.0","qty":"1","timestamp":"2024-07-19
11:34:17","price_cost":"239.0","is_gift":"0","amount_discount":"0.0"}],"type":"INSERT"}
diff --git
a/regression-test/suites/load_p0/stream_load/load_object_array_json.groovy
b/regression-test/suites/load_p0/stream_load/load_object_array_json.groovy
new file mode 100644
index 000..aeb7f9297cc
--- /dev/null
+++ b/regression-test/suites/load_p0/stream_load/load_object_array_json.groovy
@@ -0,0 +1,100 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distr
[PR] [fix](routine load) fix show routine load task result incorrect [doris]
sollhui opened a new pull request, #38523: URL: https://github.com/apache/doris/pull/38523 ### Bug report: Create a job: ``` CREATE ROUTINE LOAD testShow ON test_show_routine_load COLUMNS TERMINATED BY "," PROPERTIES ( "max_batch_interval" = "5", "max_batch_rows" = "30", "max_batch_size" = "209715200" ) FROM KAFKA ( "kafka_broker_list" = "127.0.0.1:19092", "kafka_topic" = "test_show_routine_load", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` show routine load task: ``` SHOW ROUTINE LOAD TASK WHERE JobName = "testShow"; ``` result: ``` ERROR 1105 (HY000): errCode = 2, detailMessage = The job named testShowdoes not exists or job state is stopped or cancelled ``` ### Solution Do not use `toLowerCase` method; -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](routine load) fix show routine load task result incorrect [doris]
doris-robot commented on PR #38523: URL: https://github.com/apache/doris/pull/38523#issuecomment-2257706740 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](routine load) fix show routine load task result incorrect [doris]
sollhui commented on PR #38523: URL: https://github.com/apache/doris/pull/38523#issuecomment-2257706947 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](simdjson) fix simdjson reader will parse given root twice when data is object array [doris]
xiaokang merged PR #38490: URL: https://github.com/apache/doris/pull/38490 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improment]publish workload to BE by tag [doris]
github-actions[bot] commented on PR #38486: URL: https://github.com/apache/doris/pull/38486#issuecomment-2257709776 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](paimon)add support for 'in' and 'not in' [doris]
morningman merged PR #38390: URL: https://github.com/apache/doris/pull/38390 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compatibility) type toSql should return lowercase string (#38012) [doris]
doris-robot commented on PR #38521: URL: https://github.com/apache/doris/pull/38521#issuecomment-2257713042 TPC-H: Total hot run time: 49733 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ddd1cbf228a7f88742aebe873474cdf3be563ea8, data reload: false -- Round 1 -- q1 17568 438343364336 q2 2092153 145 145 q3 10262 191219411912 q4 10320 127913351279 q5 8400381339133813 q6 258 124 124 124 q7 2027156916271569 q8 9270271626922692 q9 10541 10306 10218 10218 q10 8626352435173517 q11 411 249 238 238 q12 464 313 308 308 q13 18333 398240333982 q14 358 331 329 329 q15 510 447 460 447 q16 681 576 568 568 q17 1117970 955 955 q18 7384683868606838 q19 1796164916101610 q20 517 297 318 297 q21 4475412741114111 q22 531 450 445 445 Total cold run time: 115941 ms Total hot run time: 49733 ms - Round 2, with runtime_filter_mode=off - q1 4329429743224297 q2 314 228 219 219 q3 4208415941594159 q4 2776275527702755 q5 7095708570497049 q6 236 121 118 118 q7 3225283928182818 q8 4386446844804468 q9 16831 16675 16726 16675 q10 4247427942274227 q11 775 705 731 705 q12 1020840 881 840 q13 6763376237333733 q14 442 423 420 420 q15 492 458 455 455 q16 749 682 692 682 q17 3846392838233823 q18 8761878887988788 q19 1716171316511651 q20 2371211921092109 q21 8479852985378529 q22 1049975 972 972 Total cold run time: 84110 ms Total hot run time: 79492 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](routine load) fix show routine load task result incorrect [doris]
sollhui commented on PR #38523: URL: https://github.com/apache/doris/pull/38523#issuecomment-2257713073 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [bugfix](paimon)add support for 'in' and 'not in' (#38390)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 65c24f114c6 [bugfix](paimon)add support for 'in' and 'not in' (#38390)
65c24f114c6 is described below
commit 65c24f114c696edc0d700f9b6fdc55a466363591
Author: wuwenchi
AuthorDate: Tue Jul 30 15:57:07 2024 +0800
[bugfix](paimon)add support for 'in' and 'not in' (#38390)
## Proposed changes
add support for `in` and `not in`:
```
select * from tb where partition_column in ('a','b','c');
select * from tb where partition_column not in ('a','b','c');
```
---
.../paimon/source/PaimonPredicateConverter.java| 39 +++
.../datasource/paimon/source/PaimonScanNode.java | 23 +++-
.../paimon/test_paimon_predict.groovy | 127 +
3 files changed, 183 insertions(+), 6 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java
index 605bc1b321a..9e46474898d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java
@@ -21,9 +21,11 @@ import org.apache.doris.analysis.CastExpr;
import org.apache.doris.analysis.CompoundPredicate;
import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.FunctionCallExpr;
+import org.apache.doris.analysis.InPredicate;
import org.apache.doris.analysis.IsNullPredicate;
import org.apache.doris.analysis.LiteralExpr;
import org.apache.doris.analysis.SlotRef;
+import org.apache.doris.analysis.Subquery;
import org.apache.doris.thrift.TExprOpcode;
import org.apache.paimon.data.BinaryString;
@@ -85,11 +87,48 @@ public class PaimonPredicateConverter {
default:
return null;
}
+} else if (dorisExpr instanceof InPredicate) {
+return doInPredicate((InPredicate) dorisExpr);
} else {
return binaryExprDesc(dorisExpr);
}
}
+private Predicate doInPredicate(InPredicate predicate) {
+// InPredicate, only support a in (1,2,3)
+if (predicate.contains(Subquery.class)) {
+return null;
+}
+
+SlotRef slotRef = convertDorisExprToSlotRef(predicate.getChild(0));
+if (slotRef == null) {
+return null;
+}
+String colName = slotRef.getColumnName();
+int idx = fieldNames.indexOf(colName);
+DataType dataType = paimonFieldTypes.get(idx);
+List valueList = new ArrayList<>();
+for (int i = 1; i < predicate.getChildren().size(); i++) {
+if (!(predicate.getChild(i) instanceof LiteralExpr)) {
+return null;
+}
+LiteralExpr literalExpr =
convertDorisExprToLiteralExpr(predicate.getChild(i));
+Object value = dataType.accept(new
PaimonValueConverter(literalExpr));
+if (value == null) {
+return null;
+}
+valueList.add(value);
+}
+
+if (predicate.isNotIn()) {
+// not in
+return builder.notIn(idx, valueList);
+} else {
+// in
+return builder.in(idx, valueList);
+}
+}
+
private Predicate binaryExprDesc(Expr dorisExpr) {
TExprOpcode opcode = dorisExpr.getOpcode();
// Make sure the col slot is always first
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
index aeecbd7eba2..45516fd2841 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
@@ -317,15 +317,26 @@ public class PaimonScanNode extends FileQueryScanNode {
@Override
public String getNodeExplainString(String prefix, TExplainLevel
detailLevel) {
-String result = super.getNodeExplainString(prefix, detailLevel)
-+ String.format("%spaimonNativeReadSplits=%d/%d\n",
-prefix, rawFileSplitNum, (paimonSplitNum +
rawFileSplitNum));
+StringBuilder sb = new
StringBuilder(super.getNodeExplainString(prefix, detailLevel));
+sb.append(String.format("%spaimonNativeReadSplits=%d/%d\n",
+prefix, rawFileSplitNum, (paimonSplitNum + rawFileSplitNum)));
+
+sb.append(prefix).append("predicatesFromPaimon:");
+if (predicates.isEmpty()) {
+sb.append(" NONE\n");
+} else {
+s
[PR] [chore](bdbtool) Skip read checksum and log escaped value [doris]
w41ter opened a new pull request, #38524: URL: https://github.com/apache/doris/pull/38524 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](bdbtool) Skip read checksum and log escaped value [doris]
doris-robot commented on PR #38524: URL: https://github.com/apache/doris/pull/38524#issuecomment-2257715066 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](bdbtool) Skip read checksum and log escaped value [doris]
w41ter commented on PR #38524: URL: https://github.com/apache/doris/pull/38524#issuecomment-2257715863 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improment]publish workload to BE by tag [doris]
github-actions[bot] commented on PR #38486: URL: https://github.com/apache/doris/pull/38486#issuecomment-2257719727 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] (improvement)(jdbc catalog) add some jdbc catalog faq [doris-website]
zy-kkk opened a new pull request, #928: URL: https://github.com/apache/doris-website/pull/928 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)hive cache add partitionId to partitionName Map [doris]
doris-robot commented on PR #38525: URL: https://github.com/apache/doris/pull/38525#issuecomment-2257732293 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)hive cache add partitionId to partitionName Map [doris]
zddr commented on PR #38525: URL: https://github.com/apache/doris/pull/38525#issuecomment-2257732486 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](nereids) fix insert stmt throw [doris]
924060929 opened a new pull request, #38526: URL: https://github.com/apache/doris/pull/38526 ## Proposed changes fix insert ... with ... select ..., which some cte does not use ``` errCode = 2, detailMessage = class org.apache.doris.planner.MultiCastDataSink cannot be cast to class org.apache.doris.planner.DataStreamSink (org.apache.doris.planner.MultiCastDataSink and org.apache.doris.planner.DataStreamSink are in unnamed module of loader 'app') ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix insert stmt throw [doris]
doris-robot commented on PR #38526: URL: https://github.com/apache/doris/pull/38526#issuecomment-2257734415 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compatibility) type toSql should return lowercase string (#38012) [doris]
doris-robot commented on PR #38521: URL: https://github.com/apache/doris/pull/38521#issuecomment-2257738862 TPC-DS: Total hot run time: 204494 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ddd1cbf228a7f88742aebe873474cdf3be563ea8, data reload: false query1 990 438 386 386 query2 6529276926782678 query3 6914203 199 199 query4 21554 18136 18211 18136 query5 19723 652265676522 query6 293 216 232 216 query7 4154304 308 304 query8 454 407 492 407 query9 3167271626592659 query10 413 313 279 279 query11 11374 10934 10770 10770 query12 125 79 76 76 query13 5608729 715 715 query14 18920 13727 13380 13380 query15 363 251 252 251 query16 6479292 266 266 query17 17231455901 901 query18 2292423 413 413 query19 215 151 149 149 query20 84 81 80 80 query21 195 95 93 93 query22 5166502250025002 query23 33264 32610 32314 32314 query24 7028649364976493 query25 535 430 438 430 query26 533 161 165 161 query27 1810297 305 297 query28 6150243023642364 query29 2984274127592741 query30 245 170 162 162 query31 916 744 758 744 query32 68 65 62 62 query33 400 259 251 251 query34 860 483 482 482 query35 1115895 923 895 query36 1347115112461151 query37 91 62 62 62 query38 3112290529412905 query39 1369133913301330 query40 206 98 100 98 query41 46 46 45 45 query42 91 97 90 90 query43 674 735 676 676 query44 1133714 728 714 query45 250 239 233 233 query46 1258972 973 972 query47 1835186317171717 query48 1027743 739 739 query49 619 382 374 374 query50 856 626 645 626 query51 4735469946794679 query52 88 89 78 78 query53 456 337 326 326 query54 2693245924802459 query55 96 82 87 82 query56 251 213 211 211 query57 1196125911351135 query58 213 206 194 194 query59 4168393539043904 query60 224 216 204 204 query61 97 93 98 93 query62 868 509 527 509 query63 491 351 342 342 query64 2511155813511351 query65 3666357135313531 query66 770 380 374 374 query67 17988 15359 15198 15198 query68 9260652 668 652 query69 597 366 364 364 query70 1508147414431443 query71 420 313 313 313 query72 6710346934733469 query73 734 341 330 330 query74 6317585159405851 query75 5417370437533704 query76 5737110111521101 query77 982 260 261 260 query78 12746 11646 12399 11646 query79 10471 640 657 640 query80 1107403 395 395 query81 494 234 233 233 query82 1498103 99 99 query83 169 133 124 124 query84 259 73 72 72 query85 874 333 334 333 query86 332 293 272 272 query87 3241300430103004 query88 4936233923372337 query89 481 287 317 287 query90 1947218 198 198 query91 175 143 149 143 query92 61 54 55 54 query93 6355561 582 561 query94 701 207 203 203 query95 1107106010421042 query96 660 333 328 328 query97 6485632663626326 query98 187 179 178 178 query99 2834852 854 852 Total cold run time: 321870 ms Total hot run time: 204494 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [typo]Fix typo in arrow flight documentation [doris-website]
DongLiang-0 commented on PR #901: URL: https://github.com/apache/doris-website/pull/901#issuecomment-2257740648 > Please also fix it in branch-3.0 done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](catalog)fix db name may be null in NotificationEvent [doris]
doris-robot commented on PR #38421: URL: https://github.com/apache/doris/pull/38421#issuecomment-2257744681 TPC-H: Total hot run time: 41433 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit edd6bd6dee5adfaec2a0ac1007d570c75e37b8aa, data reload: false -- Round 1 -- q1 17711 405440274027 q2 2015204 200 200 q3 10456 124213281242 q4 10193 796 914 796 q5 7628298029362936 q6 218 138 135 135 q7 1040613 606 606 q8 9449187519221875 q9 8479658665566556 q10 8739384738103810 q11 430 256 240 240 q12 439 231 229 229 q13 17766 295429552954 q14 278 245 254 245 q15 528 488 493 488 q16 530 395 382 382 q17 960 911 895 895 q18 7919732873427328 q19 1472121312061206 q20 564 336 334 334 q21 5295467147614671 q22 351 282 278 278 Total cold run time: 112460 ms Total hot run time: 41433 ms - Round 2, with runtime_filter_mode=off - q1 4088402940404029 q2 326 229 222 222 q3 3008300730853007 q4 1961200620082006 q5 5523549554565456 q6 219 139 131 131 q7 2093177718241777 q8 3321337733383338 q9 8630870287158702 q10 3948404739893989 q11 548 440 446 440 q12 748 583 593 583 q13 15249 311230653065 q14 295 278 270 270 q15 552 490 501 490 q16 466 418 406 406 q17 1779169016951690 q18 8156773776287628 q19 1798174717071707 q20 2049185918801859 q21 5673547553735373 q22 525 450 485 450 Total cold run time: 70955 ms Total hot run time: 56618 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (cloud-merge) Fix the coredump because of change_cache_type to origin_type [doris]
Lchangliang commented on PR #38518: URL: https://github.com/apache/doris/pull/38518#issuecomment-2257746080 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (cloud-merge) Fix the coredump because of change_cache_type to origin_type [doris]
Lchangliang commented on PR #38518: URL: https://github.com/apache/doris/pull/38518#issuecomment-2257746736 run cloud_p1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](S3) Use allocator for s3 buffer's allocation [doris]
ByteYue commented on PR #38519: URL: https://github.com/apache/doris/pull/38519#issuecomment-2257746770 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](partial update) Persist partial_update_info in RocksDB in case of BE restart after a partial update has commited [doris]
zhannngchen commented on code in PR #38331:
URL: https://github.com/apache/doris/pull/38331#discussion_r1691351024
##
be/src/olap/rowset/rowset_meta_manager.cpp:
##
@@ -533,4 +533,74 @@ Status RowsetMetaManager::load_json_rowset_meta(OlapMeta*
meta,
return status;
}
+Status RowsetMetaManager::save_partial_update_info(
+OlapMeta* meta, int64_t tablet_id, const RowsetId& rowset_id,
+const PartialUpdateInfoPB& partial_update_info_pb) {
+std::string key =
+fmt::format("{}{}_{}", PARTIAL_UPDATE_INFO_PREFIX, tablet_id,
rowset_id.to_string());
+std::string value;
+if (!partial_update_info_pb.SerializeToString(&value)) {
+return Status::Error(
+"serialize partial update info failed. key:{}", key);
+}
+LOG_INFO("save partial update info, key={}, value_size={}", key,
value.size());
Review Comment:
change to VLOG
##
be/src/olap/rowset/rowset_meta_manager.cpp:
##
@@ -533,4 +533,74 @@ Status RowsetMetaManager::load_json_rowset_meta(OlapMeta*
meta,
return status;
}
+Status RowsetMetaManager::save_partial_update_info(
+OlapMeta* meta, int64_t tablet_id, const RowsetId& rowset_id,
+const PartialUpdateInfoPB& partial_update_info_pb) {
+std::string key =
+fmt::format("{}{}_{}", PARTIAL_UPDATE_INFO_PREFIX, tablet_id,
rowset_id.to_string());
Review Comment:
tablet_id + txn_id might be better?
the cost of key compare is lower
##
be/src/olap/storage_engine.cpp:
##
@@ -1022,6 +1027,39 @@ void StorageEngine::_clean_unused_pending_publish_info()
{
}
}
+void StorageEngine::_clean_unused_partial_update_info() {
+std::vector> remove_infos;
+auto unused_partial_update_info_collector = [this, &remove_infos](
+int64_t tablet_id,
RowsetId rowset_id,
+std::string_view
value) -> bool {
+TabletSharedPtr tablet = _tablet_manager->get_tablet(tablet_id);
+if (tablet == nullptr) {
+remove_infos.emplace_back(tablet_id, rowset_id);
+return true;
+}
+RowsetSharedPtr rowset = tablet->get_rowset(rowset_id);
+if (rowset == nullptr) {
Review Comment:
should check txn id, rather than rowset
Whether the rowset is in the tablet or not, there is no way to confirm that
this partial update info is useless, only use the txn id can confirm that. And
you can't find a rowset in the tablet until it's published.
##
be/src/olap/txn_manager.cpp:
##
@@ -514,6 +553,21 @@ Status TxnManager::publish_txn(OlapMeta* meta,
TPartitionId partition_id,
return status;
}
+if (tablet_txn_info->unique_key_merge_on_write &&
tablet_txn_info->partial_update_info &&
+tablet_txn_info->partial_update_info->is_partial_update) {
+status =
+RowsetMetaManager::remove_partial_update_info(meta, tablet_id,
rowset->rowset_id());
Review Comment:
should not remove here，it better to clear PartialUpdateInfo at the same time
with the clear of rowset(RowsetMetaManager::remove)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improment]publish workload to BE by tag [doris]
github-actions[bot] commented on PR #38486: URL: https://github.com/apache/doris/pull/38486#issuecomment-2257748632 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compatibility) type toSql should return lowercase string (#38012) [doris]
doris-robot commented on PR #38521: URL: https://github.com/apache/doris/pull/38521#issuecomment-2257749653 ClickBench: Total hot run time: 30.82 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ddd1cbf228a7f88742aebe873474cdf3be563ea8, data reload: false query1 0.020.020.02 query2 0.070.020.02 query3 0.260.050.04 query4 1.790.060.07 query5 0.540.520.53 query6 1.230.640.62 query7 0.010.010.00 query8 0.030.020.03 query9 0.530.490.50 query10 0.540.540.55 query11 0.120.080.08 query12 0.120.090.10 query13 0.620.620.62 query14 0.800.770.80 query15 0.780.760.77 query16 0.390.370.39 query17 1.001.021.01 query18 0.220.260.26 query19 1.921.861.85 query20 0.020.010.01 query21 15.55 0.560.55 query22 2.172.701.54 query23 17.33 0.870.94 query24 8.051.061.05 query25 0.380.110.06 query26 0.840.160.15 query27 0.040.040.04 query28 5.010.780.75 query29 12.65 2.362.29 query30 0.600.510.53 query31 2.820.400.38 query32 3.360.490.50 query33 3.103.083.08 query34 15.24 4.794.81 query35 4.864.864.86 query36 1.061.001.02 query37 0.060.050.05 query38 0.040.020.02 query39 0.020.020.01 query40 0.150.140.15 query41 0.070.020.01 query42 0.020.010.02 query43 0.020.020.02 Total cold run time: 104.45 s Total hot run time: 30.82 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](schema scan) Use async scanner for schema scanners [doris]
doris-robot commented on PR #38403: URL: https://github.com/apache/doris/pull/38403#issuecomment-2257751374 TPC-H: Total hot run time: 41671 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 5e46648cc6e0080eef54b6034415b6b71f49f463, data reload: false -- Round 1 -- q1 18035 418241984182 q2 2404206 207 206 q3 10469 131713771317 q4 10191 817 924 817 q5 7586292329882923 q6 221 137 138 137 q7 1020615 622 615 q8 9443185519321855 q9 8495658166086581 q10 8708382038483820 q11 438 249 245 245 q12 406 227 227 227 q13 17771 298229672967 q14 273 245 239 239 q15 538 480 481 480 q16 480 388 398 388 q17 965 921 921 921 q18 8135728972587258 q19 1431121512161215 q20 567 333 350 333 q21 5245466546954665 q22 347 283 280 280 Total cold run time: 113168 ms Total hot run time: 41671 ms - Round 2, with runtime_filter_mode=off - q1 4025402340074007 q2 332 219 218 218 q3 2987299929672967 q4 1898188218391839 q5 5229523452445234 q6 221 129 131 129 q7 2058165617211656 q8 3237327333043273 q9 8327836783018301 q10 3758380238913802 q11 555 454 465 454 q12 710 561 537 537 q13 13951 294729712947 q14 301 270 255 255 q15 514 485 475 475 q16 455 405 411 405 q17 1740166116841661 q18 7790727072907270 q19 1674166616571657 q20 1975176917661766 q21 5511519751475147 q22 529 487 455 455 Total cold run time: 6 ms Total hot run time: 54455 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cast) differences between FE and BE in parsing strings to integers. [doris]
doris-robot commented on PR #38511: URL: https://github.com/apache/doris/pull/38511#issuecomment-2257751523 TPC-H: Total hot run time: 41285 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit aaa4f59cd075d51a8b23749fa95d610264bb9eb7, data reload: false -- Round 1 -- q1 17616 405040164016 q2 2016210 202 202 q3 10429 125912991259 q4 10164 757 966 757 q5 7639289729572897 q6 214 139 140 139 q7 1031610 619 610 q8 9432187619451876 q9 8622657965976579 q10 8742382238373822 q11 433 253 248 248 q12 421 229 226 226 q13 17757 293129432931 q14 270 247 250 247 q15 532 482 493 482 q16 530 400 382 382 q17 958 945 884 884 q18 7926720672367206 q19 1471121612111211 q20 544 324 334 324 q21 5274470247174702 q22 357 285 285 285 Total cold run time: 112378 ms Total hot run time: 41285 ms - Round 2, with runtime_filter_mode=off - q1 4077400640214006 q2 326 224 226 224 q3 2960299930382999 q4 1980197519771975 q5 5508552654615461 q6 224 130 136 130 q7 2149177817821778 q8 3313335333463346 q9 8669870687158706 q10 3982403739423942 q11 538 446 459 446 q12 727 588 621 588 q13 16491 313531123112 q14 296 286 270 270 q15 533 490 485 485 q16 470 410 415 410 q17 1743171617081708 q18 8027779176417641 q19 1734171516941694 q20 2047185818811858 q21 5665538654225386 q22 544 484 475 475 Total cold run time: 72003 ms Total hot run time: 56640 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip the alignment process of some rowsets in partial update [doris]
github-actions[bot] commented on PR #38487: URL: https://github.com/apache/doris/pull/38487#issuecomment-2257752152 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [add]The kafka connector document adds enable.2pc, avro.topic2schema.… [doris-website]
DongLiang-0 commented on PR #890: URL: https://github.com/apache/doris-website/pull/890#issuecomment-2257754721 > Please also fix it in 3.0-branch done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compatibility) type toSql should return lowercase string (#38012) [doris]
doris-robot commented on PR #38521: URL: https://github.com/apache/doris/pull/38521#issuecomment-2257756017 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit ddd1cbf228a7f88742aebe873474cdf3be563ea8 with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.4 seconds inserted 1000 Rows, about 467K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](load) new insgestion load [doris]
gnehil commented on PR #36862: URL: https://github.com/apache/doris/pull/36862#issuecomment-2257761673 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](catalog)fix db name may be null in NotificationEvent [doris]
doris-robot commented on PR #38421: URL: https://github.com/apache/doris/pull/38421#issuecomment-2257765707 TPC-DS: Total hot run time: 169541 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit edd6bd6dee5adfaec2a0ac1007d570c75e37b8aa, data reload: false query1 912 375 362 362 query2 6463172816531653 query3 6668213 223 213 query4 19995 17427 17163 17163 query5 3703497 526 497 query6 279 168 166 166 query7 4588303 300 300 query8 247 200 198 198 query9 8488236623602360 query10 424 270 258 258 query11 10613 996499709964 query12 117 86 87 86 query13 1617371 367 367 query14 9549698970176989 query15 197 166 162 162 query16 6813430 447 430 query17 934 549 552 549 query18 1917281 274 274 query19 200 148 139 139 query20 90 88 85 85 query21 203 100 95 95 query22 4045390838503850 query23 33692 33831 33805 33805 query24 10201 315532053155 query25 684 390 416 390 query26 1734159 149 149 query27 3120279 275 275 query28 7629200120062001 query29 1324438 447 438 query30 244 153 156 153 query31 965 791 782 782 query32 107 62 62 62 query33 717 323 327 323 query34 914 478 514 478 query35 905 787 773 773 query36 1053895 896 895 query37 293 80 81 80 query38 2991285327932793 query39 878 814 832 814 query40 291 114 112 112 query41 48 44 43 43 query42 129 97 106 97 query43 464 435 423 423 query44 1175735 752 735 query45 214 178 173 173 query46 1082808 753 753 query47 1800167817221678 query48 385 296 283 283 query49 1005420 414 414 query50 906 431 433 431 query51 6774657366736573 query52 98 95 89 89 query53 266 181 177 177 query54 617 460 466 460 query55 76 78 76 76 query56 279 264 261 261 query57 1135102910471029 query58 286 276 290 276 query59 2567249823512351 query60 305 289 287 287 query61 120 116 119 116 query62 883 654 667 654 query63 213 179 185 179 query64 5869204019021902 query65 3154310231033102 query66 1336331 328 328 query67 15290 14916 14917 14916 query68 4365549 563 549 query69 485 293 299 293 query70 1101103710651037 query71 448 276 278 276 query72 7934272925022502 query73 763 319 336 319 query74 6112558756775587 query75 3608275627332733 query76 2454129813971298 query77 683 310 305 305 query78 9535892889418928 query79 1962547 535 535 query80 2018518 496 496 query81 554 226 219 219 query82 1043133 129 129 query83 293 174 176 174 query84 274 80 81 80 query85 1309362 302 302 query86 441 306 288 288 query87 3292313630973097 query88 2896240923812381 query89 377 294 287 287 query90 1761195 197 195 query91 128 102 102 102 query92 60 49 55 49 query93 2067598 607 598 query94 897 289 287 287 query95 381 263 270 263 query96 605 276 277 276 query97 3223304230283028 query98 220 200 202 200 query99 1662128412861284 Total cold run time: 265231 ms Total hot run time: 169541 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [feature](load) new insgestion load [doris]
github-actions[bot] commented on PR #36862: URL: https://github.com/apache/doris/pull/36862#issuecomment-2257771878 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cast) differences between FE and BE in parsing strings to integers. [doris]
doris-robot commented on PR #38511: URL: https://github.com/apache/doris/pull/38511#issuecomment-2257772309 TPC-DS: Total hot run time: 171132 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit aaa4f59cd075d51a8b23749fa95d610264bb9eb7, data reload: false query1 910 373 372 372 query2 6498170016791679 query3 6672210 220 210 query4 20082 17552 17527 17527 query5 3642501 527 501 query6 280 166 171 166 query7 4597300 293 293 query8 266 196 189 189 query9 8517237323602360 query10 436 277 265 265 query11 10521 10084 10014 10014 query12 131 96 87 87 query13 1642378 379 378 query14 9067836185098361 query15 213 164 171 164 query16 6802444 428 428 query17 980 584 561 561 query18 1866295 291 291 query19 206 149 151 149 query20 91 87 90 87 query21 205 101 99 99 query22 4164401540274015 query23 34018 33952 33398 33398 query24 9753311731513117 query25 659 436 415 415 query26 1579152 152 152 query27 2938284 276 276 query28 7306200719741974 query29 1115440 436 436 query30 231 150 152 150 query31 940 753 804 753 query32 102 59 60 59 query33 694 346 340 340 query34 923 490 502 490 query35 888 773 760 760 query36 1057860 895 860 query37 166 74 84 74 query38 3033284428262826 query39 872 835 812 812 query40 249 114 112 112 query41 46 43 45 43 query42 126 94 101 94 query43 458 432 421 421 query44 1180730 745 730 query45 214 177 178 177 query46 1088818 768 768 query47 1823176017241724 query48 358 294 287 287 query49 915 430 420 420 query50 896 421 420 420 query51 6768659566116595 query52 100 89 86 86 query53 250 182 177 177 query54 604 453 446 446 query55 80 77 76 76 query56 279 248 250 248 query57 1123102810421028 query58 273 276 266 266 query59 2698242622812281 query60 304 272 276 272 query61 103 97 98 97 query62 874 658 655 655 query63 204 184 182 182 query64 5633192118791879 query65 3163309830943094 query66 1039352 348 348 query67 15415 14822 14849 14822 query68 4333550 561 550 query69 446 310 299 299 query70 1103107710751075 query71 419 274 273 273 query72 7106270025702570 query73 759 329 327 327 query74 6001567256135613 query75 3360276127252725 query76 2425134513971345 query77 425 305 303 303 query78 9436894789708947 query79 1486539 535 535 query80 1432502 491 491 query81 581 238 241 238 query82 1066129 131 129 query83 252 168 171 168 query84 266 80 76 76 query85 1245316 378 316 query86 413 293 295 293 query87 3290313330723072 query88 2897238423922384 query89 380 292 288 288 query90 1713188 187 187 query91 127 102 107 102 query92 55 51 55 51 query93 1388604 602 602 query94 785 303 272 272 query95 381 265 279 265 query96 593 284 278 278 query97 3291305030493049 query98 214 209 195 195 query99 1611127812971278 Total cold run time: 259604 ms Total hot run time: 171132 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [Improvement](schema scan) Use async scanner for schema scanners [doris]
doris-robot commented on PR #38403: URL: https://github.com/apache/doris/pull/38403#issuecomment-2257772184 TPC-DS: Total hot run time: 169408 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 5e46648cc6e0080eef54b6034415b6b71f49f463, data reload: false query1 909 373 369 369 query2 6475170016701670 query3 6678212 223 212 query4 19795 17609 17497 17497 query5 4287499 521 499 query6 296 185 165 165 query7 4612296 289 289 query8 256 195 191 191 query9 8498237923752375 query10 435 276 269 269 query11 10508 10225 10120 10120 query12 134 90 90 90 query13 1636379 386 379 query14 8967698579646985 query15 205 164 164 164 query16 7121426 426 426 query17 969 568 566 566 query18 1929298 294 294 query19 203 151 151 151 query20 91 91 87 87 query21 208 100 101 100 query22 4302404740514047 query23 33745 33024 33238 33024 query24 10313 308831303088 query25 701 398 386 386 query26 1830148 145 145 query27 3036272 268 268 query28 7047198019611961 query29 1360424 422 422 query30 291 151 151 151 query31 907 757 745 745 query32 99 55 54 54 query33 730 313 310 310 query34 907 484 473 473 query35 857 739 726 726 query36 978 863 882 863 query37 303 85 80 80 query38 2839281827412741 query39 860 822 800 800 query40 287 111 120 111 query41 49 44 54 44 query42 120 105 102 102 query43 469 420 425 420 query44 1176730 729 729 query45 209 180 180 180 query46 1095799 773 773 query47 1838168716971687 query48 367 289 284 284 query49 1179418 426 418 query50 884 432 435 432 query51 6745670766226622 query52 98 91 91 91 query53 254 179 179 179 query54 624 459 447 447 query55 74 76 74 74 query56 284 255 262 255 query57 1136103410421034 query58 284 270 265 265 query59 2745252225262522 query60 327 265 278 265 query61 103 97 96 96 query62 922 656 665 656 query63 220 182 190 182 query64 5901191619501916 query65 3167308830793079 query66 1437336 332 332 query67 15359 14863 14861 14861 query68 4449558 559 558 query69 463 299 298 298 query70 1113105310611053 query71 414 273 287 273 query72 7536273424982498 query73 766 329 325 325 query74 6014564656315631 query75 3364274027092709 query76 2643131813781318 query77 518 307 309 307 query78 9342894488818881 query79 1866535 526 526 query80 1184507 498 498 query81 553 231 226 226 query82 1071130 128 128 query83 245 169 167 167 query84 271 80 79 79 query85 1382320 306 306 query86 386 306 300 300 query87 3266314530693069 query88 2947242324652423 query89 384 281 280 280 query90 1807192 191 191 query91 127 103 102 102 query92 66 50 49 49 query93 1619605 600 600 query94 906 295 304 295 query95 378 261 266 261 query96 617 283 289 283 query97 3212308930493049 query98 213 198 198 198 query99 1627131412891289 Total cold run time: 263419 ms Total hot run time: 169408 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [doc]Flink connector document adds schema-change-mode parameter configuration [doris-website]
DongLiang-0 commented on PR #891: URL: https://github.com/apache/doris-website/pull/891#issuecomment-2257775389 > Please also fix it in 3.0-branch done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feat](nereids)Support date_trunc function in partition prune [doris]
doris-robot commented on PR #38025: URL: https://github.com/apache/doris/pull/38025#issuecomment-2257774344 TPC-H: Total hot run time: 41509 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0dae4302649e22fdff5a4fd4c1cd49fd56f55c48, data reload: false -- Round 1 -- q1 17628 408840234023 q2 2016212 203 203 q3 10455 124213681242 q4 10169 866 950 866 q5 7600295329672953 q6 225 140 140 140 q7 1032614 612 612 q8 9446189419331894 q9 8433654865826548 q10 8747381038183810 q11 435 250 245 245 q12 430 235 232 232 q13 17767 297729722972 q14 267 245 245 245 q15 519 480 477 477 q16 512 393 396 393 q17 953 904 914 904 q18 7946725272077207 q19 1576121111941194 q20 548 324 340 324 q21 5347481847454745 q22 350 290 280 280 Total cold run time: 112401 ms Total hot run time: 41509 ms - Round 2, with runtime_filter_mode=off - q1 4046398940263989 q2 319 222 226 222 q3 2976295231182952 q4 2017198819831983 q5 5564549354495449 q6 219 138 135 135 q7 2105179618231796 q8 3329340233533353 q9 8703863886878638 q10 4030404240364036 q11 538 453 451 451 q12 747 592 573 573 q13 16532 320531173117 q14 301 290 279 279 q15 542 489 492 489 q16 487 420 397 397 q17 1751172217361722 q18 8117769176567656 q19 1715170316851685 q20 2069188518391839 q21 5721550053895389 q22 539 449 469 449 Total cold run time: 72367 ms Total hot run time: 56599 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improvement](mtmv) Support to partition prune when query rewrite by sync materialized view [doris]
seawinde opened a new pull request, #38527:
URL: https://github.com/apache/doris/pull/38527
## Proposed changes
Support to partition prune when query rewrite by sync materialized view
such as table def is as following:
```sql
CREATE TABLE IF NOT EXISTS test_unique (
`time` DATETIME NULL COMMENT '查询时间',
`app_name` VARCHAR(64) NULL COMMENT '标识',
`event_id` VARCHAR(128) NULL COMMENT '标识',
`decision` VARCHAR(32) NULL COMMENT '枚举值',
`id` VARCHAR(35) NOT NULL COMMENT 'od',
`code` VARCHAR(64) NULL COMMENT '标识',
`event_type` VARCHAR(32) NULL COMMENT '事件类型'
)
UNIQUE KEY(time)
PARTITION BY RANGE(time)
(
FROM ("2024-07-01 00:00:00") TO ("2024-07-15 00:00:00") INTERVAL 1
HOUR
)
DISTRIBUTED BY HASH(time)
BUCKETS 3 PROPERTIES ("replication_num" = "1");
```
sync materialized view def is
```sql
create materialized view as
select
app_name,
event_id,
time,
count(*)
from
test_duplicate
group by
app_name,
event_id,
time;
```
if your query is following, if rewritten by sync materialized view
successfully, should partition prune
```sql
select
app_name,
event_id,
time,
count(*)
from
test_duplicate
where time < '2024-07-05 01:00:00'
group by
app_name,
time,
event_id;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](mtmv) Support to partition prune when query rewrite by sync materialized view [doris]
doris-robot commented on PR #38527: URL: https://github.com/apache/doris/pull/38527#issuecomment-2257774959 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix insert stmt throw [doris]
924060929 commented on PR #38526: URL: https://github.com/apache/doris/pull/38526#issuecomment-2257776133 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](mtmv) Support to partition prune when query rewrite by sync materialized view [doris]
seawinde commented on PR #38527: URL: https://github.com/apache/doris/pull/38527#issuecomment-2257776131 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](catalog)fix db name may be null in NotificationEvent [doris]
doris-robot commented on PR #38421: URL: https://github.com/apache/doris/pull/38421#issuecomment-2257776362 ClickBench: Total hot run time: 29.94 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit edd6bd6dee5adfaec2a0ac1007d570c75e37b8aa, data reload: false query1 0.050.040.03 query2 0.070.030.04 query3 0.230.040.04 query4 1.680.070.08 query5 0.490.480.49 query6 1.140.720.71 query7 0.020.010.02 query8 0.050.040.05 query9 0.570.500.52 query10 0.570.580.57 query11 0.160.120.11 query12 0.140.120.12 query13 0.610.590.60 query14 0.780.790.80 query15 0.890.850.86 query16 0.340.350.35 query17 1.021.021.02 query18 0.220.220.21 query19 1.841.771.79 query20 0.020.010.01 query21 15.41 0.760.66 query22 3.968.361.23 query23 18.02 1.311.31 query24 2.240.230.22 query25 0.180.080.08 query26 0.320.220.22 query27 0.470.230.24 query28 13.16 0.990.97 query29 12.51 3.313.27 query30 0.250.060.06 query31 2.890.410.40 query32 3.230.490.48 query33 2.932.952.93 query34 15.42 4.234.26 query35 4.304.284.29 query36 0.670.480.49 query37 0.190.160.17 query38 0.170.160.15 query39 0.040.050.04 query40 0.160.130.12 query41 0.100.050.04 query42 0.060.040.05 query43 0.040.040.04 Total cold run time: 107.61 s Total hot run time: 29.94 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix insert stmt throw [doris]
924060929 commented on PR #38526: URL: https://github.com/apache/doris/pull/38526#issuecomment-2257779916 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](schema scan) Use async scanner for schema scanners [doris]
doris-robot commented on PR #38403: URL: https://github.com/apache/doris/pull/38403#issuecomment-2257783292 ClickBench: Total hot run time: 30.44 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 5e46648cc6e0080eef54b6034415b6b71f49f463, data reload: false query1 0.040.030.03 query2 0.080.040.04 query3 0.230.050.05 query4 1.690.070.07 query5 0.480.470.49 query6 1.140.710.72 query7 0.020.010.01 query8 0.050.040.05 query9 0.550.520.50 query10 0.570.560.56 query11 0.160.110.12 query12 0.140.130.13 query13 0.620.610.60 query14 0.770.790.78 query15 0.880.860.87 query16 0.350.360.35 query17 1.021.021.00 query18 0.220.200.20 query19 1.821.721.79 query20 0.010.010.01 query21 15.42 0.770.65 query22 3.927.251.77 query23 18.01 1.321.20 query24 2.270.220.22 query25 0.180.080.08 query26 0.330.210.22 query27 0.460.230.24 query28 13.49 1.030.98 query29 12.84 3.343.32 query30 0.260.060.06 query31 2.860.430.41 query32 3.220.480.48 query33 2.962.972.95 query34 15.45 4.284.27 query35 4.304.314.27 query36 0.670.490.49 query37 0.200.170.16 query38 0.160.150.15 query39 0.040.040.04 query40 0.170.130.13 query41 0.100.050.05 query42 0.050.040.05 query43 0.040.040.04 Total cold run time: 108.24 s Total hot run time: 30.44 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cast) differences between FE and BE in parsing strings to integers. [doris]
doris-robot commented on PR #38511: URL: https://github.com/apache/doris/pull/38511#issuecomment-2257783473 ClickBench: Total hot run time: 30.03 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit aaa4f59cd075d51a8b23749fa95d610264bb9eb7, data reload: false query1 0.050.030.03 query2 0.080.040.04 query3 0.230.050.05 query4 1.690.070.07 query5 0.490.490.49 query6 1.120.710.72 query7 0.020.010.01 query8 0.050.040.05 query9 0.570.520.51 query10 0.570.550.55 query11 0.160.110.11 query12 0.160.130.12 query13 0.620.600.60 query14 0.780.790.79 query15 0.880.860.86 query16 0.350.350.36 query17 0.991.021.03 query18 0.210.200.22 query19 1.851.781.73 query20 0.020.010.01 query21 15.45 0.790.66 query22 4.057.501.20 query23 17.95 1.501.37 query24 2.250.220.22 query25 0.180.100.08 query26 0.320.210.21 query27 0.450.240.24 query28 13.52 0.991.00 query29 12.52 3.283.27 query30 0.260.060.06 query31 2.850.410.42 query32 3.230.490.49 query33 2.923.002.94 query34 15.46 4.284.25 query35 4.334.294.29 query36 0.680.480.48 query37 0.180.150.16 query38 0.170.160.15 query39 0.040.040.04 query40 0.160.120.14 query41 0.100.050.04 query42 0.060.040.05 query43 0.050.040.04 Total cold run time: 108.07 s Total hot run time: 30.03 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improment]publish workload to BE by tag [doris]
github-actions[bot] commented on PR #38486: URL: https://github.com/apache/doris/pull/38486#issuecomment-2257786080 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](routine load) fix show routine load task result incorrect [doris]
doris-robot commented on PR #38523: URL: https://github.com/apache/doris/pull/38523#issuecomment-2257790758 TPC-H: Total hot run time: 42246 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit e168993ad838f71491ca9cb8912f94692d6e73a0, data reload: false -- Round 1 -- q1 17855 412740724072 q2 2022200 198 198 q3 10862 132013761320 q4 11322 833 958 833 q5 7650303329872987 q6 222 141 145 141 q7 1052615 613 613 q8 9435188019101880 q9 8532664666306630 q10 8799386438533853 q11 429 250 251 250 q12 431 236 228 228 q13 1 310630973097 q14 278 261 251 251 q15 520 489 499 489 q16 534 430 420 420 q17 962 907 968 907 q18 9799787576017601 q19 3737108012251080 q20 547 332 339 332 q21 5317478348074783 q22 349 281 286 281 Total cold run time: 118431 ms Total hot run time: 42246 ms - Round 2, with runtime_filter_mode=off - q1 4097405140264026 q2 344 230 227 227 q3 3130317931443144 q4 2031200720012001 q5 5471541154585411 q6 223 133 136 133 q7 2172180618001800 q8 3311336533513351 q9 8640875086758675 q10 3980394040303940 q11 560 464 464 464 q12 764 546 654 546 q13 16311 312130943094 q14 309 263 273 263 q15 531 492 482 482 q16 464 424 421 421 q17 1761171217261712 q18 8171781676747674 q19 1911172217071707 q20 2055183418271827 q21 5922538954385389 q22 555 469 482 469 Total cold run time: 72713 ms Total hot run time: 56756 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix insert stmt throw MultiCastDataSink cannot be cast to DataStreamSink [doris]
github-actions[bot] commented on PR #38526: URL: https://github.com/apache/doris/pull/38526#issuecomment-2257792038 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix insert stmt throw MultiCastDataSink cannot be cast to DataStreamSink [doris]
github-actions[bot] commented on PR #38526: URL: https://github.com/apache/doris/pull/38526#issuecomment-2257792105 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
