[PR] [fix](statistics)Remove old partition stats in the next analyze after insert overwrite. [doris]
Jibing-Li opened a new pull request, #38175: URL: https://github.com/apache/doris/pull/38175 Remove old partition stats in the next analyze after insert overwrite. Because insert overwrite will use new partitions to replace old ones, which means all partitions are new. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](statistics)Remove old partition stats in the next analyze after insert overwrite. [doris]
doris-robot commented on PR #38175: URL: https://github.com/apache/doris/pull/38175#issuecomment-2241503422 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](statistics)Remove old partition stats in the next analyze after insert overwrite. [doris]
Jibing-Li commented on PR #38175: URL: https://github.com/apache/doris/pull/38175#issuecomment-2241503514 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth [doris]
doris-robot commented on PR #38168: URL: https://github.com/apache/doris/pull/38168#issuecomment-2241507503 TPC-H: Total hot run time: 39876 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ceca9286bd4caa2984f06314c6d73dc644dd314e, data reload: false -- Round 1 -- q1 17631 435342434243 q2 2018191 186 186 q3 10448 121111161116 q4 10183 752 720 720 q5 7534271126572657 q6 224 140 140 140 q7 963 591 601 591 q8 9218208220502050 q9 8847659165866586 q10 8817376137863761 q11 455 240 234 234 q12 507 230 227 227 q13 17883 297729762976 q14 281 242 238 238 q15 528 477 486 477 q16 534 378 376 376 q17 964 725 701 701 q18 8015743075097430 q19 6024147213521352 q20 650 327 334 327 q21 4949328032053205 q22 346 283 283 283 Total cold run time: 117019 ms Total hot run time: 39876 ms - Round 2, with runtime_filter_mode=off - q1 4334424042354235 q2 389 267 280 267 q3 2970279528632795 q4 1993168416521652 q5 5681551654995499 q6 218 133 129 129 q7 2186180318901803 q8 3275335134193351 q9 8756873388718733 q10 4068396937563756 q11 598 497 503 497 q12 803 637 631 631 q13 17108 319632323196 q14 330 288 279 279 q15 518 481 489 481 q16 497 430 439 430 q17 1824152515091509 q18 8240784778017801 q19 1730169615781578 q20 2136187118331833 q21 5204507647094709 q22 609 524 499 499 Total cold run time: 73467 ms Total hot run time: 55663 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth [doris]
doris-robot commented on PR #38168: URL: https://github.com/apache/doris/pull/38168#issuecomment-2241510501 TPC-DS: Total hot run time: 175157 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ceca9286bd4caa2984f06314c6d73dc644dd314e, data reload: false query1 918 369 371 369 query2 6433188918421842 query3 6660208 224 208 query4 28092 17467 17237 17237 query5 3637477 487 477 query6 272 172 157 157 query7 4581285 279 279 query8 227 194 195 194 query9 8593245324362436 query10 440 296 275 275 query11 12334 10184 10105 10105 query12 116 91 88 88 query13 1649371 372 371 query14 10205 770776417641 query15 226 175 170 170 query16 7750451 458 451 query17 1586586 540 540 query18 1968283 280 280 query19 200 153 156 153 query20 90 87 86 86 query21 211 128 127 127 query22 4229411239453945 query23 33957 33977 33489 33489 query24 11189 296429712964 query25 621 404 415 404 query26 1162153 154 153 query27 2257280 284 280 query28 6856208220822082 query29 884 626 654 626 query30 250 154 150 150 query31 999 760 764 760 query32 88 56 57 56 query33 753 344 343 343 query34 907 486 520 486 query35 850 748 728 728 query36 1128964 976 964 query37 145 82 82 82 query38 2957287128732871 query39 910 856 885 856 query40 212 121 121 121 query41 46 44 44 44 query42 118 103 93 93 query43 488 459 451 451 query44 1165746 732 732 query45 196 161 161 161 query46 1084733 713 713 query47 1876177217751772 query48 366 304 284 284 query49 850 415 409 409 query50 794 399 392 392 query51 7013685668826856 query52 107 89 94 89 query53 360 290 296 290 query54 964 450 452 450 query55 80 73 73 73 query56 297 273 272 272 query57 1159104710261026 query58 256 247 245 245 query59 2964271826392639 query60 296 270 274 270 query61 96 92 94 92 query62 801 654 634 634 query63 331 287 287 287 query64 9519224574072245 query65 3136310931703109 query66 837 336 330 330 query67 15381 15092 14883 14883 query68 8749551 561 551 query69 704 444 379 379 query70 1383114811331133 query71 490 282 286 282 query72 7605541057285410 query73 1701330 323 323 query74 6027571956325632 query75 4129274027242724 query76 4505930 904 904 query77 680 311 315 311 query78 959413684 96019601 query79 8647537 530 530 query80 819 483 473 473 query81 572 221 223 221 query82 293 137 131 131 query83 207 174 169 169 query84 273 85 87 85 query85 710 377 315 315 query86 365 316 273 273 query87 3370308230953082 query88 4281238723552355 query89 466 395 384 384 query90 2042200 201 200 query91 132 100 104 100 query92 72 48 54 48 query93 1041515 511 511 query94 1248291 292 291 query95 410 322 321 321 query96 578 268 270 268 query97 3209303730143014 query98 229 197 192 192 query99 1534127912571257 Total cold run time: 293187 ms Total hot run time: 175157 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [fix](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth [doris]
doris-robot commented on PR #38168: URL: https://github.com/apache/doris/pull/38168#issuecomment-2241511995 ClickBench: Total hot run time: 30.98 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ceca9286bd4caa2984f06314c6d73dc644dd314e, data reload: false query1 0.040.040.04 query2 0.080.040.04 query3 0.220.050.05 query4 1.680.070.07 query5 0.500.480.49 query6 1.130.740.72 query7 0.020.010.01 query8 0.050.040.05 query9 0.540.490.50 query10 0.550.560.55 query11 0.150.110.11 query12 0.140.120.13 query13 0.600.590.57 query14 0.760.790.80 query15 0.870.810.81 query16 0.380.350.36 query17 0.950.960.98 query18 0.220.220.22 query19 1.861.821.84 query20 0.020.010.01 query21 15.39 0.760.66 query22 4.227.022.12 query23 18.31 1.451.31 query24 2.090.220.22 query25 0.160.080.08 query26 0.300.220.21 query27 0.450.230.23 query28 13.32 1.021.00 query29 12.62 3.303.26 query30 0.250.060.05 query31 2.860.400.39 query32 3.290.460.47 query33 2.882.992.86 query34 17.26 4.344.34 query35 4.444.474.44 query36 0.640.500.50 query37 0.190.160.16 query38 0.160.150.15 query39 0.040.040.03 query40 0.150.120.14 query41 0.100.050.05 query42 0.050.050.05 query43 0.050.040.04 Total cold run time: 109.98 s Total hot run time: 30.98 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](statistics)Remove old partition stats in the next analyze after insert overwrite. [doris]
doris-robot commented on PR #38175: URL: https://github.com/apache/doris/pull/38175#issuecomment-2241514682 TPC-H: Total hot run time: 39752 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 419994054f07520bd0749c6805c53f4a6cb74ce2, data reload: false -- Round 1 -- q1 17629 436542984298 q2 2004201 188 188 q3 10440 113310631063 q4 10188 824 792 792 q5 7520278426492649 q6 219 135 138 135 q7 952 599 604 599 q8 9213206020712060 q9 8621655865456545 q10 8848379637873787 q11 504 237 236 236 q12 426 227 225 225 q13 17761 297930032979 q14 281 228 253 228 q15 536 481 494 481 q16 505 393 390 390 q17 971 678 701 678 q18 8146748473247324 q19 3682153513521352 q20 696 326 318 318 q21 4892322931373137 q22 349 293 288 288 Total cold run time: 114383 ms Total hot run time: 39752 ms - Round 2, with runtime_filter_mode=off - q1 4322421342204213 q2 366 276 274 274 q3 3008273928382739 q4 1980170317091703 q5 5663561855205520 q6 239 139 137 137 q7 2195184718671847 q8 3273343033943394 q9 8794875388618753 q10 4056396138283828 q11 631 479 503 479 q12 789 633 680 633 q13 16096 319431613161 q14 320 298 282 282 q15 533 501 486 486 q16 470 430 451 430 q17 1815150415041504 q18 8199791379517913 q19 2388157516261575 q20 2162193818631863 q21 5031487047974797 q22 562 484 505 484 Total cold run time: 72892 ms Total hot run time: 56015 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](statistics)Remove old partition stats in the next analyze after insert overwrite. [doris]
doris-robot commented on PR #38175: URL: https://github.com/apache/doris/pull/38175#issuecomment-2241517399 TPC-DS: Total hot run time: 173447 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 419994054f07520bd0749c6805c53f4a6cb74ce2, data reload: false query1 913 378 369 369 query2 6439187518071807 query3 6630208 228 208 query4 28395 17525 17371 17371 query5 3619474 491 474 query6 277 177 175 175 query7 4588286 294 286 query8 247 200 194 194 query9 8583246224462446 query10 448 300 273 273 query11 11732 996798699869 query12 121 89 84 84 query13 1651376 368 368 query14 10225 757875427542 query15 227 159 166 159 query16 7760507 438 438 query17 1558558 521 521 query18 1926282 271 271 query19 193 147 149 147 query20 87 84 102 84 query21 206 132 132 132 query22 4301406839703970 query23 34092 33747 33695 33695 query24 11257 288029722880 query25 648 394 383 383 query26 1132153 155 153 query27 2473274 279 274 query28 7468209821102098 query29 900 637 637 637 query30 259 154 152 152 query31 961 735 776 735 query32 98 59 56 56 query33 750 347 334 334 query34 940 488 500 488 query35 872 778 770 770 query36 1145999 947 947 query37 158 92 86 86 query38 2989285928172817 query39 951 825 811 811 query40 207 117 121 117 query41 44 44 43 43 query42 121 98 97 97 query43 509 472 474 472 query44 1219732 734 732 query45 192 163 160 160 query46 1084732 757 732 query47 1825175417791754 query48 364 287 295 287 query49 837 430 413 413 query50 788 388 383 383 query51 7045686867356735 query52 104 97 94 94 query53 357 291 286 286 query54 918 447 451 447 query55 77 78 75 75 query56 298 279 265 265 query57 1115102510431025 query58 261 253 252 252 query59 2905262728742627 query60 302 280 325 280 query61 98 94 96 94 query62 786 645 636 636 query63 318 285 290 285 query64 9425221416841684 query65 3163309930903090 query66 742 323 329 323 query67 15416 14893 14894 14893 query68 4551540 528 528 query69 719 446 369 369 query70 1160111711521117 query71 420 284 284 284 query72 7409537956305379 query73 756 326 321 321 query74 6067568856425642 query75 3447268127292681 query76 2850959 963 959 query77 630 314 309 309 query78 10011 890389438903 query79 2311531 525 525 query80 1315475 550 475 query81 589 222 217 217 query82 814 138 132 132 query83 271 159 165 159 query84 239 88 85 85 query85 1507328 294 294 query86 466 323 316 316 query87 3263306930713069 query88 3893235823662358 query89 475 386 372 372 query90 1753190 194 190 query91 130 100 100 100 query92 59 48 51 48 query93 2205517 516 516 query94 1310290 276 276 query95 401 304 319 304 query96 596 277 270 270 query97 3268306030513051 query98 226 203 195 195 query99 1551125912181218 Total cold run time: 282401 ms Total hot run time: 173447 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [fix](statistics)Remove old partition stats in the next analyze after insert overwrite. [doris]
doris-robot commented on PR #38175: URL: https://github.com/apache/doris/pull/38175#issuecomment-2241519299 ClickBench: Total hot run time: 30.48 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 419994054f07520bd0749c6805c53f4a6cb74ce2, data reload: false query1 0.040.030.04 query2 0.080.040.04 query3 0.220.050.05 query4 1.680.090.07 query5 0.500.480.48 query6 1.130.720.72 query7 0.020.010.01 query8 0.050.040.05 query9 0.560.500.50 query10 0.560.560.54 query11 0.150.110.11 query12 0.150.120.12 query13 0.610.580.58 query14 0.750.780.78 query15 0.870.810.81 query16 0.370.370.37 query17 1.051.051.05 query18 0.220.220.22 query19 1.911.831.81 query20 0.020.010.01 query21 15.41 0.750.65 query22 4.667.091.73 query23 18.28 1.411.17 query24 2.080.240.22 query25 0.140.090.09 query26 0.300.210.21 query27 0.460.240.23 query28 13.23 1.021.01 query29 12.65 3.273.24 query30 0.250.060.06 query31 2.880.390.38 query32 3.270.480.46 query33 2.892.912.88 query34 17.06 4.384.35 query35 4.414.404.43 query36 0.660.480.47 query37 0.190.170.16 query38 0.160.140.15 query39 0.040.030.04 query40 0.150.120.12 query41 0.090.050.05 query42 0.060.050.06 query43 0.060.040.04 Total cold run time: 110.32 s Total hot run time: 30.48 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
doris-robot commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241523525 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
morningman opened a new pull request, #38176: URL: https://github.com/apache/doris/pull/38176 bp #35397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
github-actions[bot] commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241525801 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
github-actions[bot] commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241527484 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
morningman commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241531018 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] Add regression case for group commit [doris]
Yukang-Lian commented on PR #38171: URL: https://github.com/apache/doris/pull/38171#issuecomment-2241537100 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
github-actions[bot] commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241537677 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
github-actions[bot] commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241537687 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
morningman commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241540843 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
github-actions[bot] commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241542443 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Chore](brpc) add gc for abafreelist to avoid eagain and set brpc tim… [doris]
BiteThet opened a new pull request, #38177: URL: https://github.com/apache/doris/pull/38177 pick from #37888 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](brpc) add gc for abafreelist to avoid eagain and set brpc tim… [doris]
BiteThet commented on PR #38177: URL: https://github.com/apache/doris/pull/38177#issuecomment-2241544929 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](brpc) add gc for abafreelist to avoid eagain and set brpc tim… [doris]
github-actions[bot] commented on PR #38177: URL: https://github.com/apache/doris/pull/38177#issuecomment-2241546593 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth [doris]
yiguolei commented on code in PR #38168:
URL: https://github.com/apache/doris/pull/38168#discussion_r1685696855
##
be/src/runtime/memory/mem_tracker_limiter.h:
##
@@ -202,6 +202,18 @@ class MemTrackerLimiter final : public MemTracker {
return querytid;
}
+void set_weighted_memory(int64_t weighted_limit, double weighted_ratio) {
+std::lock_guard l(_weighted_mem_lock);
+_weighted_limit = weighted_limit;
Review Comment:
我感觉这两个变量还是放到query ctx 中比较好，
memtracker 只要把memory track 准就可以了，不要把这些预估的计算也放到memtracker 里。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth [doris]
yiguolei commented on code in PR #38168:
URL: https://github.com/apache/doris/pull/38168#discussion_r1685697259
##
be/src/runtime/memory/thread_mem_tracker_mgr.h:
##
@@ -291,6 +291,9 @@ inline bool ThreadMemTrackerMgr::try_reserve(int64_t size) {
_limiter_tracker_raw->release(size); // rollback
return false;
}
+if (_limiter_tracker_raw->wg_refresh_interval_memory_growth != nullptr) {
Review Comment:
我们最终还是要在thread context 中放一个workload group的weak ptr。比如我们做IO 隔离，我们会在page io
那个地方就要获取当前在哪个wg 里，获得wg 相关的io limiter，此时也得依赖这个获得。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth [doris]
yiguolei commented on code in PR #38168:
URL: https://github.com/apache/doris/pull/38168#discussion_r1685697259
##
be/src/runtime/memory/thread_mem_tracker_mgr.h:
##
@@ -291,6 +291,9 @@ inline bool ThreadMemTrackerMgr::try_reserve(int64_t size) {
_limiter_tracker_raw->release(size); // rollback
return false;
}
+if (_limiter_tracker_raw->wg_refresh_interval_memory_growth != nullptr) {
Review Comment:
我们最终还是要在thread context 中放一个workload group的weak ptr。比如我们做IO 隔离，我们会在page io
那个地方就要获取当前在哪个wg 里，获得wg 相关的io limiter，此时也得依赖这个获得。
一旦我们再thread context 中有了workload group，那么reserve 的时候，直接在wg 里放一个memory
growth的counter 感觉是合理的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth [doris]
yiguolei commented on code in PR #38168:
URL: https://github.com/apache/doris/pull/38168#discussion_r1685696855
##
be/src/runtime/memory/mem_tracker_limiter.h:
##
@@ -202,6 +202,18 @@ class MemTrackerLimiter final : public MemTracker {
return querytid;
}
+void set_weighted_memory(int64_t weighted_limit, double weighted_ratio) {
+std::lock_guard l(_weighted_mem_lock);
+_weighted_limit = weighted_limit;
Review Comment:
我感觉这两个变量还是放到query ctx 中比较好，
memtracker 只要把memory track 准就可以了，不要把这些预估的计算也放到memtracker 里。
可能放到queryctx 中也不对，但是当前我觉得往memtracker 里移动是不对的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
doris-robot commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241549309 TPC-H: Total hot run time: 50183 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b5dbef43cbc5431a8029d3504bc6574b99eda7d4, data reload: false -- Round 1 -- q1 18078 447543754375 q2 2082169 155 155 q3 10485 192919431929 q4 10369 124113221241 q5 8653400539593959 q6 241 127 148 127 q7 2073163816301630 q8 9288274927292729 q9 11089 10498 10326 10326 q10 8702352835323528 q11 424 256 238 238 q12 483 303 315 303 q13 18356 396440823964 q14 347 328 337 328 q15 509 481 469 469 q16 686 577 584 577 q17 1153960 989 960 q18 7278701368476847 q19 1782165116081608 q20 525 314 316 314 q21 4445413541394135 q22 534 441 448 441 Total cold run time: 117582 ms Total hot run time: 50183 ms - Round 2, with runtime_filter_mode=off - q1 4417429743074297 q2 322 231 223 223 q3 4237415141494149 q4 2764273027652730 q5 7268715771487148 q6 237 123 118 118 q7 3254290628972897 q8 4400452745174517 q9 17472 17113 17036 17036 q10 4289429142894289 q11 771 706 703 703 q12 1047842 859 842 q13 7091376937493749 q14 452 426 449 426 q15 503 470 453 453 q16 740 709 677 677 q17 3924384639313846 q18 8790874788018747 q19 1763171016611661 q20 2388212021262120 q21 8519853786138537 q22 1052968 979 968 Total cold run time: 85700 ms Total hot run time: 80133 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] 2.1.5-rc02 [doris]
yiguolei merged PR #38167: URL: https://github.com/apache/doris/pull/38167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: 2.1.5-rc02 (#38167)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new b6dd7bcc6da 2.1.5-rc02 (#38167)
b6dd7bcc6da is described below
commit b6dd7bcc6da11e0c4550a498838f755902d61961
Author: yiguolei <676222...@qq.com>
AuthorDate: Sun Jul 21 17:59:02 2024 +0800
2.1.5-rc02 (#38167)
## Proposed changes
Issue Number: close #xxx
Co-authored-by: yiguolei
---
gensrc/script/gen_build_version.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/gensrc/script/gen_build_version.sh
b/gensrc/script/gen_build_version.sh
index c866816eb98..4ab0b13f003 100755
--- a/gensrc/script/gen_build_version.sh
+++ b/gensrc/script/gen_build_version.sh
@@ -31,7 +31,7 @@ build_version_prefix="doris"
build_version_major=2
build_version_minor=1
build_version_patch=5
-build_version_rc_version="rc01"
+build_version_rc_version="rc02"
build_version="${build_version_prefix}-${build_version_major}.${build_version_minor}.${build_version_patch}-${build_version_rc_version}"
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Refactor](common) refactor the Exception code [doris]
github-actions[bot] commented on PR #38172: URL: https://github.com/apache/doris/pull/38172#issuecomment-2241550930 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Refactor](common) refactor the Exception code [doris]
github-actions[bot] commented on PR #38172: URL: https://github.com/apache/doris/pull/38172#issuecomment-2241550920 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
doris-robot commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241553042 TPC-DS: Total hot run time: 205012 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b5dbef43cbc5431a8029d3504bc6574b99eda7d4, data reload: false query1 937 427 386 386 query2 6549279427542754 query3 6919213 210 210 query4 20148 18150 17982 17982 query5 19747 656966026569 query6 351 212 227 212 query7 4658309 317 309 query8 423 413 415 413 query9 3181272826652665 query10 417 334 298 298 query11 11402 10721 10719 10719 query12 124 74 76 74 query13 5606721 721 721 query14 18455 13696 13416 13416 query15 365 249 261 249 query16 6437303 261 261 query17 13591563886 886 query18 2255429 415 415 query19 232 153 156 153 query20 81 81 80 80 query21 188 96 91 91 query22 5246488348984883 query23 32714 31798 31952 31798 query24 6822652564736473 query25 532 423 443 423 query26 532 170 164 164 query27 1861300 301 300 query28 6188238823432343 query29 2914265527462655 query30 244 166 173 166 query31 921 749 752 749 query32 72 67 63 63 query33 413 258 259 258 query34 858 502 491 491 query35 1119972 943 943 query36 1344114312091143 query37 93 60 61 60 query38 3115293829492938 query39 1395135113161316 query40 212 98 100 98 query41 50 44 44 44 query42 87 78 86 78 query43 707 703 729 703 query44 1148728 731 728 query45 247 242 238 238 query46 1240988 974 974 query47 1844176717031703 query48 1030718 752 718 query49 625 368 383 368 query50 884 644 631 631 query51 4803469647114696 query52 108 80 93 80 query53 463 327 318 318 query54 2702250224682468 query55 83 79 88 79 query56 236 248 210 210 query57 1201106410651064 query58 226 218 186 186 query59 4440414637033703 query60 224 216 223 216 query61 99 94 98 94 query62 798 471 474 471 query63 499 346 349 346 query64 2591157215421542 query65 3629355535743555 query66 812 385 391 385 query67 16103 15766 17358 15766 query68 8974677 663 663 query69 589 351 347 347 query70 1718160916091609 query71 416 310 317 310 query72 6515353435253525 query73 739 330 341 330 query74 6299588459235884 query75 5292375537513751 query76 5462118711971187 query77 925 259 263 259 query78 12765 12395 11819 11819 query79 12710 651 632 632 query80 863 406 426 406 query81 494 235 238 235 query82 723 103 101 101 query83 178 139 140 139 query84 257 72 71 71 query85 932 343 344 343 query86 349 310 314 310 query87 3282302230213021 query88 5538235023332333 query89 409 311 285 285 query90 2640231 220 220 query91 197 140 143 140 query92 57 53 53 53 query93 5533571 542 542 query94 1291211 205 205 query95 1106107310481048 query96 677 338 333 333 query97 6670639764576397 query98 193 187 175 175 query99 3125848 848 848 Total cold run time: 320226 ms Total hot run time: 205012 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
doris-robot commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241554571 ClickBench: Total hot run time: 30.88 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b5dbef43cbc5431a8029d3504bc6574b99eda7d4, data reload: false query1 0.030.020.02 query2 0.060.030.02 query3 0.250.040.05 query4 1.810.110.07 query5 0.540.520.51 query6 1.230.630.63 query7 0.020.010.01 query8 0.030.030.02 query9 0.520.500.48 query10 0.550.540.56 query11 0.120.090.09 query12 0.110.100.09 query13 0.620.620.61 query14 0.810.790.80 query15 0.790.760.77 query16 0.370.370.37 query17 0.991.010.99 query18 0.240.240.25 query19 1.911.851.86 query20 0.020.010.01 query21 15.46 0.560.55 query22 1.952.192.16 query23 17.35 1.000.94 query24 5.890.601.08 query25 0.370.090.06 query26 0.570.150.14 query27 0.040.040.04 query28 7.570.790.71 query29 12.61 2.282.18 query30 0.560.520.48 query31 2.820.400.38 query32 3.360.500.48 query33 3.093.083.11 query34 15.25 4.834.81 query35 4.874.904.88 query36 1.071.001.02 query37 0.060.040.05 query38 0.030.020.02 query39 0.020.020.01 query40 0.160.140.14 query41 0.060.010.02 query42 0.020.010.02 query43 0.020.020.02 Total cold run time: 104.22 s Total hot run time: 30.88 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
doris-robot commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241555400 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit b5dbef43cbc5431a8029d3504bc6574b99eda7d4 with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.3 seconds inserted 1000 Rows, about 469K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](statistics)Remove old partition stats in the next analyze after insert overwrite. [doris]
Jibing-Li commented on PR #38175: URL: https://github.com/apache/doris/pull/38175#issuecomment-2241561069 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](statistics)Remove old partition stats in the next analyze after insert overwrite. [doris]
Jibing-Li commented on PR #38175: URL: https://github.com/apache/doris/pull/38175#issuecomment-2241561117 run cloud_p1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](inverted index) fixed in_list condition not indexed on pipelinex [doris]
zzzxl1993 opened a new pull request, #38178: URL: https://github.com/apache/doris/pull/38178 ## Proposed changes https://github.com/apache/doris/pull/36565 https://github.com/apache/doris/pull/37842 https://github.com/apache/doris/pull/37921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](inverted index) fixed in_list condition not indexed on pipelinex [doris]
doris-robot commented on PR #38178: URL: https://github.com/apache/doris/pull/38178#issuecomment-2241610549 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](inverted index) fixed in_list condition not indexed on pipelinex [doris]
zzzxl1993 commented on PR #38178: URL: https://github.com/apache/doris/pull/38178#issuecomment-2241610594 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](inverted index) fixed in_list condition not indexed on pipelinex [doris]
github-actions[bot] commented on code in PR #38178:
URL: https://github.com/apache/doris/pull/38178#discussion_r1685740584
##
be/src/pipeline/exec/scan_operator.cpp:
##
@@ -1090,7 +1078,7 @@ void
ScanLocalState::_normalize_compound_predicate(
template
template
-Status ScanLocalState::_normalize_binary_in_compound_predicate(
+Status ScanLocalState::_normalize_binary_compound_predicate(
Review Comment:
warning: function '_normalize_binary_compound_predicate' has cognitive
complexity of 54 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status ScanLocalState::_normalize_binary_compound_predicate(
^
```
Additional context
**be/src/pipeline/exec/scan_operator.cpp:1084:** +1, including nesting
penalty of 0, nesting level increased to 1
```cpp
if (TExprNodeType::BINARY_PRED == expr->node_type()) {
^
```
**be/src/pipeline/exec/scan_operator.cpp:1085:** nesting level increased to 2
```cpp
auto eq_checker = [](const std::string& fn_name) { return fn_name ==
"eq"; };
^
```
**be/src/pipeline/exec/scan_operator.cpp:1086:** nesting level increased to 2
```cpp
auto ne_checker = [](const std::string& fn_name) { return fn_name ==
"ne"; };
^
```
**be/src/pipeline/exec/scan_operator.cpp:1087:** nesting level increased to 2
```cpp
auto noneq_checker = [](const std::string& fn_name) {
^
```
**be/src/pipeline/exec/scan_operator.cpp:1088:** +1
```cpp
return fn_name != "ne" && fn_name != "eq" && fn_name !=
"eq_for_null";
^
```
**be/src/pipeline/exec/scan_operator.cpp:1096:** +2, including nesting
penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_should_push_down_binary_predicate(
^
```
**be/src/common/status.h:609:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/pipeline/exec/scan_operator.cpp:1096:** +3, including nesting
penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_should_push_down_binary_predicate(
^
```
**be/src/common/status.h:611:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/pipeline/exec/scan_operator.cpp:1099:** +2, including nesting
penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_should_push_down_binary_predicate(
^
```
**be/src/common/status.h:609:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/pipeline/exec/scan_operator.cpp:1099:** +3, including nesting
penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_should_push_down_binary_predicate(
^
```
**be/src/common/status.h:611:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/pipeline/exec/scan_operator.cpp:1102:** +2, including nesting
penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_should_push_down_binary_predicate(
^
```
**be/src/common/status.h:609:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/pipeline/exec/scan_operator.cpp:1102:** +3, including nesting
penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_should_push_down_binary_predicate(
^
```
**be/src/common/status.h:611:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/pipeline/exec/scan_operator.cpp:1105:** +2, including nesting
penalty of 1, nesting level increased to 2
```cpp
if (eq_pdt == vectorized::VScanNode::PushDownType::UNACCEPTABLE &&
^
```
**be/src/pipeline/exec/scan_operator.cpp:1106:** +1
```cpp
ne_pdt == vectorized::VScanNode::PushDownType::UNACCEPTABLE &&
^
```
**be/src/pipeline/exec/scan_operator.cpp:1113:** +2, including nesting
penalty of 1, nesting level increased to 2
```cpp
if (eq_pdt == vectorized::VScanNode::PushDownType::ACCEPTABLE ||
^
```
**be/src/pipeline/exec/scan_operator.cpp:1114:** +1
```cpp
ne_pdt == vectorized::VScanNode::PushDownType::ACCEPTABLE ||
^
```
**be/src/pipeline/exec/scan_operator.cpp:1116:** +3, including nesting
penalty of 2, nesting level increased to 3
```cpp
if (value.data != nullpt
Re: [PR] [opt](inverted index) opt value extraction from column to string [doris]
doris-robot commented on PR #38179: URL: https://github.com/apache/doris/pull/38179#issuecomment-2241614340 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](inverted index) opt value extraction from column to string [doris]
zzzxl1993 opened a new pull request, #38179: URL: https://github.com/apache/doris/pull/38179 ## Proposed changes https://github.com/apache/doris/pull/37395 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](nereids) fix normalize repeat alias rewrite [doris]
feiniaofeiafei commented on PR #38166: URL: https://github.com/apache/doris/pull/38166#issuecomment-2241615767 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](statistics)Remove old partition stats in the next analyze after insert overwrite. [doris]
Jibing-Li commented on PR #38175: URL: https://github.com/apache/doris/pull/38175#issuecomment-2241615635 run cloud_p1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) opt value extraction from column to string [doris]
github-actions[bot] commented on PR #38179: URL: https://github.com/apache/doris/pull/38179#issuecomment-2241616259 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](inverted index) mow supports index optimization [doris]
zzzxl1993 opened a new pull request, #38180: URL: https://github.com/apache/doris/pull/38180 ## Proposed changes https://github.com/apache/doris/pull/37428 https://github.com/apache/doris/pull/37429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) mow supports index optimization [doris]
doris-robot commented on PR #38180: URL: https://github.com/apache/doris/pull/38180#issuecomment-2241618257 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](inverted index) fixed in_list condition not indexed on pipelinex [doris]
zzzxl1993 commented on PR #38178: URL: https://github.com/apache/doris/pull/38178#issuecomment-2241618672 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) opt value extraction from column to string [doris]
zzzxl1993 commented on PR #38179: URL: https://github.com/apache/doris/pull/38179#issuecomment-2241618794 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
morningman commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241618955 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) mow supports index optimization [doris]
zzzxl1993 commented on PR #38180: URL: https://github.com/apache/doris/pull/38180#issuecomment-2241619515 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) mow supports index optimization [doris]
github-actions[bot] commented on PR #38180: URL: https://github.com/apache/doris/pull/38180#issuecomment-2241620357 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
github-actions[bot] commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241621087 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](nereids) fix normalize repeat alias rewrite [doris]
doris-robot commented on PR #38166: URL: https://github.com/apache/doris/pull/38166#issuecomment-2241627310 TPC-H: Total hot run time: 39674 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ad9968523f6fd69cb190a79fc0023bcb1ea7c71b, data reload: false -- Round 1 -- q1 17620 432742884288 q2 2017186 189 186 q3 10473 118711041104 q4 10183 838 740 740 q5 7554267826902678 q6 222 135 136 135 q7 950 615 609 609 q8 9226207820362036 q9 8805652765066506 q10 8843381637953795 q11 446 241 234 234 q12 445 233 225 225 q13 17769 297330122973 q14 275 234 236 234 q15 521 475 470 470 q16 491 404 378 378 q17 944 660 688 660 q18 8096755273867386 q19 7819125612271227 q20 685 338 333 333 q21 4868319031983190 q22 350 290 287 287 Total cold run time: 118602 ms Total hot run time: 39674 ms - Round 2, with runtime_filter_mode=off - q1 4390420942464209 q2 368 273 267 267 q3 2985294829272927 q4 2000171917471719 q5 5553551454995499 q6 223 136 141 136 q7 2219186318691863 q8 3249343333983398 q9 8718884088528840 q10 4043381938153815 q11 599 501 512 501 q12 794 644 645 644 q13 15894 314132163141 q14 312 283 301 283 q15 525 489 490 489 q16 496 447 445 445 q17 1815155615091509 q18 7979786178057805 q19 1713153016521530 q20 2193185918831859 q21 4997497847354735 q22 603 495 482 482 Total cold run time: 71668 ms Total hot run time: 56096 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
doris-robot commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241629304 TPC-H: Total hot run time: 49938 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ab59851ae4af3582a18081b55c96728fd1f437ed, data reload: false -- Round 1 -- q1 17596 442943564356 q2 2068157 151 151 q3 10467 188619121886 q4 10308 126513121265 q5 8404391539593915 q6 229 153 126 126 q7 2044160916091609 q8 9268270426902690 q9 10934 10359 10467 10359 q10 8636352934993499 q11 424 247 236 236 q12 473 300 307 300 q13 18324 397140223971 q14 347 349 322 322 q15 503 456 457 456 q16 664 570 578 570 q17 1131946 962 946 q18 7306681268856812 q19 1808164716361636 q20 565 294 318 294 q21 4438410341004100 q22 538 439 446 439 Total cold run time: 116475 ms Total hot run time: 49938 ms - Round 2, with runtime_filter_mode=off - q1 4311434543774345 q2 320 229 223 223 q3 4188414141614141 q4 2775274027432740 q5 7201711771087108 q6 236 120 122 120 q7 3247285828602858 q8 4350459044434443 q9 17414 16966 17219 16966 q10 4217422342504223 q11 764 723 696 696 q12 1042862 895 862 q13 6288378037643764 q14 453 424 449 424 q15 500 465 451 451 q16 731 675 682 675 q17 3901387938823879 q18 8758880188288801 q19 1703171816881688 q20 2384212621182118 q21 8642837684268376 q22 1039987 988 987 Total cold run time: 84464 ms Total hot run time: 79888 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) opt value extraction from column to string [doris]
doris-robot commented on PR #38179: URL: https://github.com/apache/doris/pull/38179#issuecomment-2241629326 TeamCity be ut coverage result: Function Coverage: 36.45% (9236/25341) Line Coverage: 27.99% (75499/269754) Region Coverage: 26.80% (38807/144790) Branch Coverage: 23.56% (19712/83680) Coverage Report: http://coverage.selectdb-in.cc/coverage/d0f247f6c1930a3a52d15593f903c5be2d319d0d_d0f247f6c1930a3a52d15593f903c5be2d319d0d/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](nereids) fix normalize repeat alias rewrite [doris]
doris-robot commented on PR #38166: URL: https://github.com/apache/doris/pull/38166#issuecomment-2241630569 TPC-DS: Total hot run time: 174194 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ad9968523f6fd69cb190a79fc0023bcb1ea7c71b, data reload: false query1 920 366 368 366 query2 6441192118741874 query3 6654208 215 208 query4 28490 17528 17579 17528 query5 3641474 477 474 query6 270 169 160 160 query7 4578291 279 279 query8 247 188 184 184 query9 8600247224582458 query10 449 285 274 274 query11 10535 10087 10030 10030 query12 118 95 83 83 query13 1642379 373 373 query14 10274 832478527852 query15 228 169 168 168 query16 7522500 460 460 query17 1608568 544 544 query18 1690285 288 285 query19 204 161 157 157 query20 91 83 80 80 query21 203 134 132 132 query22 4346419939823982 query23 34038 33827 33542 33542 query24 10664 290029262900 query25 614 422 412 412 query26 717 157 158 157 query27 2243288 282 282 query28 5907212720982098 query29 911 726 625 625 query30 253 157 154 154 query31 1009791 757 757 query32 92 53 54 53 query33 656 352 331 331 query34 900 480 503 480 query35 881 777 761 761 query36 1144966 965 965 query37 146 80 86 80 query38 2940280927872787 query39 905 823 800 800 query40 195 120 117 117 query41 43 47 43 43 query42 121 100 102 100 query43 516 482 456 456 query44 1103733 732 732 query45 192 161 162 161 query46 1081759 739 739 query47 1824176817331733 query48 360 300 291 291 query49 834 403 399 399 query50 792 391 388 388 query51 6768659968156599 query52 108 91 96 91 query53 355 285 312 285 query54 876 451 443 443 query55 77 73 72 72 query56 283 263 256 256 query57 1104104810721048 query58 259 249 252 249 query59 2893280826722672 query60 296 268 266 266 query61 96 90 98 90 query62 811 636 643 636 query63 315 288 286 286 query64 9191221516361636 query65 3155309931053099 query66 745 327 328 327 query67 15455 14977 15014 14977 query68 4582544 543 543 query69 565 362 339 339 query70 1177114511561145 query71 429 278 282 278 query72 8176560955195519 query73 744 323 331 323 query74 6120566556095609 query75 3834267826822678 query76 2666989 869 869 query77 629 310 305 305 query78 10382 906090259025 query79 3360533 522 522 query80 2669471 470 470 query81 600 223 224 223 query82 1348136 136 136 query83 316 166 166 166 query84 326 88 88 88 query85 1521309 287 287 query86 484 312 326 312 query87 3352312931123112 query88 4697237523802375 query89 501 397 376 376 query90 1993191 189 189 query91 127 101 100 100 query92 63 52 53 52 query93 4162509 500 500 query94 1396290 281 281 query95 414 312 315 312 query96 609 272 273 272 query97 3227310730293029 query98 229 209 193 193 query99 1637124012781240 Total cold run time: 284954 ms Total hot run time: 174194 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [Fix](nereids) fix normalize repeat alias rewrite [doris]
doris-robot commented on PR #38166: URL: https://github.com/apache/doris/pull/38166#issuecomment-2241632062 ClickBench: Total hot run time: 30.58 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ad9968523f6fd69cb190a79fc0023bcb1ea7c71b, data reload: false query1 0.050.040.04 query2 0.090.050.04 query3 0.220.040.04 query4 1.690.070.07 query5 0.500.500.49 query6 1.150.730.71 query7 0.020.010.02 query8 0.050.050.04 query9 0.540.500.48 query10 0.540.540.55 query11 0.160.120.12 query12 0.140.110.12 query13 0.590.580.59 query14 0.770.780.78 query15 0.870.810.81 query16 0.360.370.36 query17 1.021.041.02 query18 0.220.210.21 query19 1.851.821.84 query20 0.010.010.01 query21 15.41 0.770.64 query22 4.367.821.94 query23 18.27 1.421.21 query24 2.130.210.22 query25 0.160.090.08 query26 0.300.210.21 query27 0.450.230.22 query28 13.36 1.021.00 query29 12.64 3.273.27 query30 0.250.060.05 query31 2.860.390.39 query32 3.260.470.47 query33 2.872.912.90 query34 17.12 4.304.34 query35 4.374.384.37 query36 0.650.460.46 query37 0.180.160.15 query38 0.150.150.15 query39 0.050.040.04 query40 0.140.120.12 query41 0.090.040.05 query42 0.050.040.04 query43 0.050.050.04 Total cold run time: 110.01 s Total hot run time: 30.58 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
doris-robot commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241632915 TPC-DS: Total hot run time: 203469 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ab59851ae4af3582a18081b55c96728fd1f437ed, data reload: false query1 917 425 376 376 query2 6551272426122612 query3 6918223 209 209 query4 20231 18099 18033 18033 query5 19727 653265296529 query6 288 224 232 224 query7 4169296 309 296 query8 427 403 398 398 query9 3121270426312631 query10 407 319 309 309 query11 11334 10835 10746 10746 query12 120 78 73 73 query13 5606694 702 694 query14 17742 13409 13172 13172 query15 359 247 243 243 query16 6460286 267 267 query17 17401460880 880 query18 2325407 411 407 query19 203 148 154 148 query20 83 79 79 79 query21 192 96 91 91 query22 5177500249984998 query23 32369 31876 31987 31876 query24 6959651364726472 query25 525 426 421 421 query26 524 165 160 160 query27 1866295 302 295 query28 6252237323382338 query29 2902269128132691 query30 252 166 170 166 query31 915 761 738 738 query32 72 67 60 60 query33 389 265 246 246 query34 863 477 488 477 query35 1124934 915 915 query36 1294117213171172 query37 87 61 61 61 query38 3101301129012901 query39 1394131913321319 query40 205 99 97 97 query41 51 43 44 43 query42 90 83 85 83 query43 685 765 699 699 query44 1126711 723 711 query45 252 241 227 227 query46 1228923 995 923 query47 1830167517381675 query48 1018721 713 713 query49 615 366 366 366 query50 858 584 598 584 query51 4729464846604648 query52 90 87 75 75 query53 442 337 326 326 query54 2656244924892449 query55 96 86 87 86 query56 245 219 211 211 query57 1229112811621128 query58 212 198 205 198 query59 4257388437303730 query60 230 214 200 200 query61 100 93 95 93 query62 820 456 467 456 query63 488 340 337 337 query64 2464155114341434 query65 3627354935483548 query66 791 386 384 384 query67 19010 15493 16394 15493 query68 8276660 643 643 query69 569 355 347 347 query70 1665133813051305 query71 397 312 310 310 query72 6689357135153515 query73 732 333 319 319 query74 6279581659285816 query75 4544373236933693 query76 4639118411941184 query77 555 250 258 250 query78 12835 11669 11755 11669 query79 8192636 649 636 query80 2373407 407 407 query81 519 234 236 234 query82 149599 96 96 query83 186 134 131 131 query84 260 71 73 71 query85 1426332 339 332 query86 375 303 291 291 query87 3272304230663042 query88 5223234423582344 query89 362 301 282 282 query90 1745211 210 210 query91 191 143 139 139 query92 57 54 53 53 query93 4734552 567 552 query94 887 208 210 208 query95 1081106110411041 query96 645 328 328 328 query97 6571652363686368 query98 198 178 170 170 query99 2982843 899 843 Total cold run time: 314383 ms Total hot run time: 203469 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [improvement](nereids)Set index initial row count to -1. [doris]
doris-robot commented on PR #38181: URL: https://github.com/apache/doris/pull/38181#issuecomment-2241634175 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](nereids)Set index initial row count to -1. [doris]
Jibing-Li commented on PR #38181: URL: https://github.com/apache/doris/pull/38181#issuecomment-2241634207 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improvement](nereids)Set index initial row count to -1. [doris]
Jibing-Li opened a new pull request, #38181: URL: https://github.com/apache/doris/pull/38181 Set index initial row count to -1. So when a new table created, we can tell the row count of a index is reported at least once or not. When a new table's row count of a index is never reported, the row count of that index would be -1 instead of 0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
doris-robot commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241634458 ClickBench: Total hot run time: 30.57 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ab59851ae4af3582a18081b55c96728fd1f437ed, data reload: false query1 0.020.020.02 query2 0.070.020.02 query3 0.260.050.05 query4 1.800.070.06 query5 0.540.520.52 query6 1.330.620.62 query7 0.010.000.00 query8 0.040.020.02 query9 0.520.510.49 query10 0.550.540.54 query11 0.120.090.09 query12 0.110.090.09 query13 0.640.610.61 query14 0.780.790.82 query15 0.790.770.76 query16 0.380.390.38 query17 1.011.011.02 query18 0.240.260.25 query19 1.871.861.89 query20 0.020.010.00 query21 15.49 0.560.55 query22 2.052.541.57 query23 16.71 0.990.86 query24 5.900.881.12 query25 0.360.120.07 query26 0.630.170.15 query27 0.030.040.04 query28 7.250.760.74 query29 12.68 2.312.14 query30 0.640.540.53 query31 2.810.390.38 query32 3.390.500.49 query33 3.073.073.10 query34 15.27 4.844.80 query35 4.894.854.86 query36 1.051.021.02 query37 0.060.040.05 query38 0.040.020.02 query39 0.020.020.01 query40 0.160.140.14 query41 0.060.020.02 query42 0.020.010.01 query43 0.020.010.02 Total cold run time: 103.7 s Total hot run time: 30.57 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](s3) auto retry when meeting 429 error (#35397) [doris]
doris-robot commented on PR #38176: URL: https://github.com/apache/doris/pull/38176#issuecomment-2241635503 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit ab59851ae4af3582a18081b55c96728fd1f437ed with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.9 seconds inserted 1000 Rows, about 456K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](nereids)Set index initial row count to -1. [doris]
Jibing-Li commented on PR #38181: URL: https://github.com/apache/doris/pull/38181#issuecomment-2241637051 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](nereids)Set index initial row count to -1. [doris]
doris-robot commented on PR #38181: URL: https://github.com/apache/doris/pull/38181#issuecomment-2241660028 TPC-H: Total hot run time: 39917 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 96a61c0b79846ea1da36c8c83253d4f5c8addba0, data reload: false -- Round 1 -- q1 17675 484043674367 q2 2024209 189 189 q3 10498 120710971097 q4 10218 799 810 799 q5 7610275626772677 q6 221 135 134 134 q7 957 596 602 596 q8 9220208220962082 q9 8717657866166578 q10 8872381837733773 q11 457 237 235 235 q12 400 221 226 221 q13 17840 295029562950 q14 275 232 246 232 q15 529 485 494 485 q16 514 382 377 377 q17 990 725 720 720 q18 8167757474107410 q19 8322140612141214 q20 881 340 330 330 q21 4914316432413164 q22 349 292 287 287 Total cold run time: 119650 ms Total hot run time: 39917 ms - Round 2, with runtime_filter_mode=off - q1 4413423342464233 q2 376 261 273 261 q3 3105293829622938 q4 1980175417301730 q5 5600553454605460 q6 219 134 132 132 q7 2242187218241824 q8 3274344334193419 q9 8853887388258825 q10 4155376937943769 q11 600 486 508 486 q12 814 631 629 629 q13 15866 319732253197 q14 337 276 294 276 q15 534 486 485 485 q16 495 428 442 428 q17 1816153514781478 q18 8182795277917791 q19 1739166316521652 q20 2070187518841875 q21 9432474347384738 q22 586 535 513 513 Total cold run time: 76688 ms Total hot run time: 56139 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](tvf)Support enclose and escape parameters when reading csv files using tvf. [doris]
hubgeter commented on PR #37919: URL: https://github.com/apache/doris/pull/37919#issuecomment-2241665252 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](nereids)Set index initial row count to -1. [doris]
Jibing-Li commented on PR #38181: URL: https://github.com/apache/doris/pull/38181#issuecomment-2241669628 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](tvf)Support enclose and escape parameters when reading csv files using tvf. [doris]
github-actions[bot] commented on PR #37919: URL: https://github.com/apache/doris/pull/37919#issuecomment-2241671514 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](tvf)Support enclose and escape parameters when reading csv files using tvf. [doris]
doris-robot commented on PR #37919: URL: https://github.com/apache/doris/pull/37919#issuecomment-2241690666 TPC-H: Total hot run time: 40214 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 514166ec7e8ce87de89a863c342c778973c0f92e, data reload: false -- Round 1 -- q1 17616 441043394339 q2 2024194 191 191 q3 10446 119710711071 q4 10186 803 825 803 q5 7540271927002700 q6 223 141 141 141 q7 956 609 603 603 q8 9231207620912076 q9 9611659566146595 q10 8857388038273827 q11 460 246 249 246 q12 487 229 234 229 q13 19134 303529972997 q14 280 238 240 238 q15 524 511 484 484 q16 532 393 385 385 q17 982 667 673 667 q18 8040742275017422 q19 1599139614281396 q20 658 338 318 318 q21 5078320132613201 q22 354 299 285 285 Total cold run time: 114818 ms Total hot run time: 40214 ms - Round 2, with runtime_filter_mode=off - q1 4358426742464246 q2 375 266 269 266 q3 3003277127442744 q4 1862164616101610 q5 5294535953245324 q6 219 129 133 129 q7 2133172217381722 q8 3189335333403340 q9 8411839584058395 q10 3895369137323691 q11 584 495 483 483 q12 812 614 632 614 q13 16197 295230592952 q14 301 266 256 256 q15 520 475 472 472 q16 475 433 417 417 q17 1820150314741474 q18 7590737174217371 q19 1658149515721495 q20 2024182417951795 q21 4967473147514731 q22 555 493 480 480 Total cold run time: 70242 ms Total hot run time: 54007 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](tvf)Support enclose and escape parameters when reading csv files using tvf. [doris]
doris-robot commented on PR #37919: URL: https://github.com/apache/doris/pull/37919#issuecomment-2241695250 TPC-DS: Total hot run time: 174462 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 514166ec7e8ce87de89a863c342c778973c0f92e, data reload: false query1 910 387 375 375 query2 6430191618841884 query3 6657212 226 212 query4 27652 17971 17522 17522 query5 4199505 484 484 query6 271 181 168 168 query7 4593297 302 297 query8 238 195 201 195 query9 8619250324642464 query10 437 298 271 271 query11 11694 10153 10075 10075 query12 127 97 83 83 query13 1649364 357 357 query14 10429 776474517451 query15 222 169 168 168 query16 7580499 518 499 query17 1418554 527 527 query18 1802276 277 276 query19 199 151 150 150 query20 91 86 82 82 query21 205 131 141 131 query22 4280420241314131 query23 33965 33292 33348 33292 query24 12133 287828752875 query25 651 362 367 362 query26 1815152 151 151 query27 2930272 276 272 query28 7335205220402040 query29 1058626 634 626 query30 292 149 149 149 query31 949 721 734 721 query32 93 52 55 52 query33 751 326 329 326 query34 884 478 486 478 query35 835 760 735 735 query36 1110911 922 911 query37 183 79 84 79 query38 2854272927672729 query39 859 800 827 800 query40 271 122 120 120 query41 51 46 44 44 query42 131 104 107 104 query43 523 475 480 475 query44 1179735 738 735 query45 191 161 167 161 query46 1100728 732 728 query47 1872176917631763 query48 374 296 301 296 query49 1220427 445 427 query50 800 404 406 404 query51 6919663866856638 query52 103 95 101 95 query53 362 295 301 295 query54 937 468 458 458 query55 81 79 78 78 query56 314 291 302 291 query57 1180105310471047 query58 276 260 271 260 query59 2824277627682768 query60 338 300 306 300 query61 118 113 118 113 query62 842 667 675 667 query63 324 302 292 292 query64 10566 226917621762 query65 3170313431343134 query66 1405344 340 340 query67 15973 15039 15171 15039 query68 6135568 559 559 query69 647 473 391 391 query70 1201116611261126 query71 466 289 293 289 query72 7626603557535753 query73 783 327 328 327 query74 6086576057165716 query75 3849270126372637 query76 3615959 860 860 query77 687 328 324 324 query78 10697 896098458960 query79 8631528 531 528 query80 2477495 496 495 query81 613 229 217 217 query82 1428140 130 130 query83 306 171 166 166 query84 273 87 86 86 query85 1593313 307 307 query86 474 332 295 295 query87 3251310931223109 query88 5158238724182387 query89 509 385 376 376 query90 1846194 194 194 query91 131 101 99 99 query92 65 52 53 52 query93 5825517 523 517 query94 1006293 279 279 query95 404 322 321 321 query96 620 275 269 269 query97 3191302130293021 query98 219 199 194 194 query99 1757124413051244 Total cold run time: 303412 ms Total hot run time: 174462 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [feature](tvf)Support enclose and escape parameters when reading csv files using tvf. [doris]
doris-robot commented on PR #37919: URL: https://github.com/apache/doris/pull/37919#issuecomment-2241696645 ClickBench: Total hot run time: 30.43 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 514166ec7e8ce87de89a863c342c778973c0f92e, data reload: false query1 0.050.030.04 query2 0.080.040.04 query3 0.230.050.06 query4 1.670.100.09 query5 0.500.480.48 query6 1.130.720.73 query7 0.020.010.01 query8 0.050.040.04 query9 0.560.490.49 query10 0.550.550.55 query11 0.170.110.12 query12 0.150.130.13 query13 0.600.590.59 query14 0.770.770.78 query15 0.840.810.82 query16 0.360.370.36 query17 1.051.060.97 query18 0.220.220.22 query19 1.841.681.75 query20 0.010.020.01 query21 15.40 0.770.65 query22 4.477.121.67 query23 18.27 1.411.21 query24 2.180.230.21 query25 0.170.100.09 query26 0.300.220.21 query27 0.450.240.24 query28 13.24 1.021.01 query29 12.64 3.333.31 query30 0.260.060.06 query31 2.860.390.40 query32 3.260.480.47 query33 2.952.862.92 query34 17.14 4.424.41 query35 4.504.424.45 query36 0.650.480.49 query37 0.180.150.16 query38 0.150.150.15 query39 0.040.030.04 query40 0.160.130.12 query41 0.100.050.05 query42 0.060.060.05 query43 0.050.040.04 Total cold run time: 110.33 s Total hot run time: 30.43 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](nereids)Set index initial row count to -1. [doris]
doris-robot commented on PR #38181: URL: https://github.com/apache/doris/pull/38181#issuecomment-2241696847 TPC-H: Total hot run time: 39551 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0993bf5371121d75300d8e89ef89f92fc283c266, data reload: false -- Round 1 -- q1 17635 435842344234 q2 2008186 201 186 q3 10448 119311131113 q4 10198 832 800 800 q5 7548264827172648 q6 216 136 135 135 q7 960 588 593 588 q8 9209205420532053 q9 8426652664976497 q10 8812378837453745 q11 489 243 232 232 q12 439 213 227 213 q13 17782 298429972984 q14 279 226 234 226 q15 525 491 502 491 q16 502 383 381 381 q17 956 756 600 600 q18 7971744573347334 q19 8054137013681368 q20 743 315 316 315 q21 4921315631333133 q22 343 284 275 275 Total cold run time: 118464 ms Total hot run time: 39551 ms - Round 2, with runtime_filter_mode=off - q1 4376423842454238 q2 368 270 264 264 q3 2987288229132882 q4 2034166117311661 q5 5566549354825482 q6 221 134 141 134 q7 2238188118161816 q8 3276340833793379 q9 8773885188198819 q10 4075384337653765 q11 585 489 483 483 q12 807 634 637 634 q13 16116 315531773155 q14 294 282 289 282 q15 528 480 489 480 q16 501 448 439 439 q17 1811154714971497 q18 8008793478627862 q19 1778142816851428 q20 2959185118641851 q21 5097471946714671 q22 548 512 510 510 Total cold run time: 72946 ms Total hot run time: 55732 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](nereids)Set index initial row count to -1. [doris]
doris-robot commented on PR #38181: URL: https://github.com/apache/doris/pull/38181#issuecomment-2241700390 TPC-DS: Total hot run time: 174001 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0993bf5371121d75300d8e89ef89f92fc283c266, data reload: false query1 912 375 367 367 query2 6459191417961796 query3 6631204 218 204 query4 23263 17574 17525 17525 query5 3786471 486 471 query6 273 177 176 176 query7 4586292 293 292 query8 242 193 201 193 query9 8568247124532453 query10 443 296 261 261 query11 11407 10173 10091 10091 query12 118 88 83 83 query13 1652367 378 367 query14 10326 756379617563 query15 230 170 164 164 query16 7040506 481 481 query17 1579558 509 509 query18 1691274 280 274 query19 191 151 150 150 query20 88 82 82 82 query21 202 128 127 127 query22 4266399839733973 query23 33862 33901 33558 33558 query24 11234 292728432843 query25 607 387 389 387 query26 1207149 152 149 query27 2676277 275 275 query28 6927208720842084 query29 907 646 629 629 query30 253 157 162 157 query31 973 761 748 748 query32 101 55 55 55 query33 750 329 344 329 query34 930 485 506 485 query35 877 751 750 750 query36 1127989 995 989 query37 147 80 80 80 query38 2945288928212821 query39 897 853 832 832 query40 213 121 119 119 query41 47 44 44 44 query42 110 99 104 99 query43 516 485 473 473 query44 1231735 724 724 query45 198 157 159 157 query46 1083736 751 736 query47 1890179418301794 query48 372 286 293 286 query49 842 398 427 398 query50 776 391 401 391 query51 6859671166636663 query52 111 95 93 93 query53 353 281 286 281 query54 924 440 436 436 query55 75 74 71 71 query56 279 252 277 252 query57 1128105110541051 query58 243 244 252 244 query59 2718272726342634 query60 303 282 270 270 query61 95 96 97 96 query62 826 640 636 636 query63 324 299 293 293 query64 9917228718051805 query65 3152312631063106 query66 914 324 322 322 query67 15506 15025 14943 14943 query68 4463536 541 536 query69 465 340 335 335 query70 1202113611451136 query71 371 284 272 272 query72 7012554657035546 query73 736 322 324 322 query74 6071566956215621 query75 3450269726872687 query76 2619915 918 915 query77 434 299 298 298 query78 10100 909089518951 query79 2267530 530 530 query80 2084467 470 467 query81 584 220 216 216 query82 882 135 135 135 query83 280 167 163 163 query84 275 89 85 85 query85 2052381 311 311 query86 482 330 320 320 query87 3263312330873087 query88 4198236523522352 query89 475 386 374 374 query90 1789191 189 189 query91 126 98 99 98 query92 60 47 47 47 query93 2487505 502 502 query94 1190300 296 296 query95 404 315 311 311 query96 595 285 274 274 query97 3204299830412998 query98 217 195 198 195 query99 1608126112421242 Total cold run time: 276823 ms Total hot run time: 174001 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [improvement](nereids)Set index initial row count to -1. [doris]
doris-robot commented on PR #38181: URL: https://github.com/apache/doris/pull/38181#issuecomment-2241701901 ClickBench: Total hot run time: 30.13 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 0993bf5371121d75300d8e89ef89f92fc283c266, data reload: false query1 0.040.040.04 query2 0.080.040.04 query3 0.230.050.05 query4 1.680.080.08 query5 0.490.490.49 query6 1.140.720.72 query7 0.020.010.01 query8 0.050.040.04 query9 0.550.510.50 query10 0.540.540.54 query11 0.150.110.12 query12 0.140.110.12 query13 0.600.590.59 query14 0.750.780.78 query15 0.860.810.81 query16 0.360.370.35 query17 1.030.960.96 query18 0.220.210.21 query19 1.821.681.72 query20 0.010.010.01 query21 15.39 0.730.64 query22 4.367.581.52 query23 18.27 1.381.25 query24 2.080.230.23 query25 0.180.090.08 query26 0.290.220.21 query27 0.450.240.23 query28 13.33 1.020.99 query29 12.57 3.333.29 query30 0.250.060.05 query31 2.860.390.39 query32 3.280.470.47 query33 2.902.912.94 query34 17.10 4.334.39 query35 4.514.384.40 query36 0.650.460.48 query37 0.180.160.15 query38 0.150.150.16 query39 0.040.030.04 query40 0.160.120.13 query41 0.090.040.05 query42 0.060.050.06 query43 0.050.040.04 Total cold run time: 109.96 s Total hot run time: 30.13 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 02/26: [Improvement](scan) use loop to instead recursion on Level1Iterator::_normal_next (#38005)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0cc64a3a63c0b423864eb32e43b752013daf884b
Author: Pxl
AuthorDate: Fri Jul 19 10:15:04 2024 +0800
[Improvement](scan) use loop to instead recursion on
Level1Iterator::_normal_next (#38005)
## Proposed changes
use loop to instead recursion on Level1Iterator::_normal_next
---
be/src/vec/olap/vcollect_iterator.cpp | 18 +-
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/be/src/vec/olap/vcollect_iterator.cpp
b/be/src/vec/olap/vcollect_iterator.cpp
index 61050979b84..ceeede7c7c3 100644
--- a/be/src/vec/olap/vcollect_iterator.cpp
+++ b/be/src/vec/olap/vcollect_iterator.cpp
@@ -857,18 +857,18 @@ Status
VCollectIterator::Level1Iterator::_merge_next(Block* block) {
Status VCollectIterator::Level1Iterator::_normal_next(Block* block) {
SCOPED_RAW_TIMER(&_reader->_stats.collect_iterator_normal_next_timer);
auto res = _cur_child->next(block);
+
+while (res.is() && !_children.empty()) {
+_cur_child = std::move(*(_children.begin()));
+_children.pop_front();
+res = _cur_child->next(block);
+}
+
if (LIKELY(res.ok())) {
return Status::OK();
} else if (res.is()) {
-// current child has been read, to read next
-if (!_children.empty()) {
-_cur_child = std::move(*(_children.begin()));
-_children.pop_front();
-return _normal_next(block);
-} else {
-_cur_child.reset();
-return Status::Error("");
-}
+_cur_child.reset();
+return Status::Error("");
} else {
_cur_child.reset();
LOG(WARNING) << "failed to get next from child, res=" << res;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/26: [regression](limit) Add group by limit regression test case (#37940)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 19e5fd77c25239fe0428a7f5d821a9b2a5704f9f
Author: HappenLee
AuthorDate: Fri Jul 19 09:54:55 2024 +0800
[regression](limit) Add group by limit regression test case (#37940)
Add group by limit regression test case
---
be/src/common/config.cpp | 2 +
be/src/common/config.h | 4 ++
be/src/pipeline/exec/aggregation_sink_operator.cpp | 3 +-
.../data/query_p0/limit/test_group_by_limit.out| 66 ++
.../query_p0/limit/test_group_by_limit.groovy | 64 +
5 files changed, 138 insertions(+), 1 deletion(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 3e9203987c2..b152111011e 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1343,6 +1343,8 @@ DEFINE_mBool(ignore_not_found_file_in_external_table,
"true");
DEFINE_mBool(enable_hdfs_mem_limiter, "true");
+DEFINE_mInt16(topn_agg_limit_multiplier, "2");
+
// clang-format off
#ifdef BE_TEST
// test s3
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 1ce9c66939c..f4ed1decaa0 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1435,6 +1435,10 @@ DECLARE_mBool(ignore_not_found_file_in_external_table);
DECLARE_mBool(enable_hdfs_mem_limiter);
+// Define how many percent data in hashtable bigger than limit
+// we should do agg limit opt
+DECLARE_mInt16(topn_agg_limit_multiplier);
+
#ifdef BE_TEST
// test s3
DECLARE_String(test_s3_resource);
diff --git a/be/src/pipeline/exec/aggregation_sink_operator.cpp
b/be/src/pipeline/exec/aggregation_sink_operator.cpp
index 79ca07281d9..f3a6942c33f 100644
--- a/be/src/pipeline/exec/aggregation_sink_operator.cpp
+++ b/be/src/pipeline/exec/aggregation_sink_operator.cpp
@@ -503,7 +503,8 @@ Status
AggSinkLocalState::_execute_with_serialized_key_helper(vectorized::Block*
_shared_state->reach_limit =
hash_table_size >=
(_shared_state->do_sort_limit
- ? Base::_parent->template
cast()._limit * 5
+ ? Base::_parent->template
cast()._limit *
+ config::topn_agg_limit_multiplier
: Base::_parent->template
cast()._limit);
if (_shared_state->reach_limit &&
_shared_state->do_sort_limit) {
_shared_state->build_limit_heap(hash_table_size);
diff --git a/regression-test/data/query_p0/limit/test_group_by_limit.out
b/regression-test/data/query_p0/limit/test_group_by_limit.out
new file mode 100644
index 000..d9ac2a2481a
--- /dev/null
+++ b/regression-test/data/query_p0/limit/test_group_by_limit.out
@@ -0,0 +1,66 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select --
+253967024 8491AIR
+259556658 8641FOB
+260402265 8669MAIL
+
+-- !select --
+449872500 15000 1
+386605746 12900 2
+320758616 10717 3
+
+-- !select --
+198674527 65880.0
+198679731 65630.01
+198501055 66220.02
+
+-- !select --
+27137 1 1992-02-02
+45697 1 1992-02-04
+114452 5 1992-02-05
+
+-- !select --
+27137 1 1992-02-02T00:00
+45697 1 1992-02-04T00:00
+114452 5 1992-02-05T00:00
+
+-- !select --
+139015016 46321
+130287219 43132
+162309750 53343
+
+-- !select --
+64774969 2166AIR 1
+54166166 1804AIR 2
+45538267 1532AIR 3
+
+-- !select --
+6882631228 AIR 1 0.0
+6756423228 AIR 1 0.01
+7920028254 AIR 1 0.02
+
+-- !select --
+7618 1 AIR 1 0.0 1992-02-06
+2210 1 AIR 1 0.0 1992-03-24
+16807 1 AIR 1 0.0 1992-03-29
+
+-- !select --
+6882631228 AIR 1 0.0
+6756423228 AIR 1 0.01
+7920028254 AIR 1 0.02
+
+-- !select --
+6882631228 AIR 1 0.0
+6756423228 AIR 1 0.01
+7920028254 AIR 1 0.02
+
+-- !select --
+7707018238 TRUCK 1 0.0
+7467045233 TRUCK 1 0.01
+6927206245 TRUCK 1 0.02
+
+-- !select --
+7661562249 TRUCK 1 0.08
+6673139228 TRUCK 1 0.07
+8333862265 TRUCK 1 0.06
+
diff --git a/regression-test/suites/query_p0/limit/test_group_by_limit.groovy
b/regression-test/suites/query_p0/limit/test_group_by_limit.groovy
new file mode 100644
index 000..271619c4a93
--- /dev/null
+++ b/regression-test/suites/query_p0/limit/test_group_by_limit.groovy
@@ -
(doris) 05/26: [fix](cloud) To ensure compatibility with the cloud upgrade (#37708)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2261a094be66250d71cc3c1eb38dce4a1d190c14
Author: deardeng <565620...@qq.com>
AuthorDate: Fri Jul 19 12:03:37 2024 +0800
[fix](cloud) To ensure compatibility with the cloud upgrade (#37708)
Due to the cloud version being significantly outdated, to ensure
compatibility with the cloud version, the version value has been lowered
to 123
---
.../org/apache/doris/mysql/privilege/Role.java | 94 --
1 file changed, 52 insertions(+), 42 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
index 2cea07bec0b..4146da80827 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
@@ -1102,48 +1102,58 @@ public class Role implements Writable,
GsonPostProcessable {
}
private void compatibilityErrEnum() {
-if (Env.getCurrentEnvJournalVersion() >= FeMetaVersion.VERSION_129) {
-if (Config.isNotCloudMode()) {
-// not cloud mode,
-// SHOW_VIEW_PRIV_DEPRECATED -> SHOW_VIEW_PRIV (9 -> 14)
-tblPatternToPrivs.values().forEach(privBitSet -> {
-if
(privBitSet.containsPrivs(Privilege.SHOW_VIEW_PRIV_DEPRECATED)) {
-// remove SHOW_VIEW_PRIV_DEPRECATED
-
privBitSet.unset(Privilege.SHOW_VIEW_PRIV_DEPRECATED.getIdx());
-// add SHOW_VIEW_PRIV
-privBitSet.set(Privilege.SHOW_VIEW_PRIV.getIdx());
-}
-});
-} else {
-// cloud mode
-// CLUSTER_USAGE_PRIV_DEPRECATED -> CLUSTER_USAGE_PRIV (9 ->
12)
-clusterPatternToPrivs.values().forEach(privBitSet -> {
-if
(privBitSet.containsPrivs(Privilege.CLUSTER_USAGE_PRIV_DEPRECATED)) {
-// remove CLUSTER_USAGE_PRIV_DEPRECATED
-
privBitSet.unset(Privilege.CLUSTER_USAGE_PRIV_DEPRECATED.getIdx());
-// add CLUSTER_USAGE_PRIV
-privBitSet.set(Privilege.CLUSTER_USAGE_PRIV.getIdx());
-}
-});
-// STAGE_USAGE_PRIV_DEPRECATED -> STAGE_USAGE_PRIV (10 -> 13)
-stagePatternToPrivs.values().forEach(privBitSet -> {
-if
(privBitSet.containsPrivs(Privilege.STAGE_USAGE_PRIV_DEPRECATED)) {
-// remove CLUSTER_USAGE_PRIV_DEPRECATED
-
privBitSet.unset(Privilege.STAGE_USAGE_PRIV_DEPRECATED.getIdx());
-// add CLUSTER_USAGE_PRIV
-privBitSet.set(Privilege.STAGE_USAGE_PRIV.getIdx());
-}
-});
-// SHOW_VIEW_PRIV_CLOUD_DEPRECATED -> SHOW_VIEW_PRIV (11 -> 14)
-tblPatternToPrivs.values().forEach(privBitSet -> {
-if
(privBitSet.containsPrivs(Privilege.SHOW_VIEW_PRIV_CLOUD_DEPRECATED)) {
-// remove SHOW_VIEW_PRIV_CLOUD_DEPRECATED
-
privBitSet.unset(Privilege.SHOW_VIEW_PRIV_CLOUD_DEPRECATED.getIdx());
-// add SHOW_VIEW_PRIV
-privBitSet.set(Privilege.SHOW_VIEW_PRIV.getIdx());
-}
-});
-}
+int currentVersion = Env.getCurrentEnvJournalVersion();
+if (currentVersion < FeMetaVersion.VERSION_123) {
+// For versions lower than VERSION_123, neither the community nor
the cloud requires compatibility logic.
+return;
+}
+
+LOG.info("auth into compatibility logic, currentVersion={}",
currentVersion);
+if (Config.isNotCloudMode() && currentVersion >=
FeMetaVersion.VERSION_129) {
+// not cloud mode,
+// For versions greater than VERSION_123,
+// the community requires versions above VERSION_129 to follow
compatibility logic.
+
+// SHOW_VIEW_PRIV_DEPRECATED -> SHOW_VIEW_PRIV (9 -> 14)
+tblPatternToPrivs.values().forEach(privBitSet -> {
+if
(privBitSet.containsPrivs(Privilege.SHOW_VIEW_PRIV_DEPRECATED)) {
+// remove SHOW_VIEW_PRIV_DEPRECATED
+
privBitSet.unset(Privilege.SHOW_VIEW_PRIV_DEPRECATED.getIdx());
+// add SHOW_VIEW_PRIV
+privBitSet.set(Privilege.SHOW_VIEW_PRIV.getIdx());
+}
+});
+} else {
+// cloud mode
+// For versions greater than VERSION_123, the cloud requires
compatibility logic.
+
+// CLUSTER_USAGE_PRIV_
(doris) 07/26: [Fix](function) fix some date function impl in FE with special dates (#37766)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 467821a5d96f6da9dd2d810fe17cf73d05a14ed8
Author: zclllhhjj
AuthorDate: Fri Jul 19 14:07:18 2024 +0800
[Fix](function) fix some date function impl in FE with special dates
(#37766)
make `dayofyear`, `dayofweek`, `weekofyear` result of FE folding
constant result same with MySQL. mainly about date in BC01 (year ``)
before:
```sql
mysql> select
DAYOFWEEK('-01-01'),DAYOFWEEK('-01-02'),DAYOFWEEK('-01-03'),DAYOFWEEK('-01-04'),DAYOFWEEK('-01-05'),DAYOFWEEK('-01-06'),DAYOFWEEK('-01-07'),DAYOFWEEK('-01-08');
+--+--+--+--+--+--+--+--+
| 7| 1| 2| 3| 4| 5| 6| 7|
+--+--+--+--+--+--+--+--+
|7 |1 |2 |3 |4 |5 |6 |7 |
+--+--+--+--+--+--+--+--+
mysql> select
DAYOFYEAR('-02-27'),DAYOFYEAR('-02-28'),DAYOFYEAR('-03-01'),DAYOFYEAR('-03-02');
+--+--+--+--+
| 58 | 59 | 61 | 62 |
+--+--+--+--+
| 58 | 59 | 61 | 62 |
+--+--+--+--+
mysql> select
WEEKOFYEAR('-01-01'),WEEKOFYEAR('-01-02'),WEEKOFYEAR('-01-03'),WEEKOFYEAR('-01-04'),WEEKOFYEAR('-01-05'),WEEKOFYEAR('-01-06'),WEEKOFYEAR('-01-07'),WEEKOFYEAR('-01-08');
+--+--+--+--+--+--+--+--+
| 52 | 52 | 1| 1| 1| 1| 1| 1|
+--+--+--+--+--+--+--+--+
| 52 | 52 |1 |1 |1 |1 |1 |1 |
+--+--+--+--+--+--+--+--+
mysql> select
WEEKOFYEAR('-02-25'),WEEKOFYEAR('-02-26'),WEEKOFYEAR('-02-27'),WEEKOFYEAR('-02-28'),WEEKOFYEAR('-03-01'),WEEKOFYEAR('-03-02'),WEEKOFYEAR('2022-03-03');
+--+--+--+--+--+--+--+
| 8| 8| 8| 9| 9| 9| 9|
+--+--+--+--+--+--+--+
|8 |8 |8 |9 |9 |9 |9 |
+--+--+--+--+--+--+--+
```
after:
```sql
mysql> select
DAYOFWEEK('-01-01'),DAYOFWEEK('-01-02'),DAYOFWEEK('-01-03'),DAYOFWEEK('-01-04'),DAYOFWEEK('-01-05'),DAYOFWEEK('-01-06'),DAYOFWEEK('-01-07'),DAYOFWEEK('-01-08');
+--+--+--+--+--+--+--+--+
| 1| 2| 3| 4| 5| 6| 7| 1|
+--+--+--+--+--+--+--+--+
|1 |2 |3 |4 |5 |6 |7 |1 |
+--+--+--+--+--+--+--+--+
mysql> select
DAYOFYEAR('-02-27'),DAYOFYEAR('-02-28'),DAYOFYEAR('-03-01'),DAYOFYEAR('-03-02');
+--+--+--+--+
| 58 | 59 | 60 | 61 |
+--+--+--+--+
| 58 | 59 | 60 | 61 |
+--+--+--+--+
mysql> select
WEEKOFYEAR('-01-01'),WEEKOFYEAR('-01-02'),WEEKOFYEAR('-01-03'),WEEKOFYEAR('-01-04'),WEEKOFYEAR('-01-05'),WEEKOFYEAR('-01-06'),WEEKOFYEAR('-01-07'),WEEKOFYEAR('-01-08');
+--+--+--+--+--+--+--+--+
| 52 | 1| 1| 1| 1| 1| 1| 1|
+--+--+--+--+--+--+--+--+
| 52 |1 |1 |1 |1 |1 |1 |1 |
+--+--+--+--+--+--+--+--+
mysql> select
WEEKOFYEAR('-02-25'),WEEKOFYEAR('-02-26'),WEEKOFYEAR('-02-27'),WEEKOFYEAR('-02-28'),WEEKOFYEAR('-03-01'),WEEKOFYEAR('-03-02'),WEEKOFYEAR('2022-03-03');
+--+--+--+--+--+--+--+
| 8| 8| 9| 9| 9| 9| 9|
+--+--+--+--+--+--+--+
|8 |8 |9 |9 |9 |9 |9 |
+--+--+--+--+--+--+--+
```
---
.../executable/DateTimeExtractAndTransform.java| 45 ++---
.../trees/expressions/literal/DateLiteral.java | 15 ++
.../functions/DateTimeExtractAndTransformTest.java | 57 ++
.../data/correctness_p0/test_cast_date_decimal.out | 5 +-
.../correctness_p0/test_cast_date_decimal.groovy | 6 ++-
5 files changed, 118 insertions(+), 10 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/DateTimeExtractAndTransform.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/DateTimeExtractAndTransform.java
index c14b372f201..9742602a07a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/tre
(doris) 08/26: [fix](inverted index)Support rename column with inverted index (#38079)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8aac782421f8954d1d06bafb7cefc79ad1403b38
Author: qiye
AuthorDate: Fri Jul 19 14:28:40 2024 +0800
[fix](inverted index)Support rename column with inverted index (#38079)
Copy columns and properties to make them mutable.
close #38078
---
fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
index bfe1beb1dd1..03cb3f41ca0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
@@ -29,6 +29,8 @@ import org.apache.doris.proto.OlapFile;
import org.apache.doris.thrift.TIndexType;
import org.apache.doris.thrift.TOlapTableIndex;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
import com.google.gson.annotations.SerializedName;
import org.apache.commons.lang3.StringUtils;
@@ -68,9 +70,9 @@ public class Index implements Writable {
IndexDef.IndexType indexType, Map properties,
String comment) {
this.indexId = indexId;
this.indexName = indexName;
-this.columns = columns;
+this.columns = columns == null ? Lists.newArrayList() :
Lists.newArrayList(columns);
this.indexType = indexType;
-this.properties = properties;
+this.properties = properties == null ? Maps.newHashMap() :
Maps.newHashMap(properties);
this.comment = comment;
if (indexType == IndexDef.IndexType.INVERTED) {
if (this.properties != null && !this.properties.isEmpty()) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 09/26: [fix](trino-connector) Refactor some code of trino-connector jni reader (#37302)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d27992f84411e2b6d2d4c6265e5c64a8081547db
Author: Tiewei Fang <43782773+bepppo...@users.noreply.github.com>
AuthorDate: Fri Jul 19 14:37:51 2024 +0800
[fix](trino-connector) Refactor some code of trino-connector jni reader
(#37302)
Previously, TrinoConnectorJniScanner converted Trino types to Hive
types, and then to Doris types.
In fact, the Hive type can be directly obtained from `columns_types`.
In this way, the step of converting Trino types to Hive types can be
deleted.
---
.../format/table/trino_connector_jni_reader.cpp| 5 +-
.../trinoconnector/TrinoConnectorJniScanner.java | 16 +--
.../TrinoTypeToHiveTypeTranslator.java | 115 -
3 files changed, 12 insertions(+), 124 deletions(-)
diff --git a/be/src/vec/exec/format/table/trino_connector_jni_reader.cpp
b/be/src/vec/exec/format/table/trino_connector_jni_reader.cpp
index b72da757629..eaced8572b9 100644
--- a/be/src/vec/exec/format/table/trino_connector_jni_reader.cpp
+++ b/be/src/vec/exec/format/table/trino_connector_jni_reader.cpp
@@ -43,9 +43,11 @@ TrinoConnectorJniReader::TrinoConnectorJniReader(
RuntimeProfile* profile, const TFileRangeDesc& range)
: JniReader(file_slot_descs, state, profile) {
std::vector column_names;
+std::vector column_types;
for (const auto& desc : _file_slot_descs) {
std::string field = desc->col_name();
column_names.emplace_back(field);
+column_types.emplace_back(JniConnector::get_jni_type(desc->type()));
}
std::map params = {
{"catalog_name",
range.table_format_params.trino_connector_params.catalog_name},
@@ -63,7 +65,8 @@ TrinoConnectorJniReader::TrinoConnectorJniReader(
range.table_format_params.trino_connector_params.trino_connector_predicate},
{"trino_connector_trascation_handle",
range.table_format_params.trino_connector_params.trino_connector_trascation_handle},
-{"required_fields", join(column_names, ",")}};
+{"required_fields", join(column_names, ",")},
+{"columns_types", join(column_types, "#")}};
// Used to create trino connector options
for (const auto& kv :
diff --git
a/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorJniScanner.java
b/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorJniScanner.java
index af20668081d..6b15abfa413 100644
---
a/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorJniScanner.java
+++
b/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorJniScanner.java
@@ -129,11 +129,16 @@ public class TrinoConnectorJniScanner extends JniScanner {
public TrinoConnectorJniScanner(int batchSize, Map params)
{
-catalogNameString = params.get("catalog_name");
-super.batchSize = batchSize;
-super.fields = params.get("required_fields").split(",");
+String[] requiredFields = params.get("required_fields").split(",");
+String[] requiredTypes = params.get("columns_types").split("#");
+ColumnType[] columnTypes = new ColumnType[requiredTypes.length];
+for (int i = 0; i < requiredTypes.length; i++) {
+columnTypes[i] = ColumnType.parseType(requiredFields[i],
requiredTypes[i]);
+}
+initTableInfo(columnTypes, requiredFields, batchSize);
appendDataTimeNs = new long[fields.length];
+catalogNameString = params.get("catalog_name");
connectorSplitString = params.get("trino_connector_split");
connectorTableHandleString =
params.get("trino_connector_table_handle");
connectorColumnHandleString =
params.get("trino_connector_column_handles");
@@ -336,7 +341,6 @@ public class TrinoConnectorJniScanner extends JniScanner {
}
private void parseRequiredTypes() {
-ColumnType[] columnTypes = new ColumnType[fields.length];
appendDataTimeNs = new long[fields.length];
trinoTypeList = Lists.newArrayList();
for (int i = 0; i < fields.length; i++) {
@@ -345,12 +349,8 @@ public class TrinoConnectorJniScanner extends JniScanner {
throw new RuntimeException(String.format("Cannot find field %s
in schema %s",
fields[i], trinoConnectorAllFieldNames));
}
-
trinoTypeList.add(columnMetadataList.get(index).getType());
-String hiveType =
TrinoTypeToHiveTypeTranslator.fromTrinoTypeToHiveType(trinoTypeList.get(i));
-columnTypes[i] = ColumnType.parseType(fields[i], hiveType);
}
-super.types = columnTypes;
(doris) 10/26: [Regression](inverted index) fix build index case for wrong debug point (#38112)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6acfc8de691cff301abafbddb030b91fd275a0e1
Author: airborne12
AuthorDate: Fri Jul 19 14:53:02 2024 +0800
[Regression](inverted index) fix build index case for wrong debug point
(#38112)
## Proposed changes
Fix wrong debug point in test case
---
.../test_index_builder_drop_index_fault_injection.groovy | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/regression-test/suites/fault_injection_p0/test_index_builder_drop_index_fault_injection.groovy
b/regression-test/suites/fault_injection_p0/test_index_builder_drop_index_fault_injection.groovy
index e4a620aa9d1..f8501f0e851 100644
---
a/regression-test/suites/fault_injection_p0/test_index_builder_drop_index_fault_injection.groovy
+++
b/regression-test/suites/fault_injection_p0/test_index_builder_drop_index_fault_injection.groovy
@@ -31,7 +31,7 @@ suite("test_index_builder_drop_index_fault_injection",
"nonConcurrent") {
assertEquals(show_result[3].Key_name, "index_k5")
try {
-
GetDebugPoint().enableDebugPointForAllBEs("segment_iterator._read_columns_by_index",
[indexes_count: 3])
+
GetDebugPoint().enableDebugPointForAllBEs("index_builder.update_inverted_index_info.drop_index",
[indexes_count: 3])
sql "DROP INDEX index_int ON ${indexTbName}"
show_result = sql_return_maparray "show index from ${indexTbName}"
logger.info("show index from " + indexTbName + " result: " +
show_result)
@@ -44,7 +44,7 @@ suite("test_index_builder_drop_index_fault_injection",
"nonConcurrent") {
}
try {
-
GetDebugPoint().enableDebugPointForAllBEs("segment_iterator._read_columns_by_index",
[indexes_count: 2])
+
GetDebugPoint().enableDebugPointForAllBEs("index_builder.update_inverted_index_info.drop_index",
[indexes_count: 2])
sql "DROP INDEX index_str_k2 ON ${indexTbName}"
show_result = sql_return_maparray "show index from ${indexTbName}"
logger.info("show index from " + indexTbName + " result: " +
show_result)
@@ -56,7 +56,7 @@ suite("test_index_builder_drop_index_fault_injection",
"nonConcurrent") {
}
try {
-
GetDebugPoint().enableDebugPointForAllBEs("segment_iterator._read_columns_by_index",
[indexes_count: 1])
+
GetDebugPoint().enableDebugPointForAllBEs("index_builder.update_inverted_index_info.drop_index",
[indexes_count: 1])
sql "DROP INDEX index_str_k4 ON ${indexTbName}"
show_result = sql_return_maparray "show index from ${indexTbName}"
logger.info("show index from " + indexTbName + " result: " +
show_result)
@@ -67,7 +67,7 @@ suite("test_index_builder_drop_index_fault_injection",
"nonConcurrent") {
}
try {
-
GetDebugPoint().enableDebugPointForAllBEs("segment_iterator._read_columns_by_index",
[indexes_count: 0])
+
GetDebugPoint().enableDebugPointForAllBEs("index_builder.update_inverted_index_info.drop_index",
[indexes_count: 0])
sql "DROP INDEX index_k5 ON ${indexTbName}"
show_result = sql_return_maparray "show index from ${indexTbName}"
logger.info("show index from " + indexTbName + " result: " +
show_result)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 17/26: [fix](test) use waitForSchemaChangeDone instead of custom sleeping (#38105)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2cc901d3b9badeec371c04c4a812835a0e06302b
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Fri Jul 19 16:34:45 2024 +0800
[fix](test) use waitForSchemaChangeDone instead of custom sleeping (#38105)
---
.../test_alter_table_add_columns.groovy| 22 ++
.../test_alter_table_drop_column.groovy| 30 +++-
.../test_alter_table_modify_column.groovy | 28 ++-
.../test_schema_change_duplicate.groovy| 86 ++
.../schema_change_p2/test_schema_change.groovy | 19 +
5 files changed, 49 insertions(+), 136 deletions(-)
diff --git
a/regression-test/suites/schema_change_p0/test_alter_table_add_columns.groovy
b/regression-test/suites/schema_change_p0/test_alter_table_add_columns.groovy
index 33484f95bf8..f5da136bdd6 100644
---
a/regression-test/suites/schema_change_p0/test_alter_table_add_columns.groovy
+++
b/regression-test/suites/schema_change_p0/test_alter_table_add_columns.groovy
@@ -56,26 +56,14 @@ suite("test_alter_table_add_columns") {
qt_order """ select * from ${tbName} order by siteid"""
// Add one value column light schema change is false
-sleep(1000)
+
sql """ alter table ${tbName} ADD COLUMN (new_k1 INT DEFAULT '1', new_k2
INT DEFAULT '2');"""
-def waitSchemaChangeJob = { String tableName /* param */ ->
-int tryTimes = 30
-while (tryTimes-- > 0) {
-def jobResult = sql """SHOW ALTER TABLE COLUMN WHERE
IndexName='${tableName}' ORDER BY createtime DESC LIMIT 1 """
-def jobState = jobResult[0][9].toString()
-if ('cancelled'.equalsIgnoreCase(jobState)) {
-logger.info("jobResult:{}", jobResult)
-throw new IllegalStateException("${tableName}'s job has been
cancelled")
-}
-if ('finished'.equalsIgnoreCase(jobState)) {
-logger.info("jobResult:{}", jobResult)
-break
-}
-sleep(1)
-}
+
+waitForSchemaChangeDone {
+sql """ SHOW ALTER TABLE COLUMN WHERE TableName='${tbName}' ORDER BY
createtime DESC LIMIT 1 """
+time 600
}
-waitSchemaChangeJob(tbName)
qt_sql """ DESC ${tbName}"""
sql """ INSERT INTO ${tbName} VALUES
diff --git
a/regression-test/suites/schema_change_p0/test_alter_table_drop_column.groovy
b/regression-test/suites/schema_change_p0/test_alter_table_drop_column.groovy
index 163f1267aae..bd600d81698 100644
---
a/regression-test/suites/schema_change_p0/test_alter_table_drop_column.groovy
+++
b/regression-test/suites/schema_change_p0/test_alter_table_drop_column.groovy
@@ -75,26 +75,12 @@ suite("test_alter_table_drop_column") {
exception "Can not drop key column in Unique data model table"
}
-def waitSchemaChangeJob = { String tableName /* param */ ->
-int tryTimes = 30
-while (tryTimes-- > 0) {
-def jobResult = sql """SHOW ALTER TABLE COLUMN WHERE
IndexName='${tableName}' ORDER BY createtime DESC LIMIT 1 """
-def jobState = jobResult[0][9].toString()
-if ('cancelled'.equalsIgnoreCase(jobState)) {
-logger.info("jobResult:{}", jobResult)
-throw new IllegalStateException("${tableName}'s job has been
cancelled")
-}
-if ('finished'.equalsIgnoreCase(jobState)) {
-logger.info("jobResult:{}", jobResult)
-return
-}
-sleep(1)
-}
-assertTrue(false)
-}
-
sql """ alter table ${uniqueTableName} drop COLUMN pv from
${uniqueTableRollupName};"""
-waitSchemaChangeJob(uniqueTableName)
+
+waitForSchemaChangeDone {
+sql """ SHOW ALTER TABLE COLUMN WHERE TableName='${uniqueTableName}'
ORDER BY createtime DESC LIMIT 1 """
+time 600
+}
qt_order """ select * from ${uniqueTableName} order by siteid"""
qt_order """ select * from ${uniqueTableName} order by citycode"""
@@ -138,7 +124,11 @@ suite("test_alter_table_drop_column") {
}
sql """ alter table ${aggTableName} drop COLUMN pv from
${aggTableRollupName};"""
-waitSchemaChangeJob(aggTableName)
+
+waitForSchemaChangeDone {
+sql """ SHOW ALTER TABLE COLUMN WHERE TableName='${aggTableName}'
ORDER BY createtime DESC LIMIT 1 """
+time 600
+}
qt_order """ select * from ${aggTableName} order by siteid"""
qt_order """ select * from ${aggTableName} order by citycode"""
diff --git
a/regression-test/suites/schema_change_p0/test_alter_table_modify_column.groovy
b/regression-test/suites/schema_change_p0/test_alter_table_modify_column.groovy
index 405b28fefe3..3920086bf6c 100644
---
a/regression-test/suites/schema_change_p0/test_alter_table_modi
(doris) 15/26: [enhance](mtmv)mtmv date trunc support hour (#37678)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit cf689d705498a258d80c8c1dd6eb34f82609ec7d
Author: zhangdong <493738...@qq.com>
AuthorDate: Fri Jul 19 15:48:08 2024 +0800
[enhance](mtmv)mtmv date trunc support hour (#37678)
before support:year", "month", "day"
now add:hour
---
.../doris/mtmv/MTMVPartitionExprDateTrunc.java | 5 ++-
.../mtmv_p0/test_rollup_partition_mtmv.groovy | 39 --
2 files changed, 32 insertions(+), 12 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionExprDateTrunc.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionExprDateTrunc.java
index a42fae22585..f16252cede0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionExprDateTrunc.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionExprDateTrunc.java
@@ -47,7 +47,7 @@ import java.util.Optional;
import java.util.Set;
public class MTMVPartitionExprDateTrunc implements MTMVPartitionExprService {
-private static Set timeUnits = ImmutableSet.of("year", "month",
"day");
+private static Set timeUnits = ImmutableSet.of("year", "month",
"day", "hour");
private String timeUnit;
public MTMVPartitionExprDateTrunc(FunctionCallExpr functionCallExpr)
throws AnalysisException {
@@ -202,6 +202,9 @@ public class MTMVPartitionExprDateTrunc implements
MTMVPartitionExprService {
case "day":
result = value.plusDays(1L);
break;
+case "hour":
+result = value.plusHours(1L);
+break;
default:
throw new AnalysisException(
"async materialized view partition roll up not support
timeUnit: " + timeUnit);
diff --git a/regression-test/suites/mtmv_p0/test_rollup_partition_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_rollup_partition_mtmv.groovy
index c24bc5aa910..21c77bc1e24 100644
--- a/regression-test/suites/mtmv_p0/test_rollup_partition_mtmv.groovy
+++ b/regression-test/suites/mtmv_p0/test_rollup_partition_mtmv.groovy
@@ -661,21 +661,21 @@ suite("test_rollup_partition_mtmv") {
log.info(e.getMessage())
}
-// not support trunc hour
+ // not support trunc minute
sql """drop table if exists `${tableName}`"""
sql """drop materialized view if exists ${mvName};"""
sql """
CREATE TABLE `${tableName}` (
`k1` LARGEINT NOT NULL COMMENT '\"用户id\"',
- `k2` DATE NOT NULL COMMENT '\"数据灌入日期时间\"'
+ `k2` DATETIME NOT NULL COMMENT '\"数据灌入日期时间\"'
) ENGINE=OLAP
DUPLICATE KEY(`k1`)
COMMENT 'OLAP'
PARTITION BY range(`k2`)
(
-PARTITION p_20200101 VALUES [("2020-01-01"),("2020-01-02")),
-PARTITION p_20200102 VALUES [("2020-01-02"),("2020-01-03")),
-PARTITION p_20200201 VALUES [("2020-02-01"),("2020-02-02"))
+PARTITION p_1 VALUES [("2020-01-01 00:00:00"),("2020-01-01 00:30:00")),
+PARTITION p_2 VALUES [("2020-01-01 00:30:00"),("2020-01-01 01:00:00")),
+PARTITION p_3 VALUES [("2020-01-01 01:00:00"),("2020-01-01 01:30:00"))
)
DISTRIBUTED BY HASH(`k1`) BUCKETS 2
PROPERTIES ('replication_num' = '1') ;
@@ -685,7 +685,7 @@ suite("test_rollup_partition_mtmv") {
sql """
CREATE MATERIALIZED VIEW ${mvName}
BUILD DEFERRED REFRESH AUTO ON MANUAL
-partition by (date_trunc(`k2`,'hour'))
+partition by (date_trunc(`k2`,'minute'))
DISTRIBUTED BY RANDOM BUCKETS 2
PROPERTIES (
'replication_num' = '1'
@@ -698,23 +698,40 @@ suite("test_rollup_partition_mtmv") {
log.info(e.getMessage())
}
+// support hour
+sql """
+CREATE MATERIALIZED VIEW ${mvName}
+BUILD DEFERRED REFRESH AUTO ON MANUAL
+partition by (date_trunc(`k2`,'hour'))
+DISTRIBUTED BY RANDOM BUCKETS 2
+PROPERTIES (
+'replication_num' = '1'
+)
+AS
+SELECT * FROM ${tableName};
+"""
+
+def hour_partitions = sql """show partitions from ${mvName}"""
+logger.info("hour_partitions: " + hour_partitions.toString())
+assertEquals(2, hour_partitions.size())
+
sql """drop materialized view if exists ${mvName};"""
try {
sql """
CREATE MATERIALIZED VIEW ${mvName}
BUILD DEFERRED REFRESH AUTO ON MANUAL
-partition by (hour_alias)
+partition by (minute_alias)
DISTRIBUTED BY RANDOM BUCKETS 2
PROPERTIES (
'replication_num' = '1'
)
AS
-SELECT date_trunc(`k2`,'hour') as hour_alias, *
(doris) 12/26: [fix](auth) fix create mv not need alter priv of table (#38011)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5c3ba5b1312bb98ec5bd3a316458e0309c6fc6d3
Author: zhangdong <493738...@qq.com>
AuthorDate: Fri Jul 19 15:35:00 2024 +0800
[fix](auth) fix create mv not need alter priv of table (#38011)
case will add in another pr
---
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index 55c553d1ecd..e9fef3ca661 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -979,9 +979,8 @@ public class StmtExecutor {
syncJournalIfNeeded();
analyzer = new Analyzer(context.getEnv(), context);
parsedStmt.analyze(analyzer);
-parsedStmt.checkPriv();
}
-
+parsedStmt.checkPriv();
if (prepareStmt instanceof PrepareStmt && !isExecuteStmt) {
handlePrepareStmt();
return;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 24/26: [Bug](function) fix mod function cause core dump (#37999)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e789bf256dd98230fd31ba31a5800ea8822de85f
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Sat Jul 20 16:00:09 2024 +0800
[Bug](function) fix mod function cause core dump (#37999)
## Proposed changes
as (-9223372036854775808 % -1) will cause coredump directly by fpe,
so check this case and throw exception.
---
be/src/vec/functions/modulo.cpp| 18
.../sql_functions/math_functions/test_mod.out | 6 +++
.../sql_functions/math_functions/test_mod.groovy | 49 ++
3 files changed, 73 insertions(+)
diff --git a/be/src/vec/functions/modulo.cpp b/be/src/vec/functions/modulo.cpp
index 66376b66019..7a2dfc004ef 100644
--- a/be/src/vec/functions/modulo.cpp
+++ b/be/src/vec/functions/modulo.cpp
@@ -33,6 +33,20 @@
namespace doris::vectorized {
+template
+inline void throw_if_division_leads_to_FPE(A a, B b) {
+// http://avva.livejournal.com/2548306.html
+// (-9223372036854775808 % -1) will cause coredump directly, so check this
case to throw exception, or maybe could return 0 as result
+if constexpr (std::is_signed_v && std::is_signed_v) {
+if (b == -1 && a == std::numeric_limits::min()) {
+throw Exception(ErrorCode::INVALID_ARGUMENT,
+"Division of minimal signed number by minus one is
an undefined "
+"behavior, {} % {}. ",
+a, b);
+}
+}
+}
+
template
struct ModuloImpl {
using ResultType = typename NumberTraits::ResultOfModulo::Type;
@@ -51,6 +65,7 @@ struct ModuloImpl {
if constexpr (std::is_floating_point_v) {
c[i] = std::fmod((double)a[i], (double)b);
} else {
+throw_if_division_leads_to_FPE(a[i], b);
c[i] = a[i] % b;
}
}
@@ -65,6 +80,7 @@ struct ModuloImpl {
if constexpr (std::is_floating_point_v) {
return std::fmod((double)a, (double)b);
} else {
+throw_if_division_leads_to_FPE(a, b);
return a % b;
}
}
@@ -94,6 +110,7 @@ struct PModuloImpl {
if constexpr (std::is_floating_point_v) {
c[i] = std::fmod(std::fmod((double)a[i], (double)b) +
(double)b, double(b));
} else {
+throw_if_division_leads_to_FPE(a[i], b);
c[i] = (a[i] % b + b) % b;
}
}
@@ -108,6 +125,7 @@ struct PModuloImpl {
if constexpr (std::is_floating_point_v) {
return std::fmod(std::fmod((double)a, (double)b) + (double)b,
(double)b);
} else {
+throw_if_division_leads_to_FPE(a, b);
return (a % b + b) % b;
}
}
diff --git
a/regression-test/data/query_p0/sql_functions/math_functions/test_mod.out
b/regression-test/data/query_p0/sql_functions/math_functions/test_mod.out
new file mode 100644
index 000..6e3ce6219de
--- /dev/null
+++ b/regression-test/data/query_p0/sql_functions/math_functions/test_mod.out
@@ -0,0 +1,6 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !sql --
+-21474836484 -1
+1 2 3
+5 -9223372036854775808-1
+
diff --git
a/regression-test/suites/query_p0/sql_functions/math_functions/test_mod.groovy
b/regression-test/suites/query_p0/sql_functions/math_functions/test_mod.groovy
new file mode 100644
index 000..027983ff749
--- /dev/null
+++
b/regression-test/suites/query_p0/sql_functions/math_functions/test_mod.groovy
@@ -0,0 +1,49 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_mod") {
+def tableName = "test_mod"
+sql "set enable_fold_constant_by_be = false;"
+sql """DROP TABLE IF EXISTS `test_mod`"""
+sql """ CREATE TABL
(doris) 04/26: [improvement](cloud) Accelerate cloud rebalance by batch editlog (#37787)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 02dff85679115b8da747b39e203858308870a249
Author: deardeng <565620...@qq.com>
AuthorDate: Fri Jul 19 11:29:58 2024 +0800
[improvement](cloud) Accelerate cloud rebalance by batch editlog (#37787)
1. use `JournalBatch` to batch editlogs
2. same partition, different tablets use one editlog
env：
in docker cloud mode, 3fe 3be.
3be expansion to 4be, trigger cloud rebalance
table, 1860 partitions, 48 buckets, every rebalance loop min balance 12
and close pre cache
result:
```
before improvement
2024-07-16 16:51:01,371 INFO (cloud tablet rebalancer|77)
[CloudTabletRebalancer.runAfterCatalogReady():228]
finished to rebalancer. cost: 58471 ms
after imprevement
2024-07-16 17:10:20,699 INFO (cloud tablet rebalancer|77)
[CloudTabletRebalancer.runAfterCatalogReady():235]
finished to rebalancer. cost: 28687 ms
```
---
.../analysis/ShowReplicaDistributionStmt.java | 6 +-
.../apache/doris/cloud/catalog/CloudReplica.java | 6 +-
.../doris/cloud/catalog/CloudTabletRebalancer.java | 158 +---
.../cloud/datasource/CloudInternalCatalog.java | 7 +-
.../cloud/persist/UpdateCloudReplicaInfo.java | 53 ++-
.../org/apache/doris/journal/JournalBatch.java | 6 +
.../java/org/apache/doris/persist/EditLog.java | 24 ++-
.../org/apache/doris/regression/suite/Suite.groovy | 11 ++
.../cloud_p0/multi_cluster/test_rebalance.groovy | 165 +
9 files changed, 372 insertions(+), 64 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowReplicaDistributionStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowReplicaDistributionStmt.java
index 6d598be727a..58d2ac1052f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowReplicaDistributionStmt.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowReplicaDistributionStmt.java
@@ -82,6 +82,10 @@ public class ShowReplicaDistributionStmt extends ShowStmt {
@Override
public RedirectStatus getRedirectStatus() {
-return RedirectStatus.FORWARD_NO_SYNC;
+if (ConnectContext.get().getSessionVariable().getForwardToMaster()) {
+return RedirectStatus.FORWARD_NO_SYNC;
+} else {
+return RedirectStatus.NO_FORWARD;
+}
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudReplica.java
b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudReplica.java
index d2e173dfd53..be0c510559e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudReplica.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudReplica.java
@@ -25,6 +25,7 @@ import org.apache.doris.cloud.system.CloudSystemInfoService;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.io.Text;
+import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.system.Backend;
@@ -221,7 +222,10 @@ public class CloudReplica extends Replica {
return backendId;
}
}
-
+if
(DebugPointUtil.isEnable("CloudReplica.getBackendIdImpl.clusterToBackends")) {
+LOG.info("Debug Point enable
CloudReplica.getBackendIdImpl.clusterToBackends");
+return -1;
+}
return hashReplicaToBe(clusterId, false);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java
b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java
index 30149de0d56..73ddbe4c455 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java
@@ -45,6 +45,7 @@ import org.apache.doris.thrift.TStatusCode;
import org.apache.doris.thrift.TWarmUpCacheAsyncRequest;
import org.apache.doris.thrift.TWarmUpCacheAsyncResponse;
+import com.google.common.base.Preconditions;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -57,6 +58,7 @@ import java.util.Random;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.LinkedBlockingQueue;
+import java.util.stream.Collectors;
public class CloudTabletRebalancer extends MasterDaemon {
private static final Logger LOG =
LogManager.getLogger(CloudTabletRebalancer.class);
@@ -181,8 +183,13 @@ public class CloudTabletRebalancer extends MasterDaemon {
// 2 complete route info
replicaInfos = new ArrayList();
completeRouteInfo();
-for (UpdateCloudReplicaInfo info : replicaInfos) {
-
(doris) 19/26: [Fix]add min scan thread num for workload group's scan thread (#38096)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 7d70c1739c0bae681c01fb544e62f9eabaedc2d5
Author: wangbo
AuthorDate: Fri Jul 19 18:05:17 2024 +0800
[Fix]add min scan thread num for workload group's scan thread (#38096)
## Proposed changes
Set workload group's and non-workload group's remote scan min thread to
reduce thread num, prevent Be core for thread Exhaustion.
before:
https://github.com/user-attachments/assets/3a861191-c5a9-4b73-8a08-0aec0bed1cd5";>
after:
https://github.com/user-attachments/assets/4024bbc8-d9d3-45bd-a895-07a6d87a6fd8";>
---
be/src/common/config.cpp | 1 +
be/src/common/config.h | 1 +
be/src/runtime/workload_group/workload_group.cpp | 5 +-
be/src/util/s3_util.cpp | 2 +-
be/src/vec/exec/scan/scanner_scheduler.cpp | 60 +---
be/src/vec/exec/scan/scanner_scheduler.h | 13 +++--
6 files changed, 38 insertions(+), 44 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index b152111011e..5222100170e 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -250,6 +250,7 @@ DEFINE_Validator(doris_scanner_thread_pool_thread_num,
[](const int config) -> b
}
return true;
});
+DEFINE_Int32(doris_scanner_min_thread_pool_thread_num, "8");
DEFINE_Int32(remote_split_source_batch_size, "10240");
DEFINE_Int32(doris_max_remote_scanner_thread_pool_thread_num, "-1");
// number of olap scanner thread pool queue size
diff --git a/be/src/common/config.h b/be/src/common/config.h
index f4ed1decaa0..53261ab2fb9 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -299,6 +299,7 @@
DECLARE_mInt64(doris_blocking_priority_queue_wait_timeout_ms);
// number of scanner thread pool size for olap table
// and the min thread num of remote scanner thread pool
DECLARE_mInt32(doris_scanner_thread_pool_thread_num);
+DECLARE_mInt32(doris_scanner_min_thread_pool_thread_num);
// number of batch size to fetch the remote split source
DECLARE_mInt32(remote_split_source_batch_size);
// max number of remote scanner thread pool size
diff --git a/be/src/runtime/workload_group/workload_group.cpp
b/be/src/runtime/workload_group/workload_group.cpp
index fd45093758e..f4d1e0d4f7e 100644
--- a/be/src/runtime/workload_group/workload_group.cpp
+++ b/be/src/runtime/workload_group/workload_group.cpp
@@ -316,7 +316,7 @@ WorkloadGroupInfo WorkloadGroupInfo::parse_topic_info(
}
// 11 min remote scan thread num
-int min_remote_scan_thread_num =
vectorized::ScannerScheduler::get_remote_scan_thread_num();
+int min_remote_scan_thread_num =
config::doris_scanner_min_thread_pool_thread_num;
if (tworkload_group_info.__isset.min_remote_scan_thread_num &&
tworkload_group_info.min_remote_scan_thread_num > 0) {
min_remote_scan_thread_num =
tworkload_group_info.min_remote_scan_thread_num;
@@ -415,7 +415,8 @@ void
WorkloadGroup::upsert_task_scheduler(WorkloadGroupInfo* tg_info, ExecEnv* e
std::unique_ptr
remote_scan_scheduler =
std::make_unique("RScan_"
+ tg_name,
cg_cpu_ctl_ptr);
-Status ret = remote_scan_scheduler->start(remote_max_thread_num,
remote_max_thread_num,
+Status ret = remote_scan_scheduler->start(remote_max_thread_num,
+
config::doris_scanner_min_thread_pool_thread_num,
remote_scan_thread_queue_size);
if (ret.ok()) {
_remote_scan_task_sched = std::move(remote_scan_scheduler);
diff --git a/be/src/util/s3_util.cpp b/be/src/util/s3_util.cpp
index d7a83fa2cff..ffb93c2d9d9 100644
--- a/be/src/util/s3_util.cpp
+++ b/be/src/util/s3_util.cpp
@@ -257,7 +257,7 @@ std::shared_ptr
S3ClientFactory::_create_s3_client(
aws_config.maxConnections =
config::doris_scanner_thread_pool_thread_num;
#else
aws_config.maxConnections =
-
ExecEnv::GetInstance()->scanner_scheduler()->remote_thread_pool_max_size();
+
ExecEnv::GetInstance()->scanner_scheduler()->remote_thread_pool_max_thread_num();
#endif
}
diff --git a/be/src/vec/exec/scan/scanner_scheduler.cpp
b/be/src/vec/exec/scan/scanner_scheduler.cpp
index 4d07e66917d..351f5d4e275 100644
--- a/be/src/vec/exec/scan/scanner_scheduler.cpp
+++ b/be/src/vec/exec/scan/scanner_scheduler.cpp
@@ -80,28 +80,33 @@ void ScannerScheduler::stop() {
_is_closed = true;
-_local_scan_thread_pool->shutdown();
-_remote_scan_thread_pool->shutdown();
_limited_scan_thread_pool->shutdown();
-
-_local_scan_thread_pool->join();
-_remote_scan_thread_pool->join();
_limited_scan_thread_pool->wait();
+
(doris) 23/26: [Enhancement](group commit)Optimize be select for group commit (#35558)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c11a46dd829c5510df0e50ccf9d522586ef63d6c
Author: abmdocrt
AuthorDate: Sat Jul 20 15:46:49 2024 +0800
[Enhancement](group commit)Optimize be select for group commit (#35558)
1. Streamload and insert into, if batched and sent to the master FE,
should use a consistent BE strategy (previously, insert into reused the
first selected BE, while streamload used round robin). First, a map
records a fixed be id for a certain table. The first
time a table is imported, a BE is randomly selected, and this table id
and be id are recorded in the map permanently. Subsequently, all data
imported into this table will select the BE corresponding to the table
id recorded in the map. This ensures that batching is maximized to a
single BE.
To address the issue of excessive load on a single BE, a variable
similar to a bvar window is used to monitor the total data volume sent
to a specific BE for a specific table during the batch interval (default
10 seconds). A second map is used to track
this. If a new import finds that its corresponding BE's window variable
is less than a certain value (e.g., 1G), the new import continues to be
sent to the corresponding BE according to map1. If it exceeds this
value, the new import is sent to another BE with the smallest window
variable value, and map1 is updated. If every BE exceeds this value, the
one with the smallest value is still chosen. This helps to alleviate
excessive pressure on a single BE.
2. For streamload, if batched and sent to a BE, it will batch directly
on this BE and will commit the transaction at the end of the import. At
this point, a request is sent to the FE, which records the size of this
import and adds it to the window variable.
3. Streamload sent to observer FE, as well as insert into sent to
observer FE, follow the logic in 1 by RPC, passing the table id to the
master FE to obtain the selected be id.
---
be/src/runtime/group_commit_mgr.cpp| 7 +
.../cloud/planner/CloudGroupCommitPlanner.java | 43 +
.../doris/common/util/SlidingWindowCounter.java| 73
.../org/apache/doris/httpv2/rest/LoadAction.java | 42 +++--
.../org/apache/doris/load/GroupCommitManager.java | 196 +
.../insert/OlapGroupCommitInsertExecutor.java | 13 ++
.../apache/doris/planner/GroupCommitPlanner.java | 29 +--
.../main/java/org/apache/doris/qe/Coordinator.java | 25 ++-
.../java/org/apache/doris/qe/MasterOpExecutor.java | 49 ++
.../apache/doris/service/FrontendServiceImpl.java | 31
gensrc/thrift/FrontendService.thrift | 15 ++
11 files changed, 446 insertions(+), 77 deletions(-)
diff --git a/be/src/runtime/group_commit_mgr.cpp
b/be/src/runtime/group_commit_mgr.cpp
index 5f989da023b..30885fa1ac9 100644
--- a/be/src/runtime/group_commit_mgr.cpp
+++ b/be/src/runtime/group_commit_mgr.cpp
@@ -442,6 +442,13 @@ Status GroupCommitTable::_finish_group_commit_load(int64_t
db_id, int64_t table_
request.__set_db_id(db_id);
request.__set_table_id(table_id);
request.__set_txnId(txn_id);
+request.__set_groupCommit(true);
+request.__set_receiveBytes(state->num_bytes_load_total());
+if (_exec_env->master_info()->__isset.backend_id) {
+request.__set_backendId(_exec_env->master_info()->backend_id);
+} else {
+LOG(WARNING) << "_exec_env->master_info not set backend_id";
+}
if (state) {
request.__set_commitInfos(state->tablet_commit_infos());
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/cloud/planner/CloudGroupCommitPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/cloud/planner/CloudGroupCommitPlanner.java
index 6978ec65e72..782f78e6bc4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/cloud/planner/CloudGroupCommitPlanner.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/cloud/planner/CloudGroupCommitPlanner.java
@@ -20,24 +20,18 @@ package org.apache.doris.cloud.planner;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.OlapTable;
-import org.apache.doris.cloud.system.CloudSystemInfoService;
import org.apache.doris.common.DdlException;
-import org.apache.doris.common.ErrorCode;
-import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.LoadException;
import org.apache.doris.common.UserException;
import org.apache.doris.planner.GroupCommitPlanner;
import org.apache.doris.qe.ConnectContext;
-import org.apache.doris.system.Backend;
import org.apache.doris.thrift.TUniqueId;
-import com.google.common.base.Strings;
import org.apache.logging.log4j.LogManager;
impo
(doris) 13/26: [fix](case)partition num will be 3 in daily pipeline (#37945)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ad206d5040d31b294cb340b98d46f6ac2b232e70
Author: zhangdong <493738...@qq.com>
AuthorDate: Fri Jul 19 15:38:49 2024 +0800
[fix](case)partition num will be 3 in daily pipeline (#37945)
partition num will be 3 in daily pipeline, because
force_olap_table_replication_num be set to 3
---
.../suites/external_table_p0/tvf/test_partitions_tvf.groovy | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/regression-test/suites/external_table_p0/tvf/test_partitions_tvf.groovy
b/regression-test/suites/external_table_p0/tvf/test_partitions_tvf.groovy
index 0939528c314..cab85625c84 100644
--- a/regression-test/suites/external_table_p0/tvf/test_partitions_tvf.groovy
+++ b/regression-test/suites/external_table_p0/tvf/test_partitions_tvf.groovy
@@ -54,8 +54,8 @@
suite("test_partitions_tvf","p0,external,tvf,external_docker") {
// assertEquals(1, res[0][9]);
// StorageMedium
assertEquals("HDD", res[0][10]);
-// ReplicaAllocation
-assertEquals("tag.location.default: 1", res[0][16]);
+// ReplicaAllocation: if force_olap_table_replication_num is set to 3,here
will be 3
+// assertEquals("tag.location.default: 1", res[0][16]);
// IsMutable
assertEquals(true, res[0][17]);
// SyncWithBaseTables
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 21/26: [fix](test) do not throw exception when a stream load consums more time (#38107)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2aa54ba7a5ee74aa9eb0108c2690f63581f521fb
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Sat Jul 20 14:58:45 2024 +0800
[fix](test) do not throw exception when a stream load consums more time
(#38107)
than expected
---
.../org/apache/doris/regression/action/StreamLoadAction.groovy| 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/StreamLoadAction.groovy
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/StreamLoadAction.groovy
index 606b9bc4ac8..70a7f41e82b 100644
---
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/StreamLoadAction.groovy
+++
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/StreamLoadAction.groovy
@@ -366,10 +366,10 @@ class StreamLoadAction implements SuiteAction {
if (time > 0) {
long elapsed = endTime - startTime
-try{
-Assert.assertTrue("Expect elapsed <= ${time}, but meet
${elapsed}", elapsed <= time)
-} catch (Throwable t) {
-throw new IllegalStateException("Expect elapsed <=
${time}, but meet ${elapsed}")
+if (elapsed > time) {
+log.info("Stream load consums more time than expected,
elapsed ${elapsed} ms, expect ${time} ms")
+} else {
+log.info("Stream load consums time elapsed ${elapsed} ms,
expect ${time} ms")
}
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 03/26: [improvement](statistics) Update table/index row count in one operation. (#38059)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 191910ee0bc9497a73d54b5748526ef3898cb207
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Fri Jul 19 10:38:04 2024 +0800
[improvement](statistics) Update table/index row count in one operation.
(#38059)
Atomic update table/index row count. Avoid the row count in partial
updated status.
---
.../main/java/org/apache/doris/analysis/ShowDataStmt.java | 10 ++
.../java/org/apache/doris/catalog/CloudTabletStatMgr.java | 7 ---
.../src/main/java/org/apache/doris/catalog/OlapTable.java | 15 +--
.../main/java/org/apache/doris/catalog/TabletStatMgr.java | 5 -
.../java/org/apache/doris/analysis/ShowDataStmtTest.java | 2 +-
regression-test/suites/statistics/test_analyze_mv.groovy | 9 -
6 files changed, 32 insertions(+), 16 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataStmt.java
index 51690f9ce87..b87c0f0788f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataStmt.java
@@ -90,6 +90,7 @@ public class ShowDataStmt extends ShowStmt {
.addColumn(new Column("ReplicaCount",
ScalarType.createVarchar(20)))
.addColumn(new Column("RowCount",
ScalarType.createVarchar(20)))
.addColumn(new Column("RemoteSize",
ScalarType.createVarchar(30)))
+.addColumn(new Column("StatsRowCount",
ScalarType.createVarchar(20)))
.build();
public static final ImmutableList SHOW_TABLE_DATA_META_DATA_ORIGIN
=
@@ -288,6 +289,7 @@ public class ShowDataStmt extends ShowStmt {
long indexReplicaCount = 0;
long indexRowCount = 0;
long indexRemoteSize = 0;
+long indexStatsRowCount =
olapTable.getRowCountForIndex(indexId);
for (Partition partition : olapTable.getAllPartitions()) {
MaterializedIndex mIndex = partition.getIndex(indexId);
indexSize += mIndex.getDataSize(false);
@@ -300,7 +302,7 @@ public class ShowDataStmt extends ShowStmt {
//
.add("TableName").add("IndexName").add("Size").add("ReplicaCount").add("RowCount")
// .add("RemoteSize")
List row = Arrays.asList(tableName, indexName,
indexSize, indexReplicaCount,
- indexRowCount, indexRemoteSize);
+ indexRowCount, indexRemoteSize,
indexStatsRowCount);
totalRowsObject.add(row);
totalSize += indexSize;
@@ -333,11 +335,11 @@ public class ShowDataStmt extends ShowStmt {
if (index == 0) {
result = Arrays.asList(tableName.getTbl(),
String.valueOf(row.get(1)),
readableSize, String.valueOf(row.get(3)),
-String.valueOf(row.get(4)),
remoteReadableSize);
+String.valueOf(row.get(4)),
remoteReadableSize, String.valueOf(row.get(6)));
} else {
result = Arrays.asList("", String.valueOf(row.get(1)),
readableSize, String.valueOf(row.get(3)),
-String.valueOf(row.get(4)),
remoteReadableSize);
+String.valueOf(row.get(4)),
remoteReadableSize, String.valueOf(row.get(6)));
}
totalRows.add(result);
}
@@ -349,7 +351,7 @@ public class ShowDataStmt extends ShowStmt {
String remoteReadableSize =
DebugUtil.DECIMAL_FORMAT_SCALE_3.format(totalRemoteSizePair.first) + " "
+ totalRemoteSizePair.second;
List row = Arrays.asList("", "Total", readableSize,
String.valueOf(totalReplicaCount), "",
- remoteReadableSize);
+ remoteReadableSize, "");
totalRows.add(row);
} finally {
olapTable.readUnlock();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java
index 3a75862f8d3..e762f678b79 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java
@@ -159,7 +159,7 @@ public class CloudTabletStatMgr extends MasterDaemon {
if (!table.writeLockIfExist()) {
continue;
}
-
+
(doris) 16/26: [fix](cloud) Fix auth compatibility logic run errer mode (#38136)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c5d780e456cdcf9d2c0956b3df053484b9fca217
Author: deardeng <565620...@qq.com>
AuthorDate: Fri Jul 19 16:33:42 2024 +0800
[fix](cloud) Fix auth compatibility logic run errer mode (#38136)
```
Caused by: java.lang.IllegalStateException: 11
at
com.google.common.base.Preconditions.checkState(Preconditions.java:512)
~[guava-32.1.2-jre.jar:?]
at
org.apache.doris.mysql.privilege.PrivBitSet.get(PrivBitSet.java:63)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.mysql.privilege.PrivBitSet.containsPrivs(PrivBitSet.java:116)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.mysql.privilege.Role.lambda$compatibilityErrEnum$4(Role.java:1150)
~[doris-fe.jar:1.2-SNAPSHOT]
at
java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:647) ~[?:?]
at
org.apache.doris.mysql.privilege.Role.compatibilityErrEnum(Role.java:1149)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.mysql.privilege.Role.gsonPostProcess(Role.java:1100)
~[doris-fe.jar:1.2-SNAPSHOT]
```
## Proposed changes
Issue Number: close #xxx
---
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
index 4146da80827..8354c655e2a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
@@ -1123,7 +1123,7 @@ public class Role implements Writable,
GsonPostProcessable {
privBitSet.set(Privilege.SHOW_VIEW_PRIV.getIdx());
}
});
-} else {
+} else if (Config.isCloudMode()) {
// cloud mode
// For versions greater than VERSION_123, the cloud requires
compatibility logic.
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 18/26: [fix](compatibility) Enhance SSL and Long Flag Handling in MySQL Handshake (#38086)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 976e43729fa18674e2eb232f75a6cabb7a5e82da
Author: zy-kkk
AuthorDate: Fri Jul 19 16:41:38 2024 +0800
[fix](compatibility) Enhance SSL and Long Flag Handling in MySQL Handshake
(#38086)
Issue Number: close #38065
This PR introduces a fix to the SSL and CLIENT_LONG_FLAG handling in the
MySQL handshake process. It ensures that the flags are correctly set
according to the protocol requirements and resolves the issues related
to decimal data fetching when SSL is enabled.
---
fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
index b3d4793eeb2..cbfa88038be 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
@@ -80,7 +80,7 @@ public class MysqlCapability {
private static final int SSL_FLAGS = Flag.CLIENT_PROTOCOL_41.getFlagBit()
| Flag.CLIENT_CONNECT_WITH_DB.getFlagBit() |
Flag.CLIENT_SECURE_CONNECTION.getFlagBit()
| Flag.CLIENT_PLUGIN_AUTH.getFlagBit() |
Flag.CLIENT_LOCAL_FILES.getFlagBit()
-| Flag.CLIENT_SSL.getFlagBit();
+| Flag.CLIENT_LONG_FLAG.getFlagBit() |
Flag.CLIENT_SSL.getFlagBit();
public static final MysqlCapability DEFAULT_CAPABILITY = new
MysqlCapability(DEFAULT_FLAGS);
public static final MysqlCapability SSL_CAPABILITY = new
MysqlCapability(SSL_FLAGS);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 22/26: [Improment]Query queued by be memory (#37559)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 7c3da4271bf93d039a00ceade460140977f48f26
Author: wangbo
AuthorDate: Sat Jul 20 15:39:19 2024 +0800
[Improment]Query queued by be memory (#37559)
## Proposed changes
Add a mechanism of back pressure based on memory usage.
when any BE's mem usage bigger than ```query_queue_by_be_used_memory```,
query could queue in FE.
---
be/src/service/backend_service.cpp | 10 ++
be/src/service/backend_service.h | 3 +
.../main/java/org/apache/doris/common/Config.java | 11 ++
.../main/java/org/apache/doris/catalog/Env.java| 9 ++
.../main/java/org/apache/doris/qe/Coordinator.java | 20 ++-
.../java/org/apache/doris/qe/QeProcessorImpl.java | 7 +-
.../apache/doris/resource/AdmissionControl.java| 156 +
.../doris/resource/workloadgroup/QueryQueue.java | 90 +++-
.../doris/resource/workloadgroup/QueueToken.java | 36 +++--
.../resource/workloadgroup/WorkloadGroup.java | 6 +-
.../doris/tablefunction/MetadataGenerator.java | 11 +-
.../org/apache/doris/common/GenericPoolTest.java | 7 +
.../apache/doris/utframe/MockedBackendFactory.java | 7 +
gensrc/thrift/BackendService.thrift| 14 ++
.../workload_manager_p0/test_curd_wlg.groovy | 2 +-
15 files changed, 325 insertions(+), 64 deletions(-)
diff --git a/be/src/service/backend_service.cpp
b/be/src/service/backend_service.cpp
index 1d85a2bca69..4effc225110 100644
--- a/be/src/service/backend_service.cpp
+++ b/be/src/service/backend_service.cpp
@@ -1297,4 +1297,14 @@ void
BaseBackendService::get_realtime_exec_status(TGetRealtimeExecStatusResponse
response.__set_report_exec_status_params(*report_exec_status_params);
}
+void BaseBackendService::get_be_resource(TGetBeResourceResult& result,
+ const TGetBeResourceRequest& request)
{
+int64_t mem_usage = PerfCounters::get_vm_rss();
+int64_t mem_limit = MemInfo::mem_limit();
+TGlobalResourceUsage global_resource_usage;
+global_resource_usage.__set_mem_limit(mem_limit);
+global_resource_usage.__set_mem_usage(mem_usage);
+result.__set_global_resource_usage(global_resource_usage);
+}
+
} // namespace doris
diff --git a/be/src/service/backend_service.h b/be/src/service/backend_service.h
index 4a04f16e853..0ada1bf5393 100644
--- a/be/src/service/backend_service.h
+++ b/be/src/service/backend_service.h
@@ -138,6 +138,9 @@ public:
void get_realtime_exec_status(TGetRealtimeExecStatusResponse& response,
const TGetRealtimeExecStatusRequest&
request) override;
+void get_be_resource(TGetBeResourceResult& result,
+ const TGetBeResourceRequest& request) override;
+
// begin cloud backend functions
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 39e8e4f1c17..7af0da4f9d6 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -1767,6 +1767,17 @@ public class Config extends ConfigBase {
@ConfField(mutable = true, varType = VariableAnnotation.EXPERIMENTAL)
public static boolean enable_cpu_hard_limit = false;
+@ConfField(mutable = true, description = {
+"当BE内存用量大于该值时，查询会进入排队逻辑，默认值为-1，代表该值不生效。取值范围0~1的小数",
+"When be memory usage bigger than this value, query could queue, "
++ "default value is -1, means this value not work. Decimal
value range from 0 to 1"})
+public static double query_queue_by_be_used_memory = -1;
+
+@ConfField(mutable = true, description = {"基于内存反压场景FE定时拉取BE内存用量的时间间隔",
+"In the scenario of memory backpressure, "
++ "the time interval for obtaining BE memory usage at
regular intervals"})
+public static long get_be_resource_usage_interval_ms = 1;
+
@ConfField(mutable = false, masterOnly = true)
public static int backend_rpc_timeout_ms = 6; // 1 min
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index c999024d359..49157b19ffb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -242,6 +242,7 @@ import org.apache.doris.qe.JournalObservable;
import org.apache.doris.qe.QueryCancelWorker;
import org.apache.doris.qe.SessionVariable;
import org.apache.doris.qe.VariableMgr;
+import org.apache.doris.resource.Ad
(doris) 20/26: [fix](test) fix test which conflict between #37219 and #37951 (#38157)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git commit a1d3328cc09f5eed30e295078d905bdc8362fe9f Author: 924060929 <924060...@qq.com> AuthorDate: Sat Jul 20 00:24:03 2024 +0800 [fix](test) fix test which conflict between #37219 and #37951 (#38157) fix test which conflict between #37219 and #37951 --- .../tpcds_sf100/noStatsRfPrune/query14.out | 8 ++-- .../tpcds_sf100/noStatsRfPrune/query58.out | 18 .../tpcds_sf100/noStatsRfPrune/query95.out | 28 ++-- .../tpcds_sf100/no_stats_shape/query14.out | 8 ++-- .../tpcds_sf100/no_stats_shape/query58.out | 24 +-- .../tpcds_sf100/no_stats_shape/query95.out | 28 ++-- .../new_shapes_p0/tpcds_sf100/rf_prune/query14.out | 12 +++--- .../new_shapes_p0/tpcds_sf100/rf_prune/query16.out | 4 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query23.out | 12 +++--- .../new_shapes_p0/tpcds_sf100/rf_prune/query33.out | 34 +++ .../new_shapes_p0/tpcds_sf100/rf_prune/query35.out | 50 +++--- .../new_shapes_p0/tpcds_sf100/rf_prune/query69.out | 42 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query94.out | 4 +- .../new_shapes_p0/tpcds_sf100/shape/query14.out| 14 +++--- .../new_shapes_p0/tpcds_sf100/shape/query16.out| 4 +- .../new_shapes_p0/tpcds_sf100/shape/query23.out| 12 +++--- .../new_shapes_p0/tpcds_sf100/shape/query33.out| 26 +-- .../new_shapes_p0/tpcds_sf100/shape/query35.out| 50 +++--- .../new_shapes_p0/tpcds_sf100/shape/query69.out| 42 +- .../new_shapes_p0/tpcds_sf100/shape/query94.out| 4 +- .../new_shapes_p0/tpcds_sf1000/shape/query23.out | 12 +++--- .../new_shapes_p0/tpcds_sf1000/shape/query33.out | 26 +-- .../new_shapes_p0/tpcds_sf1000/shape/query35.out | 44 +-- .../new_shapes_p0/tpcds_sf1000/shape/query51.out | 8 ++-- .../new_shapes_p0/tpcds_sf1000/shape/query58.out | 16 +++ .../new_shapes_p0/tpcds_sf1000/shape/query69.out | 42 +- .../new_shapes_p0/tpcds_sf1000/shape/query83.out | 16 +++ .../new_shapes_p0/tpch_sf1000/rf_prune/q16.out | 2 +- .../data/new_shapes_p0/tpch_sf1000/shape/q16.out | 2 +- 29 files changed, 296 insertions(+), 296 deletions(-) diff --git a/regression-test/data/new_shapes_p0/tpcds_sf100/noStatsRfPrune/query14.out b/regression-test/data/new_shapes_p0/tpcds_sf100/noStatsRfPrune/query14.out index fdf405c4b80..bde878b33ca 100644 --- a/regression-test/data/new_shapes_p0/tpcds_sf100/noStatsRfPrune/query14.out +++ b/regression-test/data/new_shapes_p0/tpcds_sf100/noStatsRfPrune/query14.out @@ -138,16 +138,16 @@ PhysicalCteAnchor ( cteId=CTEId#0 ) PhysicalDistribute[DistributionSpecHash] --hashAgg[LOCAL] PhysicalProject ---hashJoin[INNER_JOIN broadcast] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF17 d_date_sk->[ws_sold_date_sk] +--hashJoin[INNER_JOIN broadcast] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF18 d_date_sk->[ws_sold_date_sk] PhysicalProject ---hashJoin[LEFT_SEMI_JOIN bucketShuffle] hashCondition=((web_sales.ws_item_sk = cross_items.ss_item_sk)) otherCondition=() +--hashJoin[RIGHT_SEMI_JOIN shuffleBucket] hashCondition=((web_sales.ws_item_sk = cross_items.ss_item_sk)) otherCondition=() build RFs:RF17 ws_item_sk->[ss_item_sk] +PhysicalCteConsumer ( cteId=CTEId#0 ) apply RFs: RF17 PhysicalProject --hashJoin[INNER_JOIN shuffleBucket] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=() PhysicalProject ---PhysicalOlapScan[web_sales] apply RFs: RF17 +--PhysicalOlapScan[web_sales] apply RFs: RF18 PhysicalProject --PhysicalOlapScan[item] -PhysicalCteConsumer ( cteId=CTEId#0 ) PhysicalProject --filter((date_dim.d_moy = 11) and (date_dim.d_year = 2002)) PhysicalOlapScan[date_dim] diff --git a/regression-test/data/new_shapes_p0/tpcds_sf100/noStatsRfPrune/query58.out b/re
(doris) 11/26: [Chore](brpc) add gc for abafreelist to avoid eagain and set brpc timeout_ms to 2s (#37888)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f735df9f079b03b6fa411d42581ea0a080544e46
Author: Pxl
AuthorDate: Fri Jul 19 15:12:39 2024 +0800
[Chore](brpc) add gc for abafreelist to avoid eagain and set brpc
timeout_ms to 2s (#37888)
## Proposed changes
1. add gc for abafreelist to avoid eagain
```
[CANCELLED]failed to send brpc when exchange, error=Resource temporarily
unavailable, error_text=[E11]Resource temporarily unavailable
```
2. set brpc timeout_ms to 2s
---
be/src/util/brpc_client_cache.h| 1 +
9.0-add-gc-on-abalist-and-adjust-gc-size.patch | 280 +
2 files changed, 281 insertions(+)
diff --git a/be/src/util/brpc_client_cache.h b/be/src/util/brpc_client_cache.h
index 290f2cc3e04..ebef80f4a6b 100644
--- a/be/src/util/brpc_client_cache.h
+++ b/be/src/util/brpc_client_cache.h
@@ -126,6 +126,7 @@ public:
options.connection_group = connection_group;
}
options.connect_timeout_ms = 2000;
+options.timeout_ms = 2000;
options.max_retry = 10;
std::unique_ptr channel(new brpc::Channel());
diff --git
a/thirdparty/patches/brpc-1.9.0-add-gc-on-abalist-and-adjust-gc-size.patch
b/thirdparty/patches/brpc-1.9.0-add-gc-on-abalist-and-adjust-gc-size.patch
new file mode 100644
index 000..7287b4a536d
--- /dev/null
+++ b/thirdparty/patches/brpc-1.9.0-add-gc-on-abalist-and-adjust-gc-size.patch
@@ -0,0 +1,280 @@
+From fb53ab9245e8570d44a2eeea2ab536841a7876ec Mon Sep 17 00:00:00 2001
+From: BiteThet
+Date: Tue, 16 Jul 2024 14:21:21 +0800
+Subject: [PATCH] add gc on abalist and adjust gc size
+
+---
+ src/bthread/bthread.cpp | 3 +-
+ src/bthread/id.cpp | 1 +
+ src/bthread/list_of_abafree_id.h | 161 ---
+ 3 files changed, 151 insertions(+), 14 deletions(-)
+
+diff --git a/src/bthread/bthread.cpp b/src/bthread/bthread.cpp
+index 5ac0c3b1..86fd2c90 100644
+--- a/src/bthread/bthread.cpp
b/src/bthread/bthread.cpp
+@@ -147,7 +147,8 @@ start_from_non_worker(bthread_t* __restrict tid,
+
+ struct TidTraits {
+ static const size_t BLOCK_SIZE = 63;
+-static const size_t MAX_ENTRIES = 65536;
++static const size_t MAX_ENTRIES = 655360;
++static const size_t INIT_GC_SIZE = 65536;
+ static const bthread_t ID_INIT;
+ static bool exists(bthread_t id) { return bthread::TaskGroup::exists(id);
}
+ };
+diff --git a/src/bthread/id.cpp b/src/bthread/id.cpp
+index 41c49a3f..ba77580a 100644
+--- a/src/bthread/id.cpp
b/src/bthread/id.cpp
+@@ -291,6 +291,7 @@ void id_pool_status(std::ostream &os) {
+ struct IdTraits {
+ static const size_t BLOCK_SIZE = 63;
+ static const size_t MAX_ENTRIES = 10;
++static const size_t INIT_GC_SIZE = 4096;
+ static const bthread_id_t ID_INIT;
+ static bool exists(bthread_id_t id)
+ { return bthread::id_exists_with_true_negatives(id); }
+diff --git a/src/bthread/list_of_abafree_id.h
b/src/bthread/list_of_abafree_id.h
+index ac2b2234..45043acc 100644
+--- a/src/bthread/list_of_abafree_id.h
b/src/bthread/list_of_abafree_id.h
+@@ -22,8 +22,10 @@
+ #ifndef BTHREAD_LIST_OF_ABAFREE_ID_H
+ #define BTHREAD_LIST_OF_ABAFREE_ID_H
+
+-#include
+ #include
++#include
++
++#include "butil/macros.h"
+
+ namespace bthread {
+
+@@ -48,6 +50,9 @@ namespace bthread {
+ // // Max #entries. Often has close relationship with concurrency, 65536
+ // // is "huge" for most apps.
+ // static const size_t MAX_ENTRIES = 65536;
++// // Initial GC length, when the number of blocks reaches this length,
++// // start to initiate list GC operation. It will release useless blocks
++// static const size_t INIT_GC_SIZE = 4096;
+ //
+ // // Initial value of id. Id with the value is treated as invalid.
+ // static const Id ID_INIT = ...;
+@@ -64,13 +69,14 @@ class ListOfABAFreeId {
+ public:
+ ListOfABAFreeId();
+ ~ListOfABAFreeId();
+-
++
+ // Add an identifier into the list.
+ int add(Id id);
+-
++
+ // Apply fn(id) to all identifiers.
+-template void apply(const Fn& fn);
+-
++template
++void apply(const Fn& fn);
++
+ // Put #entries of each level into `counts'
+ // Returns #levels.
+ size_t get_sizes(size_t* counts, size_t n);
+@@ -82,19 +88,31 @@ private:
+ IdBlock* next;
+ };
+ void forward_index();
++
++struct TempIdBlock {
++IdBlock* block;
++uint32_t index;
++uint32_t nblock;
++};
++
++int gc();
++int add_to_temp_list(TempIdBlock* temp_list, Id id);
++template
++int for_each(const Fn& fn);
++void free_list(IdBlock* block);
++
+ IdBlock* _cur_block;
+ uint32_t _cur_index;
+ uint32_t _nblock;
+ IdBlock _head_block;
++uint32_t _next_gc_size;
+ };
+
+ // [impl.]
+
+ tem
(doris) 14/26: [opt](nereids) refine left semi/anti cost under short-cut opt (#37951)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 18a69dfa04cef4a86e96094804643d0f1382b016
Author: xzj7019 <13794+xzj7...@users.noreply.github.com>
AuthorDate: Fri Jul 19 15:39:54 2024 +0800
[opt](nereids) refine left semi/anti cost under short-cut opt (#37951)
Refine left semi/anti cost computing under short-cut opt, for the case
whose semi/anti join has the small left side and big right side, which
original solution can't support. This pr reduce the left style cost by
reduce the right side cost and improve the possibility of choosing left
style joins.
Pass the performance test on tpch/tpcds/usercase.
previous work: #37060
---
.../org/apache/doris/nereids/cost/CostModelV1.java | 9 ++--
.../java/org/apache/doris/qe/SessionVariable.java | 2 +-
.../shape/query23.out | 12 +++---
.../shape/query33.out | 26 +--
.../shape/query35.out | 44 +--
.../shape/query58.out | 16 +++
.../shape/query69.out | 42 +-
.../shape/query83.out | 16 +++
.../noStatsRfPrune/query14.out | 8 ++--
.../noStatsRfPrune/query58.out | 18
.../noStatsRfPrune/query95.out | 28 ++--
.../no_stats_shape/query14.out | 10 ++---
.../no_stats_shape/query58.out | 24 +--
.../no_stats_shape/query95.out | 28 ++--
.../rf_prune/query14.out | 46 ++--
.../rf_prune/query16.out | 4 +-
.../rf_prune/query23.out | 12 +++---
.../rf_prune/query33.out | 34 +++
.../rf_prune/query35.out | 50 +++---
.../rf_prune/query69.out | 42 +-
.../rf_prune/query94.out | 4 +-
.../nereids_tpcds_shape_sf100_p0/shape/query14.out | 46 ++--
.../nereids_tpcds_shape_sf100_p0/shape/query16.out | 4 +-
.../nereids_tpcds_shape_sf100_p0/shape/query23.out | 12 +++---
.../nereids_tpcds_shape_sf100_p0/shape/query33.out | 26 +--
.../nereids_tpcds_shape_sf100_p0/shape/query35.out | 50 +++---
.../nereids_tpcds_shape_sf100_p0/shape/query69.out | 42 +-
.../nereids_tpcds_shape_sf100_p0/shape/query94.out | 4 +-
.../nereids_tpch_shape_sf1000_p0/rf_prune/q16.out | 2 +-
.../nereids_tpch_shape_sf1000_p0/shape/q16.out | 2 +-
30 files changed, 330 insertions(+), 333 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
index 68ddbac13aa..4ef38138755 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
@@ -387,7 +387,9 @@ class CostModelV1 extends PlanVisitor {
);
}
double probeShortcutFactor = 1.0;
-if (physicalHashJoin.getJoinType().isLeftSemiOrAntiJoin()
+if (ConnectContext.get() != null &&
ConnectContext.get().getStatementContext() != null
+&&
!ConnectContext.get().getStatementContext().isHasUnknownColStats()
+&& physicalHashJoin.getJoinType().isLeftSemiOrAntiJoin()
&& physicalHashJoin.getOtherJoinConjuncts().isEmpty()
&& physicalHashJoin.getMarkJoinConjuncts().isEmpty()) {
// left semi/anti has short-cut opt, add probe side factor for
distinguishing from the right ones
@@ -414,15 +416,14 @@ class CostModelV1 extends PlanVisitor {
}
}
return CostV1.of(context.getSessionVariable(),
-leftRowCount * probeShortcutFactor
-+ rightRowCount * buildSideFactor
+leftRowCount * probeShortcutFactor + rightRowCount *
probeShortcutFactor * buildSideFactor
+ outputRowCount * probeSideFactor,
rightRowCount,
0
);
}
return CostV1.of(context.getSessionVariable(),
-leftRowCount * probeShortcutFactor + rightRowCount +
outputRowCount,
+leftRowCount * probeShortcutFactor + rightRowCount *
probeShortcutFactor + outputRowCount,
rightRowCount, 0
);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 4daa717f8b
(doris) 26/26: [fix](cloud) fix be core when using stream load record in cloud mode (#37903)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5be89b9d571d2bc583d0d87084f74d9792e70345
Author: hui lai <1353307...@qq.com>
AuthorDate: Sat Jul 20 20:59:41 2024 +0800
[fix](cloud) fix be core when using stream load record in cloud mode
(#37903)
---
be/src/cloud/cloud_backend_service.cpp | 8 +++
be/src/cloud/cloud_backend_service.h | 3 +
be/src/cloud/cloud_storage_engine.cpp | 4 ++
be/src/http/action/http_stream.cpp | 8 ++-
be/src/http/action/stream_load.cpp | 8 ++-
be/src/olap/storage_engine.cpp | 41 ++--
be/src/olap/storage_engine.h | 16 ++---
be/src/service/backend_service.cpp | 48 --
be/src/service/backend_service.h | 4 ++
.../org/apache/doris/load/StreamLoadRecordMgr.java | 28
.../pipeline/cloud_p0/conf/be_custom.conf | 2 +
regression-test/pipeline/p0/conf/be.conf | 1 +
.../stream_load/test_stream_load_record.groovy | 76 ++
13 files changed, 178 insertions(+), 69 deletions(-)
diff --git a/be/src/cloud/cloud_backend_service.cpp
b/be/src/cloud/cloud_backend_service.cpp
index f576b60045d..d91e9e416b8 100644
--- a/be/src/cloud/cloud_backend_service.cpp
+++ b/be/src/cloud/cloud_backend_service.cpp
@@ -29,6 +29,8 @@
#include "common/status.h"
#include "io/cache/block_file_cache_downloader.h"
#include "io/cache/block_file_cache_factory.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "runtime/stream_load/stream_load_recorder.h"
#include "util/brpc_client_cache.h" // BrpcClientCache
#include "util/thrift_server.h"
@@ -186,4 +188,10 @@ void
CloudBackendService::check_warm_up_cache_async(TCheckWarmUpCacheAsyncRespon
response.status = t_status;
}
+void CloudBackendService::get_stream_load_record(TStreamLoadRecordResult&
result,
+ int64_t
last_stream_record_time) {
+BaseBackendService::get_stream_load_record(result, last_stream_record_time,
+
_engine.get_stream_load_recorder());
+}
+
} // namespace doris
diff --git a/be/src/cloud/cloud_backend_service.h
b/be/src/cloud/cloud_backend_service.h
index 88f0099fe73..358cb4d1f0b 100644
--- a/be/src/cloud/cloud_backend_service.h
+++ b/be/src/cloud/cloud_backend_service.h
@@ -53,6 +53,9 @@ public:
void check_warm_up_cache_async(TCheckWarmUpCacheAsyncResponse& response,
const TCheckWarmUpCacheAsyncRequest&
request) override;
+void get_stream_load_record(TStreamLoadRecordResult& result,
+int64_t last_stream_record_time) override;
+
private:
CloudStorageEngine& _engine;
};
diff --git a/be/src/cloud/cloud_storage_engine.cpp
b/be/src/cloud/cloud_storage_engine.cpp
index b1b455d2007..de4bbac7b3e 100644
--- a/be/src/cloud/cloud_storage_engine.cpp
+++ b/be/src/cloud/cloud_storage_engine.cpp
@@ -196,6 +196,10 @@ Status CloudStorageEngine::open() {
_tablet_hotspot = std::make_unique();
+RETURN_NOT_OK_STATUS_WITH_WARN(
+
init_stream_load_recorder(ExecEnv::GetInstance()->store_paths()[0].path),
+"init StreamLoadRecorder failed");
+
return ThreadPoolBuilder("SyncLoadForTabletsThreadPool")
.set_max_threads(config::sync_load_for_tablets_thread)
.set_min_threads(config::sync_load_for_tablets_thread)
diff --git a/be/src/http/action/http_stream.cpp
b/be/src/http/action/http_stream.cpp
index a3439969e60..87cc2f694eb 100644
--- a/be/src/http/action/http_stream.cpp
+++ b/be/src/http/action/http_stream.cpp
@@ -30,6 +30,7 @@
#include
#include
+#include "cloud/cloud_storage_engine.h"
#include "cloud/config.h"
#include "common/config.h"
#include "common/consts.h"
@@ -119,7 +120,7 @@ void HttpStreamAction::handle(HttpRequest* req) {
// add new line at end
str = str + '\n';
HttpChannel::send_reply(req, str);
-if (config::enable_stream_load_record && !config::is_cloud_mode()) {
+if (config::enable_stream_load_record) {
str = ctx->prepare_stream_load_record(str);
_save_stream_load_record(ctx, str);
}
@@ -364,8 +365,9 @@ Status HttpStreamAction::process_put(HttpRequest* http_req,
void
HttpStreamAction::_save_stream_load_record(std::shared_ptr
ctx,
const std::string& str) {
-auto stream_load_recorder =
-
ExecEnv::GetInstance()->storage_engine().to_local().get_stream_load_recorder();
+std::shared_ptr stream_load_recorder =
+
ExecEnv::GetInstance()->storage_engine().get_stream_load_recorder();
+
if (stream_load_recorder != nullptr) {
std::string key =
std::to_string(c
(doris) 25/26: [fix](ut) repair segcompaction ut (#38165)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b77441e055079b9b5d3be1fd6dffe6d2e313288b
Author: zhannngchen <48427519+zhannngc...@users.noreply.github.com>
AuthorDate: Sat Jul 20 19:43:47 2024 +0800
[fix](ut) repair segcompaction ut (#38165)
SegcompactionTest is disabled by #16068 due to some code clean up, it
needs to be repaired
-
Co-authored-by: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
---
be/test/CMakeLists.txt | 1 -
be/test/olap/segcompaction_test.cpp | 518 +---
2 files changed, 246 insertions(+), 273 deletions(-)
diff --git a/be/test/CMakeLists.txt b/be/test/CMakeLists.txt
index c119af01e88..5a37267a937 100644
--- a/be/test/CMakeLists.txt
+++ b/be/test/CMakeLists.txt
@@ -46,7 +46,6 @@ list(REMOVE_ITEM UT_FILES
${CMAKE_CURRENT_SOURCE_DIR}/olap/rowset/segment_v2/frame_of_reference_page_test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/olap/rowset/segment_v2/plain_page_test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/olap/rowset/segment_v2/rle_page_test.cpp
-${CMAKE_CURRENT_SOURCE_DIR}/olap/segcompaction_test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/runtime/decimal_value_test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/runtime/result_buffer_mgr_test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/util/decompress_test.cpp
diff --git a/be/test/olap/segcompaction_test.cpp
b/be/test/olap/segcompaction_test.cpp
index f612694e7e3..468b24784af 100644
--- a/be/test/olap/segcompaction_test.cpp
+++ b/be/test/olap/segcompaction_test.cpp
@@ -23,7 +23,6 @@
#include
#include "common/config.h"
-#include "env/env_posix.h"
#include "gen_cpp/AgentService_types.h"
#include "gen_cpp/olap_file.pb.h"
#include "io/fs/local_file_system.h"
@@ -47,14 +46,12 @@ namespace doris {
using namespace ErrorCode;
static const uint32_t MAX_PATH_LEN = 1024;
-static std::unique_ptr l_engine;
+static StorageEngine* l_engine;
static const std::string lTestDir = "./data_test/data/segcompaction_test";
class SegCompactionTest : public testing::Test {
public:
-SegCompactionTest() : _data_dir(std::make_unique(lTestDir)) {
-_data_dir->update_capacity();
-}
+SegCompactionTest() = default;
void SetUp() {
config::enable_segcompaction = true;
@@ -76,20 +73,24 @@ public:
doris::EngineOptions options;
options.store_paths = paths;
-Status s = doris::StorageEngine::open(options, &l_engine);
+
+auto engine = std::make_unique(options);
+l_engine = engine.get();
+ExecEnv::GetInstance()->set_storage_engine(std::move(engine));
+
+Status s = l_engine->open();
EXPECT_TRUE(s.ok()) << s.to_string();
-ExecEnv* exec_env = doris::ExecEnv::GetInstance();
+_data_dir = std::make_unique(*l_engine, lTestDir);
+static_cast(_data_dir->update_capacity());
EXPECT_TRUE(io::global_local_filesystem()->create_directory(lTestDir).ok());
-l_engine->start_bg_threads();
+s = l_engine->start_bg_threads();
+EXPECT_TRUE(s.ok()) << s.to_string();
}
-void TearDown() {
-l_engine.reset();
-config::enable_segcompaction = false;
-}
+void TearDown() { config::enable_segcompaction = false; }
protected:
OlapReaderStatistics _stats;
@@ -122,7 +123,8 @@ protected:
}
// (k1 int, k2 varchar(20), k3 int) keys (k1, k2)
-void create_tablet_schema(TabletSchemaSPtr tablet_schema, KeysType
keystype) {
+void create_tablet_schema(TabletSchemaSPtr tablet_schema, KeysType
keystype,
+ int num_value_col = 1) {
TabletSchemaPB tablet_schema_pb;
tablet_schema_pb.set_keys_type(keystype);
tablet_schema_pb.set_num_short_key_columns(2);
@@ -152,15 +154,18 @@ protected:
column_2->set_is_nullable(true);
column_2->set_is_bf_column(false);
-ColumnPB* column_3 = tablet_schema_pb.add_column();
-column_3->set_unique_id(3);
-column_3->set_name("v1");
-column_3->set_type("INT");
-column_3->set_length(4);
-column_3->set_is_key(false);
-column_3->set_is_nullable(false);
-column_3->set_is_bf_column(false);
-column_3->set_aggregation("SUM");
+for (int i = 1; i <= num_value_col; i++) {
+ColumnPB* v_column = tablet_schema_pb.add_column();
+v_column->set_unique_id(2 + i);
+v_column->set_name(fmt::format("v{}", i));
+v_column->set_type("INT");
+v_column->set_length(4);
+v_column->set_is_key(false);
+v_column->set_is_nullable(false);
+v_column->set_is_bf_column(false);
+v_column->set_default_value(std::to_string(i * 10));
+v_column->set_aggregation("SUM");
+}
ta
