Re: [PR] [chore](meta) Add more log for FE meta [doris]
doris-robot commented on PR #38828: URL: https://github.com/apache/doris/pull/38828#issuecomment-2267376581 TPC-H: Total hot run time: 41729 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 8ad86a02268c559406641ad36468b907115bce7c, data reload: false -- Round 1 -- q1 17631 438341014101 q2 2028202 196 196 q3 10607 130113171301 q4 10227 812 908 812 q5 7651302329932993 q6 224 142 139 139 q7 1045605 607 605 q8 9433195819861958 q9 8453661666026602 q10 8709386438793864 q11 428 255 248 248 q12 420 225 222 222 q13 17773 291929492919 q14 273 240 251 240 q15 515 500 493 493 q16 546 400 392 392 q17 987 941 950 941 q18 8000730772217221 q19 1485123712241224 q20 558 323 343 323 q21 5326465547804655 q22 357 283 280 280 Total cold run time: 112676 ms Total hot run time: 41729 ms - Round 2, with runtime_filter_mode=off - q1 4113400139883988 q2 326 227 217 217 q3 2979300131493001 q4 2005204019611961 q5 5627550154485448 q6 222 135 133 133 q7 2070173318271733 q8 3327339933733373 q9 8690867188278671 q10 3962407439173917 q11 565 443 456 443 q12 747 624 598 598 q13 15312 313531243124 q14 313 279 280 279 q15 529 484 485 484 q16 459 411 424 411 q17 1803175617731756 q18 8250780278547802 q19 1877173917401739 q20 2115185018691850 q21 5811538853815381 q22 537 458 471 458 Total cold run time: 71639 ms Total hot run time: 56767 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() replace thread.sleep() #37817 [doris]
github-actions[bot] commented on PR #38814: URL: https://github.com/apache/doris/pull/38814#issuecomment-2267380502 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() replace thread.sleep() #37817 [doris]
github-actions[bot] commented on PR #38814: URL: https://github.com/apache/doris/pull/38814#issuecomment-2267380481 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](meta) Add more log for FE meta [doris]
doris-robot commented on PR #38828: URL: https://github.com/apache/doris/pull/38828#issuecomment-2267382258 TPC-DS: Total hot run time: 169641 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 8ad86a02268c559406641ad36468b907115bce7c, data reload: false query1 924 376 369 369 query2 6458164716671647 query3 6671210 227 210 query4 20122 17423 17127 17127 query5 3634514 504 504 query6 268 193 174 174 query7 4607300 291 291 query8 253 194 191 191 query9 8522234923382338 query10 446 296 290 290 query11 10394 10106 10049 10049 query12 122 89 89 89 query13 1628383 381 381 query14 951172427242 query15 217 162 167 162 query16 6937439 401 401 query17 971 592 584 584 query18 1495302 285 285 query19 193 146 148 146 query20 99 93 89 89 query21 205 99 98 98 query22 4217427639473947 query23 33859 33603 33691 33603 query24 10239 309630863086 query25 684 404 405 404 query26 1627164 158 158 query27 2912285 293 285 query28 7193199619941994 query29 1065447 445 445 query30 240 156 156 156 query31 950 762 758 758 query32 105 55 56 55 query33 671 314 319 314 query34 940 507 518 507 query35 906 750 764 750 query36 1052897 882 882 query37 224 85 83 83 query38 2974287828052805 query39 875 819 798 798 query40 290 114 109 109 query41 44 43 43 43 query42 118 99 101 99 query43 466 410 411 410 query44 1186752 750 750 query45 209 181 173 173 query46 1100804 792 792 query47 1820172017471720 query48 363 300 293 293 query49 1004430 419 419 query50 908 431 430 430 query51 6766662965866586 query52 96 93 85 85 query53 253 187 182 182 query54 622 451 452 451 query55 78 77 75 75 query56 291 262 252 252 query57 1146104510331033 query58 269 282 265 265 query59 2498223322902233 query60 325 269 276 269 query61 95 94 92 92 query62 872 676 656 656 query63 213 179 176 176 query64 5704191118631863 query65 3152310930733073 query66 1055331 327 327 query67 15392 14947 14837 14837 query68 4344565 576 565 query69 443 301 299 299 query70 1117107010211021 query71 417 285 275 275 query72 7066265825272527 query73 768 336 331 331 query74 6069564756805647 query75 3380271227492712 query76 2058123012821230 query77 466 314 308 308 query78 9362895389208920 query79 2384546 539 539 query80 940 505 498 498 query81 560 224 223 223 query82 1009137 127 127 query83 252 172 167 167 query84 264 81 78 78 query85 1199314 297 297 query86 458 351 303 303 query87 3305309430973094 query88 3209254025222522 query89 388 289 287 287 query90 1693188 192 188 query91 124 99 98 98 query92 59 50 48 48 query93 1910626 632 626 query94 789 293 296 293 query95 382 268 263 263 query96 620 289 285 285 query97 3248302631063026 query98 220 203 197 197 query99 1639129212571257 Total cold run time: 260887 ms Total hot run time: 169641 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [chore](meta) Add more log for FE meta [doris]
doris-robot commented on PR #38828: URL: https://github.com/apache/doris/pull/38828#issuecomment-2267386042 ClickBench: Total hot run time: 29.75 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 8ad86a02268c559406641ad36468b907115bce7c, data reload: false query1 0.040.040.04 query2 0.080.050.04 query3 0.220.050.05 query4 1.690.060.06 query5 0.480.480.47 query6 1.140.720.72 query7 0.020.020.01 query8 0.050.050.04 query9 0.570.510.50 query10 0.550.570.56 query11 0.150.110.12 query12 0.150.120.12 query13 0.620.600.60 query14 0.770.790.79 query15 0.900.860.87 query16 0.360.350.36 query17 1.001.020.99 query18 0.210.210.21 query19 1.831.741.71 query20 0.020.010.01 query21 15.40 0.750.64 query22 3.977.281.18 query23 18.03 1.331.27 query24 2.280.230.21 query25 0.190.080.08 query26 0.310.210.21 query27 0.460.220.23 query28 13.20 1.000.98 query29 12.61 3.323.28 query30 0.250.050.06 query31 2.860.410.40 query32 3.250.480.49 query33 2.972.912.91 query34 15.48 4.264.26 query35 4.294.284.26 query36 0.690.480.48 query37 0.190.160.16 query38 0.150.160.16 query39 0.040.030.04 query40 0.170.130.12 query41 0.110.050.05 query42 0.060.050.04 query43 0.040.040.04 Total cold run time: 107.85 s Total hot run time: 29.75 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] add information_schema.table_properties #38745 [doris]
doris-robot commented on PR #38746: URL: https://github.com/apache/doris/pull/38746#issuecomment-2267404647 TPC-H: Total hot run time: 41485 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 7b55e2afa1f4a6e9a2a0da622f7ddd866b17fe88, data reload: false -- Round 1 -- q1 17630 411740564056 q2 2020203 195 195 q3 10464 128013381280 q4 10179 788 910 788 q5 7660297729562956 q6 225 139 136 136 q7 1049611 599 599 q8 9686192019571920 q9 8572663666146614 q10 8725384738423842 q11 436 244 245 244 q12 404 224 219 219 q13 17762 293329182918 q14 279 234 247 234 q15 519 488 498 488 q16 542 391 381 381 q17 960 911 919 911 q18 8149745671587158 q19 1380122312051205 q20 550 335 338 335 q21 5276472747584727 q22 354 279 288 279 Total cold run time: 112821 ms Total hot run time: 41485 ms - Round 2, with runtime_filter_mode=off - q1 4063399740323997 q2 326 217 213 213 q3 2976297631242976 q4 1973200219831983 q5 5560554954515451 q6 220 142 128 128 q7 2081173318621733 q8 3285335433193319 q9 8682864888078648 q10 3931408838993899 q11 553 444 471 444 q12 752 610 566 566 q13 15411 310430863086 q14 298 287 288 287 q15 523 495 496 495 q16 471 417 427 417 q17 1753175417231723 q18 8195771076897689 q19 1713172916941694 q20 2061184218441842 q21 5800556252325232 q22 522 451 462 451 Total cold run time: 71149 ms Total hot run time: 56273 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [enhancement]use awaitility.await() add in schema_change tc #37817 [doris]
Vallishp opened a new pull request, #38831: URL: https://github.com/apache/doris/pull/38831 ## Proposed changes Issue Number: close #xxx use awaitility in schema_change testcases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() add in schema_change tc #37817 [doris]
doris-robot commented on PR #38831: URL: https://github.com/apache/doris/pull/38831#issuecomment-2267407057 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() add in schema_change tc #37817 [doris]
Vallishp commented on PR #38831: URL: https://github.com/apache/doris/pull/38831#issuecomment-2267407127 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] add information_schema.table_properties #38745 [doris]
doris-robot commented on PR #38746: URL: https://github.com/apache/doris/pull/38746#issuecomment-2267410696 TPC-DS: Total hot run time: 169794 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 7b55e2afa1f4a6e9a2a0da622f7ddd866b17fe88, data reload: false query1 916 381 368 368 query2 6483164716561647 query3 6660214 227 214 query4 20115 17403 17590 17403 query5 3641499 501 499 query6 287 167 165 165 query7 4614313 301 301 query8 254 193 198 193 query9 8527236223792362 query10 428 279 269 269 query11 10648 10171 10198 10171 query12 123 94 90 90 query13 1637391 385 385 query14 10040 768464256425 query15 204 161 166 161 query16 6873476 457 457 query17 958 563 560 560 query18 1913291 296 291 query19 201 152 148 148 query20 93 89 86 86 query21 198 100 105 100 query22 4142414239843984 query23 33858 33537 33723 33537 query24 10365 320531803180 query25 686 398 410 398 query26 1703160 150 150 query27 2964288 294 288 query28 7511200919811981 query29 1279446 441 441 query30 234 154 165 154 query31 937 777 780 777 query32 101 57 61 57 query33 706 338 341 338 query34 947 491 503 491 query35 885 765 746 746 query36 1037923 892 892 query37 294 84 89 84 query38 2905280227922792 query39 903 842 815 815 query40 253 111 114 111 query41 46 43 45 43 query42 128 101 103 101 query43 472 403 437 403 query44 1161742 749 742 query45 207 195 187 187 query46 1080826 819 819 query47 1810170017241700 query48 385 307 296 296 query49 929 415 419 415 query50 902 440 439 439 query51 6787671466926692 query52 105 94 91 91 query53 258 187 184 184 query54 645 457 453 453 query55 73 80 78 78 query56 267 263 252 252 query57 1116106410321032 query58 268 277 265 265 query59 2595238024012380 query60 297 272 272 272 query61 96 92 90 90 query62 885 667 689 667 query63 213 183 181 181 query64 5614191319021902 query65 3196309730923092 query66 1299335 338 335 query67 15613 14882 14746 14746 query68 4311577 572 572 query69 441 307 321 307 query70 1186105110711051 query71 418 292 291 291 query72 7103267525382538 query73 763 335 336 335 query74 6025572456925692 query75 3390275927532753 query76 2423120512581205 query77 420 312 314 312 query78 9459892188348834 query79 1435549 548 548 query80 1949516 507 507 query81 594 220 229 220 query82 1068136 134 134 query83 292 169 175 169 query84 282 81 80 80 query85 1290317 303 303 query86 408 306 304 304 query87 3271311631033103 query88 3028254425172517 query89 389 298 281 281 query90 1793203 205 203 query91 124 101 100 100 query92 60 50 57 50 query93 1406624 629 624 query94 821 288 299 288 query95 380 268 265 265 query96 609 289 292 289 query97 3249310630543054 query98 209 202 199 199 query99 1630129812981298 Total cold run time: 263196 ms Total hot run time: 169794 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [Enhancement] add information_schema.table_properties #38745 [doris]
doris-robot commented on PR #38746: URL: https://github.com/apache/doris/pull/38746#issuecomment-2267414099 ClickBench: Total hot run time: 30.17 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 7b55e2afa1f4a6e9a2a0da622f7ddd866b17fe88, data reload: false query1 0.040.050.04 query2 0.060.040.03 query3 0.230.050.05 query4 1.670.070.07 query5 0.490.490.48 query6 1.150.720.71 query7 0.020.010.01 query8 0.050.040.05 query9 0.580.520.52 query10 0.560.560.57 query11 0.160.120.11 query12 0.150.120.12 query13 0.620.590.60 query14 0.770.800.78 query15 0.900.850.87 query16 0.350.350.37 query17 1.000.991.00 query18 0.220.210.21 query19 1.871.791.75 query20 0.020.010.02 query21 15.40 0.750.65 query22 3.996.591.53 query23 17.89 1.291.21 query24 2.280.230.22 query25 0.180.090.08 query26 0.320.220.21 query27 0.460.220.23 query28 13.20 1.000.98 query29 12.58 3.293.30 query30 0.260.060.06 query31 2.850.420.41 query32 3.230.490.48 query33 2.982.952.97 query34 15.45 4.264.28 query35 4.334.324.29 query36 0.680.470.49 query37 0.190.160.17 query38 0.160.140.15 query39 0.040.040.03 query40 0.150.140.13 query41 0.100.050.05 query42 0.060.040.04 query43 0.040.040.04 Total cold run time: 107.73 s Total hot run time: 30.17 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](SC) Set enable_new_tablet_do_compaction = false temporarily for… [doris]
gavinchou opened a new pull request, #38832: URL: https://github.com/apache/doris/pull/38832 … cloud_p0 We have to figure out why it cause a SEGV when run cloud_p0 ``` /root/doris/cloud/../gensrc/build/gen_cpp/cloud.pb.h:48193:33 /root/doris/cloud/src/meta-service/meta_service_job.cpp:436:9 /root/doris/cloud/src/meta-service/meta_service.h:684:13 /root/doris/cloud/src/meta-service/meta_service.h:478:9 /root/doris/gensrc/build/gen_cpp/cloud.pb.cc:0:7 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](SC) Set enable_new_tablet_do_compaction = false temporarily for… [doris]
doris-robot commented on PR #38832: URL: https://github.com/apache/doris/pull/38832#issuecomment-2267415310 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](SC) Set enable_new_tablet_do_compaction = false temporarily for cloud_p0 [doris]
gavinchou commented on PR #38832: URL: https://github.com/apache/doris/pull/38832#issuecomment-2267416294 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] add null check before accessing pointer [doris]
github-actions[bot] commented on PR #38817: URL: https://github.com/apache/doris/pull/38817#issuecomment-2267427321 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] add null check before accessing pointer [doris]
github-actions[bot] commented on PR #38817: URL: https://github.com/apache/doris/pull/38817#issuecomment-2267427339 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](meta) Add more log for FE meta and revert #37293 [doris]
gavinchou commented on PR #38828: URL: https://github.com/apache/doris/pull/38828#issuecomment-2267442793 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](delete) Modify some p2 delete cases' property [doris]
Yukang-Lian commented on PR #37689: URL: https://github.com/apache/doris/pull/37689#issuecomment-2267464655 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](delete) Modify some p2 delete cases' property [doris]
github-actions[bot] commented on PR #37689: URL: https://github.com/apache/doris/pull/37689#issuecomment-2267464852 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](SC) Set enable_new_tablet_do_compaction = false temporarily for cloud_p0 [doris]
github-actions[bot] commented on PR #38832: URL: https://github.com/apache/doris/pull/38832#issuecomment-2267493010 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](SC) Set enable_new_tablet_do_compaction = false temporarily for cloud_p0 [doris]
github-actions[bot] commented on PR #38832: URL: https://github.com/apache/doris/pull/38832#issuecomment-2267493023 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](schema-change) Fix schema change run into nullable check specified for MV (#38806)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 69857b0fc38 [fix](schema-change) Fix schema change run into nullable
check specified for MV (#38806)
69857b0fc38 is described below
commit 69857b0fc38dbe124eb3f7c6af284a9a9108322c
Author: Siyang Tang <82279870+tangsiyang2...@users.noreply.github.com>
AuthorDate: Sun Aug 4 18:33:25 2024 +0800
[fix](schema-change) Fix schema change run into nullable check specified
for MV (#38806)
## Proposed changes
Fix using `!=` to compare ColumnType as reference, introduced by #32810
Add a minimal verify case.
---
be/src/olap/schema_change.cpp | 3 +-
.../org/apache/doris/alter/SchemaChangeJobV2.java | 15 -
.../schema_change/test_column_reorder.groovy | 38 ++
3 files changed, 47 insertions(+), 9 deletions(-)
diff --git a/be/src/olap/schema_change.cpp b/be/src/olap/schema_change.cpp
index 599d9c1d142..38dbcf1c429 100644
--- a/be/src/olap/schema_change.cpp
+++ b/be/src/olap/schema_change.cpp
@@ -291,6 +291,7 @@ Status BlockChanger::change_block(vectorized::Block*
ref_block,
// swap ref_block[key] and new_block[value]
std::list> swap_idx_list;
for (int idx = 0; idx < column_size; idx++) {
+// just for MV, schema change should not run into this branch
if (_schema_mapping[idx].expr != nullptr) {
vectorized::VExprContextSPtr ctx;
RETURN_IF_ERROR(vectorized::VExpr::create_expr_tree(*_schema_mapping[idx].expr,
ctx));
@@ -367,7 +368,7 @@ Status BlockChanger::change_block(vectorized::Block*
ref_block,
return Status::OK();
}
-// This check is to prevent schema-change from causing data loss
+// This check is for MV to prevent schema-change from causing data loss
Status BlockChanger::_check_cast_valid(vectorized::ColumnPtr ref_column,
vectorized::ColumnPtr new_column,
AlterTabletType type) {
if (ref_column->size() != new_column->size()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
index 94dd2421f84..c2c0aa108f4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
@@ -79,6 +79,7 @@ import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
@@ -480,14 +481,10 @@ public class SchemaChangeJobV2 extends AlterJobV2 {
if
(indexColumnMap.containsKey(SchemaChangeHandler.SHADOW_NAME_PREFIX +
column.getName())) {
Column newColumn = indexColumnMap.get(
SchemaChangeHandler.SHADOW_NAME_PREFIX +
column.getName());
-if (newColumn.getType() != column.getType()) {
-try {
-SlotRef slot = new SlotRef(destSlotDesc);
-slot.setCol(column.getName());
-defineExprs.put(column.getName(),
slot.castTo(newColumn.getType()));
-} catch (AnalysisException e) {
-throw new
AlterCancelException(e.getMessage());
-}
+if (!Objects.equals(newColumn.getType(),
column.getType())) {
+SlotRef slot = new SlotRef(destSlotDesc);
+slot.setCol(column.getName());
+defineExprs.put(column.getName(),
slot.castTo(newColumn.getType()));
}
}
}
@@ -510,6 +507,8 @@ public class SchemaChangeJobV2 extends AlterJobV2 {
}
} // end for partitions
+} catch (AnalysisException e) {
+throw new AlterCancelException(e.getMessage());
} finally {
tbl.readUnlock();
}
diff --git
a/regression-test/suites/variant_p0/schema_change/test_column_reorder.groovy
b/regression-test/suites/variant_p0/schema_change/test_column_reorder.groovy
new file mode 100644
index 000..bb1e4137189
--- /dev/null
+++ b/regression-test/suites/variant_p0/schema_change/test_column_reorder.groovy
@@ -0,0 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF l
Re: [PR] [Fix](group commit) Fix multiple cluster group commit BE select strategy [doris]
dataroaring merged PR #38644: URL: https://github.com/apache/doris/pull/38644 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](meta) Add more log for FE meta and revert #37293 [doris]
github-actions[bot] commented on PR #38828: URL: https://github.com/apache/doris/pull/38828#issuecomment-2267493511 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](meta) Add more log for FE meta and revert #37293 [doris]
github-actions[bot] commented on PR #38828: URL: https://github.com/apache/doris/pull/38828#issuecomment-2267493518 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](schema-change) Fix schema change run into nullable check specified for MV [doris]
dataroaring merged PR #38806: URL: https://github.com/apache/doris/pull/38806 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Fix](group commit) Fix multiple cluster group commit BE select strategy (#38644)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ef4fba649cb [Fix](group commit) Fix multiple cluster group commit BE
select strategy (#38644)
ef4fba649cb is described below
commit ef4fba649cb563efe64874addcbac11204925cfc
Author: abmdocrt
AuthorDate: Sun Aug 4 18:33:55 2024 +0800
[Fix](group commit) Fix multiple cluster group commit BE select strategy
(#38644)
---
.../org/apache/doris/load/GroupCommitManager.java | 64 ++
1 file changed, 41 insertions(+), 23 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/GroupCommitManager.java
b/fe/fe-core/src/main/java/org/apache/doris/load/GroupCommitManager.java
index 20f7b9ed9be..1009c4257b8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/GroupCommitManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/GroupCommitManager.java
@@ -58,10 +58,10 @@ public class GroupCommitManager {
private Set blockedTableIds = new HashSet<>();
-// Table id to BE id map. Only for group commit.
-private Map tableToBeMap = new ConcurrentHashMap<>();
-// BE id to pressure map. Only for group commit.
-private Map tablePressureMap = new
ConcurrentHashMap<>();
+// Encoded to BE id map. Only for group commit.
+private final Map tableToBeMap = new ConcurrentHashMap<>();
+// Table id to pressure map. Only for group commit.
+private final Map tableToPressureMap = new
ConcurrentHashMap<>();
public boolean isBlock(long tableId) {
return blockedTableIds.contains(tableId);
@@ -243,13 +243,13 @@ public class GroupCommitManager {
private long selectBackendForCloudGroupCommitInternal(long tableId, String
cluster)
throws DdlException, LoadException {
-LOG.debug("cloud group commit select be info, tableToBeMap {},
tablePressureMap {}", tableToBeMap.toString(),
-tablePressureMap.toString());
+LOG.debug("cloud group commit select be info, tableToBeMap {},
tablePressureMap {}",
+tableToBeMap.toString(), tableToPressureMap.toString());
if (Strings.isNullOrEmpty(cluster)) {
ErrorReport.reportDdlException(ErrorCode.ERR_NO_CLUSTER_ERROR);
}
-Long cachedBackendId = getCachedBackend(tableId);
+Long cachedBackendId = getCachedBackend(cluster, tableId);
if (cachedBackendId != null) {
return cachedBackendId;
}
@@ -261,7 +261,7 @@ public class GroupCommitManager {
throw new LoadException("No alive backend");
}
// If the cached backend is not active or decommissioned, select a
random new backend.
-Long randomBackendId = getRandomBackend(tableId, backends);
+Long randomBackendId = getRandomBackend(cluster, tableId, backends);
if (randomBackendId != null) {
return randomBackendId;
}
@@ -274,8 +274,8 @@ public class GroupCommitManager {
private long selectBackendForLocalGroupCommitInternal(long tableId) throws
LoadException {
LOG.debug("group commit select be info, tableToBeMap {},
tablePressureMap {}", tableToBeMap.toString(),
-tablePressureMap.toString());
-Long cachedBackendId = getCachedBackend(tableId);
+tableToPressureMap.toString());
+Long cachedBackendId = getCachedBackend(null, tableId);
if (cachedBackendId != null) {
return cachedBackendId;
}
@@ -293,7 +293,7 @@ public class GroupCommitManager {
}
// If the cached backend is not active or decommissioned, select a
random new backend.
-Long randomBackendId = getRandomBackend(tableId, backends);
+Long randomBackendId = getRandomBackend(null, tableId, backends);
if (randomBackendId != null) {
return randomBackendId;
}
@@ -305,31 +305,41 @@ public class GroupCommitManager {
}
@Nullable
-private Long getCachedBackend(long tableId) {
+private Long getCachedBackend(String cluster, long tableId) {
OlapTable table = (OlapTable)
Env.getCurrentEnv().getInternalCatalog().getTableByTableId(tableId);
-if (tableToBeMap.containsKey(tableId)) {
-if (tablePressureMap.get(tableId).get() <
table.getGroupCommitDataBytes()) {
-Backend backend =
Env.getCurrentSystemInfo().getBackend(tableToBeMap.get(tableId));
+if (tableToBeMap.containsKey(encode(cluster, tableId))) {
+if (tableToPressureMap.get(tableId).get() <
table.getGroupCommitDataBytes()) {
+// There are multiple threads getting cached backends for the
same table.
+// Maybe one thread removes the tableId from the tableToBeMap.
+// Another thread gets the same tab
[PR] [enhance](server) add metrics to show thread cpu usage [doris]
cosven opened a new pull request, #38833: URL: https://github.com/apache/doris/pull/38833 ## Proposed changes Add a metric `thread_cpu_total` to show cpu usage by each thread. User can query thread pool usage by the following promql  For example, check the cpu usage of `CumuCompaction` thread pool ![Uploading image.png…]() -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
doris-robot commented on PR #38833: URL: https://github.com/apache/doris/pull/38833#issuecomment-2267496674 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
github-actions[bot] commented on code in PR #38833:
URL: https://github.com/apache/doris/pull/38833#discussion_r1703150007
##
be/src/util/thread.h:
##
@@ -291,4 +293,87 @@ class ThreadJoiner {
// Registers /threadz with the debug webserver.
void register_thread_display_page(WebPageHandler* web_page_handler);
+
+// A singleton class that tracks all live threads, and groups them together
for easy
+// auditing. Used only by Thread.
+class ThreadMgr {
+public:
+static ThreadMgr* instance();
+
+ThreadMgr() : _threads_started_metric(0), _threads_running_metric(0) {}
+~ThreadMgr() {
+std::unique_lock lock(_lock);
+_thread_categories.clear();
+}
+
+static void set_thread_name(const std::string& name, int64_t tid);
+
+#ifndef __APPLE__
+static void set_idle_sched(int64_t tid);
+
+static void set_thread_nice_value(int64_t tid);
+#endif
+
+// not the system TID, since pthread_t is less prone to being recycled.
+void add_thread(const pthread_t& pthread_id, const std::string& name,
+const std::string& category, int64_t tid);
+
+// Removes a thread from the supplied category. If the thread has
+// already been removed, this is a no-op.
+void remove_thread(const pthread_t& pthread_id, const std::string&
category);
+
+void display_thread_callback(const WebPageHandler::ArgumentMap& args,
EasyJson* ej) const;
+void update_threads_metrics();
+
+private:
+// Container class for any details we want to capture about a thread
+// TODO: Add start-time.
+// TODO: Track fragment ID.
+class ThreadDescriptor {
+public:
+ThreadDescriptor() {}
Review Comment:
warning: use '= default' to define a trivial default constructor
[modernize-use-equals-default]
```suggestion
ThreadDescriptor() = default;
```
##
be/src/util/thread.h:
##
@@ -291,4 +293,87 @@
// Registers /threadz with the debug webserver.
void register_thread_display_page(WebPageHandler* web_page_handler);
+
+// A singleton class that tracks all live threads, and groups them together
for easy
+// auditing. Used only by Thread.
+class ThreadMgr {
+public:
+static ThreadMgr* instance();
+
+ThreadMgr() : _threads_started_metric(0), _threads_running_metric(0) {}
+~ThreadMgr() {
+std::unique_lock lock(_lock);
+_thread_categories.clear();
+}
+
+static void set_thread_name(const std::string& name, int64_t tid);
+
+#ifndef __APPLE__
+static void set_idle_sched(int64_t tid);
+
+static void set_thread_nice_value(int64_t tid);
+#endif
+
+// not the system TID, since pthread_t is less prone to being recycled.
+void add_thread(const pthread_t& pthread_id, const std::string& name,
+const std::string& category, int64_t tid);
+
+// Removes a thread from the supplied category. If the thread has
+// already been removed, this is a no-op.
+void remove_thread(const pthread_t& pthread_id, const std::string&
category);
+
+void display_thread_callback(const WebPageHandler::ArgumentMap& args,
EasyJson* ej) const;
+void update_threads_metrics();
+
+private:
+// Container class for any details we want to capture about a thread
+// TODO: Add start-time.
+// TODO: Track fragment ID.
+class ThreadDescriptor {
+public:
+ThreadDescriptor() {}
+ThreadDescriptor(std::string category, std::string name, int64_t
thread_id)
+: _name(std::move(name)), _category(std::move(category)),
_thread_id(thread_id) {}
+
+const std::string& name() const { return _name; }
+const std::string& category() const { return _category; }
+int64_t thread_id() const { return _thread_id; }
+
+void register_metric();
+void update_metric();
+void deregister_metric();
+
+private:
+std::string _name;
+std::string _category;
+int64_t _thread_id;
+
+std::shared_ptr _metric_entity;
+DoubleCounter* thread_cpu_total = nullptr;
+};
+
+void summarize_thread_descriptor(const ThreadDescriptor& desc, EasyJson*
ej) const;
+
+// A ThreadCategory is a set of threads that are logically related.
+// TODO: unordered_map is incompatible with pthread_t, but would be more
+// efficient here.
+typedef std::map ThreadCategory;
+
+// All thread categories, keyed on the category name.
+typedef std::map ThreadCategoryMap;
+
+// Protects _thread_categories and thread metrics.
+mutable std::mutex _lock;
+
+// All thread categories that ever contained a thread, even if empty
+ThreadCategoryMap _thread_categories;
+
+// Counters to track all-time total number of threads, and the
+// current number of running threads.
+uint64_t _threads_started_metric;
Review Comment:
warning: use default member initializer for '_threads_started_metric'
[modernize-use-default-member-init]
Re: [PR] [docs](function) support auto_partition_name function [doris-website]
zclllyybb commented on code in PR #945:
URL: https://github.com/apache/doris-website/pull/945#discussion_r1703152554
##
docs/sql-manual/sql-functions/string-functions/auto_partition_name.md:
##
@@ -0,0 +1,113 @@
+---
+{
+"title": "AUTO_PARTITION_NAME",
+"language": "en"
+}
+---
+
+
+
+## auto_partition_name
+### description
+ Syntax
+
+`VARCHAR AUTO_PARTITION_NAME('RANGE', 'VARCHAR unit', DATETIME datetime)`
+
+`VARCHAR AUTO_PARTITION_NAME('LIST', VARCHAR,...)`
+
+Generate datetime partition names by unit following RANGE's partition name
rules
+
+Convert strings to partition names following LIST's partition name rules
+
+The datetime parameter is a legal date expression.
+
+The unit parameter is the time interval you want, the available values are:
[`second`, `minute`, `hour`, `day`, `month`, `year`].
Review Comment:
add some example about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
github-actions[bot] commented on code in PR #38833:
URL: https://github.com/apache/doris/pull/38833#discussion_r1703154792
##
be/src/util/thread.h:
##
@@ -291,4 +293,86 @@
// Registers /threadz with the debug webserver.
void register_thread_display_page(WebPageHandler* web_page_handler);
+// A singleton class that tracks all live threads, and groups them together
for easy
+// auditing. Used only by Thread.
+class ThreadMgr {
+public:
+static ThreadMgr* instance();
+
+ThreadMgr() : _threads_started_metric(0), _threads_running_metric(0) {}
+~ThreadMgr() {
+std::unique_lock lock(_lock);
+_thread_categories.clear();
+}
+
+static void set_thread_name(const std::string& name, int64_t tid);
+
+#ifndef __APPLE__
+static void set_idle_sched(int64_t tid);
+
+static void set_thread_nice_value(int64_t tid);
+#endif
+
+// not the system TID, since pthread_t is less prone to being recycled.
+void add_thread(const pthread_t& pthread_id, const std::string& name,
+const std::string& category, int64_t tid);
+
+// Removes a thread from the supplied category. If the thread has
+// already been removed, this is a no-op.
+void remove_thread(const pthread_t& pthread_id, const std::string&
category);
+
+void display_thread_callback(const WebPageHandler::ArgumentMap& args,
EasyJson* ej) const;
+void update_threads_metrics();
+
+private:
+// Container class for any details we want to capture about a thread
+// TODO: Add start-time.
+// TODO: Track fragment ID.
+class ThreadDescriptor {
+public:
+ThreadDescriptor() {}
+ThreadDescriptor(std::string category, std::string name, int64_t
thread_id)
+: _name(std::move(name)), _category(std::move(category)),
_thread_id(thread_id) {}
+
+const std::string& name() const { return _name; }
+const std::string& category() const { return _category; }
+int64_t thread_id() const { return _thread_id; }
+
+void register_metric();
+void update_metric();
+void deregister_metric();
+
+private:
+std::string _name;
+std::string _category;
+int64_t _thread_id;
+
+std::shared_ptr _metric_entity;
+DoubleCounter* thread_cpu_total = nullptr;
+};
+
+void summarize_thread_descriptor(const ThreadDescriptor& desc, EasyJson*
ej) const;
+
+// A ThreadCategory is a set of threads that are logically related.
+// TODO: unordered_map is incompatible with pthread_t, but would be more
+// efficient here.
+typedef std::map ThreadCategory;
+
+// All thread categories, keyed on the category name.
+typedef std::map ThreadCategoryMap;
+
+// Protects _thread_categories and thread metrics.
+mutable std::mutex _lock;
+
+// All thread categories that ever contained a thread, even if empty
+ThreadCategoryMap _thread_categories;
+
+// Counters to track all-time total number of threads, and the
+// current number of running threads.
+uint64_t _threads_started_metric;
+uint64_t _threads_running_metric;
+
Review Comment:
warning: use default member initializer for '_threads_running_metric'
[modernize-use-default-member-init]
be/src/util/thread.h:302:
```diff
- ThreadMgr() : _threads_started_metric(0), _threads_running_metric(0) {}
+ ThreadMgr() : _threads_started_metric(0), {}
```
```suggestion
uint64_t _threads_running_metric{0};
```
##
be/src/util/thread.h:
##
@@ -291,4 +293,86 @@
// Registers /threadz with the debug webserver.
void register_thread_display_page(WebPageHandler* web_page_handler);
+// A singleton class that tracks all live threads, and groups them together
for easy
+// auditing. Used only by Thread.
+class ThreadMgr {
+public:
+static ThreadMgr* instance();
+
+ThreadMgr() : _threads_started_metric(0), _threads_running_metric(0) {}
+~ThreadMgr() {
+std::unique_lock lock(_lock);
+_thread_categories.clear();
+}
+
+static void set_thread_name(const std::string& name, int64_t tid);
+
+#ifndef __APPLE__
+static void set_idle_sched(int64_t tid);
+
+static void set_thread_nice_value(int64_t tid);
+#endif
+
+// not the system TID, since pthread_t is less prone to being recycled.
+void add_thread(const pthread_t& pthread_id, const std::string& name,
+const std::string& category, int64_t tid);
+
+// Removes a thread from the supplied category. If the thread has
+// already been removed, this is a no-op.
+void remove_thread(const pthread_t& pthread_id, const std::string&
category);
+
+void display_thread_callback(const WebPageHandler::ArgumentMap& args,
EasyJson* ej) const;
+void update_threads_metrics();
+
+private:
+// Container class for any details we want to capture about a thread
+// TODO: Add start-time.
+// TODO: Track f
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
cosven commented on PR #38833: URL: https://github.com/apache/doris/pull/38833#issuecomment-2267507377 run performance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
github-actions[bot] commented on code in PR #38833:
URL: https://github.com/apache/doris/pull/38833#discussion_r1703155730
##
be/src/util/thread.h:
##
@@ -291,4 +293,86 @@ class ThreadJoiner {
// Registers /threadz with the debug webserver.
void register_thread_display_page(WebPageHandler* web_page_handler);
+// A singleton class that tracks all live threads, and groups them together
for easy
+// auditing. Used only by Thread.
+class ThreadMgr {
+public:
+static ThreadMgr* instance();
+
+ThreadMgr() : _threads_started_metric(0), _threads_running_metric(0) {}
+~ThreadMgr() {
+std::unique_lock lock(_lock);
+_thread_categories.clear();
+}
+
+static void set_thread_name(const std::string& name, int64_t tid);
+
+#ifndef __APPLE__
+static void set_idle_sched(int64_t tid);
+
+static void set_thread_nice_value(int64_t tid);
+#endif
+
+// not the system TID, since pthread_t is less prone to being recycled.
+void add_thread(const pthread_t& pthread_id, const std::string& name,
+const std::string& category, int64_t tid);
+
+// Removes a thread from the supplied category. If the thread has
+// already been removed, this is a no-op.
+void remove_thread(const pthread_t& pthread_id, const std::string&
category);
+
+void display_thread_callback(const WebPageHandler::ArgumentMap& args,
EasyJson* ej) const;
+void update_threads_metrics();
+
+private:
+// Container class for any details we want to capture about a thread
+// TODO: Add start-time.
+// TODO: Track fragment ID.
+class ThreadDescriptor {
+public:
+ThreadDescriptor() {}
+ThreadDescriptor(std::string category, std::string name, int64_t
thread_id)
+: _name(std::move(name)), _category(std::move(category)),
_thread_id(thread_id) {}
+
+const std::string& name() const { return _name; }
+const std::string& category() const { return _category; }
+int64_t thread_id() const { return _thread_id; }
+
+void register_metric();
+void update_metric();
+void deregister_metric();
+
+private:
+std::string _name;
+std::string _category;
+int64_t _thread_id;
+
+std::shared_ptr _metric_entity;
+DoubleCounter* thread_cpu_total = nullptr;
+};
+
+void summarize_thread_descriptor(const ThreadDescriptor& desc, EasyJson*
ej) const;
+
+// A ThreadCategory is a set of threads that are logically related.
+// TODO: unordered_map is incompatible with pthread_t, but would be more
+// efficient here.
+typedef std::map ThreadCategory;
+
+// All thread categories, keyed on the category name.
+typedef std::map ThreadCategoryMap;
+
+// Protects _thread_categories and thread metrics.
+mutable std::mutex _lock;
+
+// All thread categories that ever contained a thread, even if empty
+ThreadCategoryMap _thread_categories;
+
+// Counters to track all-time total number of threads, and the
+// current number of running threads.
+uint64_t _threads_started_metric;
Review Comment:
warning: use default member initializer for '_threads_started_metric'
[modernize-use-default-member-init]
be/src/util/thread.h:301:
```diff
- ThreadMgr() : _threads_started_metric(0), _threads_running_metric(0) {}
+ ThreadMgr() : , _threads_running_metric(0) {}
```
```suggestion
uint64_t _threads_started_metric{0};
```
##
be/src/util/thread.h:
##
@@ -291,4 +293,86 @@
// Registers /threadz with the debug webserver.
void register_thread_display_page(WebPageHandler* web_page_handler);
+// A singleton class that tracks all live threads, and groups them together
for easy
+// auditing. Used only by Thread.
+class ThreadMgr {
+public:
+static ThreadMgr* instance();
+
+ThreadMgr() : _threads_started_metric(0), _threads_running_metric(0) {}
+~ThreadMgr() {
+std::unique_lock lock(_lock);
+_thread_categories.clear();
+}
+
+static void set_thread_name(const std::string& name, int64_t tid);
+
+#ifndef __APPLE__
+static void set_idle_sched(int64_t tid);
+
+static void set_thread_nice_value(int64_t tid);
+#endif
+
+// not the system TID, since pthread_t is less prone to being recycled.
+void add_thread(const pthread_t& pthread_id, const std::string& name,
+const std::string& category, int64_t tid);
+
+// Removes a thread from the supplied category. If the thread has
+// already been removed, this is a no-op.
+void remove_thread(const pthread_t& pthread_id, const std::string&
category);
+
+void display_thread_callback(const WebPageHandler::ArgumentMap& args,
EasyJson* ej) const;
+void update_threads_metrics();
+
+private:
+// Container class for any details we want to capture about a thread
+// TODO: Add start-time.
+// TODO: Track fragment ID.
+cl
Re: [PR] [Enhancement] (doris-future) Support auto partition name function [doris]
zclllyybb commented on code in PR #34258:
URL: https://github.com/apache/doris/pull/34258#discussion_r1703155402
##
be/src/vec/functions/function_string.h:
##
@@ -386,6 +389,204 @@ class FunctionStrcmp : public IFunction {
}
};
+class FunctionAutoPartitionName : public IFunction {
+public:
+static constexpr auto name = "auto_partition_name";
+static FunctionPtr create() { return
std::make_shared(); }
+String get_name() const override { return name; }
+size_t get_number_of_arguments() const override { return 0; }
+bool is_variadic() const override { return true; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+return std::make_shared();
+}
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) const override
{
+int argument_size = arguments.size();
+if (argument_size < 2) {
+return Status::InvalidArgument(
+"auto_partition_name must contains at least two
arguments");
+}
+std::vector chars_list(argument_size);
+std::vector offsets_list(argument_size);
+std::vector is_const_args(argument_size);
+
+for (int i = 0; i < argument_size; ++i) {
+const auto& [col, is_const] =
+
unpack_if_const(block.get_by_position(arguments[i]).column);
+
+const auto* col_str = assert_cast(col.get());
+chars_list[i] = &col_str->get_chars();
+offsets_list[i] = &col_str->get_offsets();
+is_const_args[i] = is_const;
+}
+auto res = ColumnString::create();
+auto& res_data = res->get_chars();
+auto& res_offset = res->get_offsets();
+res_offset.resize(input_rows_count);
+
+const char* partition_type = chars_list[0]->raw_data();
+// partition type is list|range
+if (std::strncmp(partition_type, "list", 4) == 0) {
+int curr_len = 0;
+for (int row = 0; row < input_rows_count; row++) {
+std::string res_p = "p";
+for (int col = 1; col < argument_size; col++) {
+const auto& current_offsets = *offsets_list[col];
+const auto& current_chars = *chars_list[col];
+
+auto idx = index_check_const(row, is_const_args[col]);
+int size = current_offsets[idx] - current_offsets[idx - 1];
+const char* raw_chars =
+reinterpret_cast(¤t_chars[current_offsets[idx - 1]]);
+
+// convert string to u16string in order to convert to
unicode strings
+const std::string raw_str(raw_chars, size);
+auto u16string = _string_to_u16string(raw_str);
+res_p += _string_to_unicode(u16string) +
std::to_string(u16string.size());
+}
+
+// check the name of length
+int len = res_p.size();
+if (len > 50) [[unlikely]] {
+return Status::InvalidArgument(
+"The list partition name cannot exceed 50
characters");
+}
+curr_len += len;
+res_data.resize(curr_len);
+memcpy(&res_data[res_offset[row - 1]], res_p.c_str(), len);
+res_offset[row] = res_offset[row - 1] + len;
+}
+} else {
+const char* range_type = chars_list[1]->raw_data();
+
+res_data.resize(15 * input_rows_count);
+for (int i = 0; i < input_rows_count; i++) {
+const auto& current_offsets = *offsets_list[2];
+const auto& current_chars = *chars_list[2];
+
+auto idx = index_check_const(i, is_const_args[2]);
+int size = current_offsets[idx] - current_offsets[idx - 1];
+const char* tmp =
+reinterpret_cast(¤t_chars[current_offsets[idx - 1]]);
+std::string to_split_s(tmp, size);
+
+// check the str if it is date|datetime
+RE2 date_regex(R"(^\d{4}-\d{2}-\d{2}( \d{2}:\d{2}:\d{2})?$)");
+if (!RE2::PartialMatch(to_split_s, date_regex)) {
+return Status::InvalidArgument(
+"The range partition only support DATE|DATETIME");
+}
+
+// split date_str from (-mm-dd hh:mm:ss) to ([, mm,
dd, hh, mm, ss])
+std::vector date_str(6);
+date_str[0] = to_split_s.substr(0, 4);
+for (int i = 5, j = 1; i <= size; i += 3, j++) {
+date_str[j] = to_split_s.substr(i, 2);
+}
+int curr_len = 1;
+
+res_data[res_offset[i - 1]] = 'p';
+// raw => 202
Re: [PR] [fix](cloud) Optimize prompt messages when no cluster [doris]
doris-robot commented on PR #38834: URL: https://github.com/apache/doris/pull/38834#issuecomment-2267521107 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](cloud) Optimize prompt messages when no cluster [doris]
deardeng opened a new pull request, #38834: URL: https://github.com/apache/doris/pull/38834 ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) Optimize prompt messages when no cluster [doris]
deardeng commented on PR #38834: URL: https://github.com/apache/doris/pull/38834#issuecomment-2267521652 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
doris-robot commented on PR #38833: URL: https://github.com/apache/doris/pull/38833#issuecomment-2267531360 TPC-H: Total hot run time: 41684 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 8a37e9acaacc4dca7839c3adb60937afc13169f9, data reload: false -- Round 1 -- q1 17667 420041034103 q2 2015198 193 193 q3 10472 123813331238 q4 10174 836 934 836 q5 7617302130083008 q6 219 133 134 133 q7 1067613 604 604 q8 9430188019851880 q9 8571660866396608 q10 8739390338453845 q11 437 246 256 246 q12 408 225 222 222 q13 17757 294229422942 q14 278 248 251 248 q15 526 496 494 494 q16 543 404 385 385 q17 993 922 936 922 q18 8131738872267226 q19 1491121512231215 q20 571 330 322 322 q21 5536474947374737 q22 352 285 277 277 Total cold run time: 112994 ms Total hot run time: 41684 ms - Round 2, with runtime_filter_mode=off - q1 4125404940434043 q2 334 219 215 215 q3 3040303731943037 q4 2012204019511951 q5 5693551454615461 q6 222 130 135 130 q7 2165186718641864 q8 3348341333823382 q9 9113920391849184 q10 4051390240713902 q11 559 444 468 444 q12 791 598 586 586 q13 14368 313131103110 q14 298 276 267 267 q15 540 477 507 477 q16 452 431 420 420 q17 1790176317841763 q18 8328793677797779 q19 3068172917281728 q20 2078184118451841 q21 5753542554775425 q22 538 470 476 470 Total cold run time: 72666 ms Total hot run time: 57479 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) Optimize prompt messages when no cluster [doris]
doris-robot commented on PR #38834: URL: https://github.com/apache/doris/pull/38834#issuecomment-2267532919 TPC-H: Total hot run time: 41550 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit a6b4f5db5ffd8c2b0981d9e1e70a399b0750b4aa, data reload: false -- Round 1 -- q1 17592 413940574057 q2 2030198 197 197 q3 10472 130713091307 q4 10177 874 960 874 q5 7642296529612961 q6 222 144 139 139 q7 1078610 605 605 q8 9410183519151835 q9 8505660166326601 q10 8739381138173811 q11 433 245 251 245 q12 408 229 232 229 q13 17784 292529272925 q14 265 240 249 240 q15 527 486 489 486 q16 523 385 385 385 q17 952 939 881 881 q18 8007730174287301 q19 1871115612031156 q20 560 326 340 326 q21 5273475647034703 q22 356 286 286 286 Total cold run time: 112826 ms Total hot run time: 41550 ms - Round 2, with runtime_filter_mode=off - q1 4066401239913991 q2 333 240 223 223 q3 2966296131172961 q4 1973200919501950 q5 5563546954535453 q6 220 129 130 129 q7 2085175618001756 q8 3288337233233323 q9 9151907092459070 q10 3964400939473947 q11 559 466 461 461 q12 774 608 607 607 q13 16965 315931293129 q14 288 294 276 276 q15 530 491 486 486 q16 456 408 409 408 q17 1729175817321732 q18 8228783078747830 q19 1886171917341719 q20 2049184618401840 q21 5718528453635284 q22 547 466 470 466 Total cold run time: 73338 ms Total hot run time: 57041 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
doris-robot commented on PR #38833: URL: https://github.com/apache/doris/pull/38833#issuecomment-2267534360 TPC-DS: Total hot run time: 169710 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 8a37e9acaacc4dca7839c3adb60937afc13169f9, data reload: false query1 909 378 376 376 query2 6463179717161716 query3 6649219 238 219 query4 20519 17466 17345 17345 query5 3643530 523 523 query6 281 169 172 169 query7 4605292 301 292 query8 259 215 205 205 query9 8523236823462346 query10 425 284 275 275 query11 10671 10055 99199919 query12 126 93 90 90 query13 1639406 381 381 query14 9401708468846884 query15 215 162 161 161 query16 6980412 443 412 query17 940 559 569 559 query18 1785277 277 277 query19 193 149 144 144 query20 93 87 86 86 query21 201 107 99 99 query22 4116412039883988 query23 34281 33709 33788 33709 query24 9255315830833083 query25 656 417 405 405 query26 1505151 156 151 query27 3056288 290 288 query28 7207199119761976 query29 1112426 436 426 query30 232 153 157 153 query31 961 802 773 773 query32 103 55 57 55 query33 679 314 324 314 query34 932 495 504 495 query35 879 761 775 761 query36 1021890 915 890 query37 191 81 88 81 query38 3032292228402840 query39 870 816 832 816 query40 281 114 117 114 query41 49 44 45 44 query42 122 102 97 97 query43 487 428 441 428 query44 1162719 728 719 query45 212 185 182 182 query46 1091806 788 788 query47 1819169417191694 query48 358 292 295 292 query49 951 429 441 429 query50 899 436 437 436 query51 6862671566516651 query52 104 99 92 92 query53 258 188 188 188 query54 624 453 451 451 query55 81 74 75 74 query56 288 258 254 254 query57 1144104310371037 query58 271 262 289 262 query59 2733260124462446 query60 295 281 289 281 query61 102 90 93 90 query62 884 675 660 660 query63 218 182 183 182 query64 5603196418691869 query65 3163310031313100 query66 1021328 334 328 query67 15400 15019 14727 14727 query68 4311566 567 566 query69 438 305 299 299 query70 1071106910621062 query71 382 285 276 276 query72 7122272724542454 query73 760 324 331 324 query74 6150567255755575 query75 3384274627242724 query76 2016124712581247 query77 449 309 311 309 query78 9435894888508850 query79 1912528 522 522 query80 1114528 567 528 query81 566 230 229 229 query82 1034136 141 136 query83 253 178 180 178 query84 272 79 80 79 query85 1123315 309 309 query86 455 329 307 307 query87 3321320030803080 query88 2955244124012401 query89 387 297 289 289 query90 1693195 195 195 query91 125 101 101 101 query92 58 50 52 50 query93 1639613 616 613 query94 775 301 301 301 query95 393 289 279 279 query96 613 282 286 282 query97 3198306030573057 query98 218 205 197 197 query99 1621130012941294 Total cold run time: 260333 ms Total hot run time: 169710 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
doris-robot commented on PR #38833: URL: https://github.com/apache/doris/pull/38833#issuecomment-2267535968 ClickBench: Total hot run time: 30.16 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 8a37e9acaacc4dca7839c3adb60937afc13169f9, data reload: false query1 0.050.040.04 query2 0.080.040.04 query3 0.220.040.04 query4 1.690.060.07 query5 0.490.480.47 query6 1.130.720.71 query7 0.020.010.02 query8 0.060.040.04 query9 0.570.510.50 query10 0.560.560.59 query11 0.160.120.12 query12 0.150.120.12 query13 0.620.620.59 query14 0.770.810.80 query15 0.920.870.88 query16 0.350.350.36 query17 0.990.980.99 query18 0.230.220.21 query19 1.871.711.72 query20 0.010.000.01 query21 15.39 0.740.66 query22 3.736.791.53 query23 17.92 1.281.29 query24 2.210.220.22 query25 0.180.080.08 query26 0.320.210.21 query27 0.460.230.22 query28 13.52 1.000.98 query29 12.80 3.363.31 query30 0.260.050.05 query31 2.880.420.40 query32 3.240.510.48 query33 2.972.942.94 query34 15.44 4.274.24 query35 4.304.304.28 query36 0.670.490.47 query37 0.190.160.16 query38 0.160.150.15 query39 0.040.030.04 query40 0.170.130.13 query41 0.100.040.05 query42 0.060.040.05 query43 0.040.040.04 Total cold run time: 107.99 s Total hot run time: 30.16 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) Optimize prompt messages when no cluster [doris]
doris-robot commented on PR #38834: URL: https://github.com/apache/doris/pull/38834#issuecomment-2267535983 TPC-DS: Total hot run time: 169057 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit a6b4f5db5ffd8c2b0981d9e1e70a399b0750b4aa, data reload: false query1 908 381 373 373 query2 6398162316101610 query3 6669213 219 213 query4 20148 17518 17155 17155 query5 3682522 510 510 query6 296 166 177 166 query7 4618291 289 289 query8 250 201 188 188 query9 8517232723302327 query10 444 271 273 271 query11 10534 10063 10033 10033 query12 122 89 84 84 query13 1649391 403 391 query14 9687750868616861 query15 215 161 171 161 query16 6951485 461 461 query17 965 573 561 561 query18 1885291 287 287 query19 196 149 148 148 query20 91 87 86 86 query21 205 106 101 101 query22 4129412939543954 query23 34157 33484 33711 33484 query24 10203 319031293129 query25 684 425 401 401 query26 1596149 149 149 query27 2957291 288 288 query28 7451199719681968 query29 1169437 437 437 query30 248 160 155 155 query31 942 764 788 764 query32 106 55 61 55 query33 676 338 336 336 query34 950 521 508 508 query35 888 781 738 738 query36 1039884 882 882 query37 213 79 80 79 query38 2942287627772777 query39 902 817 815 815 query40 269 122 107 107 query41 45 46 43 43 query42 121 97 99 97 query43 458 409 412 409 query44 1197735 761 735 query45 209 177 177 177 query46 1085808 786 786 query47 1823173617261726 query48 366 296 299 296 query49 938 430 417 417 query50 892 445 430 430 query51 6934678967106710 query52 105 100 93 93 query53 253 187 186 186 query54 624 451 463 451 query55 77 74 76 74 query56 277 264 245 245 query57 1167104610391039 query58 265 282 261 261 query59 2612236322842284 query60 288 274 266 266 query61 92 91 92 91 query62 886 659 664 659 query63 211 180 183 180 query64 5673191718391839 query65 3168314030993099 query66 1319337 328 328 query67 15248 14785 15089 14785 query68 6044589 586 586 query69 719 372 301 301 query70 1126106011101060 query71 473 283 280 280 query72 7641266824672467 query73 918 331 330 330 query74 6121566755615561 query75 3666271727742717 query76 3466125412591254 query77 665 316 309 309 query78 9522884489558844 query79 1994535 526 526 query80 926 492 489 489 query81 556 223 219 219 query82 766 132 127 127 query83 197 167 173 167 query84 270 83 79 79 query85 1320342 294 294 query86 463 291 298 291 query87 3284311830913091 query88 3613250425732504 query89 403 279 287 279 query90 1795195 192 192 query91 125 98 97 97 query92 56 51 50 50 query93 1738610 624 610 query94 747 290 294 290 query95 391 274 261 261 query96 619 283 282 282 query97 3230306030363036 query98 216 198 194 194 query99 1658130812621262 Total cold run time: 267012 ms Total hot run time: 169057 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [fix](cloud) Optimize prompt messages when no cluster [doris]
doris-robot commented on PR #38834: URL: https://github.com/apache/doris/pull/38834#issuecomment-2267537692 ClickBench: Total hot run time: 30.46 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit a6b4f5db5ffd8c2b0981d9e1e70a399b0750b4aa, data reload: false query1 0.050.050.04 query2 0.070.040.04 query3 0.220.040.04 query4 1.690.070.06 query5 0.480.470.48 query6 1.150.710.72 query7 0.020.020.01 query8 0.040.050.04 query9 0.560.520.50 query10 0.560.570.57 query11 0.160.120.11 query12 0.140.120.12 query13 0.610.600.61 query14 0.780.790.79 query15 0.890.860.85 query16 0.360.360.36 query17 1.000.980.99 query18 0.220.210.22 query19 1.821.741.78 query20 0.010.000.01 query21 15.40 0.740.65 query22 3.857.301.79 query23 18.06 1.231.23 query24 2.180.230.22 query25 0.190.080.08 query26 0.310.210.22 query27 0.460.230.23 query28 13.25 1.010.97 query29 12.55 3.333.31 query30 0.250.070.07 query31 2.880.410.40 query32 3.230.480.49 query33 2.912.972.97 query34 15.42 4.274.25 query35 4.284.294.29 query36 0.680.470.50 query37 0.200.170.16 query38 0.160.150.15 query39 0.040.040.04 query40 0.160.120.13 query41 0.100.050.06 query42 0.050.060.04 query43 0.040.040.04 Total cold run time: 107.48 s Total hot run time: 30.46 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](load) Fix some spelling errors [doris-website]
liaoxin01 merged PR #946: URL: https://github.com/apache/doris-website/pull/946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [Fix](load) Fix some spelling errors (#946)
This is an automated email from the ASF dual-hosted git repository. liaoxin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 1feeafba009 [Fix](load) Fix some spelling errors (#946) 1feeafba009 is described below commit 1feeafba0097991199e23e65d11f59be51d9acee Author: Xin Liao AuthorDate: Sun Aug 4 21:31:38 2024 +0800 [Fix](load) Fix some spelling errors (#946) --- docs/data-operate/import/error-data-handling.md | 4 ++-- docs/data-operate/import/load-data-format.md | 12 ++-- .../current/data-operate/import/error-data-handling.md | 4 ++-- .../current/data-operate/import/load-data-format.md | 10 +- 4 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/data-operate/import/error-data-handling.md b/docs/data-operate/import/error-data-handling.md index 0d534e1f737..b45dfb2acfd 100644 --- a/docs/data-operate/import/error-data-handling.md +++ b/docs/data-operate/import/error-data-handling.md @@ -278,5 +278,5 @@ The default value of `max_filter_ratio` is 0, which means that if there is any e INSERT INTO my_table FROM S3/HDFS/LOCAL(); ``` :::tip -Only effective when the value of `enable_insert_strict` is `false`, used to control the maximum error rate when using `INSERT INTO FROM S3/HDFS/LOCAL()`. The default value is 1.0, which means tolerating all errors. - \ No newline at end of file +The `insert_max_filter_ratio` only takes effect when the value of `enable_insert_strict` is `false`, and it is used to control the maximum error rate of `INSERT INTO FROM S3/HDFS/LOCAL()`. The default value is 1.0, which means tolerating all errors. +::: \ No newline at end of file diff --git a/docs/data-operate/import/load-data-format.md b/docs/data-operate/import/load-data-format.md index 1cad6ae3249..86657322539 100644 --- a/docs/data-operate/import/load-data-format.md +++ b/docs/data-operate/import/load-data-format.md @@ -133,11 +133,11 @@ Doris supports importing data in JSON format. This document mainly describes the Currently, only the following import methods support data import in JSON format: -- Through [S3 table function](../../sql-manual/sql-functions/table-functions/s3.md) import statement: insert into table select * from S3(); -- Import the local JSON format file through [STREAM LOAD](../../sql-manual/sql-statements/Data-Manipulation-Statements/Load/STREAM-LOAD.md). -- Subscribe and consume JSON format in Kafka via [ROUTINE LOAD](../../sql-manual/sql-statements/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md) information. - -Other ways of importing data in JSON format are not currently supported. +- [Stream Load](./import-way/stream-load-manual.md) +- [Broker Load](./import-way/broker-load-manual.md) +- [Routine Load](./import-way/routine-load-manual.md) +- [INSERT INTO FROM S3 TVF](../../sql-manual/sql-functions/table-functions/s3) +- [INSERT INTO FROM HDFS TVF](../../sql-manual/sql-functions/table-functions/hdfs) ### Supported JSON Formats @@ -209,7 +209,7 @@ Currently only the following three JSON formats are supported: This feature requires that each row of data in the Array has exactly the same order of fields. Doris will only parse according to the field order of the first row, and then access the subsequent data in the form of subscripts. This method can improve the import efficiency by 3-5X. -## JSON Path +### JSON Path Doris supports extracting data specified in JSON through JSON Path. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/error-data-handling.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/error-data-handling.md index e668294da6c..1e404a873ce 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/error-data-handling.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/error-data-handling.md @@ -278,5 +278,5 @@ curl --location-trusted -u root -H "partial_columns:true" -H "strict_mode:true" INSERT INTO my_table FROM S3/HDFS/LOCAL(); ``` :::tip -仅当 `enable_insert_strict` 值为 `false` 时生效，用于控制使用 `INSERT INTO FROM S3/HDFS/LOCAL()` 时，设定最大错误率的。默认为 1.0 表示容忍所有错误。 - +仅当 `enable_insert_strict` 值为 `false` 时， `insert_max_filter_ratio` 才生效，只用于控制 `INSERT INTO FROM S3/HDFS/LOCAL()` 的最大错误率。默认为 1.0，表示容忍所有错误。 +::: diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/load-data-format.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/load-data-format.md index cab7adb3351..67969edba3f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/load-data-format.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/load-data-format.md @@ -29,7 +29,7 @@ Doris 支持对csv、json、parquet、orc格式的数据文件进行导入。本 ## CS
Re: [PR] [Improvement](runtime-filter) adjust always true judge rule of runtime filter apply [doris]
BiteThet commented on PR #38300: URL: https://github.com/apache/doris/pull/38300#issuecomment-2267546846 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compaction) fix mismatch between segment key and value column rows during compaction (#37960) [doris]
doris-robot commented on PR #38835: URL: https://github.com/apache/doris/pull/38835#issuecomment-2267555262 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](compaction) fix mismatch between segment key and value column rows during compaction (#37960) [doris]
luwei16 opened a new pull request, #38835: URL: https://github.com/apache/doris/pull/38835 pick master #37960 #38251 #38356 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compaction) fix mismatch between segment key and value column rows during compaction (#37960) [doris]
luwei16 commented on PR #38835: URL: https://github.com/apache/doris/pull/38835#issuecomment-2267555318 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](routine load) add routine load case with timestamp as offset(#38567) [doris]
dataroaring merged PR #38822: URL: https://github.com/apache/doris/pull/38822 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [test](routine load) add routine load case with timestamp as offset(#38567) (#38822)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 8e4fad99a1b [test](routine load) add routine load case with timestamp
as offset(#38567) (#38822)
8e4fad99a1b is described below
commit 8e4fad99a1bc981b8731ab6a754a39480ec06a9c
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Aug 4 22:05:19 2024 +0800
[test](routine load) add routine load case with timestamp as offset(#38567)
(#38822)
pick (#38567)
---
.../routine_load/test_routine_load_offset.out | 4 +
.../load_p0/routine_load/data/test_offset_time.csv | 1 +
.../routine_load/test_routine_load_offset.groovy | 113 +
3 files changed, 118 insertions(+)
diff --git
a/regression-test/data/load_p0/routine_load/test_routine_load_offset.out
b/regression-test/data/load_p0/routine_load/test_routine_load_offset.out
new file mode 100644
index 000..4c7c19cfe11
--- /dev/null
+++ b/regression-test/data/load_p0/routine_load/test_routine_load_offset.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !sql_offset --
+1 eab 2023-07-15 def 2023-07-20T05:48:31 "ghi"
+
diff --git
a/regression-test/suites/load_p0/routine_load/data/test_offset_time.csv
b/regression-test/suites/load_p0/routine_load/data/test_offset_time.csv
new file mode 100644
index 000..c712640bcde
--- /dev/null
+++ b/regression-test/suites/load_p0/routine_load/data/test_offset_time.csv
@@ -0,0 +1 @@
+1,eab,2023-07-15,def,2023-07-20:05:48:31,"ghi"
diff --git
a/regression-test/suites/load_p0/routine_load/test_routine_load_offset.groovy
b/regression-test/suites/load_p0/routine_load/test_routine_load_offset.groovy
new file mode 100644
index 000..84d0509cea3
--- /dev/null
+++
b/regression-test/suites/load_p0/routine_load/test_routine_load_offset.groovy
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.kafka.clients.admin.AdminClient
+import org.apache.kafka.clients.producer.KafkaProducer
+import org.apache.kafka.clients.producer.ProducerRecord
+import org.apache.kafka.clients.producer.ProducerConfig
+
+suite("test_routine_load_offset","p0") {
+def kafkaCsvTpoics = [
+ "test_offset_time",
+]
+String enabled = context.config.otherConfigs.get("enableKafkaTest")
+String kafka_port = context.config.otherConfigs.get("kafka_port")
+String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+def kafka_broker = "${externalEnvIp}:${kafka_port}"
+if (enabled != null && enabled.equalsIgnoreCase("true")) {
+// define kafka
+def props = new Properties()
+props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"${kafka_broker}".toString())
+props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer")
+props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer")
+// Create kafka producer
+def producer = new KafkaProducer<>(props)
+
+for (String kafkaCsvTopic in kafkaCsvTpoics) {
+def txt = new
File("""${context.file.parent}/data/${kafkaCsvTopic}.csv""").text
+def lines = txt.readLines()
+lines.each { line ->
+logger.info("=${line}")
+def record = new ProducerRecord<>(kafkaCsvTopic, null, line)
+producer.send(record)
+}
+}
+}
+
+if (enabled != null && enabled.equalsIgnoreCase("true")) {
+def tableName = "test_routine_load_offset"
+def job = "test_offset"
+
+sql """ DROP TABLE IF EXISTS ${tableName} """
+sql """
+CREATE TABLE IF NOT EXISTS ${tableName} (
+`k1` int(20) NULL,
+`k2` string NULL,
+`v1` date NULL,
+`v2` string NULL,
+`v3` datetime NULL,
+`v4` string NULL
Re: [PR] [revert](cloud-merge) Support shadow tablet to do cumulative compaction in cloud mode (apache#37293)" [doris]
dataroaring merged PR #38828: URL: https://github.com/apache/doris/pull/38828 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (ef4fba649cb -> 309f3b674ac)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ef4fba649cb [Fix](group commit) Fix multiple cluster group commit BE select strategy (#38644) add 309f3b674ac [revert](cloud-merge) Support shadow tablet to do cumulative compaction in cloud mode (apache#37293)" (#38828) No new revisions were added by this update. Summary of changes: be/src/cloud/cloud_base_compaction.cpp | 38 +-- be/src/cloud/cloud_cumulative_compaction.cpp | 46 +--- .../cloud/cloud_engine_calc_delete_bitmap_task.cpp | 18 +- be/src/cloud/cloud_meta_mgr.cpp| 5 +- be/src/cloud/cloud_schema_change_job.cpp | 104 be/src/cloud/cloud_schema_change_job.h | 3 +- be/src/cloud/cloud_storage_engine.cpp | 14 +- be/src/cloud/cloud_tablet.cpp | 32 +-- be/src/cloud/cloud_tablet.h| 4 - be/src/cloud/config.cpp| 2 - be/src/cloud/config.h | 1 - be/src/olap/base_tablet.cpp| 2 +- cloud/src/meta-service/meta_service_job.cpp| 164 ++-- cloud/test/meta_service_job_test.cpp | 264 ++- .../org/apache/doris/alter/CloudRollupJobV2.java | 14 -- .../apache/doris/alter/CloudSchemaChangeJobV2.java | 36 +-- .../java/org/apache/doris/alter/RollupJobV2.java | 2 +- .../org/apache/doris/alter/SchemaChangeJobV2.java | 8 +- .../cloud/datasource/CloudInternalCatalog.java | 61 - .../apache/doris/cloud/rpc/MetaServiceClient.java | 11 - .../apache/doris/cloud/rpc/MetaServiceProxy.java | 10 - .../java/org/apache/doris/master/MetaHelper.java | 8 +- .../org/apache/doris/persist/meta/MetaWriter.java | 4 +- gensrc/proto/cloud.proto | 4 - .../pipeline/cloud_p0/conf/be_custom.conf | 1 - .../plugins/plugin_curl_requester.groovy | 4 - .../test_schema_change_with_compaction1.groovy | 257 --- .../test_schema_change_with_compaction10.groovy| 263 --- .../test_schema_change_with_compaction11.groovy| 280 - .../test_schema_change_with_compaction2.groovy | 214 .../test_schema_change_with_compaction3.groovy | 194 -- .../test_schema_change_with_compaction4.groovy | 194 -- .../test_schema_change_with_compaction5.groovy | 260 --- .../test_schema_change_with_compaction6.groovy | 263 --- .../test_schema_change_with_compaction7.groovy | 256 --- .../test_schema_change_with_compaction8.groovy | 214 .../test_schema_change_with_compaction9.groovy | 260 --- .../cloud_p0/schema_change/ddl/date_create.sql | 24 -- .../cloud_p0/schema_change/ddl/date_delete.sql | 1 - .../cloud_p0/schema_change/ddl/date_load.sql | 6 - .../schema_change/ddl/date_unique_create.sql | 27 -- 41 files changed, 151 insertions(+), 3422 deletions(-) delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction1/test_schema_change_with_compaction1.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction10/test_schema_change_with_compaction10.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction11/test_schema_change_with_compaction11.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction2/test_schema_change_with_compaction2.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction3/test_schema_change_with_compaction3.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction4/test_schema_change_with_compaction4.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction5/test_schema_change_with_compaction5.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction6/test_schema_change_with_compaction6.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction7/test_schema_change_with_compaction7.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction8/test_schema_change_with_compaction8.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/compaction9/test_schema_change_with_compaction9.groovy delete mode 100644 regression-test/suites/cloud_p0/schema_change/ddl/date_create.sql delete mode 100644 regression-test/suites/cloud_p0/schema_change/ddl/date_delete.sql delete mode 100644 regression-test/suites/cloud_p0/schema_change/ddl/date_load.sql delete mode 100644 regression-test/suites/cloud_p0/schema_change/ddl/date_unique_create.sql -
Re: [PR] [fix](compaction) fix mismatch between segment key and value column rows during compaction (#37960) [doris]
github-actions[bot] commented on PR #38835: URL: https://github.com/apache/doris/pull/38835#issuecomment-2267557687 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](multi table) fix single stream multi table memory leak (#38255) [doris]
dataroaring merged PR #38824: URL: https://github.com/apache/doris/pull/38824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](multi table) fix single stream multi table memory leak (#38255) (#38824)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 6035edad0b1 [fix](multi table) fix single stream multi table memory
leak (#38255) (#38824)
6035edad0b1 is described below
commit 6035edad0b19a0cbe41ea97406651594d6a21d2c
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Aug 4 22:12:44 2024 +0800
[fix](multi table) fix single stream multi table memory leak (#38255)
(#38824)
pick (#38255)
We meet OOM when using single stream multi table
It exist memory leak, and heap profile like:
The stream load context will not release in some exception conditions as
plan failed for high concurrency causing timeout when obtaining read
lock. It is introduced by https://github.com/apache/doris/pull/35458
The solution effect is shown in the following figure, which can run
stably with a small amount of memory
---
be/src/io/fs/multi_table_pipe.cpp | 13 +
1 file changed, 13 insertions(+)
diff --git a/be/src/io/fs/multi_table_pipe.cpp
b/be/src/io/fs/multi_table_pipe.cpp
index fa38b6440c1..4469174211e 100644
--- a/be/src/io/fs/multi_table_pipe.cpp
+++ b/be/src/io/fs/multi_table_pipe.cpp
@@ -324,6 +324,19 @@ void MultiTablePipe::_handle_consumer_finished() {
_ctx->number_filtered_rows = _number_filtered_rows;
_ctx->number_unselected_rows = _number_unselected_rows;
_ctx->commit_infos = _tablet_commit_infos;
+
+// remove ctx to avoid memory leak.
+for (const auto& pair : _planned_tables) {
+if (pair.second) {
+
doris::ExecEnv::GetInstance()->new_load_stream_mgr()->remove(pair.second->id);
+}
+}
+for (const auto& pair : _unplanned_tables) {
+if (pair.second) {
+
doris::ExecEnv::GetInstance()->new_load_stream_mgr()->remove(pair.second->id);
+}
+}
+
LOG(INFO) << "all plan for multi-table load complete. number_total_rows="
<< _ctx->number_total_rows << " number_loaded_rows=" <<
_ctx->number_loaded_rows
<< " number_filtered_rows=" << _ctx->number_filtered_rows
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](routine load) fix enclose and escape can not set in routine load job (#38402) [doris]
dataroaring merged PR #38825: URL: https://github.com/apache/doris/pull/38825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](routine load) fix enclose and escape can not set in routine load job (#38402) (#38825)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 79b07d0b8a4 [fix](routine load) fix enclose and escape can not set in
routine load job (#38402) (#38825)
79b07d0b8a4 is described below
commit 79b07d0b8a4e179c463cf450f0b991e7e194fd60
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Aug 4 22:17:12 2024 +0800
[fix](routine load) fix enclose and escape can not set in routine load job
(#38402) (#38825)
pick (#38402)
---
.../doris/analysis/AlterRoutineLoadStmt.java | 8 +
.../doris/analysis/CreateRoutineLoadStmt.java | 32 ++--
.../doris/load/routineload/RoutineLoadJob.java | 10 +-
.../routine_load/test_routine_load_property.out| 10 ++
.../routine_load/data/test_enclose_and_escape0.csv | 1 +
.../routine_load/data/test_enclose_and_escape1.csv | 1 +
.../routine_load/test_routine_load_property.groovy | 187 +
7 files changed, 235 insertions(+), 14 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterRoutineLoadStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterRoutineLoadStmt.java
index 5a1f1ba56aa..526b2f179e8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterRoutineLoadStmt.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterRoutineLoadStmt.java
@@ -68,6 +68,8 @@ public class AlterRoutineLoadStmt extends DdlStmt {
.add(LoadStmt.STRICT_MODE)
.add(LoadStmt.TIMEZONE)
.add(CreateRoutineLoadStmt.WORKLOAD_GROUP)
+.add(LoadStmt.KEY_ENCLOSE)
+.add(LoadStmt.KEY_ESCAPE)
.build();
private final LabelName labelName;
@@ -250,6 +252,12 @@ public class AlterRoutineLoadStmt extends DdlStmt {
.getWorkloadGroup(ConnectContext.get().getCurrentUserIdentity(), workloadGroup);
analyzedJobProperties.put(CreateRoutineLoadStmt.WORKLOAD_GROUP,
String.valueOf(wgId));
}
+if (jobProperties.containsKey(LoadStmt.KEY_ENCLOSE)) {
+analyzedJobProperties.put(LoadStmt.KEY_ENCLOSE,
jobProperties.get(LoadStmt.KEY_ENCLOSE));
+}
+if (jobProperties.containsKey(LoadStmt.KEY_ESCAPE)) {
+analyzedJobProperties.put(LoadStmt.KEY_ESCAPE,
jobProperties.get(LoadStmt.KEY_ESCAPE));
+}
}
private void checkDataSourceProperties() throws UserException {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoutineLoadStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoutineLoadStmt.java
index d58b25195c7..fb794bfa7b3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoutineLoadStmt.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoutineLoadStmt.java
@@ -141,6 +141,8 @@ public class CreateRoutineLoadStmt extends DdlStmt {
.add(LOAD_TO_SINGLE_TABLET)
.add(PARTIAL_COLUMNS)
.add(WORKLOAD_GROUP)
+.add(LoadStmt.KEY_ENCLOSE)
+.add(LoadStmt.KEY_ESCAPE)
.build();
private final LabelName labelName;
@@ -178,9 +180,9 @@ public class CreateRoutineLoadStmt extends DdlStmt {
private boolean numAsString = false;
private boolean fuzzyParse = false;
-private String enclose;
+private byte enclose;
-private String escape;
+private byte escape;
private long workloadGroupId = -1;
@@ -311,11 +313,11 @@ public class CreateRoutineLoadStmt extends DdlStmt {
return jsonPaths;
}
-public String getEnclose() {
+public byte getEnclose() {
return enclose;
}
-public String getEscape() {
+public byte getEscape() {
return escape;
}
@@ -507,14 +509,24 @@ public class CreateRoutineLoadStmt extends DdlStmt {
loadToSingleTablet =
Util.getBooleanPropertyOrDefault(jobProperties.get(LoadStmt.LOAD_TO_SINGLE_TABLET),
RoutineLoadJob.DEFAULT_LOAD_TO_SINGLE_TABLET,
LoadStmt.LOAD_TO_SINGLE_TABLET + " should be a boolean");
-enclose = jobProperties.get(LoadStmt.KEY_ENCLOSE);
-if (enclose != null && enclose.length() != 1) {
-throw new AnalysisException("enclose must be single-char");
+
+String encloseStr = jobProperties.get(LoadStmt.KEY_ENCLOSE);
+if (encloseStr != null) {
+if (encloseStr.length() != 1) {
+throw new AnalysisException("enclose must be single-char");
+} else {
+enclose = encloseStr.getBytes()[0];
+}
}
-escape = jobProperties.get(LoadStmt.KEY_ESCAPE);
-if (escape != null && escape.length() != 1) {
-throw new AnalysisException("escape must be single-char");
+String escapeStr = jobProperti
(doris) branch branch-3.0 updated (bc954d423fd -> 4942677688e)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from bc954d423fd [Fix](Load)Audit logs avoid recording certain sensitive information (#38769) new ea5072058b2 [test](routine load) add routine load case with timestamp as offset (#38567) new 4942677688e [fix](routine load) fix enclose and escape can not set in routine load job (#38402) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../doris/analysis/AlterRoutineLoadStmt.java | 8 + .../doris/analysis/CreateRoutineLoadStmt.java | 32 ++-- .../doris/load/routineload/RoutineLoadJob.java | 10 +- .../routine_load/test_routine_load_offset.out | 4 + .../routine_load/test_routine_load_property.out| 10 ++ .../routine_load/data/test_enclose_and_escape0.csv | 1 + .../routine_load/data/test_enclose_and_escape1.csv | 1 + .../load_p0/routine_load/data/test_offset_time.csv | 1 + .../routine_load/test_routine_load_offset.groovy | 113 + .../routine_load/test_routine_load_property.groovy | 187 + 10 files changed, 353 insertions(+), 14 deletions(-) create mode 100644 regression-test/data/load_p0/routine_load/test_routine_load_offset.out create mode 100644 regression-test/data/load_p0/routine_load/test_routine_load_property.out create mode 100644 regression-test/suites/load_p0/routine_load/data/test_enclose_and_escape0.csv create mode 100644 regression-test/suites/load_p0/routine_load/data/test_enclose_and_escape1.csv create mode 100644 regression-test/suites/load_p0/routine_load/data/test_offset_time.csv create mode 100644 regression-test/suites/load_p0/routine_load/test_routine_load_offset.groovy create mode 100644 regression-test/suites/load_p0/routine_load/test_routine_load_property.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](routine load) fix show routine load task result incorrect (#38523) [doris]
dataroaring merged PR #38826: URL: https://github.com/apache/doris/pull/38826 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 02/02: [fix](routine load) fix enclose and escape can not set in routine load job (#38402)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4942677688e3bf722bf9e78f0d241e79addc126b
Author: hui lai <1353307...@qq.com>
AuthorDate: Tue Jul 30 14:45:24 2024 +0800
[fix](routine load) fix enclose and escape can not set in routine load job
(#38402)
---
.../doris/analysis/AlterRoutineLoadStmt.java | 8 +
.../doris/analysis/CreateRoutineLoadStmt.java | 32 ++--
.../doris/load/routineload/RoutineLoadJob.java | 10 +-
.../routine_load/test_routine_load_property.out| 10 ++
.../routine_load/data/test_enclose_and_escape0.csv | 1 +
.../routine_load/data/test_enclose_and_escape1.csv | 1 +
.../routine_load/test_routine_load_property.groovy | 187 +
7 files changed, 235 insertions(+), 14 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterRoutineLoadStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterRoutineLoadStmt.java
index 67c103ed02c..d2a0844dfb4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterRoutineLoadStmt.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterRoutineLoadStmt.java
@@ -68,6 +68,8 @@ public class AlterRoutineLoadStmt extends DdlStmt {
.add(LoadStmt.STRICT_MODE)
.add(LoadStmt.TIMEZONE)
.add(CreateRoutineLoadStmt.WORKLOAD_GROUP)
+.add(LoadStmt.KEY_ENCLOSE)
+.add(LoadStmt.KEY_ESCAPE)
.build();
private final LabelName labelName;
@@ -250,6 +252,12 @@ public class AlterRoutineLoadStmt extends DdlStmt {
.getWorkloadGroup(ConnectContext.get().getCurrentUserIdentity(), workloadGroup);
analyzedJobProperties.put(CreateRoutineLoadStmt.WORKLOAD_GROUP,
String.valueOf(wgId));
}
+if (jobProperties.containsKey(LoadStmt.KEY_ENCLOSE)) {
+analyzedJobProperties.put(LoadStmt.KEY_ENCLOSE,
jobProperties.get(LoadStmt.KEY_ENCLOSE));
+}
+if (jobProperties.containsKey(LoadStmt.KEY_ESCAPE)) {
+analyzedJobProperties.put(LoadStmt.KEY_ESCAPE,
jobProperties.get(LoadStmt.KEY_ESCAPE));
+}
}
private void checkDataSourceProperties() throws UserException {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoutineLoadStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoutineLoadStmt.java
index 95434a1fd19..13654509821 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoutineLoadStmt.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoutineLoadStmt.java
@@ -141,6 +141,8 @@ public class CreateRoutineLoadStmt extends DdlStmt {
.add(LOAD_TO_SINGLE_TABLET)
.add(PARTIAL_COLUMNS)
.add(WORKLOAD_GROUP)
+.add(LoadStmt.KEY_ENCLOSE)
+.add(LoadStmt.KEY_ESCAPE)
.build();
private final LabelName labelName;
@@ -178,9 +180,9 @@ public class CreateRoutineLoadStmt extends DdlStmt {
private boolean numAsString = false;
private boolean fuzzyParse = false;
-private String enclose;
+private byte enclose;
-private String escape;
+private byte escape;
private long workloadGroupId = -1;
@@ -311,11 +313,11 @@ public class CreateRoutineLoadStmt extends DdlStmt {
return jsonPaths;
}
-public String getEnclose() {
+public byte getEnclose() {
return enclose;
}
-public String getEscape() {
+public byte getEscape() {
return escape;
}
@@ -507,14 +509,24 @@ public class CreateRoutineLoadStmt extends DdlStmt {
loadToSingleTablet =
Util.getBooleanPropertyOrDefault(jobProperties.get(LoadStmt.LOAD_TO_SINGLE_TABLET),
RoutineLoadJob.DEFAULT_LOAD_TO_SINGLE_TABLET,
LoadStmt.LOAD_TO_SINGLE_TABLET + " should be a boolean");
-enclose = jobProperties.get(LoadStmt.KEY_ENCLOSE);
-if (enclose != null && enclose.length() != 1) {
-throw new AnalysisException("enclose must be single-char");
+
+String encloseStr = jobProperties.get(LoadStmt.KEY_ENCLOSE);
+if (encloseStr != null) {
+if (encloseStr.length() != 1) {
+throw new AnalysisException("enclose must be single-char");
+} else {
+enclose = encloseStr.getBytes()[0];
+}
}
-escape = jobProperties.get(LoadStmt.KEY_ESCAPE);
-if (escape != null && escape.length() != 1) {
-throw new AnalysisException("escape must be single-char");
+String escapeStr = jobProperties.get(LoadStmt.KEY_ESCAPE);
+if (escapeStr != null) {
+if (escapeStr.length() != 1) {
+throw new AnalysisException("enclose must be single-char");
+} else {
+escape = escapeStr.getBytes()[
(doris) 01/02: [test](routine load) add routine load case with timestamp as offset (#38567)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ea5072058b2ad62d05e1028280f3f4331d0cc0cf
Author: hui lai <1353307...@qq.com>
AuthorDate: Thu Aug 1 10:15:42 2024 +0800
[test](routine load) add routine load case with timestamp as offset (#38567)
Add routine load case with timestamp as offset.
---
.../routine_load/test_routine_load_offset.out | 4 +
.../load_p0/routine_load/data/test_offset_time.csv | 1 +
.../routine_load/test_routine_load_offset.groovy | 113 +
3 files changed, 118 insertions(+)
diff --git
a/regression-test/data/load_p0/routine_load/test_routine_load_offset.out
b/regression-test/data/load_p0/routine_load/test_routine_load_offset.out
new file mode 100644
index 000..4c7c19cfe11
--- /dev/null
+++ b/regression-test/data/load_p0/routine_load/test_routine_load_offset.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !sql_offset --
+1 eab 2023-07-15 def 2023-07-20T05:48:31 "ghi"
+
diff --git
a/regression-test/suites/load_p0/routine_load/data/test_offset_time.csv
b/regression-test/suites/load_p0/routine_load/data/test_offset_time.csv
new file mode 100644
index 000..c712640bcde
--- /dev/null
+++ b/regression-test/suites/load_p0/routine_load/data/test_offset_time.csv
@@ -0,0 +1 @@
+1,eab,2023-07-15,def,2023-07-20:05:48:31,"ghi"
diff --git
a/regression-test/suites/load_p0/routine_load/test_routine_load_offset.groovy
b/regression-test/suites/load_p0/routine_load/test_routine_load_offset.groovy
new file mode 100644
index 000..842b654cc91
--- /dev/null
+++
b/regression-test/suites/load_p0/routine_load/test_routine_load_offset.groovy
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.kafka.clients.admin.AdminClient
+import org.apache.kafka.clients.producer.KafkaProducer
+import org.apache.kafka.clients.producer.ProducerRecord
+import org.apache.kafka.clients.producer.ProducerConfig
+
+suite("test_routine_load_offset","p0") {
+def kafkaCsvTpoics = [
+ "test_offset_time",
+]
+String enabled = context.config.otherConfigs.get("enableKafkaTest")
+String kafka_port = context.config.otherConfigs.get("kafka_port")
+String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+def kafka_broker = "${externalEnvIp}:${kafka_port}"
+if (enabled != null && enabled.equalsIgnoreCase("true")) {
+// define kafka
+def props = new Properties()
+props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"${kafka_broker}".toString())
+props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer")
+props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer")
+// Create kafka producer
+def producer = new KafkaProducer<>(props)
+
+for (String kafkaCsvTopic in kafkaCsvTpoics) {
+def txt = new
File("""${context.file.parent}/data/${kafkaCsvTopic}.csv""").text
+def lines = txt.readLines()
+lines.each { line ->
+logger.info("=${line}")
+def record = new ProducerRecord<>(kafkaCsvTopic, null, line)
+producer.send(record)
+}
+}
+}
+
+if (enabled != null && enabled.equalsIgnoreCase("true")) {
+def tableName = "test_show_routine_load"
+def job = "test_offset"
+
+sql """ DROP TABLE IF EXISTS ${tableName} """
+sql """
+CREATE TABLE IF NOT EXISTS ${tableName} (
+`k1` int(20) NULL,
+`k2` string NULL,
+`v1` date NULL,
+`v2` string NULL,
+`v3` datetime NULL,
+`v4` string NULL
+) ENGINE=OLAP
+DUPLICATE KEY(`k1`)
+COMMENT 'OLAP'
+DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+PROPERTIES ("replication_allocation" = "t
(doris) branch branch-2.1 updated: [fix](routine load) fix show routine load task result incorrect (#38523) (#38826)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new f76397277ec [fix](routine load) fix show routine load task result
incorrect (#38523) (#38826)
f76397277ec is described below
commit f76397277ec44e5cb901c716644fad712500e2dd
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Aug 4 22:18:25 2024 +0800
[fix](routine load) fix show routine load task result incorrect (#38523)
(#38826)
pick (#38523)
Create a job:
```
CREATE ROUTINE LOAD testShow ON test_show_routine_load
COLUMNS TERMINATED BY ","
PROPERTIES
(
"max_batch_interval" = "5",
"max_batch_rows" = "30",
"max_batch_size" = "209715200"
)
FROM KAFKA
(
"kafka_broker_list" = "127.0.0.1:19092",
"kafka_topic" = "test_show_routine_load",
"property.kafka_default_offsets" = "OFFSET_BEGINNING"
);
```
show routine load task:
```
SHOW ROUTINE LOAD TASK WHERE JobName = "testShow";
```
result:
```
ERROR 1105 (HY000): errCode = 2, detailMessage = The job named testshowdoes
not exists or job state is stopped or cancelled
```
Do not use `toLowerCase` method;
---
.../doris/analysis/ShowRoutineLoadTaskStmt.java| 2 +-
.../routine_load/data/test_show_routine_load.csv | 1 +
.../routine_load/test_show_routine_load.groovy | 150 +
3 files changed, 152 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
index 7258822ab50..570b82d3667 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
@@ -121,7 +121,7 @@ public class ShowRoutineLoadTaskStmt extends ShowStmt {
break CHECK;
}
StringLiteral stringLiteral = (StringLiteral)
binaryPredicate.getChild(1);
-jobName = stringLiteral.getValue().toLowerCase();
+jobName = stringLiteral.getValue();
} // CHECKSTYLE IGNORE THIS LINE
if (!valid) {
diff --git
a/regression-test/suites/load_p0/routine_load/data/test_show_routine_load.csv
b/regression-test/suites/load_p0/routine_load/data/test_show_routine_load.csv
new file mode 100644
index 000..b226b99ee4e
--- /dev/null
+++
b/regression-test/suites/load_p0/routine_load/data/test_show_routine_load.csv
@@ -0,0 +1 @@
+1,eab,2023-07-15,def,2023-07-20:05:48:31,"ghi"
\ No newline at end of file
diff --git
a/regression-test/suites/load_p0/routine_load/test_show_routine_load.groovy
b/regression-test/suites/load_p0/routine_load/test_show_routine_load.groovy
new file mode 100644
index 000..6075dc20dbe
--- /dev/null
+++ b/regression-test/suites/load_p0/routine_load/test_show_routine_load.groovy
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.kafka.clients.admin.AdminClient
+import org.apache.kafka.clients.producer.KafkaProducer
+import org.apache.kafka.clients.producer.ProducerRecord
+import org.apache.kafka.clients.producer.ProducerConfig
+
+suite("test_show_routine_load","p0") {
+def kafkaCsvTpoics = [
+ "test_show_routine_load",
+]
+String enabled = context.config.otherConfigs.get("enableKafkaTest")
+String kafka_port = context.config.otherConfigs.get("kafka_port")
+String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+def kafka_broker = "${externalEnvIp}:${kafka_port}"
+if (enabled != null && enabled.equalsIgnoreCase("true")) {
+// define kafka
+def props = new Properties()
+props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"${kafka_broker}".toString())
+props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer")
+props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apac
(doris) branch branch-3.0 updated: [fix](routine load) fix show routine load task result incorrect (#38523)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 4c52883acff [fix](routine load) fix show routine load task result
incorrect (#38523)
4c52883acff is described below
commit 4c52883acff993c68a55705cebf80943fbd1060e
Author: hui lai <1353307...@qq.com>
AuthorDate: Thu Aug 1 10:15:59 2024 +0800
[fix](routine load) fix show routine load task result incorrect (#38523)
### Bug report:
Create a job:
```
CREATE ROUTINE LOAD testShow ON test_show_routine_load
COLUMNS TERMINATED BY ","
PROPERTIES
(
"max_batch_interval" = "5",
"max_batch_rows" = "30",
"max_batch_size" = "209715200"
)
FROM KAFKA
(
"kafka_broker_list" = "127.0.0.1:19092",
"kafka_topic" = "test_show_routine_load",
"property.kafka_default_offsets" = "OFFSET_BEGINNING"
);
```
show routine load task:
```
SHOW ROUTINE LOAD TASK WHERE JobName = "testShow";
```
result:
```
ERROR 1105 (HY000): errCode = 2, detailMessage = The job named testshowdoes
not exists or job state is stopped or cancelled
```
### Solution
Do not use `toLowerCase` method;
---
.../doris/analysis/ShowRoutineLoadTaskStmt.java| 2 +-
.../routine_load/data/test_show_routine_load.csv | 1 +
.../routine_load/test_show_routine_load.groovy | 167 +++--
3 files changed, 120 insertions(+), 50 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
index 7258822ab50..570b82d3667 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
@@ -121,7 +121,7 @@ public class ShowRoutineLoadTaskStmt extends ShowStmt {
break CHECK;
}
StringLiteral stringLiteral = (StringLiteral)
binaryPredicate.getChild(1);
-jobName = stringLiteral.getValue().toLowerCase();
+jobName = stringLiteral.getValue();
} // CHECKSTYLE IGNORE THIS LINE
if (!valid) {
diff --git
a/regression-test/suites/load_p0/routine_load/data/test_show_routine_load.csv
b/regression-test/suites/load_p0/routine_load/data/test_show_routine_load.csv
new file mode 100644
index 000..b226b99ee4e
--- /dev/null
+++
b/regression-test/suites/load_p0/routine_load/data/test_show_routine_load.csv
@@ -0,0 +1 @@
+1,eab,2023-07-15,def,2023-07-20:05:48:31,"ghi"
\ No newline at end of file
diff --git
a/regression-test/suites/load_p0/routine_load/test_show_routine_load.groovy
b/regression-test/suites/load_p0/routine_load/test_show_routine_load.groovy
index 9fb681576d6..6075dc20dbe 100644
--- a/regression-test/suites/load_p0/routine_load/test_show_routine_load.groovy
+++ b/regression-test/suites/load_p0/routine_load/test_show_routine_load.groovy
@@ -15,67 +15,136 @@
// specific language governing permissions and limitations
// under the License.
+import org.apache.kafka.clients.admin.AdminClient
+import org.apache.kafka.clients.producer.KafkaProducer
+import org.apache.kafka.clients.producer.ProducerRecord
+import org.apache.kafka.clients.producer.ProducerConfig
+
suite("test_show_routine_load","p0") {
+def kafkaCsvTpoics = [
+ "test_show_routine_load",
+]
+String enabled = context.config.otherConfigs.get("enableKafkaTest")
String kafka_port = context.config.otherConfigs.get("kafka_port")
String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
def kafka_broker = "${externalEnvIp}:${kafka_port}"
+if (enabled != null && enabled.equalsIgnoreCase("true")) {
+// define kafka
+def props = new Properties()
+props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"${kafka_broker}".toString())
+props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer")
+props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer")
+// Create kafka producer
+def producer = new KafkaProducer<>(props)
-try {
-sql """
-CREATE ROUTINE LOAD testShow
-COLUMNS TERMINATED BY "|"
-PROPERTIES
-(
-"max_batch_interval" = "5",
-"max_batch_rows" = "30",
-"max_batch_size" = "209715200"
-)
-FROM KAFKA
-(
-"kafka_broker_list" = "${externalEnvIp}:${kafka_port}",
-"kafka_topic" = "multi_table_load_invalid_table",
-"property.kafka_default_offsets" = "OFFSET_BEGIN
[PR] [enhancement]use awaitility.await() add in varient and schema testcase [doris]
Vallishp opened a new pull request, #38836: URL: https://github.com/apache/doris/pull/38836 ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() add in varient and schema testcase [doris]
doris-robot commented on PR #38836: URL: https://github.com/apache/doris/pull/38836#issuecomment-2267566142 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
cosven commented on PR #38833: URL: https://github.com/apache/doris/pull/38833#issuecomment-2267570707 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [enhancement](user) Support limit user connection by ip [doris]
stalary opened a new pull request, #38837: URL: https://github.com/apache/doris/pull/38837 ## Proposed changes Issue Number: close #xxx In some cases, multiple services may use a common account, but the service may not be able to control the number of connections well, and we need to restrict it in the doris. Such as prod and stage environment use common user, but stage database connection use too many due to user connection too many. This parameter can be used to refine user connection limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](user) Support limit user connection by ip [doris]
doris-robot commented on PR #38837: URL: https://github.com/apache/doris/pull/38837#issuecomment-2267573066 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](user) Support limit user connection by ip [doris]
github-actions[bot] commented on PR #38837: URL: https://github.com/apache/doris/pull/38837#issuecomment-2267575229 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](user) Support limit user connection by ip [doris]
github-actions[bot] commented on PR #38837: URL: https://github.com/apache/doris/pull/38837#issuecomment-2267575401 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() add in variant and schema testcase [doris]
Vallishp commented on PR #38836: URL: https://github.com/apache/doris/pull/38836#issuecomment-2267581793 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore] rename method name and improve logging and comments [doris]
Vallishp closed pull request #34321: [chore] rename method name and improve logging and comments URL: https://github.com/apache/doris/pull/34321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() replace thread.sleep(), neredis part [doris]
Vallishp commented on PR #38816: URL: https://github.com/apache/doris/pull/38816#issuecomment-2267587097 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() replace thread.sleep() #37817 [doris]
Vallishp commented on PR #38814: URL: https://github.com/apache/doris/pull/38814#issuecomment-2267587339 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (doris-future) Support auto partition name function [doris]
wyxxxcat commented on PR #34258: URL: https://github.com/apache/doris/pull/34258#issuecomment-2267589004 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
doris-robot commented on PR #38833: URL: https://github.com/apache/doris/pull/38833#issuecomment-2267589859 TPC-H: Total hot run time: 41895 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 8a37e9acaacc4dca7839c3adb60937afc13169f9, data reload: false -- Round 1 -- q1 17704 409941074099 q2 2014199 195 195 q3 10465 131713721317 q4 10172 856 921 856 q5 7650304030043004 q6 219 139 139 139 q7 1063616 613 613 q8 9429195419881954 q9 8575659665946594 q10 8793388738383838 q11 436 243 245 243 q12 431 232 229 229 q13 1 294829492948 q14 284 250 244 244 q15 519 481 498 481 q16 518 404 390 390 q17 980 909 919 909 q18 7961725672477247 q19 1489123912381238 q20 561 336 328 328 q21 5321475247754752 q22 353 282 277 277 Total cold run time: 112714 ms Total hot run time: 41895 ms - Round 2, with runtime_filter_mode=off - q1 4075401440434014 q2 329 226 223 223 q3 3002299231772992 q4 2026208619651965 q5 5645544954335433 q6 220 131 131 131 q7 2155185218171817 q8 3319340533883388 q9 8665877387648764 q10 4003394839803948 q11 567 457 453 453 q12 786 596 579 579 q13 14561 311230903090 q14 317 266 283 266 q15 540 510 477 477 q16 458 407 416 407 q17 1796173717481737 q18 8181786177407740 q19 1752174917341734 q20 2125185118141814 q21 5587535452595259 q22 540 461 464 461 Total cold run time: 70649 ms Total hot run time: 56692 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
doris-robot commented on PR #38833: URL: https://github.com/apache/doris/pull/38833#issuecomment-2267593205 TPC-DS: Total hot run time: 169624 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 8a37e9acaacc4dca7839c3adb60937afc13169f9, data reload: false query1 914 381 358 358 query2 6480183318041804 query3 6658217 229 217 query4 20200 17576 17121 17121 query5 3784520 509 509 query6 292 169 166 166 query7 4596301 291 291 query8 254 201 185 185 query9 8512234323312331 query10 459 278 265 265 query11 10597 10098 99329932 query12 122 92 87 87 query13 1612376 372 372 query14 8665759169266926 query15 206 167 160 160 query16 6972461 480 461 query17 984 570 556 556 query18 1925288 282 282 query19 195 150 151 150 query20 93 88 84 84 query21 207 102 101 101 query22 4208408440274027 query23 33757 33882 33541 33541 query24 10415 311830943094 query25 705 408 432 408 query26 1844162 175 162 query27 3054290 287 287 query28 7524201520012001 query29 1325460 449 449 query30 196 154 163 154 query31 961 787 779 779 query32 105 57 66 57 query33 694 330 328 328 query34 915 495 502 495 query35 875 761 767 761 query36 1072870 876 870 query37 281 84 90 84 query38 3009287228742872 query39 878 856 874 856 query40 277 127 120 120 query41 48 47 48 47 query42 105 101 106 101 query43 490 441 445 441 query44 1170732 723 723 query45 208 181 179 179 query46 1082817 790 790 query47 1809170017241700 query48 371 295 289 289 query49 965 439 422 422 query50 903 432 427 427 query51 6874670566496649 query52 200 88 86 86 query53 251 180 177 177 query54 620 442 445 442 query55 80 75 77 75 query56 271 248 251 248 query57 1142103710561037 query58 258 258 271 258 query59 2703243324202420 query60 301 273 277 273 query61 97 95 93 93 query62 890 649 673 649 query63 208 181 176 176 query64 5568191618691869 query65 3176308530753075 query66 1320327 327 327 query67 15425 15102 14813 14813 query68 7394565 576 565 query69 822 379 313 313 query70 1149101610911016 query71 554 283 275 275 query72 8066266425002500 query73 961 330 327 327 query74 6121560956485609 query75 4689271427322714 query76 5881126412751264 query77 933 318 335 318 query78 9575893189108910 query79 5064523 529 523 query80 923 498 491 491 query81 542 231 230 230 query82 765 128 134 128 query83 329 167 186 167 query84 263 79 85 79 query85 1452299 291 291 query86 429 331 315 315 query87 3289315431593154 query88 3645237624022376 query89 439 289 288 288 query90 2073193 186 186 query91 123 101 100 100 query92 62 50 52 50 query93 4676604 602 602 query94 918 285 286 285 query95 383 263 258 258 query96 624 283 275 275 query97 3220308030443044 query98 216 197 244 197 query99 1601129612851285 Total cold run time: 279538 ms Total hot run time: 169624 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [enhance](server) add metrics to show thread cpu usage [doris]
doris-robot commented on PR #38833: URL: https://github.com/apache/doris/pull/38833#issuecomment-2267594686 ClickBench: Total hot run time: 29.91 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 8a37e9acaacc4dca7839c3adb60937afc13169f9, data reload: false query1 0.040.040.04 query2 0.080.040.03 query3 0.220.040.04 query4 1.690.070.07 query5 0.510.490.48 query6 1.140.720.70 query7 0.020.010.01 query8 0.050.040.04 query9 0.570.500.52 query10 0.580.560.55 query11 0.150.120.11 query12 0.160.120.12 query13 0.610.610.60 query14 0.770.800.81 query15 0.910.870.87 query16 0.360.350.36 query17 1.011.000.97 query18 0.230.220.22 query19 1.841.761.81 query20 0.010.000.01 query21 15.42 0.730.66 query22 3.797.321.26 query23 17.96 1.381.28 query24 2.270.230.22 query25 0.180.070.08 query26 0.320.220.21 query27 0.450.230.23 query28 13.19 1.010.97 query29 12.62 3.263.26 query30 0.260.050.06 query31 2.880.400.41 query32 3.250.500.49 query33 2.942.933.01 query34 15.45 4.264.27 query35 4.294.294.30 query36 0.660.480.47 query37 0.200.160.16 query38 0.170.150.15 query39 0.040.040.04 query40 0.150.140.12 query41 0.100.050.05 query42 0.060.040.05 query43 0.040.040.04 Total cold run time: 107.64 s Total hot run time: 29.91 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Feature] (join) support sort merge join [doris]
cutiepie-10 commented on issue #20126: URL: https://github.com/apache/doris/issues/20126#issuecomment-2267598596 Hi @yiguolei, I'd like to work on this issue Since I am first timer , I could not understand exactly where this sort has to be used ?? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (doris-future) Support auto partition name function [doris]
doris-robot commented on PR #34258: URL: https://github.com/apache/doris/pull/34258#issuecomment-2267600087 TPC-H: Total hot run time: 41970 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 17b99e13439c555bdb74822bbd8fb2b27b405285, data reload: false -- Round 1 -- q1 18685 412941514129 q2 2016198 205 198 q3 10435 131713321317 q4 10285 831 940 831 q5 8598288829812888 q6 220 141 139 139 q7 1054635 623 623 q8 9432191519391915 q9 9067664966376637 q10 9060408740504050 q11 1512273 269 269 q12 576 240 242 240 q13 18317 302130213021 q14 268 247 249 247 q15 524 489 486 486 q16 536 396 413 396 q17 975 909 936 909 q18 9018735272087208 q19 2761114612031146 q20 538 319 351 319 q21 5264472947534729 q22 345 284 273 273 Total cold run time: 119486 ms Total hot run time: 41970 ms - Round 2, with runtime_filter_mode=off - q1 4123404740794047 q2 341 242 234 234 q3 3119319331403140 q4 2022204520012001 q5 5532537654815376 q6 224 134 144 134 q7 2190177217961772 q8 3316335333613353 q9 8653879986348634 q10 3893398440783984 q11 573 463 458 458 q12 745 628 598 598 q13 16566 313831763138 q14 295 278 270 270 q15 520 487 488 487 q16 465 417 428 417 q17 1730170617391706 q18 8357775576677667 q19 1711171117171711 q20 2141182318581823 q21 5851533054515330 q22 528 502 465 465 Total cold run time: 72895 ms Total hot run time: 56745 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (doris-future) Support auto partition name function [doris]
doris-robot commented on PR #34258: URL: https://github.com/apache/doris/pull/34258#issuecomment-2267602685 TPC-DS: Total hot run time: 169633 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 17b99e13439c555bdb74822bbd8fb2b27b405285, data reload: false query1 916 372 364 364 query2 6410166416031603 query3 6677209 225 209 query4 20056 17463 17381 17381 query5 3674521 508 508 query6 298 181 162 162 query7 4618311 293 293 query8 261 196 198 196 query9 8534235723582357 query10 464 286 269 269 query11 10648 10076 99569956 query12 121 97 89 89 query13 1637397 378 378 query14 9864687869726878 query15 208 170 182 170 query16 6996447 435 435 query17 956 583 601 583 query18 1932293 307 293 query19 195 152 151 151 query20 95 86 92 86 query21 212 104 97 97 query22 4147405441444054 query23 33709 33518 33519 33518 query24 9268312331283123 query25 687 413 442 413 query26 1335151 156 151 query27 2937283 280 280 query28 7422200219731973 query29 1017448 442 442 query30 241 163 154 154 query31 960 763 798 763 query32 105 57 77 57 query33 670 321 334 321 query34 911 522 515 515 query35 846 742 767 742 query36 1065916 890 890 query37 167 84 81 81 query38 3040285828372837 query39 918 873 842 842 query40 268 111 110 110 query41 46 43 44 43 query42 116 105 99 99 query43 480 412 428 412 query44 1207764 739 739 query45 212 178 177 177 query46 1083804 799 799 query47 1850169716991697 query48 368 298 306 298 query49 863 417 426 417 query50 896 438 446 438 query51 6755667265406540 query52 103 93 91 91 query53 258 185 190 185 query54 643 457 468 457 query55 80 75 76 75 query56 278 264 254 254 query57 1121104410401040 query58 261 272 267 267 query59 2501240423152315 query60 329 279 276 276 query61 96 103 94 94 query62 899 683 664 664 query63 217 194 184 184 query64 4994191518851885 query65 3165309430923092 query66 999 329 334 329 query67 15286 14851 14697 14697 query68 4350595 580 580 query69 603 439 303 303 query70 1086111710681068 query71 380 286 287 286 query72 7140264925302530 query73 775 335 332 332 query74 6054567756185618 query75 3355271027422710 query76 2163123612941236 query77 519 306 303 303 query78 9417890489658904 query79 2683529 541 529 query80 967 517 514 514 query81 552 224 228 224 query82 1021131 133 131 query83 260 172 173 172 query84 255 80 84 80 query85 1169326 301 301 query86 442 305 291 291 query87 3265310231233102 query88 3956251525102510 query89 393 286 289 286 query90 1667198 202 198 query91 199 104 101 101 query92 62 54 50 50 query93 1946619 614 614 query94 798 309 298 298 query95 384 274 274 274 query96 605 284 286 284 query97 3206302030943020 query98 228 203 202 202 query99 1647129612941294 Total cold run time: 261138 ms Total hot run time: 169633 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [Enhancement] (doris-future) Support auto partition name function [doris]
doris-robot commented on PR #34258: URL: https://github.com/apache/doris/pull/34258#issuecomment-2267604133 ClickBench: Total hot run time: 30.68 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 17b99e13439c555bdb74822bbd8fb2b27b405285, data reload: false query1 0.050.040.04 query2 0.070.030.04 query3 0.230.050.05 query4 1.680.060.06 query5 0.490.490.49 query6 1.140.710.71 query7 0.020.010.02 query8 0.050.030.04 query9 0.570.530.51 query10 0.570.570.56 query11 0.160.120.12 query12 0.150.120.12 query13 0.610.610.60 query14 0.770.800.82 query15 0.900.870.86 query16 0.350.350.35 query17 1.001.011.07 query18 0.220.210.21 query19 1.941.751.79 query20 0.020.010.01 query21 15.39 0.760.66 query22 4.156.531.85 query23 18.01 1.341.35 query24 2.270.220.22 query25 0.190.080.07 query26 0.320.220.21 query27 0.460.230.22 query28 13.20 0.990.97 query29 12.54 3.313.27 query30 0.250.060.05 query31 2.880.410.40 query32 3.240.490.49 query33 2.942.992.96 query34 15.42 4.274.28 query35 4.324.314.31 query36 0.670.470.48 query37 0.190.170.17 query38 0.150.150.15 query39 0.040.030.03 query40 0.150.130.13 query41 0.100.050.04 query42 0.060.050.04 query43 0.050.040.04 Total cold run time: 107.98 s Total hot run time: 30.68 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.1](load) fix miss writer in concurrency incremental open (#38605) [doris]
yiguolei merged PR #38793: URL: https://github.com/apache/doris/pull/38793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [branch-2.1](load) fix miss writer in concurrency incremental open (#38605) (#38793)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 8fa0710cb35 [branch-2.1](load) fix miss writer in concurrency
incremental open (#38605) (#38793)
8fa0710cb35 is described below
commit 8fa0710cb35d925d7b09c4bdee013d76d0b277f6
Author: zclllhhjj
AuthorDate: Mon Aug 5 08:56:23 2024 +0800
[branch-2.1](load) fix miss writer in concurrency incremental open (#38605)
(#38793)
pick https://github.com/apache/doris/pull/38605
---
be/src/runtime/tablets_channel.cpp | 8
be/src/runtime/tablets_channel.h | 9 -
2 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/be/src/runtime/tablets_channel.cpp
b/be/src/runtime/tablets_channel.cpp
index adaced0b76e..266d4d45f18 100644
--- a/be/src/runtime/tablets_channel.cpp
+++ b/be/src/runtime/tablets_channel.cpp
@@ -215,6 +215,7 @@ Status BaseTabletsChannel::incremental_open(const
PTabletWriterOpenRequest& para
ss << "LocalTabletsChannel txn_id: " << _txn_id << " load_id: " <<
print_id(params.id())
<< " incremental open delta writer: ";
+// every change will hold _lock. this find in under _lock too. so no need
_tablet_writers_lock again.
for (const auto& tablet : params.tablets()) {
if (_tablet_writers.find(tablet.tablet_id()) != _tablet_writers.end())
{
continue;
@@ -238,6 +239,7 @@ Status BaseTabletsChannel::incremental_open(const
PTabletWriterOpenRequest& para
_profile, _load_id);
ss << "[" << tablet.tablet_id() << "]";
{
+// here we modify _tablet_writers. so need lock.
std::lock_guard l(_tablet_writers_lock);
_tablet_writers.emplace(tablet.tablet_id(),
std::move(delta_writer));
}
@@ -479,6 +481,7 @@ Status BaseTabletsChannel::_open_all_writers(const
PTabletWriterOpenRequest& req
#endif
int tablet_cnt = 0;
+// under _lock. no need _tablet_writers_lock again.
for (const auto& tablet : request.tablets()) {
if (_tablet_writers.find(tablet.tablet_id()) != _tablet_writers.end())
{
continue;
@@ -578,6 +581,11 @@ Status BaseTabletsChannel::add_batch(const
PTabletWriterAddBlockRequest& request
std::function write_func) {
google::protobuf::RepeatedPtrField* tablet_errors =
response->mutable_tablet_errors();
+
+// add_batch may concurrency with inc_open but not under _lock.
+// so need to protect it with _tablet_writers_lock.
+std::lock_guard l(_tablet_writers_lock);
+
auto tablet_writer_it = _tablet_writers.find(tablet_id);
if (tablet_writer_it == _tablet_writers.end()) {
return Status::InternalError("unknown tablet to append data,
tablet={}", tablet_id);
diff --git a/be/src/runtime/tablets_channel.h b/be/src/runtime/tablets_channel.h
index 54438be7690..8ed4c7ab1aa 100644
--- a/be/src/runtime/tablets_channel.h
+++ b/be/src/runtime/tablets_channel.h
@@ -136,11 +136,8 @@ protected:
// id of this load channel
TabletsChannelKey _key;
-// make execute sequence
+// protect _state change. open and close. when add_batch finished, lock to
change _next_seqs also
std::mutex _lock;
-
-SpinLock _tablet_writers_lock;
-
enum State {
kInitialized,
kOpened,
@@ -166,8 +163,10 @@ protected:
// currently it's OK.
Status _close_status;
-// tablet_id -> TabletChannel
+// tablet_id -> TabletChannel. it will only be changed in open() or
inc_open()
std::unordered_map>
_tablet_writers;
+// protect _tablet_writers
+SpinLock _tablet_writers_lock;
// broken tablet ids.
// If a tablet write fails, it's id will be added to this set.
// So that following batch will not handle this tablet anymore.
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](ScanNode) Move the finalize phase of ScanNode to after the end of the Physical Translate phase [doris]
yiguolei merged PR #38604: URL: https://github.com/apache/doris/pull/38604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [Fix](ScanNode) Move the finalize phase of ScanNode to after the end of the Physical Translate phase (#38604)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 40767003c62 [Fix](ScanNode) Move the finalize phase of ScanNode to
after the end of the Physical Translate phase (#38604)
40767003c62 is described below
commit 40767003c627d4adf9d3b7e059ca9a1ee722015e
Author: Tiewei Fang <43782773+bepppo...@users.noreply.github.com>
AuthorDate: Mon Aug 5 08:58:59 2024 +0800
[Fix](ScanNode) Move the finalize phase of ScanNode to after the end of the
Physical Translate phase (#38604)
bp: #37565
Currently, Doris first obtains splits and then performs projection.
After column pruning, it calls `updateRequiredSlots` to update the
scanRange information. However, the Trino connector's column pruning
pushdown needs to be completed before obtaining splits.
Therefore, we move the finalize phase of `ScanNode` to after the end of
the `Physical Translate` phase, so that `createScanRangeLocations` can
use the final columns which have been pruning.
## Proposed changes
Issue Number: close #xxx
---
.../apache/doris/datasource/FileQueryScanNode.java | 13
.../doris/datasource/jdbc/source/JdbcScanNode.java | 9 --
.../doris/datasource/odbc/source/OdbcScanNode.java | 9 --
.../datasource/paimon/source/PaimonScanNode.java | 19
.../glue/translator/PhysicalPlanTranslator.java| 36 --
.../org/apache/doris/planner/OlapScanNode.java | 6 +++-
.../java/org/apache/doris/planner/ScanNode.java| 10 --
7 files changed, 17 insertions(+), 85 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java
index 9822855aa72..517ba8be5f8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java
@@ -19,7 +19,6 @@ package org.apache.doris.datasource;
import org.apache.doris.analysis.Analyzer;
import org.apache.doris.analysis.SlotDescriptor;
-import org.apache.doris.analysis.SlotId;
import org.apache.doris.analysis.TableSample;
import org.apache.doris.analysis.TableSnapshot;
import org.apache.doris.analysis.TupleDescriptor;
@@ -40,7 +39,6 @@ import org.apache.doris.datasource.hive.HMSExternalCatalog;
import org.apache.doris.datasource.hive.source.HiveScanNode;
import org.apache.doris.datasource.hive.source.HiveSplit;
import org.apache.doris.datasource.iceberg.source.IcebergSplit;
-import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
import org.apache.doris.planner.PlanNodeId;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.spi.Split;
@@ -80,7 +78,6 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;
-import java.util.Set;
/**
* FileQueryScanNode for querying the file access type of catalog, now only
support
@@ -182,16 +179,6 @@ public abstract class FileQueryScanNode extends
FileScanNode {
params.setSrcTupleId(-1);
}
-/**
- * Reset required_slots in contexts. This is called after Nereids planner
do the projection.
- * In the projection process, some slots may be removed. So call this to
update the slots info.
- */
-@Override
-public void updateRequiredSlots(PlanTranslatorContext
planTranslatorContext,
-Set requiredByProjectSlotIdSet) throws UserException {
-updateRequiredSlots();
-}
-
private void updateRequiredSlots() throws UserException {
params.unsetRequiredSlots();
for (SlotDescriptor slot : desc.getSlots()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java
index 0d292100fe0..a85dd4aaafb 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java
@@ -27,7 +27,6 @@ import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.ExprSubstitutionMap;
import org.apache.doris.analysis.FunctionCallExpr;
import org.apache.doris.analysis.SlotDescriptor;
-import org.apache.doris.analysis.SlotId;
import org.apache.doris.analysis.SlotRef;
import org.apache.doris.analysis.TupleDescriptor;
import org.apache.doris.catalog.Column;
@@ -39,7 +38,6 @@ import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.UserException;
import org.apache.doris.datasource.ExternalScanNode;
import org.apache.doris.datasource.jdbc.JdbcExternalTable;
-import org.apache.doris.nereids.glue.translator.PlanTrans
(doris) branch branch-2.1 updated: [opt](join) check datatype of intermediate slots in hash join (#38556) (#38792)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 53773ae6b77 [opt](join) check datatype of intermediate slots in hash
join (#38556) (#38792)
53773ae6b77 is described below
commit 53773ae6b77350890ba5d370876f9030cbb36972
Author: Jerry Hu
AuthorDate: Mon Aug 5 09:03:21 2024 +0800
[opt](join) check datatype of intermediate slots in hash join (#38556)
(#38792)
## Proposed changes
pick #38556
---
be/src/pipeline/exec/hashjoin_probe_operator.cpp | 51
1 file changed, 51 insertions(+)
diff --git a/be/src/pipeline/exec/hashjoin_probe_operator.cpp
b/be/src/pipeline/exec/hashjoin_probe_operator.cpp
index 002a79f2db2..b7dd0622fe3 100644
--- a/be/src/pipeline/exec/hashjoin_probe_operator.cpp
+++ b/be/src/pipeline/exec/hashjoin_probe_operator.cpp
@@ -21,6 +21,9 @@
#include "common/logging.h"
#include "pipeline/exec/operator.h"
+#include "runtime/descriptors.h"
+#include "vec/common/assert_cast.h"
+#include "vec/data_types/data_type_nullable.h"
namespace doris {
namespace pipeline {
@@ -637,6 +640,54 @@ Status HashJoinProbeOperatorX::prepare(RuntimeState*
state) {
_left_table_data_types =
vectorized::VectorizedUtils::get_data_types(_child_x->row_desc());
_right_table_column_names =
vectorized::VectorizedUtils::get_column_names(_build_side_child->row_desc());
+
+std::vector slots_to_check;
+for (const auto& tuple_descriptor :
_intermediate_row_desc->tuple_descriptors()) {
+for (const auto& slot : tuple_descriptor->slots()) {
+slots_to_check.emplace_back(slot);
+}
+}
+
+if (_is_mark_join) {
+const auto* last_one = slots_to_check.back();
+slots_to_check.pop_back();
+auto data_type = last_one->get_data_type_ptr();
+if (!data_type->is_nullable()) {
+return Status::InternalError(
+"The last column for mark join should be Nullable(UInt8),
not {}",
+data_type->get_name());
+}
+
+const auto& null_data_type = assert_cast(*data_type);
+if (null_data_type.get_nested_type()->get_type_id() !=
vectorized::TypeIndex::UInt8) {
+return Status::InternalError(
+"The last column for mark join should be Nullable(UInt8),
not {}",
+data_type->get_name());
+}
+}
+
+const int right_col_idx =
+(_is_right_semi_anti && !_have_other_join_conjunct) ? 0 :
_left_table_data_types.size();
+size_t idx = 0;
+for (const auto* slot : slots_to_check) {
+auto data_type = slot->get_data_type_ptr();
+auto target_data_type = idx < right_col_idx ?
_left_table_data_types[idx]
+:
_right_table_data_types[idx - right_col_idx];
+++idx;
+if (data_type->equals(*target_data_type)) {
+continue;
+}
+
+auto data_type_non_nullable = vectorized::remove_nullable(data_type);
+if (data_type_non_nullable->equals(*target_data_type)) {
+continue;
+}
+
+return Status::InternalError("intermediate slot({}) data type not
match: '{}' vs '{}'",
+ slot->id(), data_type->get_name(),
+ _left_table_data_types[idx]->get_name());
+}
+
_build_side_child.reset();
return Status::OK();
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](join) check datatype of intermediate slots in hash join (#38556) [doris]
yiguolei merged PR #38792: URL: https://github.com/apache/doris/pull/38792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch 2.1_38012 updated (27de814b0fc -> 6ea6eb8eed3)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch 2.1_38012 in repository https://gitbox.apache.org/repos/asf/doris.git discard 27de814b0fc [fix](compatibility) type toSql should return lowercase string (#38012) add 338fa32303b [pick](simdjson) fix simdjson with object array when jsonroot is not empty (#38633) add 057ee1905f8 [bugfix](hudi)add timetravel for nereids for 2.1 (#38324) (#38582) add c310b92727d [bugfix](external)Modify the default value of `pushdowncount` for 2.1 (#37754) (#38591) add 28998300d43 [Bug](fix) fix ubsan use int32_t pointer access bool value (#38621) add 63a3ff570b5 [Opt](load) print tablet id when memtable flush coredump #38618 (#38656) add 4042cdf5532 [Fix](memory) Fix allocator.h compiling failed on mac. (#38646) add b0943064e04 [fix](kerberos)fix and refactor ugi login for kerberos and simple authentication (#38607) add 9d23ccf1f2a [Improvement](schema scan) Use async scanner for schema scanners (#38… (#38666) add cafcf7acc1f [cherry-pick](SSL) Fix ssl connection close 2.1 (#38587) (#38677) add e8690b62ee8 [fix](group commit) Pick add debug log show why group commit not work; delete wal when replay success (#38611) (#38659) add 82c681595e2 [fix](local exchange) Fix local exchange blocked by a huge data block… (#38693) add 3e5255a8623 [pipeline](fix) Fix blocking task which is not triggered by 2nd RPC (… (#38694) add a4e793752fc [bugfix](iceberg)revert count(*) directly returned by fe for 2.1 (#38566) (#38655) add 60091f072a8 [fix](auth)fix create table like need create_priv of existed table (#… (#38570) add e59a866d6a3 [fix](test)Fix unstable index compaction fault injection case (#38501) (#38673) add 2562cf33a76 [fix](mtmv) Choose a valid partition column when there are both valid and invalid expressions (#38367) (#38684) add 555dccb1a4c [fix](bloom filter)Fix rename column with bloom filter (#38431) (#38662) add 4d980b82351 [feature](http action)Add http action to show nested inverted index file (#38272) (#38672) add 0da388ade5b [fix](inverted index) fix match_phrase_ edge query result error #38327 (#38740) add e140613ae1f [fix](Nereids) remove db readlock before get table from db (#38660) (#38729) add 327069fdbcc [branch-2.1](log) add tablet clear cache log (#38713) add 84d9b2fcf41 [pick](nestedtypes) support nested type with agg replace_if_not_null (#38719) add b7e1588be9a [pick](upgrade)fix log message (#38710) add f5bc65989c0 [pick](array-range)improve array_range func for large param (#38707) add 1d982ada456 [pick](array-funcs)pick array func array_enumerate_uniq bugfix (#38721) add 4f2ca439176 [minor](fe) simplfy some code in HMSExternalTable (#32344) (#38675) add b3f335ba5fb [enhancement](index compaction) Enable index compaction by default (#36812) (#38676) add f21d7e3833e [test](inverted index)Add cases for inverted index format v2 (#38132)(#38443) (#38222) add 7bcda898813 [pick](case) fix one_nested_types cases (#38723) add d8004348592 [Chore](Fe)Upgrade dependencies (#38509) (#38747) add 9b07cd20692 [pick](json-serde)pick jsonb string deserialize with spec char (#38711) add da7b2cf5783 [refactor](catalog) set "use_meta_cache" default to true (#38244)(#38352)(#38619) (#38355) add f24d55fc947 [fix](syntax) multi statements must delim with semicolon (#38670) (#38753) add 2425730609d [enhance](auth)support cache ranger datamask and row filter (#37723) (#38575) add 556f0fc784f [pick](json-keys) support json_keys function (#38631) add 64b69ed1baf [branch-2.1] Picks "[opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487" (#38682) add 74908c123a5 [fix](test)Fix unstable test drop index fault #38768 (#38772) add fe3e3d0fab6 [fix](test)Fix build index fault test (#38736) (#38762) add c0caca7c557 [fix](ES Catalog)Fix unstable test test_es_query (#38801) (#38802) add 7bdc508ac7d [Bug](fix) fix coredump case in (not null, null) execpt (not null, not null) case (#38756) add 0603ec1d9d7 [enhancement](compaction) optimizing memory usage for compaction (#37099) (#37486) add 7c70f75198f [Fix](Load)Audit logs avoid recording certain sensitive information #38769 (#38784) add eef8c87fb5e [chore](test) disable fault injection to make pipeline task check happy (#38665) (#38821) add 8e4fad99a1b [test](routine load) add routine load case with timestamp as offset(#38567) (#38822) add 6035edad0b1 [fix](multi table) fix single stream multi table memory leak (#38255) (#38824) add 79b07d0b8a4 [fix](routine load) fix enclose and escape can not set in routine load job (#38402) (#38825) add f76397277ec [fix](routine load) fix show routine load task result incorrect (#38523) (#38826) add 8fa0710cb35 [branch-2.1](load) fix miss
Re: [PR] [fix](compatibility) type toSql should return lowercase string (#38012) [doris]
yiguolei commented on PR #38517: URL: https://github.com/apache/doris/pull/38517#issuecomment-2267991384 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [feature](hive)Support reading renamed Parquet Hive and Orc Hive tables. (#38432) (#38809)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 5d02c48715f [feature](hive)Support reading renamed Parquet Hive and
Orc Hive tables. (#38432) (#38809)
5d02c48715f is described below
commit 5d02c48715ff35f22b5943433132aba08aeb162d
Author: daidai <2017501...@qq.com>
AuthorDate: Mon Aug 5 09:06:49 2024 +0800
[feature](hive)Support reading renamed Parquet Hive and Orc Hive tables.
(#38432) (#38809)
bp #38432
## Proposed changes
Add `hive_parquet_use_column_names` and `hive_orc_use_column_names`
session variables to read the table after rename column in `Hive`.
These two session variables are referenced from
`parquet_use_column_names` and `orc_use_column_names` of `Trino` hive
connector.
By default, these two session variables are true. When they are set to
false, reading orc/parquet will access the columns according to the
ordinal position in the Hive table definition.
For example:
```mysql
in Hive :
hive> create table tmp (a int , b string) stored as parquet;
hive> insert into table tmp values(1,"2");
hive> alter table tmp change column a new_a int;
hive> insert into table tmp values(2,"4");
in Doris :
mysql> set hive_parquet_use_column_names=true;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from tmp;
+---+--+
| new_a | b|
+---+--+
| NULL | 2|
| 2 | 4|
+---+--+
2 rows in set (0.02 sec)
mysql> set hive_parquet_use_column_names=false;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from tmp;
+---+--+
| new_a | b|
+---+--+
| 1 | 2|
| 2 | 4|
+---+--+
2 rows in set (0.02 sec)
```
You can use `set
parquet.column.index.access/orc.force.positional.evolution = true/false`
in hive 3 to control the results of reading the table like these two
session variables. However, for the rename struct inside column parquet
table, the effects of hive and doris are different.
---
be/src/vec/exec/format/orc/vorc_reader.cpp | 16 +-
be/src/vec/exec/format/orc/vorc_reader.h | 15 +-
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 93 -
be/src/vec/exec/format/parquet/vparquet_reader.h | 3 +-
be/src/vec/exec/scan/vfile_scanner.cpp | 21 +-
.../scripts/create_preinstalled_scripts/run64.hql | 29 ++
.../orc_table/simulation_hive1_orc/00_0| Bin 0 -> 408 bytes
.../orc_table/test_hive_rename_column_orc/00_0 | Bin 0 -> 405 bytes
.../test_hive_rename_column_orc/00_0_copy_1| Bin 0 -> 396 bytes
.../test_hive_rename_column_orc/00_0_copy_2| Bin 0 -> 554 bytes
.../test_hive_rename_column_orc/00_0_copy_3| Bin 0 -> 592 bytes
.../test_hive_rename_column_parquet/00_0 | Bin 0 -> 538 bytes
.../00_0_copy_1| Bin 0 -> 543 bytes
.../00_0_copy_2| Bin 0 -> 787 bytes
.../00_0_copy_3| Bin 0 -> 801 bytes
.../apache/doris/datasource/FileQueryScanNode.java | 4 +
.../java/org/apache/doris/qe/SessionVariable.java | 20 +
gensrc/thrift/PaloInternalService.thrift | 6 +
.../hive/test_hive_rename_column_orc_parquet.out | 435 +
.../test_hive_rename_column_orc_parquet.groovy | 196 ++
20 files changed, 803 insertions(+), 35 deletions(-)
diff --git a/be/src/vec/exec/format/orc/vorc_reader.cpp
b/be/src/vec/exec/format/orc/vorc_reader.cpp
index 547d53bd00e..7a820845ed0 100644
--- a/be/src/vec/exec/format/orc/vorc_reader.cpp
+++ b/be/src/vec/exec/format/orc/vorc_reader.cpp
@@ -279,13 +279,15 @@ Status OrcReader::init_reader(
const VExprContextSPtrs& conjuncts, bool is_acid, const
TupleDescriptor* tuple_descriptor,
const RowDescriptor* row_descriptor,
const VExprContextSPtrs* not_single_slot_filter_conjuncts,
-const std::unordered_map*
slot_id_to_filter_conjuncts) {
+const std::unordered_map*
slot_id_to_filter_conjuncts,
+const bool hive_use_column_names) {
_column_names = column_names;
_colname_to_value_range = colname_to_value_range;
_lazy_read_ctx.conjuncts = conjuncts;
_is_acid = is_acid;
_tuple_descriptor = tuple_descriptor;
_row_descriptor = row_descriptor;
+_is_hive1_orc_or_use_idx = !hive_use_column_names;
if (not_single_slot_filter_conjuncts != nullptr &&
!not_single_slot_filter_conjuncts->empty()) {
_not_single_slot_filter_conjuncts.insert(_not_single_slot_filter_conjuncts.end(),
not_single_slot_filter_conjunct
Re: [PR] [feature](hive)Support reading renamed Parquet Hive and Orc Hive tables. (#38432) [doris]
yiguolei merged PR #38809: URL: https://github.com/apache/doris/pull/38809 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cherry-pick](branch-21)fix operator do_projections should use local_state intermediate_projections (#38612) [doris]
yiguolei merged PR #38765: URL: https://github.com/apache/doris/pull/38765 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [cherry-pick](branch-21)fix operator do_projections should use local_state intermediate_projections (#38612) (#38765)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 1b3d4b4d31c [cherry-pick](branch-21)fix operator do_projections should
use local_state intermediate_projections (#38612) (#38765)
1b3d4b4d31c is described below
commit 1b3d4b4d31cd7b77dcf11769026c0f46b7c061c7
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Mon Aug 5 09:07:16 2024 +0800
[cherry-pick](branch-21)fix operator do_projections should use local_state
intermediate_projections (#38612) (#38765)
## Proposed changes
cherry-pick from master https://github.com/apache/doris/pull/38612
---
be/src/pipeline/pipeline_x/operator.cpp| 2 +-
.../data/javaudf_p0/test_javaudf_string.out| 22 ++
.../suites/javaudf_p0/test_javaudf_string.groovy | 20 +++-
3 files changed, 42 insertions(+), 2 deletions(-)
diff --git a/be/src/pipeline/pipeline_x/operator.cpp
b/be/src/pipeline/pipeline_x/operator.cpp
index a8902cf2321..6c2c2a6294d 100644
--- a/be/src/pipeline/pipeline_x/operator.cpp
+++ b/be/src/pipeline/pipeline_x/operator.cpp
@@ -220,7 +220,7 @@ Status OperatorXBase::do_projections(RuntimeState* state,
vectorized::Block* ori
vectorized::Block input_block = *origin_block;
std::vector result_column_ids;
-for (const auto& projections : _intermediate_projections) {
+for (const auto& projections : local_state->_intermediate_projections) {
result_column_ids.resize(projections.size());
for (int i = 0; i < projections.size(); i++) {
RETURN_IF_ERROR(projections[i]->execute(&input_block,
&result_column_ids[i]));
diff --git a/regression-test/data/javaudf_p0/test_javaudf_string.out
b/regression-test/data/javaudf_p0/test_javaudf_string.out
index 892c5245824..59f2f7c776d 100644
--- a/regression-test/data/javaudf_p0/test_javaudf_string.out
+++ b/regression-test/data/javaudf_p0/test_javaudf_string.out
@@ -10,6 +10,17 @@
8 8 abcdefg8poiuytre8abcdefg
9 9 abcdefg9poiuytre9abcdefg
+-- !select_default_2 --
+1 1 abcdefg1poiuytre1abcdefg
+2 2 abcdefg2poiuytre2abcdefg
+3 3 abcdefg3poiuytre3abcdefg
+4 4 abcdefg4poiuytre4abcdefg
+5 5 abcdefg5poiuytre5abcdefg
+6 6 abcdefg6poiuytre6abcdefg
+7 7 abcdefg7poiuytre7abcdefg
+8 8 abcdefg8poiuytre8abcdefg
+9 9 abcdefg9poiuytre9abcdefg
+
-- !select --
ab***fg1
ab***fg2
@@ -43,3 +54,14 @@ ab*def ab**efg
ab*def ab**efg
ab*def ab**efg
+-- !select_4 --
+ab***fg1 ab***fg1
+ab***fg2 ab***fg2
+ab***fg3 ab***fg3
+ab***fg4 ab***fg4
+ab***fg5 ab***fg5
+ab***fg6 ab***fg6
+ab***fg7 ab***fg7
+ab***fg8 ab***fg8
+ab***fg9 ab***fg9
+
diff --git a/regression-test/suites/javaudf_p0/test_javaudf_string.groovy
b/regression-test/suites/javaudf_p0/test_javaudf_string.groovy
index c6d7a7311f5..6517c4b08c2 100644
--- a/regression-test/suites/javaudf_p0/test_javaudf_string.groovy
+++ b/regression-test/suites/javaudf_p0/test_javaudf_string.groovy
@@ -29,6 +29,7 @@ suite("test_javaudf_string") {
log.info("Jar path: ${jarPath}".toString())
try {
sql """ DROP TABLE IF EXISTS ${tableName} """
+sql """ DROP TABLE IF EXISTS test_javaudf_string_2 """
sql """
CREATE TABLE IF NOT EXISTS ${tableName} (
`user_id` INT NOT NULL COMMENT "用户id",
@@ -51,7 +52,10 @@ suite("test_javaudf_string") {
sql """ INSERT INTO ${tableName} VALUES
${sb.toString()}
"""
+sql """ create table test_javaudf_string_2 like test_javaudf_string """
+sql """ insert into test_javaudf_string_2 select * from
test_javaudf_string; """
qt_select_default """ SELECT * FROM ${tableName} t ORDER BY user_id;
"""
+qt_select_default_2 """ SELECT * FROM test_javaudf_string_2 t ORDER BY
user_id; """
File path = new File(jarPath)
if (!path.exists()) {
@@ -68,9 +72,23 @@ suite("test_javaudf_string") {
qt_select """ SELECT java_udf_string_test(string_col, 2, 3) result
FROM ${tableName} ORDER BY result; """
qt_select """ SELECT java_udf_string_test('abcdef', 2, 3),
java_udf_string_test('abcdefg', 2, 3) result FROM ${tableName} ORDER BY result;
"""
-
+qt_select_4 """
+SELECT
+COALESCE(
+java_udf_string_test(test_javaudf_string.varchar_col, 2,
3),
+'not1'
+),
+COALESCE(
+java_udf_string_test(test_javaudf_string.varchar_col, 2,
3),
+'n
(doris) branch branch-2.1 updated: [pick](array-funcs)fix array with empty arg in be behavior (#38708)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 2653087843b [pick](array-funcs)fix array with empty arg in be behavior
(#38708)
2653087843b is described below
commit 2653087843b63fd2152750fd84f284ef9a9eb3fc
Author: amory
AuthorDate: Mon Aug 5 09:08:28 2024 +0800
[pick](array-funcs)fix array with empty arg in be behavior (#38708)
## Proposed changes
backport: https://github.com/apache/doris/pull/36845
Issue Number: close #xxx
---
be/src/vec/functions/array/function_array_constructor.cpp | 10 +++---
.../data/nereids_function_p0/scalar_function/Array.out | 9 -
.../suites/nereids_function_p0/scalar_function/Array.groovy| 4 +++-
3 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/be/src/vec/functions/array/function_array_constructor.cpp
b/be/src/vec/functions/array/function_array_constructor.cpp
index d19d29ec254..cec7e03b22e 100644
--- a/be/src/vec/functions/array/function_array_constructor.cpp
+++ b/be/src/vec/functions/array/function_array_constructor.cpp
@@ -36,6 +36,7 @@
#include "vec/data_types/data_type.h"
#include "vec/data_types/data_type_array.h"
#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
#include "vec/functions/function.h"
#include "vec/functions/simple_function_factory.h"
@@ -59,11 +60,14 @@ public:
bool use_default_implementation_for_nulls() const override { return false;
}
-size_t get_number_of_arguments() const override { return 1; }
+size_t get_number_of_arguments() const override { return 0; }
DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
-DCHECK(arguments.size() > 0)
-<< "function: " << get_name() << ", arguments should not be
empty";
+// we accept with empty argument, like array(), which will be treated
as array(UInt8)
+if (arguments.empty()) {
+return std::make_shared(
+make_nullable(std::make_shared()));
+}
return
std::make_shared(make_nullable(remove_nullable(arguments[0])));
}
diff --git a/regression-test/data/nereids_function_p0/scalar_function/Array.out
b/regression-test/data/nereids_function_p0/scalar_function/Array.out
index c6c32bf9b15..cfad441a499 100644
--- a/regression-test/data/nereids_function_p0/scalar_function/Array.out
+++ b/regression-test/data/nereids_function_p0/scalar_function/Array.out
@@ -14443,4 +14443,11 @@ true
["2012-03-09 09:08:09", "2012-03-09 09:08:17"]
["2012-03-10 10:09:10", "2012-03-10 10:09:19"]
["2012-03-11 11:10:11", "2012-03-11 11:10:21"]
-["2012-03-12 12:11:12", "2012-03-12 12:11:23"]
\ No newline at end of file
+["2012-03-12 12:11:12", "2012-03-12 12:11:23"]
+
+-- !array_empty_fe --
+[]
+
+-- !array_empty_be --
+[]
+
diff --git
a/regression-test/suites/nereids_function_p0/scalar_function/Array.groovy
b/regression-test/suites/nereids_function_p0/scalar_function/Array.groovy
index a6f4b0604c4..e1eb2bab515 100644
--- a/regression-test/suites/nereids_function_p0/scalar_function/Array.groovy
+++ b/regression-test/suites/nereids_function_p0/scalar_function/Array.groovy
@@ -1281,6 +1281,9 @@ suite("nereids_scalar_fn_Array") {
qt_sequence_datetime_hour """select sequence(kdtmv2s1, date_add(kdtmv2s1,
interval kint-3 hour), interval kint hour) from fn_test order by kdtmv2s1;"""
qt_sequence_datetime_minute """select sequence(kdtmv2s1,
date_add(kdtmv2s1, interval kint+1 minute), interval kint minute) from fn_test
order by kdtmv2s1;"""
qt_sequence_datetime_second """select sequence(kdtmv2s1,
date_add(kdtmv2s1, interval kint second), interval kint-1 second) from fn_test
order by kdtmv2s1;"""
+
+// with array empty
+qt_array_empty_fe """select array()"""
// make large error size
test {
sql "select array_size(sequence(kdtmv2s1, date_add(kdtmv2s1, interval
kint+1000 year), interval kint hour)) from fn_test order by kdtmv2s1;"
@@ -1305,5 +1308,4 @@ suite("nereids_scalar_fn_Array") {
logger.info(exception.message)
}
}
-
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](array-funcs)fix array with empty arg in be behavior [doris]
yiguolei merged PR #38708: URL: https://github.com/apache/doris/pull/38708 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [branch-2.1][improvement](jdbc catalog) improvement some jdbc catalog properties check order (#38770)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 9430b27e682 [branch-2.1][improvement](jdbc catalog) improvement some
jdbc catalog properties check order (#38770)
9430b27e682 is described below
commit 9430b27e6825ee8c3829181aa8d994d85fce7545
Author: zy-kkk
AuthorDate: Mon Aug 5 09:14:04 2024 +0800
[branch-2.1][improvement](jdbc catalog) improvement some jdbc catalog
properties check order (#38770)
pick (#38439)
1. Move the execution of testJdbcConnection() to checkWhenCreating
instead of the constructor
2. Move the logic of renaming lower_case_table_names to
lower_case_meta_names to setDefaultPropsIfMissing
---
.../apache/doris/datasource/CatalogFactory.java| 2 +-
.../apache/doris/datasource/CatalogProperty.java | 4 ++
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 57 +++---
.../datasource/jdbc/JdbcExternalCatalogTest.java | 26 +++---
4 files changed, 52 insertions(+), 37 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java
index fdd3b8fbe02..44e0470b069 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java
@@ -122,7 +122,7 @@ public class CatalogFactory {
catalog = new EsExternalCatalog(catalogId, name, resource,
props, comment);
break;
case "jdbc":
-catalog = new JdbcExternalCatalog(catalogId, name, resource,
props, comment, isReplay);
+catalog = new JdbcExternalCatalog(catalogId, name, resource,
props, comment);
break;
case "iceberg":
catalog =
IcebergExternalCatalogFactory.createCatalog(catalogId, name, resource, props,
comment);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java
index 50aea847e8f..a54a28d59dd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java
@@ -115,6 +115,10 @@ public class CatalogProperty implements Writable {
this.properties.put(key, val);
}
+public void deleteProperty(String key) {
+this.properties.remove(key);
+}
+
@Override
public void write(DataOutput out) throws IOException {
Text.writeString(out, GsonUtils.GSON.toJson(this));
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
index 568c9e8480c..73b6639c7b9 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
@@ -72,11 +72,10 @@ public class JdbcExternalCatalog extends ExternalCatalog {
private transient JdbcClient jdbcClient;
public JdbcExternalCatalog(long catalogId, String name, String resource,
Map props,
-String comment, boolean isReplay)
+String comment)
throws DdlException {
super(catalogId, name, InitCatalogLog.Type.JDBC, comment);
-this.catalogProperty = new CatalogProperty(resource,
processCompatibleProperties(props, isReplay));
-testJdbcConnection(isReplay);
+this.catalogProperty = new CatalogProperty(resource,
processCompatibleProperties(props));
}
@Override
@@ -99,6 +98,21 @@ public class JdbcExternalCatalog extends ExternalCatalog {
getConnectionPoolMaxWaitTime(),
getConnectionPoolMaxLifeTime());
}
+@Override
+public void setDefaultPropsIfMissing(boolean isReplay) {
+super.setDefaultPropsIfMissing(isReplay);
+// Modify lower_case_table_names to lower_case_meta_names if it exists
+if
(catalogProperty.getProperties().containsKey("lower_case_table_names") &&
isReplay) {
+String lowerCaseTableNamesValue =
catalogProperty.getProperties().get("lower_case_table_names");
+catalogProperty.addProperty("lower_case_meta_names",
lowerCaseTableNamesValue);
+catalogProperty.deleteProperty("lower_case_table_names");
+LOG.info("Modify lower_case_table_names to lower_case_meta_names,
value: {}", lowerCaseTableNamesValue);
+} else if
(catalogProperty.getProperties().containsKey("lower_case_table_names") &&
!isReplay) {
+throw new IllegalArgumentException("Jdbc catalog property
lower_case_table_names is not supported,"
+
Re: [PR] [opt](serde)Optimize the filling of fixed values into block columns without repeated deserialization. (#37377) (#38245) [doris]
yiguolei merged PR #38810: URL: https://github.com/apache/doris/pull/38810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
