Re: [PR] [feature](workloadgroup) use slot num to control memory distribution among one workload group [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38237:
URL: https://github.com/apache/doris/pull/38237#issuecomment-2247051724

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [pick](Row store) pick #38236 [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38240:
URL: https://github.com/apache/doris/pull/38240#issuecomment-2247052807

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[I] [Bug] Data Visible in Flink UI but Not Reaching Downstream Database [doris-flink-connector]

2024-07-24 Thread via GitHub


HeyChiang opened a new issue, #442:
URL: https://github.com/apache/doris-flink-connector/issues/442

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   flink-1.19.1
   flink-doris-connector-1.19-1.6.2.jar
   flink-cdc-common-3.0.1.jar
   flink-sql-connector-mysql-cdc-3.0.1.jar
   flink-sql-connector-oracle-cdc-3.0.1.jar
   
   
   ### What's Wrong?
   
   
![QQ_1721803834101](https://github.com/user-attachments/assets/b9a2a4a1-6e50-4ed3-9109-181e4d299e66)
   
   I'm observing a discrepancy between the data shown in the Flink UI and 
what's actually being written to the downstream database. The Flink UI shows a 
significant amount of data being processed and sent (as indicated by the 
colored blocks representing different tasks), but when I check the downstream 
database, no records are present. This suggests that while data is being read 
from the source and processed within Flink, it's not successfully being written 
to the target database.
   
   ### What You Expected?
   
   I expected that the data shown as processed in the Flink UI would correspond 
to actual records written to the downstream database. 
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](Row store) get the correct value of `row_store_page_size` after FE restarts [doris]

2024-07-24 Thread via GitHub


lxr599 commented on PR #38236:
URL: https://github.com/apache/doris/pull/38236#issuecomment-2247057552

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [spill](regression) add regression test cases for spill [doris]

2024-07-24 Thread via GitHub


jacktengg commented on PR #37319:
URL: https://github.com/apache/doris/pull/37319#issuecomment-2247061346

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [enhance](RateLimit) Add bvar to monitor OSS's rate limit sleep time and failure time [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2247060339

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](Row store) get the correct value of `row_store_page_size` after FE restarts [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38236:
URL: https://github.com/apache/doris/pull/38236#issuecomment-2247064646

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [pick](Row store) pick #38236 [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38240:
URL: https://github.com/apache/doris/pull/38240#issuecomment-2247066143

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](Variant) cast from jsonb with string value when enable `jsonb_string_as_string` should consider others types include date,datateime,array ... [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38291:
URL: https://github.com/apache/doris/pull/38291#issuecomment-2247067605

   
   
   TPC-DS: Total hot run time: 172971 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 0cfd6268d6d006d1f26eca127cbf794331081133, 
data reload: false
   
   query1   924 368 365 365
   query2   6357186917871787
   query3   6651200 211 200
   query4   27817   17418   17281   17281
   query5   3594487 478 478
   query6   269 163 168 163
   query7   4574290 285 285
   query8   239 186 194 186
   query9   8539241423922392
   query10  433 291 282 282
   query11  10796   996210081   9962
   query12  117 96  82  82
   query13  1647376 368 368
   query14  8662798068416841
   query15  223 172 166 166
   query16  7596431 491 431
   query17  1588551 547 547
   query18  1740288 283 283
   query19  201 156 161 156
   query20  92  83  87  83
   query21  211 130 127 127
   query22  4305423642334233
   query23  34149   33797   33636   33636
   query24  10787   297528652865
   query25  639 400 414 400
   query26  1088155 155 155
   query27  2479278 279 278
   query28  7275206320742063
   query29  931 656 648 648
   query30  254 165 232 165
   query31  976 762 744 744
   query32  92  62  54  54
   query33  752 326 324 324
   query34  939 500 497 497
   query35  871 736 750 736
   query36  1149999 983 983
   query37  149 83  88  83
   query38  2967289928382838
   query39  897 831 869 831
   query40  201 117 116 116
   query41  44  47  42  42
   query42  118 93  96  93
   query43  488 473 465 465
   query44  1207717 725 717
   query45  192 161 162 161
   query46  1079721 708 708
   query47  1862176417911764
   query48  368 293 285 285
   query49  864 396 405 396
   query50  783 378 388 378
   query51  6790670666246624
   query52  99  90  93  90
   query53  361 288 276 276
   query54  874 443 434 434
   query55  72  75  91  75
   query56  291 270 260 260
   query57  1141105410841054
   query58  251 244 257 244
   query59  2777265626922656
   query60  296 277 266 266
   query61  93  92  93  92
   query62  794 645 647 645
   query63  319 284 285 284
   query64  9513221616271627
   query65  3165308031053080
   query66  781 327 320 320
   query67  15675   15072   15024   15024
   query68  4638536 545 536
   query69  633 433 352 352
   query70  1172113611551136
   query71  438 271 273 271
   query72  8567537354715373
   query73  753 320 322 320
   query74  6192563557325635
   query75  4239266426612661
   query76  3359895 929 895
   query77  696 299 290 290
   query78  11120   925690729072
   query79  8132529 535 529
   query80  1178485 490 485
   query81  596 219 229 219
   query82  883 139 133 133
   query83  402 164 161 161
   query84  274 82  84  82
   query85  997 299 354 299
   query86  446 327 310 310
   query87  3343312930703070
   query88  5217239323762376
   query89  537 379 378 378
   query90  2014194 198 194
   query91  129 100 99  99
   query92  64  47  51  47
   query93  5442518 513 513
   query94  1289294 293 293
   query95  409 321 314 314
   query96  615 272 276 272
   query97  3155304230463042
   query98  226 194 196 194
   query99  1512129412461246
   Total cold run time: 292134 ms
   Total hot run time: 172971 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s

Re: [I] [Bug] Data Visible in Flink UI but Not Reaching Downstream Database [doris-flink-connector]

2024-07-24 Thread via GitHub


HeyChiang commented on issue #442:
URL: 
https://github.com/apache/doris-flink-connector/issues/442#issuecomment-2247069387

   
![QQ_1721804529446](https://github.com/user-attachments/assets/8ffeb9da-07b9-4b87-8d3a-086e1f2d1865)
   `
   "Process -> (BMS_BATCH_DEF: Writer -> BMS_BATCH_DEF: Committer, BMS_LOT_DEF: 
Writer -> BMS_LOT_DEF: Committer, BMS_STORER_POS: Writer -> BMS_STORER_POS: 
Committer, BMS_ST_IO_DOC: Writer -> BMS_ST_IO_DOC: Committer, BMS_ST_IO_DTL: 
Writer -> BMS_ST_IO_DTL: Committer, BMS_SU_SET_DOC: Writer -> BMS_SU_SET_DOC: 
Committer, BMS_SU_SET_DTL: Writer -> BMS_SU_SET_DTL: Committer, GPCS_INSIDER: 
Writer -> GPCS_INSIDER: Committer, GPCS_PLACEPOINT: Writer -> GPCS_PLACEPOINT: 
Committer, GPCS_PLACEPOINT_CLASS: Writer -> GPCS_PLACEPOINT_CLASS: Committer, 
GPCS_PLACEPOINT_CLASS_DTL: Writer -> GPCS_PLACEPOINT_CLASS_DTL: Committer, 
GRESA_SA_DOC: Writer -> GRESA_SA_DOC: Committer, GRESA_SA_DTL: Writer -> 
GRESA_SA_DTL: Committer, NGPCS_ALL_PRICE: Writer -> NGPCS_ALL_PRICE: Committer, 
NP_EFILES_OP_FILE: Writer -> NP_EFILES_OP_FILE: Committer, PUB_BRAND: Writer -> 
PUB_BRAND: Committer, PUB_EMPLOYEE: Writer -> PUB_EMPLOYEE: Committer, 
PUB_FACTORY: Writer -> PUB_FACTORY: Committer, PUB_GOODS: Writer -> PUB_G
 OODS: Committer, PUB_GOODS_CLASS: Writer -> PUB_GOODS_CLASS: Committer, 
PUB_GOODS_CLASS_DTL: Writer -> PUB_GOODS_CLASS_DTL: Committer) (2/3)#0" 
Id=12810 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24273330
at java.base@11.0.22/jdk.internal.misc.Unsafe.park(Native Method)
-  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24273330
at 
java.base@11.0.22/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at 
java.base@11.0.22/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at 
java.base@11.0.22/java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:417)
at 
org.apache.doris.flink.sink.writer.RecordBuffer.write(RecordBuffer.java:93)
at 
org.apache.doris.flink.sink.writer.RecordStream.write(RecordStream.java:63)
at 
org.apache.doris.flink.sink.writer.DorisStreamLoad.writeRecord(DorisStreamLoad.java:251)
at 
org.apache.doris.flink.sink.writer.DorisWriter.writeOneDorisRecord(DorisWriter.java:216)
at 
org.apache.doris.flink.sink.writer.DorisWriter.write(DorisWriter.java:186)
at 
app//org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:160)
at 
app//org.apache.flink.streaming.runtime.io.RecordProcessorUtils$$Lambda$825/0x0008007eec40.accept(Unknown
 Source)
at 
app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
at 
app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:61)
at 
app//org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:96)
at 
app//org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:94)
at 
app//org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103)
at 
org.apache.doris.flink.tools.cdc.ParsingProcessFunction.processElement(ParsingProcessFunction.java:51)
at 
org.apache.doris.flink.tools.cdc.ParsingProcessFunction.processElement(ParsingProcessFunction.java:31)
at 
app//org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
at 
app//org.apache.flink.streaming.runtime.io.RecordProcessorUtils$$Lambda$825/0x0008007eec40.accept(Unknown
 Source)
at 
app//org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:238)
at 
app//org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:157)
at 
app//org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:114)
at 
app//org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at 
app//org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:579)
at 
app//org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$746/0x0008006ff440.runDefaultAction(Unknown
 Source)
at 
app//org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at 
app//org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
at 
app//org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
at 
app//org.apache.flink.runtime.taskmanager.Task

Re: [PR] [fix](multi table) fix single stream multi table memory leak [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38255:
URL: https://github.com/apache/doris/pull/38255#issuecomment-2247070886

   
   
   TPC-DS: Total hot run time: 173558 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit b23b3901295a8bde756c30247c436c65152ce54e, 
data reload: false
   
   query1   914 382 368 368
   query2   6440189318621862
   query3   6654205 214 205
   query4   28353   17640   17276   17276
   query5   3770489 470 470
   query6   280 160 163 160
   query7   4595298 301 298
   query8   254 203 197 197
   query9   8499241923852385
   query10  435 292 272 272
   query11  11639   997910017   9979
   query12  114 82  88  82
   query13  1655381 379 379
   query14  10325   775375547554
   query15  215 166 175 166
   query16  7670474 471 471
   query17  1355542 534 534
   query18  1851277 274 274
   query19  194 144 145 144
   query20  89  82  81  81
   query21  210 129 129 129
   query22  4309414139573957
   query23  34162   34021   33647   33647
   query24  10904   287028452845
   query25  597 399 378 378
   query26  702 146 150 146
   query27  2291269 279 269
   query28  6066212821222122
   query29  905 646 649 646
   query30  252 158 154 154
   query31  1006780 798 780
   query32  98  57  53  53
   query33  656 334 342 334
   query34  952 526 508 508
   query35  893 731 750 731
   query36  1186980 978 978
   query37  142 82  82  82
   query38  2931276427542754
   query39  894 807 806 806
   query40  200 117 115 115
   query41  46  43  44  43
   query42  119 102 98  98
   query43  511 467 463 463
   query44  1082729 719 719
   query45  191 155 185 155
   query46  1090722 705 705
   query47  1849176617601760
   query48  357 312 284 284
   query49  837 394 404 394
   query50  774 379 386 379
   query51  6753674166956695
   query52  105 94  88  88
   query53  365 290 288 288
   query54  856 446 435 435
   query55  75  70  72  70
   query56  280 267 269 267
   query57  1113105110851051
   query58  236 251 247 247
   query59  2855277826072607
   query60  284 276 276 276
   query61  95  92  93  92
   query62  799 650 648 648
   query63  318 288 284 284
   query64  9134228617071707
   query65  3185310531353105
   query66  729 331 330 330
   query67  15415   15167   14961   14961
   query68  4514549 557 549
   query69  475 327 339 327
   query70  1170115411431143
   query71  378 286 283 283
   query72  6932595654465446
   query73  741 321 323 321
   query74  6146569057445690
   query75  3368270026572657
   query76  2124980 925 925
   query77  421 306 299 299
   query78  9702897997418979
   query79  2326515 522 515
   query80  2511469 532 469
   query81  599 220 228 220
   query82  807 141 135 135
   query83  306 164 171 164
   query84  252 89  81  81
   query85  2122306 289 289
   query86  360 313 330 313
   query87  3281312930913091
   query88  4031236223772362
   query89  486 380 378 378
   query90  1641190 183 183
   query91  127 100 98  98
   query92  60  49  49  49
   query93  2550514 512 512
   query94  708 288 286 286
   query95  417 327 312 312
   query96  609 269 266 266
   query97  3232301730193017
   query98  229 199 189 189
   query99  1533126812761268
   Total cold run time: 278268 ms
   Total hot run time: 173558 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s

Re: [PR] [fix] Broker already had baidu afs support but can't really use, so i fix it [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38275:
URL: https://github.com/apache/doris/pull/38275#issuecomment-2247076874

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](Variant) cast from jsonb with string value when enable `jsonb_string_as_string` should consider others types include date,datateime,array ... [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38291:
URL: https://github.com/apache/doris/pull/38291#issuecomment-2247076528

   
   
   ClickBench: Total hot run time: 30.57 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 0cfd6268d6d006d1f26eca127cbf794331081133, 
data reload: false
   
   query1   0.040.030.03
   query2   0.070.040.04
   query3   0.230.060.05
   query4   1.680.070.08
   query5   0.500.490.48
   query6   1.130.740.73
   query7   0.020.010.01
   query8   0.060.050.05
   query9   0.550.490.49
   query10  0.550.550.54
   query11  0.160.110.11
   query12  0.160.130.12
   query13  0.600.590.59
   query14  0.760.780.83
   query15  0.880.830.83
   query16  0.370.370.36
   query17  0.980.970.95
   query18  0.220.210.21
   query19  1.931.741.71
   query20  0.010.000.02
   query21  15.43   0.780.67
   query22  4.137.641.71
   query23  18.28   1.391.29
   query24  2.200.230.21
   query25  0.160.090.08
   query26  0.310.210.21
   query27  0.460.230.23
   query28  13.25   1.061.02
   query29  12.64   3.293.32
   query30  0.250.070.05
   query31  2.860.410.40
   query32  3.240.500.48
   query33  2.912.952.92
   query34  17.09   4.414.35
   query35  4.384.494.45
   query36  0.670.470.47
   query37  0.190.150.15
   query38  0.150.160.15
   query39  0.040.040.04
   query40  0.160.120.13
   query41  0.090.040.05
   query42  0.050.050.05
   query43  0.050.040.04
   Total cold run time: 109.89 s
   Total hot run time: 30.57 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix] Broker already had baidu afs support but can't really use, so i fix it [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38275:
URL: https://github.com/apache/doris/pull/38275#issuecomment-2247076938

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [branch-2.1](arrow-flight-sql) Add config arrow_flight_result_sink_buffer_size_rows [doris]

2024-07-24 Thread via GitHub


yiguolei merged PR #38223:
URL: https://github.com/apache/doris/pull/38223


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](multi table) fix single stream multi table memory leak [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38255:
URL: https://github.com/apache/doris/pull/38255#issuecomment-2247079809

   
   
   ClickBench: Total hot run time: 30.79 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit b23b3901295a8bde756c30247c436c65152ce54e, 
data reload: false
   
   query1   0.040.040.04
   query2   0.080.040.04
   query3   0.230.050.05
   query4   1.680.080.08
   query5   0.470.470.47
   query6   1.120.730.72
   query7   0.020.010.01
   query8   0.050.050.04
   query9   0.550.490.48
   query10  0.550.540.51
   query11  0.170.120.11
   query12  0.140.120.13
   query13  0.610.640.59
   query14  0.760.790.78
   query15  0.860.830.82
   query16  0.350.370.38
   query17  0.950.991.01
   query18  0.240.230.22
   query19  1.851.751.87
   query20  0.020.010.01
   query21  15.46   0.790.65
   query22  3.857.601.93
   query23  18.34   1.361.28
   query24  2.060.230.24
   query25  0.140.080.08
   query26  0.290.220.21
   query27  0.450.230.24
   query28  13.27   1.041.01
   query29  12.64   3.353.32
   query30  0.250.060.06
   query31  2.870.400.41
   query32  3.250.500.49
   query33  2.912.982.94
   query34  17.19   4.304.39
   query35  4.414.424.40
   query36  0.670.500.47
   query37  0.200.160.15
   query38  0.160.150.15
   query39  0.040.040.04
   query40  0.160.120.13
   query41  0.090.050.05
   query42  0.070.040.05
   query43  0.050.040.04
   Total cold run time: 109.56 s
   Total hot run time: 30.79 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



(doris) branch branch-2.1 updated: [branch-2.1](arrow-flight-sql) Add config arrow_flight_result_sink_buffer_size_rows (#38223)

2024-07-24 Thread yiguolei
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new 10c5c336d8e [branch-2.1](arrow-flight-sql) Add config 
arrow_flight_result_sink_buffer_size_rows (#38223)
10c5c336d8e is described below

commit 10c5c336d8e9e9436bae5a65735173ed58fe16e1
Author: Xinyi Zou 
AuthorDate: Wed Jul 24 15:15:39 2024 +0800

[branch-2.1](arrow-flight-sql) Add config 
arrow_flight_result_sink_buffer_size_rows (#38223)

pick #38221
---
 be/src/common/config.cpp  |  2 ++
 be/src/common/config.h|  3 +++
 be/src/exec/data_sink.cpp | 22 --
 be/src/pipeline/exec/result_sink_operator.cpp |  9 -
 be/src/pipeline/exec/result_sink_operator.h   |  1 +
 5 files changed, 30 insertions(+), 7 deletions(-)

diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 863d69338bc..76ce00097b0 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -617,6 +617,8 @@ DEFINE_Int32(load_process_safe_mem_permit_percent, "5");
 // result buffer cancelled time (unit: second)
 DEFINE_mInt32(result_buffer_cancelled_interval_time, "300");
 
+DEFINE_mInt32(arrow_flight_result_sink_buffer_size_rows, "32768");
+
 // the increased frequency of priority for remaining tasks in 
BlockingPriorityQueue
 DEFINE_mInt32(priority_queue_remaining_tasks_increased_frequency, "512");
 
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 9050701261c..447473e4fdd 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -675,6 +675,9 @@ DECLARE_Int32(load_process_safe_mem_permit_percent);
 // result buffer cancelled time (unit: second)
 DECLARE_mInt32(result_buffer_cancelled_interval_time);
 
+// arrow flight result sink buffer rows size, default 4096 * 8
+DECLARE_mInt32(arrow_flight_result_sink_buffer_size_rows);
+
 // the increased frequency of priority for remaining tasks in 
BlockingPriorityQueue
 DECLARE_mInt32(priority_queue_remaining_tasks_increased_frequency);
 
diff --git a/be/src/exec/data_sink.cpp b/be/src/exec/data_sink.cpp
index 054021439c5..dc651080298 100644
--- a/be/src/exec/data_sink.cpp
+++ b/be/src/exec/data_sink.cpp
@@ -78,10 +78,15 @@ Status DataSink::create_data_sink(ObjectPool* pool, const 
TDataSink& thrift_sink
 return Status::InternalError("Missing data buffer sink.");
 }
 
+int result_sink_buffer_size_rows = vectorized::RESULT_SINK_BUFFER_SIZE;
+if (!thrift_sink.result_sink.__isset.type ||
+thrift_sink.result_sink.type == 
TResultSinkType::ARROW_FLIGHT_PROTOCAL) {
+result_sink_buffer_size_rows = 
config::arrow_flight_result_sink_buffer_size_rows;
+}
+
 // TODO: figure out good buffer size based on size of output row
-sink->reset(new doris::vectorized::VResultSink(row_desc, output_exprs,
-   thrift_sink.result_sink,
-   
vectorized::RESULT_SINK_BUFFER_SIZE));
+sink->reset(new doris::vectorized::VResultSink(
+row_desc, output_exprs, thrift_sink.result_sink, 
result_sink_buffer_size_rows));
 break;
 }
 case TDataSinkType::RESULT_FILE_SINK: {
@@ -233,10 +238,15 @@ Status DataSink::create_data_sink(ObjectPool* pool, const 
TDataSink& thrift_sink
 return Status::InternalError("Missing data buffer sink.");
 }
 
+int result_sink_buffer_size_rows = vectorized::RESULT_SINK_BUFFER_SIZE;
+if (!thrift_sink.result_sink.__isset.type ||
+thrift_sink.result_sink.type == 
TResultSinkType::ARROW_FLIGHT_PROTOCAL) {
+result_sink_buffer_size_rows = 
config::arrow_flight_result_sink_buffer_size_rows;
+}
+
 // TODO: figure out good buffer size based on size of output row
-sink->reset(new doris::vectorized::VResultSink(row_desc, output_exprs,
-   thrift_sink.result_sink,
-   
vectorized::RESULT_SINK_BUFFER_SIZE));
+sink->reset(new doris::vectorized::VResultSink(
+row_desc, output_exprs, thrift_sink.result_sink, 
result_sink_buffer_size_rows));
 break;
 }
 case TDataSinkType::RESULT_FILE_SINK: {
diff --git a/be/src/pipeline/exec/result_sink_operator.cpp 
b/be/src/pipeline/exec/result_sink_operator.cpp
index 624b9ca192d..1aa7f37c1fe 100644
--- a/be/src/pipeline/exec/result_sink_operator.cpp
+++ b/be/src/pipeline/exec/result_sink_operator.cpp
@@ -19,6 +19,7 @@
 
 #include 
 
+#include "common/config.h"
 #include "common/object_pool.h"
 #include "exec/rowid_fetcher.h"
 #include "pipeline/exec/operator.h"
@@ -64,8 +65,9 @@ Status ResultSinkLocalState::init(RuntimeState* state, 

Re: [PR] [Fix](parquet-reader) Fix and optimize parquet min-max filtering. [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38277:
URL: https://github.com/apache/doris/pull/38277#issuecomment-2247082312

   
   
   TPC-H: Total hot run time: 39838 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit b42b73aef9328bb05957f36f485d49557d2f, 
data reload: false
   
   -- Round 1 --
   q1   17617   436242614261
   q2   2006188 188 188
   q3   10450   114810911091
   q4   10183   819 790 790
   q5   7537268226392639
   q6   220 138 134 134
   q7   951 592 602 592
   q8   9227206920532053
   q9   8794653265216521
   q10  8743373937343734
   q11  455 240 235 235
   q12  405 218 223 218
   q13  17757   297029722970
   q14  276 238 249 238
   q15  519 483 491 483
   q16  507 375 374 374
   q17  960 655 705 655
   q18  8040746474067406
   q19  8040145914601459
   q20  660 312 328 312
   q21  4928320132343201
   q22  351 295 284 284
   Total cold run time: 118626 ms
   Total hot run time: 39838 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   4397420542414205
   q2   370 263 259 259
   q3   3018291928932893
   q4   1992167716401640
   q5   5619549754655465
   q6   228 141 129 129
   q7   2188189618041804
   q8   3285340834053405
   q9   8667878987868786
   q10  4103381737783778
   q11  573 504 513 504
   q12  802 639 642 639
   q13  15887   317531853175
   q14  315 277 273 273
   q15  523 488 496 488
   q16  489 441 447 441
   q17  1790151015081508
   q18  8034786477487748
   q19  1718161414661466
   q20  2195189518441844
   q21  9580479446544654
   q22  609 508 519 508
   Total cold run time: 76382 ms
   Total hot run time: 55612 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](function) the bucket number arg of width_bucket should be a positive integer value (#37892) [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38295:
URL: https://github.com/apache/doris/pull/38295#issuecomment-2247092825

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to 
[doris-website](https://github.com/apache/doris-website).
   See [Doris 
Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix](function) the bucket number arg of width_bucket should be a positive integer value (#37892) [doris]

2024-07-24 Thread via GitHub


mrhhsg opened a new pull request, #38295:
URL: https://github.com/apache/doris/pull/38295

   ## Proposed changes
   
   pick #37892
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] Data Visible in Flink UI but Not Reaching Downstream Database [doris-flink-connector]

2024-07-24 Thread via GitHub


HeyChiang commented on issue #442:
URL: 
https://github.com/apache/doris-flink-connector/issues/442#issuecomment-2247094059

   > Is checkpoint enabled for the flink job?
   
   Yes. Here is the Flink command I'm using:
   
   ```bash
   bin/flink run \
-Dexecution.checkpointing.interval=2min \
-Dexecution.checkpointing.timeout=60min \
-Dexecution.checkpointing.tolerable-failed-checkpoints=3 \
-Dparallelism.default=3 \
-c org.apache.doris.flink.tools.cdc.CdcTools \
lib/flink-doris-connector-1.19-1.6.2.jar \
oracle-sync-database \
--database djwkerp \
--oracle-conf hostname=192.168.191.210 \
--oracle-conf port=1521 \
--oracle-conf username=cdc_sync_user \
--oracle-conf password="xx" \
--oracle-conf database-name=data \
--oracle-conf schema-name=DJWKERP \
--including-tables 
"NGPCS_ALL_PRICE|GPCS_PLACEPOINT|PUB_EMPLOYEE|BMS_BATCH_DEF|BMS_LOT_DEF|BMS_STORER_POS|GRESA_SA_DOC|GRESA_SA_DTL|GPCS_INSIDER|BMS_ST_IO_DOC|BMS_ST_IO_DTL|GPCS_PLACEPOINT_CLASS|GPCS_PLACEPOINT_CLASS_DTL|PUB_GOODS_CLASS|PUB_GOODS_CLASS_DTL|BMS_SU_SET_DOC|BMS_SU_SET_DTL|PUB_GOODS|PUB_FACTORY|PUB_BRAND|NP_EFILES_OP_FILE"
 \
--sink-conf fenodes=192.168.191.93:8030 \
--sink-conf username=cdc_sync_user \
--sink-conf password="xx" \
--sink-conf 
jdbc-url=jdbc:mysql://192.168.191.93:9030?allowPublicKeyRetrieval=true \
--sink-conf sink.label-prefix=label \
--table-conf replication_num=3
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](function) the bucket number arg of width_bucket should be a positive integer value (#37892) [doris]

2024-07-24 Thread via GitHub


mrhhsg commented on PR #38295:
URL: https://github.com/apache/doris/pull/38295#issuecomment-2247092902

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] rename delete-recover to recover-from-recyclebin [doris-website]

2024-07-24 Thread via GitHub


KassieZ commented on code in PR #910:
URL: https://github.com/apache/doris-website/pull/910#discussion_r1689271593


##
i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/data-admin/recyclebin.md:
##


Review Comment:
   Title is equal to #, the level 2 title .  the body start with the level 2 
title ## .
   
   {
   "title": "Here is level 1 title",
   "language": "en"
   }
   
   ## Level 2 title
   ### Level 3 title



##
i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/data-admin/recyclebin.md:
##


Review Comment:
   Title is equal to #, the level 2 title .  the body start with the level 2 
title ## .
   
   {
   "title": "Here is level 1 title",
   "language": "en"
   }
   
   ## Level 2 title
   ### Level 3 title



##
versioned_docs/version-2.0/admin-manual/data-admin/recyclebin.md:
##


Review Comment:
   Title is equal to #, the level 2 title .  the body start with the level 2 
title ## .
   
   {
   "title": "Here is level 1 title",
   "language": "en"
   }
   
   ## Level 2 title
   ### Level 3 title



##
i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/data-admin/recyclebin.md:
##


Review Comment:
   Title is equal to #, the level 2 title .  the body start with the level 2 
title ## .
   
   {
   "title": "Here is level 1 title",
   "language": "en"
   }
   
   ## Level 2 title
   ### Level 3 title



##
docs/admin-manual/data-admin/recyclebin.md:
##


Review Comment:
   1. The body start with the level 2 title ## .
   
   2. please change the relative path to sql statements :
   
[RECOVER](../../sql-manual/sql-statements/Database-Administration-Statements/RECOVER.md)
 
   
   



##
versioned_docs/version-2.1/admin-manual/data-admin/recyclebin.md:
##


Review Comment:
   Title is equal to #, the level 2 title .  the body start with the level 2 
title ## .
   
   {
   "title": "Here is level 1 title",
   "language": "en"
   }
   
   ## Level 2 title
   ### Level 3 title
   
   
   please change the relative path to sql statements 
:[RECOVER](../../sql-manual/sql-statements/Database-Administration-Statements/RECOVER.md)



##
versioned_docs/version-3.0/admin-manual/data-admin/recyclebin.md:
##


Review Comment:
   1. title without q
   https://github.com/user-attachments/assets/116652b8-a7d6-44c9-b90e-5f6ca95e4768";>
   
   2. body start with level 2 title
   
   3. please change relative path to sql statements: 
[RECOVER](../../sql-manual/sql-statements/Database-Administration-Statements/RECOVER.md)



##
i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/admin-manual/data-admin/recyclebin.md:
##


Review Comment:
   Title is equal to #, the level 2 title .  the body start with the level 2 
title ## .
   
   {
   "title": "Here is level 1 title",
   "language": "en"
   }
   
   ## Level 2 title
   ### Level 3 title



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [regression](test) check load channel and load stream before quit [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38293:
URL: https://github.com/apache/doris/pull/38293#issuecomment-2247098423

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [regression](test) check load channel and load stream before quit [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38293:
URL: https://github.com/apache/doris/pull/38293#issuecomment-2247098476

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](inverted index) fixed in_list condition not indexed on pipelinex [doris]

2024-07-24 Thread via GitHub


zzzxl1993 commented on PR #38178:
URL: https://github.com/apache/doris/pull/38178#issuecomment-2247101967

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [improvement](Nereids) add where Null rule to create empty relation as where false [doris]

2024-07-24 Thread via GitHub


LiBinfeng-01 commented on PR #38135:
URL: https://github.com/apache/doris/pull/38135#issuecomment-2247100244

   ./run external


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](parquet-reader) Fix and optimize parquet min-max filtering. [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38277:
URL: https://github.com/apache/doris/pull/38277#issuecomment-2247100928

   
   
   TPC-DS: Total hot run time: 173837 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit b42b73aef9328bb05957f36f485d49557d2f, 
data reload: false
   
   query1   911 377 366 366
   query2   6437192619071907
   query3   6645201 216 201
   query4   28565   17394   17305   17305
   query5   3597475 476 475
   query6   254 187 167 167
   query7   4586291 282 282
   query8   235 200 188 188
   query9   8539239223702370
   query10  427 322 283 283
   query11  10723   10102   10124   10102
   query12  115 83  81  81
   query13  1647383 394 383
   query14  10095   768476257625
   query15  221 167 164 164
   query16  7633512 481 481
   query17  1553566 513 513
   query18  1796270 267 267
   query19  195 153 167 153
   query20  84  80  81  80
   query21  212 133 123 123
   query22  4464425640314031
   query23  34070   33602   33655   33602
   query24  10818   287729042877
   query25  613 409 403 403
   query26  698 153 149 149
   query27  2318283 280 280
   query28  5802210520872087
   query29  878 638 616 616
   query30  251 149 160 149
   query31  985 748 776 748
   query32  95  51  52  51
   query33  659 328 345 328
   query34  913 498 506 498
   query35  900 792 793 792
   query36  1194955 965 955
   query37  136 80  81  80
   query38  2947286828402840
   query39  866 806 816 806
   query40  201 123 119 119
   query41  48  43  43  43
   query42  110 97  96  96
   query43  510 487 481 481
   query44  1049717 722 717
   query45  199 167 172 167
   query46  1079757 722 722
   query47  1905180317791779
   query48  377 289 281 281
   query49  843 400 406 400
   query50  762 384 380 380
   query51  6850669266066606
   query52  104 95  87  87
   query53  353 295 289 289
   query54  877 458 439 439
   query55  74  75  70  70
   query56  298 258 274 258
   query57  1142105410411041
   query58  245 253 274 253
   query59  2820258528342585
   query60  299 285 272 272
   query61  97  93  96  93
   query62  802 655 650 650
   query63  316 280 286 280
   query64  9121219416571657
   query65  3152309631013096
   query66  756 337 390 337
   query67  15349   15095   15050   15050
   query68  4532549 542 542
   query69  481 342 343 342
   query70  1186112311241123
   query71  416 274 280 274
   query72  7153551558805515
   query73  739 319 321 319
   query74  6124573456605660
   query75  4199268426772677
   query76  2762929 883 883
   query77  613 309 290 290
   query78  9667959089408940
   query79  3207520 522 520
   query80  1625465 465 465
   query81  598 220 216 216
   query82  832 133 142 133
   query83  347 164 170 164
   query84  272 89  85  85
   query85  1914316 305 305
   query86  485 311 316 311
   query87  3259311631323116
   query88  4563233623592336
   query89  479 377 374 374
   query90  1850192 188 188
   query91  126 170 103 103
   query92  56  49  47  47
   query93  4743515 505 505
   query94  1225295 275 275
   query95  407 316 317 316
   query96  613 273 271 271
   query97  3153302930663029
   query98  232 205 197 197
   query99  1562127012601260
   Total cold run time: 282235 ms
   Total hot run time: 173837 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the

Re: [PR] [fix](function) the bucket number arg of width_bucket should be a positive integer value (#37892) [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38295:
URL: https://github.com/apache/doris/pull/38295#issuecomment-2247103181

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](inverted index) opt value extraction from column to string [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38179:
URL: https://github.com/apache/doris/pull/38179#issuecomment-2247105891

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix])(function) add function regexp_extract_or_null [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38296:
URL: https://github.com/apache/doris/pull/38296#issuecomment-2247108395

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to 
[doris-website](https://github.com/apache/doris-website).
   See [Doris 
Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] adjust table design dir to match create table [doris-website]

2024-07-24 Thread via GitHub


KassieZ commented on code in PR #906:
URL: https://github.com/apache/doris-website/pull/906#discussion_r1689282464


##
docs/table-design/overview.md:
##


Review Comment:
   1. the language is EN docs, not zh-CN
   
   
   2. body start with level 2 title 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](inverted index) opt value extraction from column to string [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38179:
URL: https://github.com/apache/doris/pull/38179#issuecomment-2247105818

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix])(function) add function regexp_extract_or_null [doris]

2024-07-24 Thread via GitHub


suxiaogang223 opened a new pull request, #38296:
URL: https://github.com/apache/doris/pull/38296

   ## Proposed changes
   
   Add function regexp_extract_or_null to be compatible with presto. The 
function is same as regexp_extract, except that it returns null when no match 
is found.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](parquet-reader) Fix and optimize parquet min-max filtering. [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38277:
URL: https://github.com/apache/doris/pull/38277#issuecomment-2247110788

   
   
   ClickBench: Total hot run time: 30.21 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit b42b73aef9328bb05957f36f485d49557d2f, 
data reload: false
   
   query1   0.040.040.04
   query2   0.090.040.03
   query3   0.220.060.06
   query4   1.660.090.08
   query5   0.500.470.48
   query6   1.130.730.72
   query7   0.020.020.02
   query8   0.060.040.05
   query9   0.550.500.47
   query10  0.540.530.54
   query11  0.150.110.12
   query12  0.150.120.13
   query13  0.600.580.58
   query14  0.770.770.76
   query15  0.850.800.81
   query16  0.350.370.37
   query17  1.010.980.96
   query18  0.240.220.22
   query19  1.791.731.70
   query20  0.010.010.01
   query21  15.40   0.770.66
   query22  4.347.791.53
   query23  18.18   1.361.25
   query24  2.080.210.24
   query25  0.160.080.09
   query26  0.300.210.20
   query27  0.450.230.22
   query28  13.26   1.011.00
   query29  12.61   3.283.30
   query30  0.250.060.06
   query31  2.880.390.38
   query32  3.270.490.47
   query33  2.842.922.87
   query34  17.10   4.404.40
   query35  4.434.514.46
   query36  0.650.470.47
   query37  0.180.150.16
   query38  0.150.150.14
   query39  0.040.030.03
   query40  0.150.130.13
   query41  0.100.050.05
   query42  0.060.040.05
   query43  0.040.040.04
   Total cold run time: 109.65 s
   Total hot run time: 30.21 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix])(function) add function regexp_extract_or_null [doris]

2024-07-24 Thread via GitHub


suxiaogang223 commented on PR #38296:
URL: https://github.com/apache/doris/pull/38296#issuecomment-2247112237

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] Data Visible in Flink UI but Not Reaching Downstream Database [doris-flink-connector]

2024-07-24 Thread via GitHub


HeyChiang commented on issue #442:
URL: 
https://github.com/apache/doris-flink-connector/issues/442#issuecomment-2247116534

   
![QQ_1721806383826](https://github.com/user-attachments/assets/a961bf25-e16c-477a-b485-186e9fac3be2)
   FAILED


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



(doris) branch branch-2.1 updated: [opt](inverted index) opt value extraction from column to string (#38179)

2024-07-24 Thread airborne
This is an automated email from the ASF dual-hosted git repository.

airborne pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new c6cd1e2e3d4 [opt](inverted index) opt value extraction from column to 
string (#38179)
c6cd1e2e3d4 is described below

commit c6cd1e2e3d49092960ca0d674a0b677e7043266f
Author: zzzxl <33418555+zzzxl1...@users.noreply.github.com>
AuthorDate: Wed Jul 24 15:38:41 2024 +0800

[opt](inverted index) opt value extraction from column to string (#38179)

## Proposed changes

https://github.com/apache/doris/pull/37395
---
 be/src/vec/exprs/vexpr.cpp | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/be/src/vec/exprs/vexpr.cpp b/be/src/vec/exprs/vexpr.cpp
index 3a965c27c92..d788c8dc518 100644
--- a/be/src/vec/exprs/vexpr.cpp
+++ b/be/src/vec/exprs/vexpr.cpp
@@ -631,11 +631,13 @@ std::string VExpr::gen_predicate_result_sign(Block& 
block, const ColumnNumbers&
 // Generating 'result_sign' from 'inlist' requires sorting the values.
 std::set values;
 for (size_t i = 1; i < arguments.size(); i++) {
-values.insert(block.get_by_position(arguments[i]).to_string(0));
+const auto& entry = block.get_by_position(arguments[i]);
+values.insert(entry.type->to_string(*entry.column, 0));
 }
 pred_result_sign += boost::join(values, ",");
 } else {
-pred_result_sign += block.get_by_position(arguments[1]).to_string(0);
+const auto& entry = block.get_by_position(arguments[1]);
+pred_result_sign += entry.type->to_string(*entry.column, 0);
 }
 return pred_result_sign;
 }


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix](nereids) refine row count estimation for mark join [doris]

2024-07-24 Thread via GitHub


xzj7019 opened a new pull request, #38297:
URL: https://github.com/apache/doris/pull/38297

   ## Proposed changes
   
   pick from https://github.com/apache/doris/pull/38270
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids) refine row count estimation for mark join [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38297:
URL: https://github.com/apache/doris/pull/38297#issuecomment-2247120795

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to 
[doris-website](https://github.com/apache/doris-website).
   See [Doris 
Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix])(function) add function regexp_extract_or_null [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38296:
URL: https://github.com/apache/doris/pull/38296#issuecomment-2247121466

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](inverted index) opt value extraction from column to string [doris]

2024-07-24 Thread via GitHub


airborne12 merged PR #38179:
URL: https://github.com/apache/doris/pull/38179


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix])(function) add function regexp_extract_or_null [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38296:
URL: https://github.com/apache/doris/pull/38296#issuecomment-2247119031

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](group commit) group commit show error url [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38292:
URL: https://github.com/apache/doris/pull/38292#issuecomment-2247124778

   
   
   TPC-H: Total hot run time: 40027 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 1f6c62f606fbcfd0ac0e2c51986f1231765fa2d6, 
data reload: false
   
   -- Round 1 --
   q1   18679   471443084308
   q2   2023190 189 189
   q3   10554   116811711168
   q4   10249   885 765 765
   q5   7554271026442644
   q6   229 141 140 140
   q7   957 595 605 595
   q8   9218213820842084
   q9   8754658865776577
   q10  8722383737873787
   q11  450 233 231 231
   q12  403 222 227 222
   q13  17753   296630122966
   q14  268 248 241 241
   q15  523 480 473 473
   q16  498 386 378 378
   q17  1035724 695 695
   q18  8360764773737373
   q19  7082139114571391
   q20  688 318 308 308
   q21  4932321539713215
   q22  352 277 281 277
   Total cold run time: 119283 ms
   Total hot run time: 40027 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   4422429442824282
   q2   371 259 275 259
   q3   3189294929552949
   q4   1991175617211721
   q5   5576554655895546
   q6   239 135 135 135
   q7   2269190519151905
   q8   3281344134223422
   q9   8822891187878787
   q10  3977376038473760
   q11  592 515 518 515
   q12  837 636 647 636
   q13  17046   317031853170
   q14  313 300 271 271
   q15  529 495 500 495
   q16  483 439 436 436
   q17  1824154115241524
   q18  8179789577117711
   q19  1774169316361636
   q20  2966188818331833
   q21  9602472648914726
   q22  602 509 518 509
   Total cold run time: 78884 ms
   Total hot run time: 56228 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Chore](fuzzy) enable some fuzzy variables [doris]

2024-07-24 Thread via GitHub


BiteThet merged PR #38155:
URL: https://github.com/apache/doris/pull/38155


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [Improvement](profile) add catalog info in profile [doris]

2024-07-24 Thread via GitHub


Yulei-Yang opened a new pull request, #38298:
URL: https://github.com/apache/doris/pull/38298

   bp #38283


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](profile) add catalog info in profile [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38298:
URL: https://github.com/apache/doris/pull/38298#issuecomment-2247129127

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to 
[doris-website](https://github.com/apache/doris-website).
   See [Doris 
Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](profile) add catalog info in profile [doris]

2024-07-24 Thread via GitHub


Yulei-Yang commented on PR #38298:
URL: https://github.com/apache/doris/pull/38298#issuecomment-2247129446

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](profile) add catalog info in profile [doris]

2024-07-24 Thread via GitHub


Yulei-Yang commented on PR #38283:
URL: https://github.com/apache/doris/pull/38283#issuecomment-2247129206

   run cloud_p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



(doris) branch master updated: [Chore](fuzzy) enable some fuzzy variables (#38155)

2024-07-24 Thread panxiaolei
This is an automated email from the ASF dual-hosted git repository.

panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 501270d99e5 [Chore](fuzzy) enable some fuzzy variables (#38155)
501270d99e5 is described below

commit 501270d99e5652ce41851186f894aa83fcc89f3f
Author: Pxl 
AuthorDate: Wed Jul 24 15:44:24 2024 +0800

[Chore](fuzzy) enable some fuzzy variables (#38155)

## Proposed changes
enable fuzzy topnOptLimitThreshold, runtimeFilterMaxInNum,
runtimeFilterWaitTimeMs, enableSyncRuntimeFilterSize
---
 fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java 
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 12455e31c76..32212ca9d45 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -2113,10 +2113,14 @@ public class SessionVariable implements Serializable, 
Writable {
 this.rewriteOrToInPredicateThreshold = 10;
 this.enableFunctionPushdown = false;
 this.enableDeleteSubPredicateV2 = false;
+this.topnOptLimitThreshold = 0;
+this.enableSyncRuntimeFilterSize = true;
 } else {
 this.rewriteOrToInPredicateThreshold = 2;
 this.enableFunctionPushdown = true;
 this.enableDeleteSubPredicateV2 = true;
+this.topnOptLimitThreshold = 1024;
+this.enableSyncRuntimeFilterSize = false;
 }
 
 /*
@@ -2166,6 +2170,7 @@ public class SessionVariable implements Serializable, 
Writable {
 switch (Config.pull_request_id % 3) {
 case 0:
 this.fragmentTransmissionCompressionCodec = "snappy";
+this.runtimeFilterWaitTimeMs = 10;
 break;
 case 1:
 this.fragmentTransmissionCompressionCodec = "lz4";
@@ -2180,10 +2185,12 @@ public class SessionVariable implements Serializable, 
Writable {
 case 0:
 this.parallelScanMaxScannersCount = 32;
 this.parallelScanMinRowsPerScanner = 64;
+this.runtimeFilterMaxInNum = 10;
 break;
 case 1:
 this.parallelScanMaxScannersCount = 16;
 this.parallelScanMinRowsPerScanner = 128;
+this.runtimeFilterMaxInNum = 0;
 break;
 case 2:
 this.parallelScanMaxScannersCount = 8;


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction (#38224) [doris]

2024-07-24 Thread via GitHub


luwei16 opened a new pull request, #38299:
URL: https://github.com/apache/doris/pull/38299

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction (#38224) [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38299:
URL: https://github.com/apache/doris/pull/38299#issuecomment-2247130579

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to 
[doris-website](https://github.com/apache/doris-website).
   See [Doris 
Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids) refine row count estimation for mark join [doris]

2024-07-24 Thread via GitHub


xzj7019 commented on PR #38297:
URL: https://github.com/apache/doris/pull/38297#issuecomment-2247126655

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](profile) Provide more info for schedule time [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38290:
URL: https://github.com/apache/doris/pull/38290#issuecomment-2247132398

   
   
   TPC-H: Total hot run time: 40441 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 6e91ded334a0537d243ba5aad67977b275d9abde, 
data reload: false
   
   -- Round 1 --
   q1   18618   455643914391
   q2   2023191 193 191
   q3   10504   117210681068
   q4   10201   839 837 837
   q5   7562275926732673
   q6   229 145 144 144
   q7   976 608 613 608
   q8   9228212221152115
   q9   8787662366026602
   q10  8756379838203798
   q11  452 241 243 241
   q12  402 234 226 226
   q13  17773   305230273027
   q14  290 236 242 236
   q15  524 486 493 486
   q16  513 395 390 390
   q17  991 737 742 737
   q18  8219752475877524
   q19  6533136112771277
   q20  707 329 337 329
   q21  5093324333163243
   q22  371 298 309 298
   Total cold run time: 118752 ms
   Total hot run time: 40441 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   4421435742584258
   q2   372 270 266 266
   q3   3125296929602960
   q4   1950175417801754
   q5   5630553655385536
   q6   232 139 138 138
   q7   2297181718531817
   q8   3353345034263426
   q9   8806891287798779
   q10  4166380739143807
   q11  606 516 500 500
   q12  800 626 630 626
   q13  16323   320032133200
   q14  319 329 299 299
   q15  549 488 505 488
   q16  518 449 445 445
   q17  1854155215211521
   q18  8168801579547954
   q19  1787157615981576
   q20  2119190118801880
   q21  5162495749474947
   q22  613 551 518 518
   Total cold run time: 73170 ms
   Total hot run time: 56695 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](runtime-filter) adjust always true judge rule of runtime filter apply [doris]

2024-07-24 Thread via GitHub


BiteThet commented on PR #38300:
URL: https://github.com/apache/doris/pull/38300#issuecomment-2247135949

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](runtime-filter) adjust always true judge rule of runtime filter apply [doris]

2024-07-24 Thread via GitHub


BiteThet commented on PR #38300:
URL: https://github.com/apache/doris/pull/38300#issuecomment-2247138433

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction (#38224) [doris]

2024-07-24 Thread via GitHub


luwei16 commented on PR #38299:
URL: https://github.com/apache/doris/pull/38299#issuecomment-2247138920

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](Row store) get the correct value of `row_store_page_size` after FE restarts [doris]

2024-07-24 Thread via GitHub


lxr599 commented on PR #38236:
URL: https://github.com/apache/doris/pull/38236#issuecomment-2247139534

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction (#38224) [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38299:
URL: https://github.com/apache/doris/pull/38299#issuecomment-2247141633

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](group commit) group commit show error url [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38292:
URL: https://github.com/apache/doris/pull/38292#issuecomment-2247144170

   
   
   TPC-DS: Total hot run time: 174329 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 1f6c62f606fbcfd0ac0e2c51986f1231765fa2d6, 
data reload: false
   
   query1   914 370 363 363
   query2   6394204818431843
   query3   6652210 216 210
   query4   28257   17280   17418   17280
   query5   3663482 493 482
   query6   264 173 167 167
   query7   4576283 281 281
   query8   240 187 190 187
   query9   8393241723712371
   query10  420 290 267 267
   query11  12224   10246   10087   10087
   query12  121 92  83  83
   query13  1634382 370 370
   query14  10250   830976227622
   query15  220 170 166 166
   query16  7721424 465 424
   query17  1279537 521 521
   query18  1855273 272 272
   query19  191 154 147 147
   query20  87  83  89  83
   query21  203 132 132 132
   query22  4398435842124212
   query23  34110   33748   33753   33748
   query24  11229   290928622862
   query25  603 397 406 397
   query26  974 155 152 152
   query27  2438274 280 274
   query28  7015210120812081
   query29  864 641 622 622
   query30  269 158 160 158
   query31  954 759 748 748
   query32  90  52  57  52
   query33  741 353 331 331
   query34  913 498 519 498
   query35  925 769 764 764
   query36  1151989 994 989
   query37  136 82  89  82
   query38  2932277927752775
   query39  864 825 815 815
   query40  194 117 121 117
   query41  45  43  43  43
   query42  109 98  98  98
   query43  509 489 495 489
   query44  1163721 734 721
   query45  194 162 164 162
   query46  1090755 700 700
   query47  1861176117951761
   query48  364 298 297 297
   query49  835 403 413 403
   query50  778 384 395 384
   query51  6740671266886688
   query52  107 91  95  91
   query53  351 298 283 283
   query54  860 445 443 443
   query55  75  74  73  73
   query56  291 262 294 262
   query57  1133103210871032
   query58  239 240 258 240
   query59  3003267728962677
   query60  309 289 275 275
   query61  94  92  92  92
   query62  802 643 658 643
   query63  330 293 283 283
   query64  9514219716811681
   query65  3189311031133110
   query66  826 325 324 324
   query67  15517   15191   15083   15083
   query68  4532539 545 539
   query69  526 342 340 340
   query70  1192111411001100
   query71  408 281 271 271
   query72  7355575756735673
   query73  756 323 321 321
   query74  6230568356175617
   query75  3396268026812680
   query76  2226951 941 941
   query77  506 293 305 293
   query78  9622910190059005
   query79  2482534 522 522
   query80  2525467 470 467
   query81  596 220 220 220
   query82  1041139 136 136
   query83  298 164 164 164
   query84  252 87  87  87
   query85  1251319 304 304
   query86  457 337 272 272
   query87  3258311031393110
   query88  3363242523492349
   query89  466 375 397 375
   query90  1723187 189 187
   query91  131 101 103 101
   query92  60  50  50  50
   query93  2102524 512 512
   query94  1164277 295 277
   query95  416 318 306 306
   query96  606 267 272 267
   query97  3221307930043004
   query98  221 206 203 203
   query99  1508126512631263
   Total cold run time: 280576 ms
   Total hot run time: 174329 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the

Re: [PR] [fix])(function) add function regexp_extract_or_null [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38296:
URL: https://github.com/apache/doris/pull/38296#issuecomment-2247145957

   
   
   TPC-H: Total hot run time: 40242 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 8e4da4816b00e4ad44cd101cc9a7868c49671ed7, 
data reload: false
   
   -- Round 1 --
   q1   18176   444943704370
   q2   2576217 187 187
   q3   11909   118411461146
   q4   10626   944 789 789
   q5   8069272527092709
   q6   225 138 143 138
   q7   960 589 589 589
   q8   9215204320792043
   q9   8734655865536553
   q10  8808376937523752
   q11  446 245 236 236
   q12  399 237 234 234
   q13  17781   300129802980
   q14  266 241 225 225
   q15  533 485 485 485
   q16  481 394 376 376
   q17  961 748 667 667
   q18  8113753374917491
   q19  3982140314231403
   q20  709 330 330 330
   q21  4910328632533253
   q22  350 294 286 286
   Total cold run time: 118229 ms
   Total hot run time: 40242 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   4415426642414241
   q2   367 282 268 268
   q3   3004276327412741
   q4   1904162516241624
   q5   5285530152705270
   q6   218 132 129 129
   q7   2107171217011701
   q8   3188334433173317
   q9   8366834583648345
   q10  3929369537313695
   q11  558 473 486 473
   q12  765 602 618 602
   q13  16408   295829522952
   q14  310 275 272 272
   q15  515 474 470 470
   q16  474 418 423 418
   q17  1776147114591459
   q18  7635761072767276
   q19  1694147215941472
   q20  1965176117581758
   q21  4825464446234623
   q22  585 491 509 491
   Total cold run time: 70293 ms
   Total hot run time: 53597 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](workloadgroup) use slot num to control memory distribution among one workload group [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38237:
URL: https://github.com/apache/doris/pull/38237#issuecomment-2247146801

   
   
   TPC-H: Total hot run time: 40076 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 94fe659308739e065aa346c1b461ad59cc8e8b36, 
data reload: false
   
   -- Round 1 --
   q1   19371   561743124312
   q2   2024191 199 191
   q3   10546   116111221122
   q4   10245   850 834 834
   q5   7584281327202720
   q6   225 138 138 138
   q7   971 605 617 605
   q8   9219211720752075
   q9   8742658265416541
   q10  8749380237663766
   q11  520 237 251 237
   q12  401 231 227 227
   q13  18834   294129882941
   q14  281 233 244 233
   q15  523 489 496 489
   q16  494 407 406 406
   q17  985 673 689 673
   q18  8127749374757475
   q19  7048144712991299
   q20  680 324 330 324
   q21  4960318232973182
   q22  360 287 286 286
   Total cold run time: 120889 ms
   Total hot run time: 40076 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   4395424742654247
   q2   388 287 285 285
   q3   3183292328952895
   q4   2041168416611661
   q5   5590550555805505
   q6   231 137 136 136
   q7   2229182419011824
   q8   3277344434233423
   q9   8866881187918791
   q10  3973389539393895
   q11  595 519 502 502
   q12  819 649 623 623
   q13  16322   321531523152
   q14  305 297 281 281
   q15  521 495 481 481
   q16  512 452 444 444
   q17  1852154015321532
   q18  8230790778027802
   q19  1763154516411545
   q20  2142187918801879
   q21  9952467848254678
   q22  584 498 492 492
   Total cold run time: 0 ms
   Total hot run time: 56073 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](runtime-filter) adjust always true judge rule of runtime filter apply [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on code in PR #38300:
URL: https://github.com/apache/doris/pull/38300#discussion_r1689316626


##
be/src/olap/comparison_predicate.h:
##
@@ -246,14 +246,17 @@ class ComparisonPredicateBase : public ColumnPredicate {
 template 
 __attribute__((flatten)) void _evaluate_vec_internal(const 
vectorized::IColumn& column,

Review Comment:
   warning: function '_evaluate_vec_internal' has cognitive complexity of 59 
(threshold 50) [readability-function-cognitive-complexity]
   ```cpp
   __attribute__((flatten)) void _evaluate_vec_internal(const 
vectorized::IColumn& column,
 ^
   ```
   
   Additional context
   
   **be/src/olap/comparison_predicate.h:250:** +1, including nesting penalty of 
0, nesting level increased to 1
   ```cpp
   if (_can_ignore()) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:251:** +2, including nesting penalty of 
1, nesting level increased to 2
   ```cpp
   if (is_and) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:252:** +3, including nesting penalty of 
2, nesting level increased to 3
   ```cpp
   for (uint16_t i = 0; i < size; i++) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:255:** +1, nesting level increased to 2
   ```cpp
   } else {
 ^
   ```
   **be/src/olap/comparison_predicate.h:261:** +1, including nesting penalty of 
0, nesting level increased to 1
   ```cpp
   if (column.is_nullable()) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:269:** +2, including nesting penalty of 
1, nesting level increased to 2
   ```cpp
   if (nested_column.is_column_dictionary()) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:270:** +3, including nesting penalty of 
2, nesting level increased to 3
   ```cpp
   if constexpr (std::is_same_v) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:276:** +4, including nesting penalty of 
3, nesting level increased to 4
   ```cpp
   do {
   ^
   ```
   **be/src/olap/comparison_predicate.h:277:** +5, including nesting penalty of 
4, nesting level increased to 5
   ```cpp
   if constexpr (PT == PredicateType::EQ) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:278:** +6, including nesting penalty of 
5, nesting level increased to 6
   ```cpp
   if (dict_code == -2) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:288:** +1, nesting level increased to 3
   ```cpp
   } else {
 ^
   ```
   **be/src/olap/comparison_predicate.h:292:** +1, nesting level increased to 2
   ```cpp
   } else {
 ^
   ```
   **be/src/olap/comparison_predicate.h:302:** +1, nesting level increased to 1
   ```cpp
   } else {
 ^
   ```
   **be/src/olap/comparison_predicate.h:303:** +2, including nesting penalty of 
1, nesting level increased to 2
   ```cpp
   if (column.is_column_dictionary()) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:304:** +3, including nesting penalty of 
2, nesting level increased to 3
   ```cpp
   if constexpr (std::is_same_v) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:308:** +4, including nesting penalty of 
3, nesting level increased to 4
   ```cpp
   do {
   ^
   ```
   **be/src/olap/comparison_predicate.h:309:** +5, including nesting penalty of 
4, nesting level increased to 5
   ```cpp
   if constexpr (PT == PredicateType::EQ) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:310:** +6, including nesting penalty of 
5, nesting level increased to 6
   ```cpp
   if (dict_code == -2) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:319:** +1, nesting level increased to 3
   ```cpp
   } else {
 ^
   ```
   **be/src/olap/comparison_predicate.h:323:** +1, nesting level increased to 2
   ```cpp
   } else {
 ^
   ```
   **be/src/olap/comparison_predicate.h:335:** +1, including nesting penalty of 
0, nesting level increased to 1
   ```cpp
   if (_opposite) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:336:** +2, including nesting penalty of 
1, nesting level increased to 2
   ```cpp
   for (uint16_t i = 0; i < size; i++) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:341:** +1, including nesting penalty of 
0, nesting level increased to 1
   ```cpp
   if (_can_ignore()) {
   ^
   ```
   **be/src/olap/comparison_predicate.h:342:** +2, including nesting penalty of 
1, nesting 

Re: [PR] [fix](cloud tvf) Fix tvf query run in cloud multi cluster [doris]

2024-07-24 Thread via GitHub


deardeng commented on PR #37157:
URL: https://github.com/apache/doris/pull/37157#issuecomment-2247152254

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](group commit) group commit show error url [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38292:
URL: https://github.com/apache/doris/pull/38292#issuecomment-2247154398

   
   
   ClickBench: Total hot run time: 30.77 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 1f6c62f606fbcfd0ac0e2c51986f1231765fa2d6, 
data reload: false
   
   query1   0.040.030.03
   query2   0.080.040.04
   query3   0.220.040.05
   query4   1.680.070.07
   query5   0.610.470.48
   query6   1.130.720.72
   query7   0.020.010.01
   query8   0.060.040.04
   query9   0.550.500.49
   query10  0.540.560.54
   query11  0.150.120.11
   query12  0.140.110.13
   query13  0.600.580.58
   query14  0.760.780.76
   query15  0.850.810.82
   query16  0.350.350.37
   query17  0.980.980.95
   query18  0.220.210.22
   query19  1.751.721.66
   query20  0.010.010.01
   query21  15.44   0.790.67
   query22  3.967.122.26
   query23  18.29   1.351.26
   query24  2.160.230.22
   query25  0.150.100.09
   query26  0.300.210.21
   query27  0.460.220.23
   query28  13.26   1.021.00
   query29  12.56   3.283.26
   query30  0.250.060.05
   query31  2.870.380.37
   query32  3.290.480.48
   query33  2.882.882.92
   query34  17.02   4.354.35
   query35  4.424.424.39
   query36  0.650.460.48
   query37  0.190.160.16
   query38  0.160.150.15
   query39  0.040.040.04
   query40  0.160.120.12
   query41  0.090.050.05
   query42  0.060.050.04
   query43  0.050.040.04
   Total cold run time: 109.45 s
   Total hot run time: 30.77 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](Row store) get the correct value of `row_store_page_size` after FE restarts [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38236:
URL: https://github.com/apache/doris/pull/38236#issuecomment-2247153285

   
   
   TPC-H: Total hot run time: 39743 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 26e18ac623c12ffa189e17ac6897901ce8faa0c0, 
data reload: false
   
   -- Round 1 --
   q1   17650   432742804280
   q2   2023188 189 188
   q3   10401   116911321132
   q4   10187   815 708 708
   q5   7533279426472647
   q6   218 139 139 139
   q7   952 598 596 596
   q8   9216208120812081
   q9   8777658365596559
   q10  8789377837833778
   q11  491 235 237 235
   q12  397 227 222 222
   q13  17873   295529882955
   q14  273 234 234 234
   q15  523 481 497 481
   q16  521 368 375 368
   q17  965 641 763 641
   q18  8064759273587358
   q19  7370145613111311
   q20  661 329 326 326
   q21  4915321933093219
   q22  351 286 285 285
   Total cold run time: 118150 ms
   Total hot run time: 39743 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   4431423142384231
   q2   376 277 257 257
   q3   2964288429762884
   q4   1972167017621670
   q5   5647553054335433
   q6   229 138 130 130
   q7   2183184218381838
   q8   3246342534533425
   q9   8800882588978825
   q10  4138387837083708
   q11  592 516 502 502
   q12  807 661 618 618
   q13  16907   316731813167
   q14  321 301 290 290
   q15  518 500 478 478
   q16  505 436 437 436
   q17  1819152015151515
   q18  8131805778287828
   q19  1736166615751575
   q20  2936191118551855
   q21  8157489446294629
   q22  607 501 537 501
   Total cold run time: 77022 ms
   Total hot run time: 55795 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](profile) Provide more info for schedule time [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38290:
URL: https://github.com/apache/doris/pull/38290#issuecomment-2247153472

   
   
   TPC-DS: Total hot run time: 175791 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 6e91ded334a0537d243ba5aad67977b275d9abde, 
data reload: false
   
   query1   927 385 379 379
   query2   6456187618311831
   query3   6647211 219 211
   query4   25874   17556   17404   17404
   query5   3777496 498 496
   query6   276 185 183 183
   query7   4609296 289 289
   query8   236 187 193 187
   query9   8497242123982398
   query10  445 306 272 272
   query11  12167   10221   10169   10169
   query12  126 86  85  85
   query13  1638382 368 368
   query14  10365   768679027686
   query15  214 168 171 168
   query16  7764515 485 485
   query17  1607596 555 555
   query18  1986289 289 289
   query19  196 155 161 155
   query20  91  85  83  83
   query21  205 142 134 134
   query22  4571403240014001
   query23  34351   34048   33708   33708
   query24  11036   303530473035
   query25  648 425 416 416
   query26  1139172 171 171
   query27  2309286 297 286
   query28  6987209320902090
   query29  942 698 676 676
   query30  259 157 164 157
   query31  981 788 763 763
   query32  97  60  57  57
   query33  780 377 368 368
   query34  914 511 515 511
   query35  912 777 789 777
   query36  11651007981 981
   query37  150 91  92  91
   query38  3082285628812856
   query39  960 842 810 810
   query40  212 126 127 126
   query41  51  46  50  46
   query42  118 107 104 104
   query43  508 462 457 457
   query44  1177733 749 733
   query45  203 165 165 165
   query46  1110758 839 758
   query47  1820177317431743
   query48  377 300 287 287
   query49  860 408 456 408
   query50  790 414 401 401
   query51  6684667366846673
   query52  107 96  92  92
   query53  370 300 298 298
   query54  868 455 455 455
   query55  76  79  75  75
   query56  303 273 288 273
   query57  1143104110581041
   query58  252 256 270 256
   query59  2884253525902535
   query60  304 280 291 280
   query61  95  93  98  93
   query62  795 644 662 644
   query63  321 298 299 298
   query64  9511223017431743
   query65  3179313031263126
   query66  813 329 332 329
   query67  15752   15051   15206   15051
   query68  8637558 559 558
   query69  791 441 371 371
   query70  1431116710841084
   query71  530 289 293 289
   query72  9096601557365736
   query73  2200332 330 330
   query74  6257569756745674
   query75  5563269727382697
   query76  5143926 1031926
   query77  794 318 316 316
   query78  983310780   92939293
   query79  11063   557 542 542
   query80  865 491 493 491
   query81  578 222 220 220
   query82  640 149 142 142
   query83  395 173 170 170
   query84  285 87  86  86
   query85  727 301 306 301
   query86  370 327 321 321
   query87  3352314731503147
   query88  4149237623662366
   query89  507 395 398 395
   query90  2152202 201 201
   query91  133 103 104 103
   query92  68  50  52  50
   query93  5637536 539 536
   query94  1345344 284 284
   query95  433 328 326 326
   query96  609 278 276 276
   query97  3203305930353035
   query98  228 212 196 196
   query99  1560126212581258
   Total cold run time: 304643 ms
   Total hot run time: 175791 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th

Re: [PR] [Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #37257:
URL: https://github.com/apache/doris/pull/37257#issuecomment-2247159711

   TeamCity be ut coverage result:
Function Coverage: 36.40% (9243/25396) 
Line Coverage: 27.97% (75534/270071)
Region Coverage: 26.79% (38830/144960)
Branch Coverage: 23.53% (19712/83764)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/8f3272d573f85eac5bf4ac38c04a24caaa92e5bf_8f3272d573f85eac5bf4ac38c04a24caaa92e5bf/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [spill](regression) add regression test cases for spill [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #37319:
URL: https://github.com/apache/doris/pull/37319#issuecomment-2247161323

   
   
   TPC-H: Total hot run time: 39632 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 8790744140c53fe64a3ff8dc9f6a4c533ddbc5e4, 
data reload: false
   
   -- Round 1 --
   q1   17661   457942524252
   q2   2015189 194 189
   q3   10504   123810691069
   q4   10188   708 859 708
   q5   7563271926952695
   q6   225 145 140 140
   q7   959 596 597 596
   q8   9218209620962096
   q9   8732658665776577
   q10  8857382137603760
   q11  453 232 228 228
   q12  417 214 218 214
   q13  17779   294929602949
   q14  267 242 229 229
   q15  532 488 489 488
   q16  511 383 377 377
   q17  997 612 703 612
   q18  8168747774297429
   q19  6698130113871301
   q20  684 317 330 317
   q21  4987312632733126
   q22  359 287 280 280
   Total cold run time: 117774 ms
   Total hot run time: 39632 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   4441429642654265
   q2   361 272 262 262
   q3   2972292629312926
   q4   1971172616971697
   q5   5622553554995499
   q6   226 138 140 138
   q7   2252187218201820
   q8   3315345134273427
   q9   8786887088578857
   q10  4102382938653829
   q11  583 487 498 487
   q12  816 624 643 624
   q13  16526   317631793176
   q14  331 282 290 282
   q15  522 513 511 511
   q16  495 451 439 439
   q17  1815153515121512
   q18  8085789880007898
   q19  1773158515571557
   q20  2762191218271827
   q21  7826478246584658
   q22  606 491 491 491
   Total cold run time: 76188 ms
   Total hot run time: 56182 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Feature] support data mask in doris internal auth [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2247162335

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to 
[doris-website](https://github.com/apache/doris-website).
   See [Doris 
Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [Feature] support data mask in doris internal auth [doris]

2024-07-24 Thread via GitHub


qzsee opened a new pull request, #38301:
URL: https://github.com/apache/doris/pull/38301

   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](profile) Provide more info for schedule time [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38290:
URL: https://github.com/apache/doris/pull/38290#issuecomment-2247164158

   
   
   ClickBench: Total hot run time: 31.03 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 6e91ded334a0537d243ba5aad67977b275d9abde, 
data reload: false
   
   query1   0.040.030.03
   query2   0.090.040.04
   query3   0.230.070.06
   query4   1.640.090.10
   query5   0.520.490.49
   query6   1.130.740.73
   query7   0.020.010.01
   query8   0.050.040.04
   query9   0.550.480.50
   query10  0.550.530.54
   query11  0.160.120.11
   query12  0.150.130.12
   query13  0.620.590.59
   query14  0.780.790.77
   query15  0.860.820.82
   query16  0.360.370.37
   query17  0.960.951.04
   query18  0.230.210.22
   query19  1.811.701.72
   query20  0.020.010.01
   query21  15.67   0.770.64
   query22  4.316.852.27
   query23  18.28   1.421.26
   query24  2.130.220.22
   query25  0.150.090.09
   query26  0.290.220.22
   query27  0.450.230.25
   query28  13.35   1.051.00
   query29  12.62   3.303.30
   query30  0.240.060.05
   query31  2.880.400.39
   query32  3.240.480.46
   query33  2.922.892.93
   query34  17.09   4.364.36
   query35  4.414.464.42
   query36  0.650.490.49
   query37  0.200.150.16
   query38  0.160.150.15
   query39  0.050.040.04
   query40  0.150.120.14
   query41  0.090.040.05
   query42  0.060.050.05
   query43  0.040.050.04
   Total cold run time: 110.2 s
   Total hot run time: 31.03 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](group commit) group commit show error url [doris]

2024-07-24 Thread via GitHub


mymeiyi commented on PR #38292:
URL: https://github.com/apache/doris/pull/38292#issuecomment-2247166611

   run cloud_p1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix])(function) add function regexp_extract_or_null [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38296:
URL: https://github.com/apache/doris/pull/38296#issuecomment-2247167371

   
   
   TPC-DS: Total hot run time: 172205 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 8e4da4816b00e4ad44cd101cc9a7868c49671ed7, 
data reload: false
   
   query1   922 371 360 360
   query2   6458184817791779
   query3   6658206 219 206
   query4   26394   17596   17162   17162
   query5   4182477 483 477
   query6   268 174 181 174
   query7   4582299 285 285
   query8   231 182 186 182
   query9   8518243024162416
   query10  451 295 272 272
   query11  12533   997998649864
   query12  135 87  82  82
   query13  1635381 386 381
   query14  10243   749676247496
   query15  220 165 168 165
   query16  7891501 479 479
   query17  1574558 515 515
   query18  1992273 271 271
   query19  198 142 151 142
   query20  91  82  89  82
   query21  208 127 125 125
   query22  4361394342353943
   query23  33717   32894   33092   32894
   query24  12040   288028602860
   query25  668 363 384 363
   query26  1763152 151 151
   query27  3025268 273 268
   query28  7754203020012001
   query29  1126624 614 614
   query30  286 145 148 145
   query31  959 719 729 719
   query32  94  55  56  55
   query33  762 328 323 323
   query34  916 475 474 474
   query35  847 734 714 714
   query36  1071966 938 938
   query37  283 80  80  80
   query38  2861273227432732
   query39  856 798 803 798
   query40  270 116 123 116
   query41  47  44  46  44
   query42  119 98  101 98
   query43  490 460 478 460
   query44  1261731 724 724
   query45  188 193 159 159
   query46  1098718 728 718
   query47  1848175517701755
   query48  379 287 291 287
   query49  1176414 412 412
   query50  795 400 395 395
   query51  6697657866086578
   query52  108 90  95  90
   query53  355 299 290 290
   query54  974 450 445 445
   query55  75  74  74  74
   query56  287 263 265 263
   query57  1182105210811052
   query58  269 246 250 246
   query59  2795269125452545
   query60  308 286 285 285
   query61  94  95  95  95
   query62  829 647 665 647
   query63  322 289 285 285
   query64  10451   220016781678
   query65  3143310431093104
   query66  1376355 360 355
   query67  15643   15143   14873   14873
   query68  4600548 538 538
   query69  462 328 336 328
   query70  1148115611431143
   query71  402 277 293 277
   query72  7170620058785878
   query73  744 331 326 326
   query74  6062563656315631
   query75  3398261627012616
   query76  28111003935 935
   query77  458 312 308 308
   query78  9722895389138913
   query79  2717534 532 532
   query80  2080482 488 482
   query81  574 224 219 219
   query82  907 136 136 136
   query83  273 169 170 169
   query84  274 95  85  85
   query85  2095314 365 314
   query86  500 322 297 297
   query87  3288305030723050
   query88  3941238523802380
   query89  483 381 387 381
   query90  1851200 199 199
   query91  129 98  102 98
   query92  58  51  51  51
   query93  3041520 510 510
   query94  1283300 288 288
   query95  405 319 315 315
   query96  597 276 270 270
   query97  3206302230243022
   query98  216 198 192 192
   query99  1784126212731262
   Total cold run time: 288031 ms
   Total hot run time: 172205 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s

Re: [PR] [feature](workloadgroup) use slot num to control memory distribution among one workload group [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38237:
URL: https://github.com/apache/doris/pull/38237#issuecomment-2247168850

   
   
   TPC-DS: Total hot run time: 173897 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 94fe659308739e065aa346c1b461ad59cc8e8b36, 
data reload: false
   
   query1   917 374 367 367
   query2   6419184818071807
   query3   6657204 219 204
   query4   28616   17354   17308   17308
   query5   3752480 483 480
   query6   283 166 160 160
   query7   4581294 285 285
   query8   231 190 190 190
   query9   8638240423812381
   query10  418 274 278 274
   query11  12501   10099   10053   10053
   query12  115 88  85  85
   query13  1644373 357 357
   query14  10170   751583197515
   query15  220 170 164 164
   query16  7404468 471 468
   query17  1140576 518 518
   query18  1821276 279 276
   query19  195 147 154 147
   query20  91  82  85  82
   query21  204 124 123 123
   query22  4347399541553995
   query23  34024   33744   33785   33744
   query24  11032   294529062906
   query25  588 382 418 382
   query26  706 152 153 152
   query27  2412280 283 280
   query28  6075208820852085
   query29  912 628 663 628
   query30  259 160 150 150
   query31  950 782 736 736
   query32  95  53  58  53
   query33  664 335 346 335
   query34  909 511 513 511
   query35  922 764 788 764
   query36  11621007978 978
   query37  149 88  87  87
   query38  3016291327892789
   query39  902 885 856 856
   query40  210 121 127 121
   query41  47  44  45  44
   query42  115 104 101 101
   query43  507 457 473 457
   query44  1083730 735 730
   query45  191 165 163 163
   query46  1101715 733 715
   query47  1825177017871770
   query48  379 296 292 292
   query49  856 444 428 428
   query50  788 396 409 396
   query51  6859668364986498
   query52  111 93  99  93
   query53  365 300 300 300
   query54  888 457 459 457
   query55  82  75  76  75
   query56  309 295 290 290
   query57  1133106210611061
   query58  264 260 278 260
   query59  2811252527642525
   query60  322 300 307 300
   query61  121 119 114 114
   query62  792 646 669 646
   query63  327 292 294 292
   query64  9251231417081708
   query65  3204315031243124
   query66  777 337 341 337
   query67  15632   14915   14879   14879
   query68  6269562 566 562
   query69  764 467 384 384
   query70  1138110411651104
   query71  533 296 288 288
   query72  9040551057685510
   query73  829 329 324 324
   query74  6079571457685714
   query75  4733267027192670
   query76  4373930 976 930
   query77  758 308 311 308
   query78  9725917689978997
   query79  10528   541 525 525
   query80  1085501 475 475
   query81  576 223 218 218
   query82  850 171 130 130
   query83  329 166 167 166
   query84  279 86  86  86
   query85  1398317 301 301
   query86  452 301 327 301
   query87  3299314431273127
   query88  5001237923882379
   query89  550 389 390 389
   query90  1994192 195 192
   query91  130 100 105 100
   query92  71  53  49  49
   query93  7160523 518 518
   query94  1286287 321 287
   query95  413 316 329 316
   query96  620 283 275 275
   query97  3241296430262964
   query98  220 202 206 202
   query99  1579125512831255
   Total cold run time: 300793 ms
   Total hot run time: 173897 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to t

Re: [PR] [Improvement](profile) add catalog info in profile [doris]

2024-07-24 Thread via GitHub


Yulei-Yang commented on PR #38302:
URL: https://github.com/apache/doris/pull/38302#issuecomment-2247170313

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [Improvement](profile) add catalog info in profile [doris]

2024-07-24 Thread via GitHub


Yulei-Yang opened a new pull request, #38302:
URL: https://github.com/apache/doris/pull/38302

   bp #38283
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](profile) add catalog info in profile [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38302:
URL: https://github.com/apache/doris/pull/38302#issuecomment-2247169862

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to 
[doris-website](https://github.com/apache/doris-website).
   See [Doris 
Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](insert-overwrite) Fix insert overwrite auto detect transaction safe [doris]

2024-07-24 Thread via GitHub


zclllyybb commented on code in PR #38103:
URL: https://github.com/apache/doris/pull/38103#discussion_r1689333607


##
fe/fe-core/src/main/java/org/apache/doris/insertoverwrite/InsertOverwriteManager.java:
##
@@ -107,44 +107,82 @@ public void registerTaskInGroup(long groupId, long 
taskId) {
 taskGroups.get(groupId).add(taskId);
 }
 
-public List tryReplacePartitionIds(long groupId, List 
oldPartitionIds) {
+/**
+ * this func should in lock scope of getLock(groupId)
+ *
+ * @param newIds if have replaced, replace with new. otherwise itself.
+ * @return
+ */
+public boolean tryReplacePartitionIds(long groupId, List 
oldPartitionIds, List newIds) {
 Map relations = partitionPairs.get(groupId);
-List newIds = new ArrayList();
-for (Long id : oldPartitionIds) {
+boolean needReplace = false;
+for (int i = 0; i < oldPartitionIds.size(); i++) {
+long id = oldPartitionIds.get(i);
 if (relations.containsKey(id)) {
 // if we replaced it. then return new one.
 newIds.add(relations.get(id));
 } else {
-// otherwise itself. we will deal it soon.
-newIds.add(id);
+newIds.add(oldPartitionIds.get(i));

Review Comment:
   well they're same. will fix.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](function) optimization for substr with ascii string [doris]

2024-07-24 Thread via GitHub


yanyxin commented on PR #29799:
URL: https://github.com/apache/doris/pull/29799#issuecomment-2247174047

   @BiteThet Will the modifications be synchronized to 2.1.5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](Row store) get the correct value of `row_store_page_size` after FE restarts [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38236:
URL: https://github.com/apache/doris/pull/38236#issuecomment-2247175447

   
   
   TPC-DS: Total hot run time: 174056 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 26e18ac623c12ffa189e17ac6897901ce8faa0c0, 
data reload: false
   
   query1   908 368 378 368
   query2   6432196318481848
   query3   6641205 215 205
   query4   26767   17694   17447   17447
   query5   3811491 503 491
   query6   264 176 169 169
   query7   4594287 277 277
   query8   227 190 184 184
   query9   8534240023902390
   query10  432 318 280 280
   query11  11798   10086   10112   10086
   query12  111 81  80  80
   query13  1635362 357 357
   query14  9445754276987542
   query15  217 175 173 173
   query16  7738499 473 473
   query17  1364538 535 535
   query18  1907276 272 272
   query19  199 151 155 151
   query20  92  86  82  82
   query21  211 127 128 127
   query22  4499431039983998
   query23  34172   33622   33669   33622
   query24  10877   293828722872
   query25  605 403 398 398
   query26  705 149 148 148
   query27  2252276 289 276
   query28  6193208320552055
   query29  867 634 633 633
   query30  268 155 153 153
   query31  986 779 770 770
   query32  91  53  62  53
   query33  640 356 347 347
   query34  920 497 515 497
   query35  865 733 763 733
   query36  1141978 948 948
   query37  137 81  84  81
   query38  2952283027522752
   query39  877 813 831 813
   query40  200 121 117 117
   query41  46  43  45  43
   query42  112 99  97  97
   query43  501 470 463 463
   query44  1112722 723 722
   query45  190 159 160 159
   query46  1101728 756 728
   query47  1844179617981796
   query48  353 291 296 291
   query49  856 405 436 405
   query50  772 381 389 381
   query51  6776667867636678
   query52  98  95  91  91
   query53  358 289 285 285
   query54  861 456 445 445
   query55  76  74  76  74
   query56  285 265 268 265
   query57  1146104410441044
   query58  237 252 273 252
   query59  2938267025592559
   query60  322 273 265 265
   query61  98  95  95  95
   query62  798 640 651 640
   query63  321 285 287 285
   query64  9249232417851785
   query65  3167310731093107
   query66  740 321 319 319
   query67  15583   15091   14853   14853
   query68  4622549 539 539
   query69  730 426 349 349
   query70  1154115810611061
   query71  411 278 276 276
   query72  7692569654845484
   query73  775 342 333 333
   query74  6432600357215721
   query75  3448269327162693
   query76  2684953 933 933
   query77  636 312 311 311
   query78  9653920210144   9202
   query79  2993517 536 517
   query80  1360464 467 464
   query81  583 217 218 217
   query82  736 137 132 132
   query83  332 164 168 164
   query84  271 86  84  84
   query85  1666321 300 300
   query86  473 325 321 321
   query87  3268310631173106
   query88  4060237723852377
   query89  472 395 382 382
   query90  1729193 196 193
   query91  132 102 101 101
   query92  58  48  49  48
   query93  2387512 519 512
   query94  1132278 304 278
   query95  416 317 315 315
   query96  600 274 276 274
   query97  3170302430023002
   query98  230 199 205 199
   query99  1637123812701238
   Total cold run time: 278456 ms
   Total hot run time: 174056 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the

Re: [PR] [fix])(function) add function regexp_extract_or_null [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38296:
URL: https://github.com/apache/doris/pull/38296#issuecomment-2247178680

   
   
   ClickBench: Total hot run time: 30.56 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 8e4da4816b00e4ad44cd101cc9a7868c49671ed7, 
data reload: false
   
   query1   0.040.040.03
   query2   0.090.040.03
   query3   0.220.050.05
   query4   1.680.070.07
   query5   0.500.500.47
   query6   1.120.720.72
   query7   0.020.020.01
   query8   0.040.040.04
   query9   0.540.500.50
   query10  0.560.560.53
   query11  0.150.110.12
   query12  0.140.120.12
   query13  0.600.580.58
   query14  0.750.780.77
   query15  0.840.800.81
   query16  0.350.360.35
   query17  0.961.001.00
   query18  0.220.220.21
   query19  1.821.751.76
   query20  0.010.010.01
   query21  15.41   0.780.66
   query22  4.287.221.89
   query23  18.33   1.391.22
   query24  2.120.230.22
   query25  0.140.080.09
   query26  0.310.210.21
   query27  0.450.230.23
   query28  13.26   1.010.99
   query29  12.56   3.323.33
   query30  0.260.060.06
   query31  2.880.380.39
   query32  3.280.490.47
   query33  2.892.922.87
   query34  17.30   4.384.42
   query35  4.474.424.38
   query36  0.660.490.51
   query37  0.180.150.15
   query38  0.150.160.14
   query39  0.040.030.04
   query40  0.150.130.12
   query41  0.080.040.05
   query42  0.060.040.04
   query43  0.040.040.04
   Total cold run time: 109.95 s
   Total hot run time: 30.56 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](workloadgroup) use slot num to control memory distribution among one workload group [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38237:
URL: https://github.com/apache/doris/pull/38237#issuecomment-2247180109

   
   
   ClickBench: Total hot run time: 30.8 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 94fe659308739e065aa346c1b461ad59cc8e8b36, 
data reload: false
   
   query1   0.040.040.03
   query2   0.070.040.04
   query3   0.220.050.04
   query4   1.670.070.09
   query5   0.490.490.47
   query6   1.130.740.72
   query7   0.020.010.01
   query8   0.050.040.04
   query9   0.560.500.51
   query10  0.540.550.56
   query11  0.140.110.12
   query12  0.150.130.13
   query13  0.590.580.57
   query14  0.760.790.77
   query15  0.850.830.80
   query16  0.370.360.37
   query17  1.040.970.98
   query18  0.240.220.21
   query19  1.771.681.71
   query20  0.010.010.01
   query21  15.40   0.780.65
   query22  3.187.852.06
   query23  18.29   1.381.29
   query24  2.080.230.23
   query25  0.170.090.09
   query26  0.280.200.21
   query27  0.450.230.23
   query28  13.26   1.021.02
   query29  12.62   3.343.31
   query30  0.260.060.06
   query31  2.860.410.39
   query32  3.270.470.46
   query33  2.892.962.91
   query34  16.99   4.394.36
   query35  4.414.414.43
   query36  0.640.460.47
   query37  0.200.150.17
   query38  0.150.150.15
   query39  0.040.040.03
   query40  0.140.120.13
   query41  0.100.050.06
   query42  0.060.050.06
   query43  0.060.050.04
   Total cold run time: 108.51 s
   Total hot run time: 30.8 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [spill](regression) add regression test cases for spill [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #37319:
URL: https://github.com/apache/doris/pull/37319#issuecomment-2247183964

   
   
   TPC-DS: Total hot run time: 174786 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 8790744140c53fe64a3ff8dc9f6a4c533ddbc5e4, 
data reload: false
   
   query1   896 368 364 364
   query2   6440181217391739
   query3   6637206 219 206
   query4   28170   17594   17395   17395
   query5   3765475 475 475
   query6   291 176 157 157
   query7   4573287 286 286
   query8   228 187 193 187
   query9   8459238223732373
   query10  428 292 261 261
   query11  12251   10157   10165   10157
   query12  112 83  88  83
   query13  1632361 359 359
   query14  9212776879397768
   query15  239 162 163 162
   query16  7586473 474 473
   query17  1292569 519 519
   query18  1356271 271 271
   query19  197 148 147 147
   query20  85  81  81  81
   query21  206 128 131 128
   query22  4564420542724205
   query23  34039   33761   33535   33535
   query24  11229   296829982968
   query25  599 414 380 380
   query26  1220148 147 147
   query27  2746278 282 278
   query28  7953208620582058
   query29  883 652 613 613
   query30  252 155 151 151
   query31  986 770 753 753
   query32  92  51  56  51
   query33  732 336 318 318
   query34  1010505 498 498
   query35  911 764 783 764
   query36  1159961 954 954
   query37  145 85  86  85
   query38  2952283428192819
   query39  892 805 807 805
   query40  207 115 118 115
   query41  46  42  47  42
   query42  123 102 99  99
   query43  500 465 466 465
   query44  1218721 717 717
   query45  198 159 158 158
   query46  1084766 748 748
   query47  1859176217901762
   query48  371 280 292 280
   query49  831 416 409 409
   query50  780 384 386 384
   query51  6791656867056568
   query52  99  92  91  91
   query53  356 282 289 282
   query54  884 452 449 449
   query55  72  73  73  73
   query56  290 273 281 273
   query57  1112104310491043
   query58  276 251 259 251
   query59  2788265626062606
   query60  335 288 302 288
   query61  119 111 114 111
   query62  790 642 661 642
   query63  314 284 283 283
   query64  9553229417671767
   query65  3217310231043102
   query66  766 340 333 333
   query67  15459   15139   15090   15090
   query68  4631553 547 547
   query69  648 438 365 365
   query70  1210115311711153
   query71  472 287 284 284
   query72  8073580460275804
   query73  743 324 324 324
   query74  6288566956945669
   query75  4257268227062682
   query76  3292919 986 919
   query77  690 312 311 311
   query78  9581921991469146
   query79  6904515 518 515
   query80  2827478 474 474
   query81  592 227 236 227
   query82  1054136 133 133
   query83  309 164 169 164
   query84  277 86  84  84
   query85  2152311 294 294
   query86  443 312 322 312
   query87  3324309831383098
   query88  5436238123912381
   query89  506 377 370 370
   query90  2009194 189 189
   query91  127 96  101 96
   query92  58  50  46  46
   query93  5825527 519 519
   query94  1410308 262 262
   query95  409 313 307 307
   query96  616 268 269 268
   query97  3216299630482996
   query98  235 200 196 196
   query99  1531124712641247
   Total cold run time: 296002 ms
   Total hot run time: 174786 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the

Re: [PR] [Fix](multi-catalog) Fix some undefined behaviors. [doris]

2024-07-24 Thread via GitHub


morningman merged PR #38274:
URL: https://github.com/apache/doris/pull/38274


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](Row store) get the correct value of `row_store_page_size` after FE restarts [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38236:
URL: https://github.com/apache/doris/pull/38236#issuecomment-2247185958

   
   
   ClickBench: Total hot run time: 30.53 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 26e18ac623c12ffa189e17ac6897901ce8faa0c0, 
data reload: false
   
   query1   0.040.030.03
   query2   0.070.040.04
   query3   0.220.040.05
   query4   1.680.080.08
   query5   0.500.490.48
   query6   1.130.730.73
   query7   0.020.020.01
   query8   0.050.040.05
   query9   0.550.500.48
   query10  0.540.540.54
   query11  0.150.120.11
   query12  0.140.130.12
   query13  0.590.600.59
   query14  0.750.770.78
   query15  0.860.810.82
   query16  0.360.360.37
   query17  0.950.940.95
   query18  0.230.230.22
   query19  1.811.681.69
   query20  0.010.010.01
   query21  15.42   0.740.65
   query22  4.666.961.99
   query23  18.29   1.431.26
   query24  2.090.220.22
   query25  0.150.100.08
   query26  0.290.210.21
   query27  0.450.220.23
   query28  13.32   1.021.00
   query29  12.60   3.323.31
   query30  0.250.050.06
   query31  2.880.390.39
   query32  3.260.490.46
   query33  2.892.932.86
   query34  16.96   4.374.33
   query35  4.434.424.38
   query36  0.650.500.46
   query37  0.190.160.15
   query38  0.160.150.15
   query39  0.040.030.04
   query40  0.140.120.13
   query41  0.090.050.04
   query42  0.060.050.05
   query43  0.050.040.04
   Total cold run time: 109.97 s
   Total hot run time: 30.53 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



(doris) branch branch-2.1 updated: [Fix](multi-catalog) Fix some undefined behaviors. (#38274)

2024-07-24 Thread morningman
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new ef00dad680d [Fix](multi-catalog) Fix some undefined behaviors. (#38274)
ef00dad680d is described below

commit ef00dad680d1ec6da5e482c4c1a9503046b0ae2f
Author: Qi Chen 
AuthorDate: Wed Jul 24 16:14:34 2024 +0800

[Fix](multi-catalog) Fix some undefined behaviors. (#38274)

## Proposed changes

backport #37845
---
 be/src/vec/exec/format/orc/vorc_reader.cpp |  4 ++--
 .../exec/format/parquet/parquet_column_convert.h   | 26 --
 2 files changed, 16 insertions(+), 14 deletions(-)

diff --git a/be/src/vec/exec/format/orc/vorc_reader.cpp 
b/be/src/vec/exec/format/orc/vorc_reader.cpp
index 16909f0023a..49430471c9d 100644
--- a/be/src/vec/exec/format/orc/vorc_reader.cpp
+++ b/be/src/vec/exec/format/orc/vorc_reader.cpp
@@ -1586,7 +1586,7 @@ Status OrcReader::get_next_block_impl(Block* block, 
size_t* read_rows, bool* eof
 _decimal_scale_params_index = 0;
 try {
 rr = _row_reader->nextBatch(*_batch, block);
-if (rr == 0) {
+if (rr == 0 || _batch->numElements == 0) {
 *eof = true;
 *read_rows = 0;
 return Status::OK();
@@ -1656,7 +1656,7 @@ Status OrcReader::get_next_block_impl(Block* block, 
size_t* read_rows, bool* eof
 _decimal_scale_params_index = 0;
 try {
 rr = _row_reader->nextBatch(*_batch, block);
-if (rr == 0) {
+if (rr == 0 || _batch->numElements == 0) {
 *eof = true;
 *read_rows = 0;
 return Status::OK();
diff --git a/be/src/vec/exec/format/parquet/parquet_column_convert.h 
b/be/src/vec/exec/format/parquet/parquet_column_convert.h
index bc6bc232327..551bf7e14ed 100644
--- a/be/src/vec/exec/format/parquet/parquet_column_convert.h
+++ b/be/src/vec/exec/format/parquet/parquet_column_convert.h
@@ -408,18 +408,20 @@ class StringToDecimal : public PhysicalToLogicalConverter 
{
 // When Decimal in parquet is stored in byte arrays, binary and 
fixed,
 // the unscaled number must be encoded as two's complement using 
big-endian byte order.
 ValueCopyType value = 0;
-memcpy(reinterpret_cast(&value), buf + offset[i - 1], len);
-value = BitUtil::big_endian_to_host(value);
-value = value >> ((sizeof(value) - len) * 8);
-if constexpr (ScaleType == DecimalScaleParams::SCALE_UP) {
-value *= scale_params.scale_factor;
-} else if constexpr (ScaleType == DecimalScaleParams::SCALE_DOWN) {
-value /= scale_params.scale_factor;
-} else if constexpr (ScaleType == DecimalScaleParams::NO_SCALE) {
-// do nothing
-} else {
-LOG(FATAL) << "__builtin_unreachable";
-__builtin_unreachable();
+if (len > 0) {
+memcpy(reinterpret_cast(&value), buf + offset[i - 1], 
len);
+value = BitUtil::big_endian_to_host(value);
+value = value >> ((sizeof(value) - len) * 8);
+if constexpr (ScaleType == DecimalScaleParams::SCALE_UP) {
+value *= scale_params.scale_factor;
+} else if constexpr (ScaleType == 
DecimalScaleParams::SCALE_DOWN) {
+value /= scale_params.scale_factor;
+} else if constexpr (ScaleType == 
DecimalScaleParams::NO_SCALE) {
+// do nothing
+} else {
+LOG(FATAL) << "__builtin_unreachable";
+__builtin_unreachable();
+}
 }
 auto& v = reinterpret_cast(data[start_idx + i]);
 v = (DecimalType)value;


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](profile) add catalog info in profile [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38298:
URL: https://github.com/apache/doris/pull/38298#issuecomment-2247190003

   
   
   TPC-H: Total hot run time: 49948 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 1d785a78e1f0fb948f448ab50eac873f7152550e, 
data reload: false
   
   -- Round 1 --
   q1   17641   436443534353
   q2   2094156 144 144
   q3   10469   191919501919
   q4   10327   123613251236
   q5   8404386238423842
   q6   235 125 124 124
   q7   2045162416081608
   q8   9289270926752675
   q9   10517   10255   10313   10255
   q10  8624350434963496
   q11  418 250 242 242
   q12  472 307 304 304
   q13  18334   398740073987
   q14  356 324 320 320
   q15  494 460 454 454
   q16  683 569 568 568
   q17  1127957 987 957
   q18  7272705369836983
   q19  1787164315971597
   q20  538 322 311 311
   q21  4523422740964096
   q22  539 477 484 477
   Total cold run time: 116188 ms
   Total hot run time: 49948 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   4427446044204420
   q2   330 236 234 234
   q3   4204417241394139
   q4   2749274927452745
   q5   7126708871177088
   q6   241 122 118 118
   q7   3163282028472820
   q8   4368445744794457
   q9   16784   16884   16706   16706
   q10  4257422942984229
   q11  751 709 689 689
   q12  1011852 854 852
   q13  7046373337183718
   q14  448 424 431 424
   q15  491 456 456 456
   q16  734 687 681 681
   q17  3860383437933793
   q18  8710870287288702
   q19  1721167916071607
   q20  2363212520972097
   q21  8548842185478421
   q22  1041967 961 961
   Total cold run time: 84373 ms
   Total hot run time: 79357 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



(doris) branch master updated: [Bug](agg) fix collect_set function core dump without arena pool (#38234)

2024-07-24 Thread lihaopeng
This is an automated email from the ASF dual-hosted git repository.

lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new c16ea419b27 [Bug](agg) fix collect_set function core dump without 
arena pool (#38234)
c16ea419b27 is described below

commit c16ea419b27095a37d54005815f441512ef5b782
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Wed Jul 24 16:18:07 2024 +0800

[Bug](agg) fix collect_set function core dump without arena pool (#38234)

before the add_range_single_place pass nullptr as arena object,
but collect_set function need save data in arena, so will core dump
without arena pool.
---
 be/src/pipeline/exec/analytic_source_operator.cpp| 4 +---
 .../data/nereids_function_p0/agg_function/group_unique_array.out | 9 +
 .../nereids_function_p0/agg_function/group_unique_array.groovy   | 3 +++
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/be/src/pipeline/exec/analytic_source_operator.cpp 
b/be/src/pipeline/exec/analytic_source_operator.cpp
index a036481d727..406108fbc4f 100644
--- a/be/src/pipeline/exec/analytic_source_operator.cpp
+++ b/be/src/pipeline/exec/analytic_source_operator.cpp
@@ -279,8 +279,6 @@ void AnalyticLocalState::_destroy_agg_status() {
 }
 }
 
-//now is execute for lead/lag row_number/rank/dense_rank/ntile functions
-//sum min max count avg first_value last_value functions
 void AnalyticLocalState::_execute_for_win_func(int64_t partition_start, 
int64_t partition_end,
int64_t frame_start, int64_t 
frame_end) {
 for (size_t i = 0; i < _agg_functions_size; ++i) {
@@ -292,7 +290,7 @@ void AnalyticLocalState::_execute_for_win_func(int64_t 
partition_start, int64_t
 partition_start, partition_end, frame_start, frame_end,
 _fn_place_ptr +
 
_parent->cast()._offsets_of_aggregate_states[i],
-agg_columns.data(), nullptr);
+agg_columns.data(), _agg_arena_pool.get());
 
 // If the end is not greater than the start, the current window should 
be empty.
 _current_window_empty =
diff --git 
a/regression-test/data/nereids_function_p0/agg_function/group_unique_array.out 
b/regression-test/data/nereids_function_p0/agg_function/group_unique_array.out
index 036ac5ce57f..74c053e38f6 100644
--- 
a/regression-test/data/nereids_function_p0/agg_function/group_unique_array.out
+++ 
b/regression-test/data/nereids_function_p0/agg_function/group_unique_array.out
@@ -8,3 +8,12 @@
 3  ["2023-01-02"]  ["hello"]
 4  ["2023-01-02", "2023-01-03"]["sql"]
 
+-- !3 --
+["doris", "world", "hello", "sql"]
+["doris", "world", "hello", "sql"]
+["doris", "world", "hello", "sql"]
+["doris", "world", "hello", "sql"]
+["doris", "world", "hello", "sql"]
+["doris", "world", "hello", "sql"]
+["doris", "world", "hello", "sql"]
+
diff --git 
a/regression-test/suites/nereids_function_p0/agg_function/group_unique_array.groovy
 
b/regression-test/suites/nereids_function_p0/agg_function/group_unique_array.groovy
index f17eadf73a5..f110f1a50c9 100644
--- 
a/regression-test/suites/nereids_function_p0/agg_function/group_unique_array.groovy
+++ 
b/regression-test/suites/nereids_function_p0/agg_function/group_unique_array.groovy
@@ -48,4 +48,7 @@ suite("group_unique_array") {
 qt_2 """
 select k1,collect_set(k2),collect_set(k3,1) from 
test_group_unique_array_table group by k1 order by k1;
 """
+qt_3 """
+select collect_set(k3) over() from test_group_unique_array_table;
+"""
 }
\ No newline at end of file


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [spill](regression) add regression test cases for spill [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #37319:
URL: https://github.com/apache/doris/pull/37319#issuecomment-2247195231

   
   
   ClickBench: Total hot run time: 31.44 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 8790744140c53fe64a3ff8dc9f6a4c533ddbc5e4, 
data reload: false
   
   query1   0.040.040.04
   query2   0.080.040.04
   query3   0.220.040.04
   query4   1.690.070.07
   query5   0.500.500.49
   query6   1.130.730.73
   query7   0.020.020.01
   query8   0.050.050.04
   query9   0.540.470.47
   query10  0.530.550.54
   query11  0.140.110.12
   query12  0.140.110.12
   query13  0.610.590.58
   query14  0.750.770.77
   query15  0.850.810.81
   query16  0.350.350.37
   query17  0.931.010.96
   query18  0.220.210.21
   query19  1.771.751.83
   query20  0.010.000.00
   query21  15.39   0.780.66
   query22  3.577.202.84
   query23  18.30   1.401.31
   query24  2.090.250.23
   query25  0.150.080.08
   query26  0.310.210.21
   query27  0.450.230.23
   query28  13.24   1.011.00
   query29  12.58   3.273.31
   query30  0.260.050.05
   query31  2.890.390.37
   query32  3.280.470.46
   query33  2.862.852.94
   query34  17.00   4.324.42
   query35  4.394.504.39
   query36  0.660.460.47
   query37  0.190.150.17
   query38  0.150.150.15
   query39  0.040.040.04
   query40  0.150.120.13
   query41  0.100.050.05
   query42  0.060.040.05
   query43  0.050.040.04
   Total cold run time: 108.73 s
   Total hot run time: 31.44 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Bug](agg) fix collect_set function core dump without arena pool [doris]

2024-07-24 Thread via GitHub


HappenLee merged PR #38234:
URL: https://github.com/apache/doris/pull/38234


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Refactor](exec) remove the unless file and refactor the agg code [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38193:
URL: https://github.com/apache/doris/pull/38193#issuecomment-2247198464

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Refactor](exec) remove the unless file and refactor the agg code [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38193:
URL: https://github.com/apache/doris/pull/38193#issuecomment-2247198528

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 [doris]

2024-07-24 Thread via GitHub


kaka11chen commented on PR #37234:
URL: https://github.com/apache/doris/pull/37234#issuecomment-2247211729

   run buildall
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [chore](backup) Log create restore job failed [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38267:
URL: https://github.com/apache/doris/pull/38267#issuecomment-2247212687

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [chore](backup) Log create restore job failed [doris]

2024-07-24 Thread via GitHub


github-actions[bot] commented on PR #38267:
URL: https://github.com/apache/doris/pull/38267#issuecomment-2247212611

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvement](profile) add catalog info in profile [doris]

2024-07-24 Thread via GitHub


doris-robot commented on PR #38298:
URL: https://github.com/apache/doris/pull/38298#issuecomment-2247216329

   
   
   TPC-DS: Total hot run time: 204763 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 1d785a78e1f0fb948f448ab50eac873f7152550e, 
data reload: false
   
   query1   936 421 385 385
   query2   6554300429402940
   query3   6914211 214 211
   query4   20195   17999   18012   17999
   query5   19740   651664766476
   query6   274 212 225 212
   query7   4153293 310 293
   query8   420 462 439 439
   query9   3152272226652665
   query10  408 316 294 294
   query11  11420   10674   10736   10674
   query12  131 72  80  72
   query13  5602701 697 697
   query14  17579   13370   13814   13370
   query15  372 254 230 230
   query16  6489282 260 260
   query17  17121478880 880
   query18  2312415 411 411
   query19  216 157 154 154
   query20  82  83  83  83
   query21  193 98  94  94
   query22  5337501450655014
   query23  32527   32105   31977   31977
   query24  7039645264956452
   query25  517 423 433 423
   query26  534 167 162 162
   query27  1870301 299 299
   query28  6143238623182318
   query29  2920272829452728
   query30  244 174 171 171
   query31  932 730 764 730
   query32  70  63  61  61
   query33  398 247 253 247
   query34  849 467 491 467
   query35  1093920 894 894
   query36  1333103911061039
   query37  89  59  62  59
   query38  3043293629142914
   query39  1365132613041304
   query40  203 99  92  92
   query41  49  43  45  43
   query42  91  81  86  81
   query43  734 767 837 767
   query44  1126715 711 711
   query45  247 233 226 226
   query46  1239968 985 968
   query47  2262183716891689
   query48  1028703 731 703
   query49  636 372 365 365
   query50  847 618 644 618
   query51  4745469947604699
   query52  93  79  82  79
   query53  449 320 319 319
   query54  2652243124642431
   query55  86  75  81  75
   query56  236 213 222 213
   query57  1185122410761076
   query58  223 210 206 206
   query59  4229419342694193
   query60  217 222 206 206
   query61  95  91  95  91
   query62  843 471 491 471
   query63  489 348 351 348
   query64  2584154715251525
   query65  3626356135713561
   query66  777 394 372 372
   query67  17296   15616   15673   15616
   query68  9099641 676 641
   query69  591 352 369 352
   query70  1635144415761444
   query71  419 316 318 316
   query72  345634633456
   query73  742 331 322 322
   query74  6260599157845784
   query75  5489373936793679
   query76  5473114512321145
   query77  950 261 261 261
   query78  12455   11720   11760   11720
   query79  7008641 622 622
   query80  989 403 401 401
   query81  493 237 234 234
   query82  1498104 99  99
   query83  163 135 132 132
   query84  253 69  72  69
   query85  885 333 340 333
   query86  336 290 300 290
   query87  3302304230153015
   query88  4868235323472347
   query89  360 288 315 288
   query90  1949214 226 214
   query91  175 138 145 138
   query92  58  54  54  54
   query93  3680588 564 564
   query94  707 210 217 210
   query95  1106108110501050
   query96  659 340 332 332
   query97  6390633863466338
   query98  200 178 167 167
   query99  2980926 904 904
   Total cold run time: 311282 ms
   Total hot run time: 204763 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t

Re: [PR] [fix](nereids) refine row count estimation for mark join [doris]

2024-07-24 Thread via GitHub


morrySnow merged PR #38270:
URL: https://github.com/apache/doris/pull/38270


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [Bug](dry-run) make dryrun result set collect multi batch's QueryStatistics [doris]

2024-07-24 Thread via GitHub


BiteThet opened a new pull request, #38303:
URL: https://github.com/apache/doris/pull/38303

   ## Proposed changes
make dryrun result set collect multi batch's QueryStatistics


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Bug](dry-run) make dryrun result set collect multi batch's QueryStatistics [doris]

2024-07-24 Thread via GitHub


BiteThet commented on PR #38303:
URL: https://github.com/apache/doris/pull/38303#issuecomment-2247218669

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



  1   2   3   4   5   6   7   8   9   >