Re: [PR] [improve](txn insert) make sub transactions visible [doris]
github-actions[bot] commented on PR #41362: URL: https://github.com/apache/doris/pull/41362#issuecomment-2484838424 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [refactor](regression) Re-arrange cases for data-partitioning doc [doris]
doris-robot commented on PR #44248: URL: https://github.com/apache/doris/pull/44248#issuecomment-2484810027 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](brpc) coredump caused by brpc checking (#44047) [doris]
mrhhsg merged PR #44188: URL: https://github.com/apache/doris/pull/44188 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](runtime filter) Disable `build_bf_exactly` if `sync_filter_size… [doris]
github-actions[bot] commented on PR #44169: URL: https://github.com/apache/doris/pull/44169#issuecomment-2484831087 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](transaction load) transaction may publish stuck if fe restart [doris]
github-actions[bot] commented on PR #44190: URL: https://github.com/apache/doris/pull/44190#issuecomment-2484831080 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](scan) Avoid memory allocated by buffered_reader from being traced (#41921) [doris]
mrhhsg opened a new pull request, #44253: URL: https://github.com/apache/doris/pull/44253 Use OwnedSlice to replace `char*` in BufferedReader ## Proposed changes pick #41921 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) Turn off default value of `enable_cloud_tablet_report` [doris]
dataroaring merged PR #44176: URL: https://github.com/apache/doris/pull/44176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](scan) Avoid memory allocated by buffered_reader from being traced (#41921) [doris]
doris-robot commented on PR #44253: URL: https://github.com/apache/doris/pull/44253#issuecomment-2484844688 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](scan) Avoid memory allocated by buffered_reader from being traced (#41921) [doris]
mrhhsg commented on PR #44253: URL: https://github.com/apache/doris/pull/44253#issuecomment-2484844768 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](runtime filter) Disable `build_bf_exactly` if `sync_filter_size… [doris]
doris-robot commented on PR #44169: URL: https://github.com/apache/doris/pull/44169#issuecomment-2484928027 TPC-H: Total hot run time: 44986 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 7332f5bc2f590217107847da679b48b86f1297e8, data reload: false -- Round 1 -- q1 17604 750373567356 q2 2237116811821168 q3 9960119412291194 q4 10226 746 752 746 q5 7590273126302630 q6 239 146 146 146 q7 997 632 601 601 q8 9363238023812380 q9 6743640864816408 q10 7088230522962296 q11 474 260 254 254 q12 525 220 217 217 q13 17791 305930143014 q14 253 208 212 208 q15 571 537 516 516 q16 678 595 581 581 q17 994 532 527 527 q18 7472673867926738 q19 1338993 923 923 q20 2901270026882688 q21 4009329930743074 q22 1388135713211321 Total cold run time: 110441 ms Total hot run time: 44986 ms - Round 2, with runtime_filter_mode=off - q1 7293727972677267 q2 335 223 231 223 q3 2937279927972797 q4 2015170517411705 q5 5410544254485442 q6 217 138 134 134 q7 2104170317231703 q8 3257338833963388 q9 8609854786208547 q10 3506347634383438 q11 582 501 492 492 q12 775 619 588 588 q13 13663 305530563055 q14 286 263 266 263 q15 553 513 513 513 q16 699 630 624 624 q17 1846159915771577 q18 7851748474557455 q19 1664155715391539 q20 2051183918401839 q21 5376522751325132 q22 604 543 564 543 Total cold run time: 71633 ms Total hot run time: 58264 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test in olap] [doris]
doris-robot commented on PR #44262: URL: https://github.com/apache/doris/pull/44262#issuecomment-2484941110 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test in olap] [doris]
Mryange commented on PR #44262: URL: https://github.com/apache/doris/pull/44262#issuecomment-2484941219 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [only test in olap] [doris]
Mryange opened a new pull request, #44262: URL: https://github.com/apache/doris/pull/44262 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] Support move truncated old data to recycle bin [doris]
yujun777 commented on PR #43107: URL: https://github.com/apache/doris/pull/43107#issuecomment-2484936209 ok. all solved but left one thing need to change, as comment above: In InternalCatalog.replayTruncateTable, line 3789 ~ 3809 should not deleted, it's use to add partition's tablet meta into TabletInvertIndex. It's because when creatting new a partition, this partititon's tablet meta will auto add into TabletInvertIndex, but when replaying this editlog, it need to call add tablet meta function explicit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](Nereids) Support create routine load command [doris]
LiBinfeng-01 commented on PR #43930: URL: https://github.com/apache/doris/pull/43930#issuecomment-2484897474 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](hive)support hive catalog read json table. [doris]
msridhar78 commented on code in PR #43469:
URL: https://github.com/apache/doris/pull/43469#discussion_r1847815017
##
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java:
##
@@ -364,14 +364,20 @@ public TableIf getTargetTable() {
@Override
public TFileFormatType getFileFormatType() throws UserException {
TFileFormatType type = null;
-String inputFormatName =
hmsTable.getRemoteTable().getSd().getInputFormat();
+Table table = hmsTable.getRemoteTable();
+String inputFormatName = table.getSd().getInputFormat();
String hiveFormat =
HiveMetaStoreClientHelper.HiveFileFormat.getFormat(inputFormatName);
if
(hiveFormat.equals(HiveMetaStoreClientHelper.HiveFileFormat.PARQUET.getDesc()))
{
type = TFileFormatType.FORMAT_PARQUET;
} else if
(hiveFormat.equals(HiveMetaStoreClientHelper.HiveFileFormat.ORC.getDesc())) {
type = TFileFormatType.FORMAT_ORC;
} else if
(hiveFormat.equals(HiveMetaStoreClientHelper.HiveFileFormat.TEXT_FILE.getDesc()))
{
-type = TFileFormatType.FORMAT_CSV_PLAIN;
+String serDeLib =
table.getSd().getSerdeInfo().getSerializationLib();
+if (serDeLib.equals("org.apache.hive.hcatalog.data.JsonSerDe")) {
Review Comment:
Will it be better to check if serDeLib.contains("JsonSerDe") as there can be
other imports for JsonSerDe lib like org.apache.hadoop.hive.serde2.JsonSerDe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [test](auth)Add grant show_view_priv to new user [doris]
zfr9527 opened a new pull request, #44260: URL: https://github.com/apache/doris/pull/44260 [test](auth)Add grant show_view_priv to new user ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [env](compile)open compile check in data type serde files [doris]
github-actions[bot] commented on PR #42888: URL: https://github.com/apache/doris/pull/42888#issuecomment-2484943868 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [fix](config) Turn off default value of `enable_cloud_tablet_report` (#44176)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new b137aaef311 [fix](config) Turn off default value of `enable_cloud_tablet_report` (#44176) b137aaef311 is described below commit b137aaef31117b724bf01ae4e02e5d4e17a84456 Author: deardeng AuthorDate: Tue Nov 19 14:58:28 2024 +0800 [fix](config) Turn off default value of `enable_cloud_tablet_report` (#44176) --- be/src/cloud/config.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/be/src/cloud/config.cpp b/be/src/cloud/config.cpp index 32e3250f87c..bdce09ed896 100644 --- a/be/src/cloud/config.cpp +++ b/be/src/cloud/config.cpp @@ -75,5 +75,5 @@ DEFINE_mInt32(tablet_txn_info_min_expired_seconds, "120"); DEFINE_mBool(enable_use_cloud_unique_id_from_fe, "true"); -DEFINE_mBool(enable_cloud_tablet_report, "true"); +DEFINE_mBool(enable_cloud_tablet_report, "false"); } // namespace doris::config - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cleanup](load) remove unused load properties "use_new_load_scan_node" [doris]
dataroaring merged PR #43989: URL: https://github.com/apache/doris/pull/43989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](binlog) Add replace table binlog [doris]
doris-robot commented on PR #44263: URL: https://github.com/apache/doris/pull/44263#issuecomment-2484954772 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement]skip resource tag when there is not queryable replica [doris]
doris-robot commented on PR #44255: URL: https://github.com/apache/doris/pull/44255#issuecomment-2484953199 TPC-H: Total hot run time: 44590 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ae741b9d76b7e2a5e3d35be44275c37ec0d3f62e, data reload: false -- Round 1 -- q1 17604 749372507250 q2 2232116611631163 q3 9959115711821157 q4 10223 745 714 714 q5 7568269326332633 q6 241 145 144 144 q7 967 629 592 592 q8 9386236923642364 q9 6623640063586358 q10 7001230722972297 q11 471 262 259 259 q12 414 213 206 206 q13 17770 304030493040 q14 234 211 218 211 q15 589 531 525 525 q16 662 577 580 577 q17 964 535 486 486 q18 7252669067186690 q19 1347102910171017 q20 2905270826832683 q21 4071321528972897 q22 1379134913271327 Total cold run time: 109862 ms Total hot run time: 44590 ms - Round 2, with runtime_filter_mode=off - q1 7290723874087238 q2 337 232 241 232 q3 3095300229812981 q4 2040180917911791 q5 5624570057415700 q6 226 139 137 137 q7 2248186717911791 q8 3322350235023502 q9 8875890888548854 q10 3627359935913591 q11 606 499 496 496 q12 815 634 638 634 q13 12622 326632593259 q14 305 280 274 274 q15 565 527 513 513 q16 699 650 663 650 q17 1848162916101610 q18 8200760978457609 q19 1711163215571557 q20 2151188718551855 q21 5581526454495264 q22 621 543 538 538 Total cold run time: 72408 ms Total hot run time: 60076 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [feat](binlog) Add replace table binlog [doris]
w41ter opened a new pull request, #44263: URL: https://github.com/apache/doris/pull/44263 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [x] Other reason Add cases in ccr-syncer - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve] (inverted index) show index file size [doris]
gavinchou commented on code in PR #44120: URL: https://github.com/apache/doris/pull/44120#discussion_r1847840876 ## cloud/src/meta-service/meta_service.cpp: ## @@ -1642,6 +1642,8 @@ void MetaServiceImpl::get_tablet_stats(::google::protobuf::RpcController* contro #ifdef NDEBUG // Force data size >= 0 to reduce the losses caused by bugs if (tablet_stats->data_size() < 0) tablet_stats->set_data_size(0); +if (tablet_stats->index_size() < 0) tablet_stats->set_index_size(0); Review Comment: what is the relationship between data_size, index_size and segment_size do we need a separate segment_size? ## cloud/src/meta-service/meta_service.cpp: ## @@ -1642,6 +1642,8 @@ void MetaServiceImpl::get_tablet_stats(::google::protobuf::RpcController* contro #ifdef NDEBUG // Force data size >= 0 to reduce the losses caused by bugs if (tablet_stats->data_size() < 0) tablet_stats->set_data_size(0); +if (tablet_stats->index_size() < 0) tablet_stats->set_index_size(0); Review Comment: what is the relationship between data_size, index_size and segment_size -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](binlog) Add replace table binlog [doris]
github-actions[bot] commented on PR #44263: URL: https://github.com/apache/doris/pull/44263#issuecomment-2484963217 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature] (Metric) add jni metrics for jdbc connection [doris]
qzsee commented on PR #41752: URL: https://github.com/apache/doris/pull/41752#issuecomment-2484781395 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids) support refresh catalog command [doris]
vinlee19 commented on PR #43449: URL: https://github.com/apache/doris/pull/43449#issuecomment-2484753002 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated (3e445227a96 -> 15e26de31d1)
This is an automated email from the ASF dual-hosted git repository. jianliangqi pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 3e445227a96 branch-3.0: [improve](http) Save the requested url in http execution error #43855 (#44051) add 15e26de31d1 [feature](function) add approx_top_k aggregation function (#44082) No new revisions were added by this update. Summary of changes: .../vec/aggregate_functions/aggregate_function.h | 1 + .../aggregate_function_approx_top.h| 109 +++ .../aggregate_function_approx_top_k.cpp| 43 +++ .../aggregate_function_approx_top_k.h | 229 ++ .../aggregate_function_approx_top_sum.cpp | 71 + .../aggregate_function_approx_top_sum.h| 245 +++ .../aggregate_function_simple_factory.cpp | 8 +- be/src/vec/common/arena_with_free_lists.h | 104 +++ be/src/vec/common/bit_helpers.h| 45 +++ be/src/vec/common/space_saving.h | 342 + be/src/vec/exprs/vectorized_agg_fn.cpp | 16 +- be/test/common/space_saving_test.cpp | 300 ++ .../doris/catalog/BuiltinAggregateFunctions.java | 4 + .../expressions/functions/agg/ApproxTopK.java | 94 ++ .../expressions/functions/agg/ApproxTopSum.java| 94 ++ .../visitor/AggregateFunctionVisitor.java | 10 + .../inverted_index_p0/test_index_approx_top_k.out | 109 +++ .../test_index_approx_top_sum.out | 109 +++ .../test_index_approx_top_k.groovy | 153 + .../test_index_approx_top_sum.groovy | 153 + 20 files changed, 2233 insertions(+), 6 deletions(-) create mode 100644 be/src/vec/aggregate_functions/aggregate_function_approx_top.h create mode 100644 be/src/vec/aggregate_functions/aggregate_function_approx_top_k.cpp create mode 100644 be/src/vec/aggregate_functions/aggregate_function_approx_top_k.h create mode 100644 be/src/vec/aggregate_functions/aggregate_function_approx_top_sum.cpp create mode 100644 be/src/vec/aggregate_functions/aggregate_function_approx_top_sum.h create mode 100644 be/src/vec/common/arena_with_free_lists.h create mode 100644 be/src/vec/common/bit_helpers.h create mode 100644 be/src/vec/common/space_saving.h create mode 100644 be/test/common/space_saving_test.cpp create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/ApproxTopK.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/ApproxTopSum.java create mode 100644 regression-test/data/inverted_index_p0/test_index_approx_top_k.out create mode 100644 regression-test/data/inverted_index_p0/test_index_approx_top_sum.out create mode 100644 regression-test/suites/inverted_index_p0/test_index_approx_top_k.groovy create mode 100644 regression-test/suites/inverted_index_p0/test_index_approx_top_sum.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](column) ColumnArray does not implement the insert data function. [doris]
Mryange commented on PR #43926: URL: https://github.com/apache/doris/pull/43926#issuecomment-2481942978 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](log) Detail log for file cache resource limit mode [doris]
gavinchou commented on PR #44094: URL: https://github.com/apache/doris/pull/44094#issuecomment-2481905691 run buildalll -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](transaction load) transaction may publish stuck if fe restart [doris]
github-actions[bot] commented on PR #44190: URL: https://github.com/apache/doris/pull/44190#issuecomment-2484831038 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](runtime filter) Disable `build_bf_exactly` if `sync_filter_size… [doris]
github-actions[bot] commented on PR #44169: URL: https://github.com/apache/doris/pull/44169#issuecomment-2484831125 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](runtime filter) Disable `build_bf_exactly` if `sync_filter_size… [doris]
Gabriel39 commented on PR #44169: URL: https://github.com/apache/doris/pull/44169#issuecomment-2484830862 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] Update package.json [doris-website]
KassieZ opened a new pull request, #1373: URL: https://github.com/apache/doris-website/pull/1373 # Versions - [ ] dev - [ ] 3.0 - [ ] 2.1 - [ ] 2.0 # Languages - [ ] Chinese - [ ] English -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](runtime filter) Disable `build_bf_exactly` if `sync_filter_size… [doris]
github-actions[bot] commented on PR #44169: URL: https://github.com/apache/doris/pull/44169#issuecomment-2484832583 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [conf](fe) Print jvm `ClassHistogram` in fe gc log after full gc #44010 [doris]
doris-robot commented on PR #44245: URL: https://github.com/apache/doris/pull/44245#issuecomment-2484832226 TPC-H: Total hot run time: 42713 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit d4d4acab5afb2bec4c6bd55626a7d0337d8d4ec4, data reload: false -- Round 1 -- q1 17586 737372377237 q2 2060160 159 159 q3 10724 106011331060 q4 10546 790 781 781 q5 7725279827432743 q6 234 143 141 141 q7 959 598 599 598 q8 9570192219681922 q9 7755640764036403 q10 6987227322912273 q11 447 249 257 249 q12 400 214 206 206 q13 17773 297629772976 q14 230 217 213 213 q15 561 528 532 528 q16 688 605 609 605 q17 958 532 549 532 q18 7282638766246387 q19 16121056993 993 q20 2837266826552655 q21 3891307831123078 q22 1130997 974 974 Total cold run time: 111955 ms Total hot run time: 42713 ms - Round 2, with runtime_filter_mode=off - q1 7289721872797218 q2 346 242 239 239 q3 3036288329832883 q4 2007175417871754 q5 5588571857635718 q6 220 141 138 138 q7 2131176217781762 q8 3237343334713433 q9 8863886088308830 q10 3539350735333507 q11 591 502 496 496 q12 771 576 608 576 q13 16619 315131383138 q14 296 282 288 282 q15 575 511 503 503 q16 719 668 672 668 q17 1848160616011601 q18 7995764974027402 q19 4302153714721472 q20 2060182118381821 q21 5326530253245302 q22 1101104610361036 Total cold run time: 78459 ms Total hot run time: 59779 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated (eb0054f5bd0 -> aff8d7c0096)
This is an automated email from the ASF dual-hosted git repository. zouxinyi pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from eb0054f5bd0 Revert "[fix](inverted index) Fix the incorrect type usage in the test_storage_page_size_fault case." (#44232) add aff8d7c0096 branch-3.0: [chore](sink) `enable_parallel_result_sink` default value is changed to false #43933 (#44241) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/qe/SessionVariable.java| 2 +- .../java/org/apache/doris/utframe/AnotherDemoTest.java| 15 ++- .../src/test/java/org/apache/doris/utframe/DemoTest.java | 15 ++- 3 files changed, 29 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](profile) only printed for non-sink nodes in the merge profile. #44040 [doris]
doris-robot commented on PR #44251: URL: https://github.com/apache/doris/pull/44251#issuecomment-2484824276 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-44040-branch-3.0 created (now eb0054f5bd0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-44040-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at eb0054f5bd0 Revert "[fix](inverted index) Fix the incorrect type usage in the test_storage_page_size_fault case." (#44232) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-44040-branch-3.0 updated (eb0054f5bd0 -> 650316c75d6)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-44040-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from eb0054f5bd0 Revert "[fix](inverted index) Fix the incorrect type usage in the test_storage_page_size_fault case." (#44232) add 650316c75d6 [fix](profile) only printed for non-sink nodes in the merge profile. (#44040) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/common/util/RuntimeProfile.java | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-44242-branch-3.0 created (now eb0054f5bd0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-44242-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at eb0054f5bd0 Revert "[fix](inverted index) Fix the incorrect type usage in the test_storage_page_size_fault case." (#44232) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](profile) only printed for non-sink nodes in the merge profile. #44040 [doris]
doris-robot commented on PR #44252: URL: https://github.com/apache/doris/pull/44252#issuecomment-2484824763 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](s3) do not replace https scheme if specified #44242 [doris]
doris-robot commented on PR #44250: URL: https://github.com/apache/doris/pull/44250#issuecomment-2484821741 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids) support refresh catalog command [doris]
doris-robot commented on PR #43449: URL: https://github.com/apache/doris/pull/43449#issuecomment-2484837462 TPC-H: Total hot run time: 45129 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit a203ae133ff46dceb6baf6660269ed008a8ddeb0, data reload: false -- Round 1 -- q1 17581 736672737273 q2 2251116211631162 q3 9976115211681152 q4 10216 730 752 730 q5 7594267427492674 q6 236 148 144 144 q7 989 605 607 605 q8 9385235623572356 q9 6554640464746404 q10 7045226823042268 q11 469 272 259 259 q12 418 217 217 217 q13 17792 308830763076 q14 243 213 230 213 q15 586 531 524 524 q16 664 578 587 578 q17 950 541 555 541 q18 7307672567966725 q19 1339106211101062 q20 2913274426752675 q21 3985321631483148 q22 1398134313671343 Total cold run time: 109891 ms Total hot run time: 45129 ms - Round 2, with runtime_filter_mode=off - q1 7270719174047191 q2 339 237 247 237 q3 3129307230153015 q4 2060179217901790 q5 5639568357825683 q6 223 141 135 135 q7 2254186317391739 q8 3410353135313531 q9 8912888688658865 q10 3630355735773557 q11 599 501 489 489 q12 849 636 633 633 q13 11224 324433053244 q14 334 272 274 272 q15 573 521 512 512 q16 683 680 643 643 q17 1845163716261626 q18 8152781678597816 q19 1681161814471447 q20 2118188718461846 q21 5608560555355535 q22 624 540 543 540 Total cold run time: 71156 ms Total hot run time: 60346 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](case) fix test_schema_change_ck (#41654) [doris]
mymeiyi commented on PR #44254: URL: https://github.com/apache/doris/pull/44254#issuecomment-2484849303 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Revert "[opt](store) allow to set storage_page_size for tables (#43690)" [doris]
github-actions[bot] commented on PR #44246: URL: https://github.com/apache/doris/pull/44246#issuecomment-2484814077 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement] Support TCP Keep-Alive on MySQL connections [doris]
lide-reed commented on PR #43904: URL: https://github.com/apache/doris/pull/43904#issuecomment-2484851435 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](cloud-schema-change) Rowset [0-1] do not engage in delete bitmap calculation during MOW SC [doris]
github-actions[bot] commented on PR #44171: URL: https://github.com/apache/doris/pull/44171#issuecomment-2484850015 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](runtime filter) Disable `build_bf_exactly` if `sync_filter_size… [doris]
doris-robot commented on PR #44169: URL: https://github.com/apache/doris/pull/44169#issuecomment-2484864362 TPC-H: Total hot run time: 45343 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 7332f5bc2f590217107847da679b48b86f1297e8, data reload: false -- Round 1 -- q1 17592 745873447344 q2 2225116511731165 q3 9972170512871287 q4 10374 807 680 680 q5 7616275226912691 q6 243 148 146 146 q7 990 632 604 604 q8 9379238123732373 q9 6886651664746474 q10 7081229323602293 q11 485 255 272 255 q12 535 216 227 216 q13 17787 308330873083 q14 256 206 215 206 q15 565 529 536 529 q16 705 584 597 584 q17 988 547 583 547 q18 7438674466956695 q19 13411089941 941 q20 2929270227192702 q21 4021334731943194 q22 1399133413491334 Total cold run time: 110807 ms Total hot run time: 45343 ms - Round 2, with runtime_filter_mode=off - q1 7342723772137213 q2 326 226 227 226 q3 2908282127902790 q4 1978178117151715 q5 5391543954955439 q6 221 136 133 133 q7 2097168916861686 q8 3301346434423442 q9 8555859685508550 q10 3511346534443444 q11 585 487 492 487 q12 797 599 579 579 q13 14495 306430783064 q14 300 254 262 254 q15 566 518 507 507 q16 648 643 627 627 q17 1824157815791578 q18 7903752773737373 q19 1690145815201458 q20 2075184118221822 q21 5258521951865186 q22 619 544 583 544 Total cold run time: 72390 ms Total hot run time: 58117 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement]skip resource tag when there is not queryable replica [doris]
doris-robot commented on PR #44255: URL: https://github.com/apache/doris/pull/44255#issuecomment-2484864164 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](test) logging result of show tablet to investigate problem [doris]
github-actions[bot] commented on PR #44224: URL: https://github.com/apache/doris/pull/44224#issuecomment-2484857795 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature] (Metric) add jni metrics for jdbc connection [doris]
hello-stephen commented on PR #41752: URL: https://github.com/apache/doris/pull/41752#issuecomment-2484859941 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) make sub transactions visible [doris]
github-actions[bot] commented on PR #41362: URL: https://github.com/apache/doris/pull/41362#issuecomment-2484838475 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cluster key) fix some cluster key problems [doris]
doris-robot commented on PR #43574: URL: https://github.com/apache/doris/pull/43574#issuecomment-2484860981 TeamCity be ut coverage result: Function Coverage: 38.01% (9899/26042) Line Coverage: 29.20% (82810/283565) Region Coverage: 28.33% (42525/150083) Branch Coverage: 24.90% (21561/86602) Coverage Report: http://coverage.selectdb-in.cc/coverage/ba47e1b782fcb291320ac54c34433f5e04df57f4_ba47e1b782fcb291320ac54c34433f5e04df57f4/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](s3) do not replace https scheme if specified #44242 [doris]
doris-robot commented on PR #44249: URL: https://github.com/apache/doris/pull/44249#issuecomment-2484862318 TPC-H: Total hot run time: 42622 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 5b71b1d8ed9c65e7fc62822c4dd727715281046a, data reload: false -- Round 1 -- q1 17590 732772827282 q2 2053162 184 162 q3 10695 102211831022 q4 10547 799 731 731 q5 7763281027082708 q6 233 143 144 143 q7 962 614 594 594 q8 9567190219761902 q9 6905634263856342 q10 6947225322752253 q11 454 264 261 261 q12 406 210 212 210 q13 17811 293829352935 q14 246 201 211 201 q15 547 519 522 519 q16 675 615 611 611 q17 957 557 553 553 q18 7217652063876387 q19 1493108510511051 q20 2858267126592659 q21 3898317431203120 q22 10731016976 976 Total cold run time: 110897 ms Total hot run time: 42622 ms - Round 2, with runtime_filter_mode=off - q1 7228716472527164 q2 334 247 241 241 q3 3020288529492885 q4 1987180517061706 q5 5592569857655698 q6 225 135 141 135 q7 2126172717721727 q8 3303339334443393 q9 8842890187638763 q10 3535352735173517 q11 606 514 488 488 q12 822 628 619 619 q13 16628 319131383138 q14 316 279 277 277 q15 561 520 512 512 q16 731 670 652 652 q17 1830162016051605 q18 8052783273967396 q19 8389163815931593 q20 2040185918311831 q21 5260528853405288 q22 10571027999 999 Total cold run time: 82484 ms Total hot run time: 59627 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Improvement]skip resource tag when there is not queryable replica [doris]
wangbo opened a new pull request, #44255: URL: https://github.com/apache/doris/pull/44255 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Add a session variable skip_resource_tag_bad_replica , ```set skip_resource_tag_bad_replica = true```, it means when resource tag has no a queryable replica,then resource tag is disabled and try to find a queryable replica. In some scenarios, query success rate takes precedence over resource isolation. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](nereids)extract in-predicate from filter conditions [doris]
englefly commented on PR #44023: URL: https://github.com/apache/doris/pull/44023#issuecomment-2484866173 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](test) logging result of show tablet to investigate problem [doris]
github-actions[bot] commented on PR #44224: URL: https://github.com/apache/doris/pull/44224#issuecomment-2484869425 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement]skip resource tag when there is not queryable replica [doris]
wangbo commented on PR #44255: URL: https://github.com/apache/doris/pull/44255#issuecomment-2484869493 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [download] Fix 2.0.15 download link [doris-website]
KassieZ opened a new pull request, #1374: URL: https://github.com/apache/doris-website/pull/1374 # Versions - [ ] dev - [ ] 3.0 - [ ] 2.1 - [ ] 2.0 # Languages - [ ] Chinese - [ ] English -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
svn commit: r73145 - in /dev/doris/2.0.15.1: ./ apache-doris-2.0.15.1-src.tar.gz apache-doris-2.0.15.1-src.tar.gz.asc apache-doris-2.0.15.1-src.tar.gz.sha512
Author: kxiao Date: Tue Nov 19 07:17:05 2024 New Revision: 73145 Log: prepare vote for release doris-2.0.15.1 Added: dev/doris/2.0.15.1/ dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz (with props) dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz.asc (with props) dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz.sha512 Added: dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz == Binary file - no diff available. Propchange: dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz -- svn:mime-type = application/x-gzip Added: dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz.asc == Binary file - no diff available. Propchange: dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz.sha512 == --- dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz.sha512 (added) +++ dev/doris/2.0.15.1/apache-doris-2.0.15.1-src.tar.gz.sha512 Tue Nov 19 07:17:05 2024 @@ -0,0 +1 @@ +4aa834fea58f01b9eb27b9eea3aad3014ddcb8240681d6634ab606b0be50106e18704e2cd9709aba006c05a98d7b986038118780ac8c3bc328121f0c557b0bb4 apache-doris-2.0.15.1-src.tar.gz - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](test) logging result of show tablet to investigate problem [doris]
dataroaring merged PR #44224: URL: https://github.com/apache/doris/pull/44224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](branch-2.1) pick #43960 #43929 [doris]
xinyiZzz commented on PR #44240: URL: https://github.com/apache/doris/pull/44240#issuecomment-2484645566 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](branch-2.1) pick #43960 #43929 [doris]
doris-robot commented on PR #44240: URL: https://github.com/apache/doris/pull/44240#issuecomment-2484645546 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. (#42004) [doris]
github-actions[bot] commented on PR #44239: URL: https://github.com/apache/doris/pull/44239#issuecomment-2484647114 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [pick](branch-2.1) pick #43960 #43929 [doris]
xinyiZzz opened a new pull request, #44240: URL: https://github.com/apache/doris/pull/44240 pick #43960 #43929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] Doris 冷存HDFS频繁出现block读取失败导致SQL执行失败[Bug] [doris]
fdf1779 commented on issue #44236: URL: https://github.com/apache/doris/issues/44236#issuecomment-2484638082 > 2、增加在冷存HDFS完成的tablet进行完整性校验，保证上报的完整性 还有就是在冷存HDFS时，如果冷存前是多副本存储在冷存后会出现HDFS和Doris各存在一个副本的情况，因此希望能够自动去重本地磁盘的副本，只保存HDFS远端存储的文件，而且实际的be元数据最好还是保留多副本(在两个节点保留元数据)。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids) support refresh catalog command [doris]
vinlee19 commented on PR #43449: URL: https://github.com/apache/doris/pull/43449#issuecomment-2484645753 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](branch-2.1) pick #43960 #43929 [doris]
xinyiZzz commented on PR #44240: URL: https://github.com/apache/doris/pull/44240#issuecomment-2484651959 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](filesystem)Use simple authentication directly in S3FileSystem (#43636) [doris]
wuwenchi commented on PR #44238: URL: https://github.com/apache/doris/pull/44238#issuecomment-2484632854 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](arrow-flight-sql) Fix conf `public_host` and `arrow_flight_sql_proxy_port` [doris]
xinyiZzz merged PR #44177: URL: https://github.com/apache/doris/pull/44177 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](arrow-flight-sql) Fix conf `public_host` and `arrow_flight_sql_proxy_port` (#44177)
This is an automated email from the ASF dual-hosted git repository.
zouxinyi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ed556b6b3bd [fix](arrow-flight-sql) Fix conf `public_host` and
`arrow_flight_sql_proxy_port` (#44177)
ed556b6b3bd is described below
commit ed556b6b3bda81b0802bcd1e5f624b282e349a9d
Author: Xinyi Zou
AuthorDate: Tue Nov 19 11:50:59 2024 +0800
[fix](arrow-flight-sql) Fix conf `public_host` and
`arrow_flight_sql_proxy_port` (#44177)
### What problem does this PR solve?
Problem Summary:
Rename `public_access_ip` to `public_host` and `public_access_port` to
`arrow_flight_sql_proxy_port`, they do not have to be used together.
---
be/src/common/config.cpp | 25 +--
be/src/common/config.h | 28 ++
be/src/service/internal_service.cpp| 8 ---
.../arrowflight/DorisFlightSqlProducer.java| 11 +++--
.../arrowflight/FlightSqlConnectProcessor.java | 8 ---
5 files changed, 65 insertions(+), 15 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 796e9af62ce..d9b9a02260a 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -63,8 +63,29 @@ DEFINE_Int32(brpc_port, "8060");
DEFINE_Int32(arrow_flight_sql_port, "-1");
-DEFINE_mString(public_access_ip, "");
-DEFINE_Int32(public_access_port, "-1");
+// If the external client cannot directly access priority_networks, set
public_host to be accessible
+// to external client.
+// There are usually two usage scenarios:
+// 1. in production environment, it is often inconvenient to expose Doris BE
nodes to the external network.
+// However, a reverse proxy (such as Nginx) can be added to all Doris BE
nodes, and the external client will be
+// randomly routed to a Doris BE node when connecting to Nginx. set
public_host to the host of Nginx.
+// 2. if priority_networks is an internal network IP, and BE node has its own
independent external IP,
+// but Doris currently does not support modifying priority_networks, setting
public_host to the real external IP.
+DEFINE_mString(public_host, "");
+
+// If the BE node is connected to the external network through a reverse proxy
like Nginx
+// and need to use Arrow Flight SQL, should add a server in Nginx to reverse
proxy
+// `Nginx:arrow_flight_sql_proxy_port` to
`BE_priority_networks:arrow_flight_sql_port`. For example:
+// upstream arrowflight {
+//server 10.16.10.8:8069;
+//server 10.16.10.8:8068;
+//}
+// server {
+//listen 8167 http2;
+//listen [::]:8167 http2;
+//server_name doris.arrowflight.com;
+// }
+DEFINE_Int32(arrow_flight_sql_proxy_port, "-1");
// the number of bthreads for brpc, the default value is set to -1,
// which means the number of bthreads is #cpu-cores
diff --git a/be/src/common/config.h b/be/src/common/config.h
index c40875728a3..7f18406 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -100,11 +100,29 @@ DECLARE_Int32(brpc_port);
// Default -1, do not start arrow flight sql server.
DECLARE_Int32(arrow_flight_sql_port);
-// If priority_networks is incorrect but cannot be modified, set
public_access_ip as BE’s real IP.
-// For ADBC client fetch result, default is empty, the ADBC client uses the
backend ip to fetch the result.
-// If ADBC client cannot access the backend ip, can set public_access_ip to
modify the fetch result ip.
-DECLARE_mString(public_access_ip);
-DECLARE_Int32(public_access_port);
+// If the external client cannot directly access priority_networks, set
public_host to be accessible
+// to external client.
+// There are usually two usage scenarios:
+// 1. in production environment, it is often inconvenient to expose Doris BE
nodes to the external network.
+// However, a reverse proxy (such as Nginx) can be added to all Doris BE
nodes, and the external client will be
+// randomly routed to a Doris BE node when connecting to Nginx. set
public_host to the host of Nginx.
+// 2. if priority_networks is an internal network IP, and BE node has its own
independent external IP,
+// but Doris currently does not support modifying priority_networks, setting
public_host to the real external IP.
+DECLARE_mString(public_host);
+
+// If the BE node is connected to the external network through a reverse proxy
like Nginx
+// and need to use Arrow Flight SQL, should add a server in Nginx to reverse
proxy
+// `Nginx:arrow_flight_sql_proxy_port` to
`BE_priority_networks:arrow_flight_sql_port`. For example:
+// upstream arrowflight {
+//server 10.16.10.8:8069;
+//server 10.16.10.8:8068;
+//}
+// server {
+//listen 8167 http2;
+//listen [::]:8167 http2;
+//server_name doris.arrowflight.com;
+// }
+DECLARE_Int32(arrow_flight_sql_proxy_port);
// the number of bthrea
Re: [PR] branch-3.0: [fix](arrow-flight-sql) Fix Doris NULL column conversion to arrow batch #43929 [doris]
dataroaring closed pull request #44231: branch-3.0: [fix](arrow-flight-sql) Fix Doris NULL column conversion to arrow batch #43929 URL: https://github.com/apache/doris/pull/44231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 02/03: [fix](agg) Aggregating string types with null values may result in incorrect result (#42067)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to tag 2.0.15.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4a6b26e1921d72ca9c13c593485cc2261a197a37
Author: Xin Liao
AuthorDate: Mon Oct 21 11:29:17 2024 +0800
[fix](agg) Aggregating string types with null values may result in
incorrect result (#42067)
Aggregating string types with null values may result in incorrect result
because using the replace_column_data function can cause incorrect
offsets in the column.
A reproducible case：
```
CREATE TABLE `test_scan_keys_with_bool_type` (
`col1` tinyint NOT NULL,
`col2` int NOT NULL,
`col3` tinyint NOT NULL,
`col5` boolean REPLACE NOT NULL,
`col4` datetime(2) REPLACE NOT NULL,
`col6` double REPLACE_IF_NOT_NULL NULL,
`col7` varchar(100) REPLACE_IF_NOT_NULL NULL
) ENGINE=OLAP
AGGREGATE KEY(`col1`, `col2`, `col3`)
DISTRIBUTED BY HASH(`col1`, `col2`, `col3`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"disable_auto_compaction" = "true"
);
insert into test_scan_keys_with_bool_type values
( -100 ,1 , -82 ,1 , '2024-02-16 04:37:37.00' , -1299962421.904282
, NULL ),
( -100 ,0 , -82 ,1 , '2024-02-16 04:37:37.00' , -1299962421.904282
, "hi" ),
( -100 ,1 , 92 ,1 , '2024-02-16 04:37:37.00' , 23423423.0324234
, NULL );
insert into test_scan_keys_with_bool_type values
( -100 ,1 , 1 ,1 , '2024-02-16 04:37:37.00' , -1299962421.904282 ,
"doris" );
MySQL [test]> select * from test_scan_keys_with_bool_type;
+--+--+--+--++-+---+
| col1 | col2 | col3 | col5 | col4 | col6
| col7 |
+--+--+--+--++-+---+
| -100 |0 | -82 |1 | 2024-02-16 04:37:37.00 | -1299962421.9042821
| hi|
| -100 |1 | -82 |1 | 2024-02-16 04:37:37.00 | -1299962421.9042821
| NULL |
| -100 |1 |1 |1 | 2024-02-16 04:37:37.00 | -1299962421.9042821
| hidor |
| -100 |1 | 92 |1 | 2024-02-16 04:37:37.00 |23423423.0324234
| NULL |
+--+--+--+--++-+---+
4 rows in set (0.057 sec)
MySQL [test]> set skip_storage_engine_merge = true; select * from
test_scan_keys_with_bool_type;
+--+--+--+--++-+---+
| col1 | col2 | col3 | col5 | col4 | col6
| col7 |
+--+--+--+--++-+---+
| -100 |0 | -82 |1 | 2024-02-16 04:37:37.00 | -1299962421.9042821
| hi|
| -100 |1 | -82 |1 | 2024-02-16 04:37:37.00 | -1299962421.9042821
| NULL |
| -100 |1 | 92 |1 | 2024-02-16 04:37:37.00 |23423423.0324234
| NULL |
| -100 |1 |1 |1 | 2024-02-16 04:37:37.00 | -1299962421.9042821
| doris |
+--+--+--+--++-+---+
4 rows in set (0.023 sec)
```
https://github.com/apache/doris/pull/33493 By supporting variant type
aggregation, this issue has been resolved.So versions after 2.1 do not
have this issue.
---
be/src/vec/olap/block_reader.cpp | 4 +--
.../aggregate/test_aggregate_table.out | 6
.../aggregate/test_aggregate_table.groovy | 32 ++
3 files changed, 40 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/olap/block_reader.cpp b/be/src/vec/olap/block_reader.cpp
index d69efa181e8..5ff2890ab37 100644
--- a/be/src/vec/olap/block_reader.cpp
+++ b/be/src/vec/olap/block_reader.cpp
@@ -470,10 +470,10 @@ size_t BlockReader::_copy_agg_data() {
auto& dst_column = _stored_data_columns[idx];
if (_stored_has_variable_length_tag[idx]) {
//variable length type should replace ordered
+dst_column->clear();
for (size_t i = 0; i < copy_size; i++) {
auto& ref = _stored_row_ref[i];
-
dst_column->replace_column_data(*ref.block->get_by_position(idx).column,
-ref.row_pos, i);
+
dst_column->insert_from(*ref.block->get_by_position(idx).column, ref.row_pos);
}
} else {
for (auto& it : _temp_ref_map) {
diff --git
a/regression-test/data/data_model_p0/aggregate/test_aggregate_table.out
b/regression-test/data/data_model_p0/aggregate/test_aggregate_table.out
index 1a7ed71eb2e..8da75c78275 100644
--- a/regression-test/data/data_model_p0/aggregate/test_aggregate_table.out
+++ b/regression-test/data/data_model_p0/aggregate/test_aggregate
(doris) 03/03: [fix](agg) Aggregating string types with null values may result in incorrect result when compaction (#42512)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to tag 2.0.15.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 64bde86ec6028aed8f52d7eb5ac6c2ee01d2a982
Author: Xin Liao
AuthorDate: Mon Oct 28 10:24:17 2024 +0800
[fix](agg) Aggregating string types with null values may result in
incorrect result when compaction (#42512)
Follow-up for #42067.
---
be/src/vec/olap/vertical_block_reader.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/olap/vertical_block_reader.cpp
b/be/src/vec/olap/vertical_block_reader.cpp
index ee2f52462f8..dd2c167324f 100644
--- a/be/src/vec/olap/vertical_block_reader.cpp
+++ b/be/src/vec/olap/vertical_block_reader.cpp
@@ -317,10 +317,10 @@ size_t VerticalBlockReader::_copy_agg_data() {
auto& dst_column = _stored_data_columns[idx];
if (_stored_has_variable_length_tag[idx]) {
//variable length type should replace ordered
+dst_column->clear();
for (size_t i = 0; i < copy_size; i++) {
auto& ref = _stored_row_ref[i];
-
dst_column->replace_column_data(*ref.block->get_by_position(idx).column,
-ref.row_pos, i);
+
dst_column->insert_from(*ref.block->get_by_position(idx).column, ref.row_pos);
}
} else {
for (auto& it : _temp_ref_map) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/03: Revert "[pick](nestedtype)support nested type with agg replace_if_not_null #38304 (#38716)"
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to tag 2.0.15.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0f65853e657dfbd5ac9f8e0d62ededd350a7ccb2
Author: Kang
AuthorDate: Tue Nov 19 11:56:36 2024 +0800
Revert "[pick](nestedtype)support nested type with agg replace_if_not_null
#38304 (#38716)"
This reverts commit 992520484fd96fce1c0be6a98a9defa5e81d9719.
---
be/src/vec/columns/column_nullable.h | 2 +
.../org/apache/doris/analysis/CreateTableStmt.java | 11 +-
.../create_nestedtypes_with_replaceIfNOtNull.out | 46 ---
...create_nestedtypes_with_replaceIfNOtNull.groovy | 325 -
4 files changed, 7 insertions(+), 377 deletions(-)
diff --git a/be/src/vec/columns/column_nullable.h
b/be/src/vec/columns/column_nullable.h
index 96ac8a8355d..89cd27e8bd0 100644
--- a/be/src/vec/columns/column_nullable.h
+++ b/be/src/vec/columns/column_nullable.h
@@ -353,6 +353,8 @@ public:
if (!nullable_rhs.is_null_at(row)) {
nested_column->replace_column_data(*nullable_rhs.nested_column,
row, self_row);
+} else {
+nested_column->replace_column_data_default(self_row);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
index 98cb7b3e9f2..b2120a3133c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
@@ -494,12 +494,11 @@ public class CreateTableStmt extends DdlStmt {
columnDef.analyze(engineName.equals("olap"));
if (columnDef.getType().isComplexType() &&
engineName.equals("olap")) {
-if (columnDef.getAggregateType() != null &&
columnDef.getAggregateType() != AggregateType.NONE
-&& columnDef.getAggregateType() !=
AggregateType.REPLACE
-&& columnDef.getAggregateType() !=
AggregateType.REPLACE_IF_NOT_NULL) {
-throw new AnalysisException(
-columnDef.getType().getPrimitiveType() + " column
can't support aggregation "
-+ columnDef.getAggregateType());
+if (columnDef.getAggregateType() != null
+&& columnDef.getAggregateType() != AggregateType.NONE
+&& columnDef.getAggregateType() !=
AggregateType.REPLACE) {
+throw new
AnalysisException(columnDef.getType().getPrimitiveType()
++ " column can't support aggregation " +
columnDef.getAggregateType());
}
if (columnDef.isKey()) {
throw new
AnalysisException(columnDef.getType().getPrimitiveType()
diff --git
a/regression-test/data/datatype_p0/nested_types/ddl/create_nestedtypes_with_replaceIfNOtNull.out
b/regression-test/data/datatype_p0/nested_types/ddl/create_nestedtypes_with_replaceIfNOtNull.out
deleted file mode 100644
index 7513390d37c..000
---
a/regression-test/data/datatype_p0/nested_types/ddl/create_nestedtypes_with_replaceIfNOtNull.out
+++ /dev/null
@@ -1,46 +0,0 @@
--- This file is automatically generated. You should know what you did if you
want to edit this
--- !sql_all --
-1 [1, 0] [1, 2] [100, 200] [1000, 2000][1, 2]
[10, 20][1.1, 2.2] [1.11, 2.22][10.01, 20.02] [30.03,
40.04] ["2024-01-01", "2024-01-02"]["2024-01-01 12:00:00", "2024-01-02
12:00:00"] ["2024-01-03", "2024-01-04"]["2024-01-03 12:00:00", "2024-01-04
12:00:00"] ["char1", "char2"] ["a", "b"] ["string1", "string2"]
-5 \N \N \N \N \N \N \N \N \N
\N \N \N \N \N \N \N \N
-7 [1, 1, 0] [12, 13, 14][1200, 1300, 1400] [12000, 13000,
14000] [12, 13, 14][120, 130, 140]
[12.12, 13.13, 14.14] [12.12, 13.13, 14.14] [120.12, 130.13, 140.14]
[140.14, 150.15, 160.16]["2024-07-01", "2024-07-02", "2024-07-03"]
["2024-07-01 12:00:00", "2024-07-02 12:00:00", "2024-07-03 12:00:00"]
["2024-07-03", "2024-07-04", "2024-07-05"] ["2024-07-03 12:00:00",
"2024-07-04 12:00:00", "2024-07-05 12:00:00"] ["char12", "char13", "char14"]
["l", "m", "n"] ["stri [...]
-
--- !sql_replace_1 --
-1 [1, 1, 0] [12, 13, 14][1200, 1300, 1400] [12000, 13000,
14000] [12, 13, 14][120, 130, 140]
[12.12, 13.13, 14.14] [12.12, 13.13, 14.14] [120.12, 130.13, 140.14]
[140.14, 150.15, 160.16]["2024-07-01", "2024-07-02", "2024-07-03"]
["2024-07-01 12:00:00", "2024-07-02 12:00:00", "2024-07-03 12:00:00"]
["2024-07-03", "2024-07-04", "2024-07-05"] ["2024-07-03 12:00:00
Re: [PR] [feat](Nereids) support refresh catalog command [doris]
vinlee19 commented on PR #43449: URL: https://github.com/apache/doris/pull/43449#issuecomment-2484655046 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](local shuffle) Fix unbalanced data distribution #44137 [doris]
dataroaring closed pull request #44234: branch-3.0: [fix](local shuffle) Fix unbalanced data distribution #44137 URL: https://github.com/apache/doris/pull/44234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. (#42004) [doris]
doris-robot commented on PR #44239: URL: https://github.com/apache/doris/pull/44239#issuecomment-2484641295 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](arrow-flight-sql) Fix Doris NULL column conversion to arrow batch #43929 [doris]
doris-robot commented on PR #44233: URL: https://github.com/apache/doris/pull/44233#issuecomment-2484638452 TeamCity be ut coverage result: Function Coverage: 36.44% (9511/26104) Line Coverage: 27.94% (78321/280317) Region Coverage: 26.59% (40151/150975) Branch Coverage: 23.40% (20408/87206) Coverage Report: http://coverage.selectdb-in.cc/coverage/9abe93d6b9d6fbe9a65bc577f0d978698a23ec19_9abe93d6b9d6fbe9a65bc577f0d978698a23ec19/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve][ms] Bvars add the FDB get_count_normalized indicator [doris]
Sjwhello commented on PR #44036: URL: https://github.com/apache/doris/pull/44036#issuecomment-2484655843 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) tag 2.0.15.1 updated (b066ba7c7de -> 64bde86ec60)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a change to tag 2.0.15.1 in repository https://gitbox.apache.org/repos/asf/doris.git *** WARNING: tag 2.0.15.1 was modified! *** from b066ba7c7de (commit) to 64bde86ec60 (commit) discard b066ba7c7de change version to 2.0.15.1 discard f00b3ab6afe [fix](agg) Aggregating string types with null values may result in incorrect result when compaction (#42512) discard bd4fc3f7d42 [fix](agg) Aggregating string types with null values may result in incorrect result (#42067) new 0f65853e657 Revert "[pick](nestedtype)support nested type with agg replace_if_not_null #38304 (#38716)" new 4a6b26e1921 [fix](agg) Aggregating string types with null values may result in incorrect result (#42067) new 64bde86ec60 [fix](agg) Aggregating string types with null values may result in incorrect result when compaction (#42512) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the tag are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (b066ba7c7de) \ N -- N -- N refs/tags/2.0.15.1 (64bde86ec60) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/vec/columns/column_nullable.h | 2 + .../org/apache/doris/analysis/CreateTableStmt.java | 11 +- gensrc/script/gen_build_version.sh | 2 +- .../create_nestedtypes_with_replaceIfNOtNull.out | 46 --- ...create_nestedtypes_with_replaceIfNOtNull.groovy | 325 - 5 files changed, 8 insertions(+), 378 deletions(-) delete mode 100644 regression-test/data/datatype_p0/nested_types/ddl/create_nestedtypes_with_replaceIfNOtNull.out delete mode 100644 regression-test/suites/datatype_p0/nested_types/ddl/create_nestedtypes_with_replaceIfNOtNull.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](branch-2.1) pick #43960 #43929 #44177 [doris]
github-actions[bot] commented on PR #44240: URL: https://github.com/apache/doris/pull/44240#issuecomment-2484657747 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](nereids)should set isForwardedToMaster and redirectStatus to null before fallback to legacy planner #44163 [doris]
doris-robot commented on PR #44237: URL: https://github.com/apache/doris/pull/44237#issuecomment-2484627914 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-44163-branch-2.1 created (now ab7f5ba2acb)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-44163-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git at ab7f5ba2acb branch-2.1: [fix](mtmv)After failure, one should quickly return to avoid repeated failures #44174 (#44204) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](local shuffle) Fix unbalanced data distribution [doris]
Gabriel39 opened a new pull request, #44137: URL: https://github.com/apache/doris/pull/44137 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](arrow-flight-sql) Fix Doris NULL column conversion to arrow batch #43929 [doris]
xinyiZzz closed pull request #44231: branch-3.0: [fix](arrow-flight-sql) Fix Doris NULL column conversion to arrow batch #43929 URL: https://github.com/apache/doris/pull/44231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](debug) Support dump const nullable Column from Block [doris]
zclllyybb commented on PR #44029: URL: https://github.com/apache/doris/pull/44029#issuecomment-2484713878 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](branch-2.1) pick #43960 #43929 #44177 [doris]
github-actions[bot] commented on PR #44240: URL: https://github.com/apache/doris/pull/44240#issuecomment-2484736353 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [performance](move-memtable) async close tablet streams (#41156 & #43813) (#44128)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new b4e136bfccd [performance](move-memtable) async close tablet streams
(#41156 & #43813) (#44128)
b4e136bfccd is described below
commit b4e136bfccdc9636a8df9307d05631deb8e008ac
Author: Kaijie Chen
AuthorDate: Tue Nov 19 14:14:53 2024 +0800
[performance](move-memtable) async close tablet streams (#41156 & #43813)
(#44128)
backport #41156 and #43813
---
be/src/runtime/load_stream.cpp| 133 +-
be/src/runtime/load_stream.h | 5 +-
be/src/runtime/load_stream_writer.cpp | 12 ++-
be/src/runtime/load_stream_writer.h | 9 +++
4 files changed, 91 insertions(+), 68 deletions(-)
diff --git a/be/src/runtime/load_stream.cpp b/be/src/runtime/load_stream.cpp
index a2d9fd2f611..79427703873 100644
--- a/be/src/runtime/load_stream.cpp
+++ b/be/src/runtime/load_stream.cpp
@@ -62,7 +62,6 @@ TabletStream::TabletStream(PUniqueId load_id, int64_t id,
int64_t txn_id,
_txn_id(txn_id),
_load_stream_mgr(load_stream_mgr) {
load_stream_mgr->create_tokens(_flush_tokens);
-_status = Status::OK();
_profile = profile->create_child(fmt::format("TabletStream {}", id), true,
true);
_append_data_timer = ADD_TIMER(_profile, "AppendDataTime");
_add_segment_timer = ADD_TIMER(_profile, "AddSegmentTime");
@@ -71,7 +70,7 @@ TabletStream::TabletStream(PUniqueId load_id, int64_t id,
int64_t txn_id,
inline std::ostream& operator<<(std::ostream& ostr, const TabletStream&
tablet_stream) {
ostr << "load_id=" << tablet_stream._load_id << ", txn_id=" <<
tablet_stream._txn_id
- << ", tablet_id=" << tablet_stream._id << ", status=" <<
tablet_stream._status;
+ << ", tablet_id=" << tablet_stream._id << ", status=" <<
tablet_stream._status.status();
return ostr;
}
@@ -89,19 +88,19 @@ Status
TabletStream::init(std::shared_ptr schema, int64_t
_load_stream_writer = std::make_shared(&req, _profile);
DBUG_EXECUTE_IF("TabletStream.init.uninited_writer", {
-_status = Status::Uninitialized("fault injection");
-return _status;
+_status.update(Status::Uninitialized("fault injection"));
+return _status.status();
});
-_status = _load_stream_writer->init();
+_status.update(_load_stream_writer->init());
if (!_status.ok()) {
LOG(INFO) << "failed to init rowset builder due to " << *this;
}
-return _status;
+return _status.status();
}
Status TabletStream::append_data(const PStreamHeader& header, butil::IOBuf*
data) {
if (!_status.ok()) {
-return _status;
+return _status.status();
}
// dispatch add_segment request
@@ -150,8 +149,8 @@ Status TabletStream::append_data(const PStreamHeader&
header, butil::IOBuf* data
if (eos && st.ok()) {
st = _load_stream_writer->close_segment(new_segid);
}
-if (!st.ok() && _status.ok()) {
-_status = st;
+if (!st.ok()) {
+_status.update(st);
LOG(WARNING) << "write data failed " << st << ", " << *this;
}
};
@@ -167,11 +166,11 @@ Status TabletStream::append_data(const PStreamHeader&
header, butil::IOBuf* data
timer.start();
while (flush_token->num_tasks() >= load_stream_flush_token_max_tasks) {
if (timer.elapsed_time() / 1000 / 1000 >=
load_stream_max_wait_flush_token_time_ms) {
-_status = Status::Error(
-"wait flush token back pressure time is more than "
-"load_stream_max_wait_flush_token_time {}",
-load_stream_max_wait_flush_token_time_ms);
-return _status;
+_status.update(
+Status::Error("wait flush token back pressure time
is more than "
+"load_stream_max_wait_flush_token_time
{}",
+
load_stream_max_wait_flush_token_time_ms));
+return _status.status();
}
bthread_usleep(2 * 1000); // 2ms
}
@@ -181,14 +180,14 @@ Status TabletStream::append_data(const PStreamHeader&
header, butil::IOBuf* data
g_load_stream_flush_running_threads << 1;
auto st = flush_token->submit_func(flush_func);
if (!st.ok()) {
-_status = st;
+_status.update(st);
}
-return _status;
+return _status.status();
}
Status TabletStream::add_segment(const PStreamHeader& header, butil::IOBuf*
data) {
if (!_status.ok()) {
-return _status;
+return _status.status();
}
SCOPED_TIMER(_add_segment_timer);
@@ -207,17 +206,17 @@ Status TabletStream::add_segment(const PStreamHeader&
header, butil::IOBuf* data
{
std::lock_g
Re: [PR] Revert "[opt](store) allow to set storage_page_size for tables (#43690)" [doris]
zzzxl1993 commented on PR #44246: URL: https://github.com/apache/doris/pull/44246#issuecomment-2484805580 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Revert "[opt](store) allow to set storage_page_size for tables (#43690)" [doris]
doris-robot commented on PR #44246: URL: https://github.com/apache/doris/pull/44246#issuecomment-2484805518 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: Revert "[fix](inverted index) Fix the incorrect type usage in the test_storage_page_size_fault case." (#44232)
This is an automated email from the ASF dual-hosted git repository.
jianliangqi pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new eb0054f5bd0 Revert "[fix](inverted index) Fix the incorrect type usage
in the test_storage_page_size_fault case." (#44232)
eb0054f5bd0 is described below
commit eb0054f5bd08289ec60bf3b6e6e684531d705b09
Author: zzzxl
AuthorDate: Tue Nov 19 14:26:31 2024 +0800
Revert "[fix](inverted index) Fix the incorrect type usage in the
test_storage_page_size_fault case." (#44232)
Reverts apache/doris#44081
---
be/src/olap/rowset/segment_v2/segment_writer.cpp | 4 ++--
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/segment_writer.cpp
b/be/src/olap/rowset/segment_v2/segment_writer.cpp
index d238ea0313d..3d192dee6e9 100644
--- a/be/src/olap/rowset/segment_v2/segment_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_writer.cpp
@@ -253,10 +253,10 @@ Status SegmentWriter::_create_column_writer(uint32_t cid,
const TabletColumn& co
opts.data_page_size = storage_page_size;
}
DBUG_EXECUTE_IF("VerticalSegmentWriter._create_column_writer.storage_page_size",
{
-auto table_id =
DebugPoints::instance()->get_debug_param_or_default(
+auto table_id =
DebugPoints::instance()->get_debug_param_or_default(
"VerticalSegmentWriter._create_column_writer.storage_page_size", "table_id",
INT_MIN);
-auto target_data_page_size =
DebugPoints::instance()->get_debug_param_or_default(
+auto target_data_page_size =
DebugPoints::instance()->get_debug_param_or_default(
"VerticalSegmentWriter._create_column_writer.storage_page_size",
"storage_page_size", INT_MIN);
if (table_id == INT_MIN || target_data_page_size == INT_MIN) {
diff --git a/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp
b/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp
index 8df223d6a37..a84095fa4ef 100644
--- a/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp
@@ -246,10 +246,10 @@ Status
VerticalSegmentWriter::_create_column_writer(uint32_t cid, const TabletCo
opts.data_page_size = storage_page_size;
}
DBUG_EXECUTE_IF("VerticalSegmentWriter._create_column_writer.storage_page_size",
{
-auto table_id =
DebugPoints::instance()->get_debug_param_or_default(
+auto table_id =
DebugPoints::instance()->get_debug_param_or_default(
"VerticalSegmentWriter._create_column_writer.storage_page_size", "table_id",
INT_MIN);
-auto target_data_page_size =
DebugPoints::instance()->get_debug_param_or_default(
+auto target_data_page_size =
DebugPoints::instance()->get_debug_param_or_default(
"VerticalSegmentWriter._create_column_writer.storage_page_size",
"storage_page_size", INT_MIN);
if (table_id == INT_MIN || target_data_page_size == INT_MIN) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Revert "[fix](inverted index) Fix the incorrect type usage in the test_storage_page_size_fault case." [doris]
qidaye merged PR #44232: URL: https://github.com/apache/doris/pull/44232 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] Revert "[opt](store) allow to set storage_page_size for tables (#43690)" [doris]
zzzxl1993 opened a new pull request, #44246: URL: https://github.com/apache/doris/pull/44246 Reverts apache/doris#43918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](nereids) stats calculator based on runtime info [doris]
xzj7019 opened a new pull request, #44247: URL: https://github.com/apache/doris/pull/44247 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) stats calculator based on runtime info [doris]
doris-robot commented on PR #44247: URL: https://github.com/apache/doris/pull/44247#issuecomment-2484808384 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](s3) do not replace https scheme if specified [doris]
github-actions[bot] commented on PR #44242: URL: https://github.com/apache/doris/pull/44242#issuecomment-2484810621 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-2.1: [refactor](regression) Re-arrange cases for data-partitioning doc [doris]
bobhan1 opened a new pull request, #44248: URL: https://github.com/apache/doris/pull/44248 pick https://github.com/apache/doris/pull/43373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43933-branch-3.0 deleted (was 0bdeb044a00)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43933-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git was 0bdeb044a00 [chore](sink) `enable_parallel_result_sink` default value is changed to false (#43933) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
