[GitHub] [doris] Mryange commented on pull request #19811: [feature](decimal)support cast rounding half up and div precision increment in decimalv3.
Mryange commented on PR #19811: URL: https://github.com/apache/doris/pull/19811#issuecomment-1569605422 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on a diff in pull request #19540: [Improvement](planner) expand sql-block-rule to make it can be used on all kinds of sql stmt
Yulei-Yang commented on code in PR #19540:
URL: https://github.com/apache/doris/pull/19540#discussion_r1211181653
##
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##
@@ -468,6 +472,10 @@ private void executeByNereids(TUniqueId queryId) throws
Exception {
context.setStartTime();
profile.getSummaryProfile().setQueryBeginTime();
context.setStmtId(STMT_ID_GENERATOR.incrementAndGet());
+
+if (checkBlockRules()) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
github-actions[bot] commented on PR #19819: URL: https://github.com/apache/doris/pull/19819#issuecomment-1569612313 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu commented on pull request #20227: [Fix](hive-catalog) Fallback to refresh catalog when hms events are missing
dutyu commented on PR #20227: URL: https://github.com/apache/doris/pull/20227#issuecomment-1569612586 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
wsjz commented on PR #19819: URL: https://github.com/apache/doris/pull/19819#issuecomment-1569616634 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19811: [feature](decimal)support cast rounding half up and div precision increment in decimalv3.
github-actions[bot] commented on PR #19811: URL: https://github.com/apache/doris/pull/19811#issuecomment-1569616934 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20258: [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property
github-actions[bot] commented on PR #20258: URL: https://github.com/apache/doris/pull/20258#issuecomment-1569618275 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu commented on pull request #20227: [Fix](hive-catalog) Fallback to refresh catalog when hms events are missing
dutyu commented on PR #20227: URL: https://github.com/apache/doris/pull/20227#issuecomment-1569619604 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19811: [feature](decimal)support cast rounding half up and div precision increment in decimalv3.
github-actions[bot] commented on PR #19811: URL: https://github.com/apache/doris/pull/19811#issuecomment-1569622917 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19811: [feature](decimal)support cast rounding half up and div precision increment in decimalv3.
github-actions[bot] commented on PR #19811: URL: https://github.com/apache/doris/pull/19811#issuecomment-1569626907 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
github-actions[bot] commented on PR #19819: URL: https://github.com/apache/doris/pull/19819#issuecomment-1569630786 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiongjx751 commented on pull request #20153: [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments
xiongjx751 commented on PR #20153: URL: https://github.com/apache/doris/pull/20153#issuecomment-1569634512 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20249: [Enhancement] Change Create Resource Group Grammar
github-actions[bot] commented on PR #20249: URL: https://github.com/apache/doris/pull/20249#issuecomment-1569636652 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20249: [Enhancement] Change Create Resource Group Grammar
github-actions[bot] commented on PR #20249: URL: https://github.com/apache/doris/pull/20249#issuecomment-1569636709 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #20249: [Enhancement] Change Create Resource Group Grammar
yiguolei merged PR #20249: URL: https://github.com/apache/doris/pull/20249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #20170: [Enhancement] Change Create Resource Group Grammar
yiguolei closed issue #20170: [Enhancement] Change Create Resource Group Grammar URL: https://github.com/apache/doris/issues/20170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #20258: [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property
Tanya-W commented on PR #20258: URL: https://github.com/apache/doris/pull/20258#issuecomment-1569639250 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-shade] Cai-Yao opened a new pull request, #19: Rename com.google.flatbuffers
Cai-Yao opened a new pull request, #19: URL: https://github.com/apache/doris-shade/pull/19 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #20153: [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments
github-actions[bot] commented on code in PR #20153: URL: https://github.com/apache/doris/pull/20153#discussion_r1211210391 ## be/test/olap/delete_bitmap_calculator_test.cpp: ## @@ -0,0 +1,301 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "olap/delete_bitmap_calculator.h" Review Comment: warning: 'olap/delete_bitmap_calculator.h' file not found [clang-diagnostic-error] ```cpp #include "olap/delete_bitmap_calculator.h" ^ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #19540: [Improvement](planner) expand sql-block-rule to make it can be used on all kinds of sql stmt
Yulei-Yang commented on PR #19540: URL: https://github.com/apache/doris/pull/19540#issuecomment-1569647038 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #20258: [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property
Tanya-W commented on PR #20258: URL: https://github.com/apache/doris/pull/20258#issuecomment-1569648427 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #20008: [Fix](Planner)fix cast date/datev2/datetime to float/double return null.
sohardforaname commented on PR #20008: URL: https://github.com/apache/doris/pull/20008#issuecomment-1569650035 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 opened a new pull request, #20264: [refactor](stats) Persist status of analyze task to FE meta data
Kikyou1997 opened a new pull request, #20264: URL: https://github.com/apache/doris/pull/20264 ## Proposed changes Issue Number: close #xxx <--Describe your changes.--> ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20258: [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property
github-actions[bot] commented on PR #20258: URL: https://github.com/apache/doris/pull/20258#issuecomment-1569656663 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #20183: [enhancement](ldap) Support refresh ldap cache
yiguolei merged PR #20183: URL: https://github.com/apache/doris/pull/20183 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (f9dfcb923d -> a1e3f49fb5)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from f9dfcb923d [Enhancement] Change Create Resource Group Grammar (#20249)
add a1e3f49fb5 [enhancement](ldap) Support refresh ldap cache (#20183)
No new revisions were added by this update.
Summary of changes:
docs/en/docs/admin-manual/privilege-ldap/ldap.md | 3 ++
.../REFRESH-LDAP.md} | 46 ++
.../zh-CN/docs/admin-manual/privilege-ldap/ldap.md | 3 ++
.../{REFRESH.md => REFRESH-LDAP.md}| 36 ++-
fe/fe-core/src/main/cup/sql_parser.cup | 11 +
.../{CleanLabelStmt.java => RefreshLdapStmt.java} | 54 --
.../java/org/apache/doris/ldap/LdapManager.java| 40 +++-
.../org/apache/doris/ldap/LdapPrivsChecker.java| 2 +-
.../java/org/apache/doris/ldap/LdapUserInfo.java | 17 +++
.../org/apache/doris/mysql/privilege/Auth.java | 15 --
.../apache/doris/mysql/privilege/UserManager.java | 2 +-
.../main/java/org/apache/doris/qe/DdlExecutor.java | 3 ++
fe/fe-core/src/main/jflex/sql_scanner.flex | 1 +
.../apache/doris/ldap/LdapPrivsCheckerTest.java| 8 +++-
14 files changed, 153 insertions(+), 88 deletions(-)
copy
docs/en/docs/sql-manual/sql-reference/{Show-Statements/SHOW-CREATE-TABLE.md =>
Utility-Statements/REFRESH-LDAP.md} (59%)
copy docs/zh-CN/docs/sql-manual/sql-reference/Utility-Statements/{REFRESH.md
=> REFRESH-LDAP.md} (58%)
copy fe/fe-core/src/main/java/org/apache/doris/analysis/{CleanLabelStmt.java
=> RefreshLdapStmt.java} (56%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr opened a new pull request, #20265: [feature-wip](MTMV)Avoid running multiple tasks in one materialized view
zddr opened a new pull request, #20265: URL: https://github.com/apache/doris/pull/20265 ## Proposed changes Issue Number: close #xxx <--Describe your changes.--> Change pendingTaskMap data structure to mvId -> jobId -> pending tasks ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo commented on a diff in pull request #20242: [feature-wip](workload-group) Support for workload group Authentication
wangbo commented on code in PR #20242:
URL: https://github.com/apache/doris/pull/20242#discussion_r1211224684
##
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java:
##
@@ -55,10 +56,12 @@ public class Role implements Writable, GsonPostProcessable {
public static Role OPERATOR = new Role(OPERATOR_ROLE,
TablePattern.ALL, PrivBitSet.of(Privilege.NODE_PRIV,
Privilege.ADMIN_PRIV),
-ResourcePattern.ALL, PrivBitSet.of(Privilege.NODE_PRIV,
Privilege.ADMIN_PRIV));
+ResourcePattern.ALL, PrivBitSet.of(Privilege.NODE_PRIV,
Privilege.ADMIN_PRIV), WorkloadGroupPattern.ALL,
+PrivBitSet.of(Privilege.USAGE_PRIV));
public static Role ADMIN = new Role(ADMIN_ROLE,
TablePattern.ALL, PrivBitSet.of(Privilege.ADMIN_PRIV),
-ResourcePattern.ALL, PrivBitSet.of(Privilege.ADMIN_PRIV));
+ResourcePattern.ALL, PrivBitSet.of(Privilege.ADMIN_PRIV),
WorkloadGroupPattern.ALL,
+PrivBitSet.of(Privilege.USAGE_PRIV));
Review Comment:
Admin only has USAGE_PRIV? not ADMIN_PRIV?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20258: [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property
github-actions[bot] commented on PR #20258: URL: https://github.com/apache/doris/pull/20258#issuecomment-1569665913 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
github-actions[bot] commented on PR #19863: URL: https://github.com/apache/doris/pull/19863#issuecomment-1569669698 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
github-actions[bot] commented on PR #19863: URL: https://github.com/apache/doris/pull/19863#issuecomment-1569669640 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #20264: [refactor](stats) Persist status of analyze task to FE meta data
Kikyou1997 commented on PR #20264: URL: https://github.com/apache/doris/pull/20264#issuecomment-1569669053 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo commented on a diff in pull request #20242: [feature-wip](workload-group) Support for workload group Authentication
wangbo commented on code in PR #20242:
URL: https://github.com/apache/doris/pull/20242#discussion_r1211239509
##
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/WorkloadGroupPrivEntry.java:
##
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mysql.privilege;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.PatternMatcherException;
+import org.apache.doris.common.io.Text;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+public class WorkloadGroupPrivEntry extends PrivEntry {
+protected static final String ANY_WORKLOAD_GROUP = "*";
+
+protected PatternMatcher workloadGroupPattern;
+protected String origWorkloadGroupName;
+protected boolean isAnyWorkloadGroup;
+
+protected WorkloadGroupPrivEntry() {
+}
+
+protected WorkloadGroupPrivEntry(PatternMatcher workloadGroupPattern,
+String origWorkloadGroupName, PrivBitSet privSet) {
+super(privSet);
+this.workloadGroupPattern = workloadGroupPattern;
+this.origWorkloadGroupName = origWorkloadGroupName;
+if (origWorkloadGroupName.equals(ANY_WORKLOAD_GROUP)) {
+isAnyWorkloadGroup = true;
+}
+}
+
+public static WorkloadGroupPrivEntry create(String workloadGroupName,
PrivBitSet privs)
+throws AnalysisException, PatternMatcherException {
+PatternMatcher workloadGroupPattern =
PatternMatcher.createMysqlPattern(
+workloadGroupName.equals(ANY_WORKLOAD_GROUP) ? "%" :
workloadGroupName,
+CaseSensibility.WORKLOAD_GROUP.getCaseSensibility());
+if (privs.containsNodePriv() || privs.containsDbTablePriv()) {
+throw new AnalysisException(
+"Workload group privilege can not contains node or db
table privileges: " + privs);
+}
+return new WorkloadGroupPrivEntry(workloadGroupPattern,
+workloadGroupName, privs);
+}
+
+public PatternMatcher getWorkloadGroupPattern() {
+return workloadGroupPattern;
Review Comment:
it seems ```workloadGroupPatternMatcher``` is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #19863: [Fix](Nereids) fold constant result is wrong on functions relative to timezone
morrySnow merged PR #19863: URL: https://github.com/apache/doris/pull/19863 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Fix](Nereids) fold constant result is wrong on functions relative to timezone (#19863)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b53c42636e [Fix](Nereids) fold constant result is wrong on functions
relative to timezone (#19863)
b53c42636e is described below
commit b53c42636ecb1522c64066ddd293e986df0f395e
Author: mch_ucchi <41606806+sohardforan...@users.noreply.github.com>
AuthorDate: Wed May 31 15:52:40 2023 +0800
[Fix](Nereids) fold constant result is wrong on functions relative to
timezone (#19863)
---
.../expression/rules/FoldConstantRuleOnFE.java | 4 --
.../nereids/trees/expressions/Expression.java | 5 +++
.../trees/expressions/ExpressionEvaluator.java | 2 +-
.../doris/nereids/trees/expressions/Foldable.java | 24 +++
.../functions/executable/DateTimeAcquire.java | 39 --
.../executable/DateTimeExtractAndTransform.java| 25 ++--
.../expressions/functions/scalar/CurrentDate.java | 3 +-
.../expressions/functions/scalar/CurrentTime.java | 3 +-
.../org/apache/doris/nereids/util/DateUtils.java | 9 +
.../nereids_p0/datatype/test_date_acquire.groovy | 46 ++
10 files changed, 121 insertions(+), 39 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnFE.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnFE.java
index 2451b72be5..7c60c8df79 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnFE.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnFE.java
@@ -328,10 +328,6 @@ public class FoldConstantRuleOnFE extends
AbstractExpressionRewriteRule {
@Override
public Expression visitBoundFunction(BoundFunction boundFunction,
ExpressionRewriteContext context) {
boundFunction = rewriteChildren(boundFunction, context);
-//functions, like current_date, do not have arg
-if (boundFunction.getArguments().isEmpty()) {
-return boundFunction;
-}
Optional checkedExpr = preProcess(boundFunction);
if (checkedExpr.isPresent()) {
return checkedExpr.get();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java
index 0d2772d768..e58795e112 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java
@@ -20,6 +20,7 @@ package org.apache.doris.nereids.trees.expressions;
import org.apache.doris.nereids.analyzer.Unbound;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.trees.AbstractTreeNode;
+import org.apache.doris.nereids.trees.expressions.functions.BoundFunction;
import org.apache.doris.nereids.trees.expressions.functions.ExpressionTrait;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
@@ -139,6 +140,10 @@ public abstract class Expression extends
AbstractTreeNode implements
*/
public boolean isConstant() {
if (this instanceof LeafExpression) {
+if (this instanceof BoundFunction) {
+BoundFunction function = ((BoundFunction) this);
+return function instanceof Foldable;
+}
return this instanceof Literal;
} else {
return children().stream().allMatch(Expression::isConstant);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ExpressionEvaluator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ExpressionEvaluator.java
index 80eb82f3a0..b6b2441c99 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ExpressionEvaluator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ExpressionEvaluator.java
@@ -137,11 +137,11 @@ public enum ExpressionEvaluator {
ImmutableMultimap.Builder mapBuilder =
new ImmutableMultimap.Builder();
List classes = ImmutableList.of(
+DateTimeAcquire.class,
DateTimeExtractAndTransform.class,
ExecutableFunctions.class,
DateLiteral.class,
DateTimeArithmetic.class,
-DateTimeAcquire.class,
NumericArithmetic.class
);
for (Class cls : classes) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Foldable.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Foldab
[GitHub] [doris] shuke987 commented on pull request #20222: [fix][regression-test] set timeout of curl in regression test to avoid hanged when be crashed.
shuke987 commented on PR #20222: URL: https://github.com/apache/doris/pull/20222#issuecomment-1569679076 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20233: [Bug](memtable) fix a bug occurred when we were inserting data into duplicate table without keys
github-actions[bot] commented on PR #20233: URL: https://github.com/apache/doris/pull/20233#issuecomment-1569683199 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo commented on a diff in pull request #20242: [feature-wip](workload-group) Support for workload group Authentication
wangbo commented on code in PR #20242:
URL: https://github.com/apache/doris/pull/20242#discussion_r1211246015
##
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/WorkloadGroupPrivEntry.java:
##
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mysql.privilege;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.PatternMatcherException;
+import org.apache.doris.common.io.Text;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+public class WorkloadGroupPrivEntry extends PrivEntry {
+protected static final String ANY_WORKLOAD_GROUP = "*";
+
+protected PatternMatcher workloadGroupPattern;
+protected String origWorkloadGroupName;
+protected boolean isAnyWorkloadGroup;
+
+protected WorkloadGroupPrivEntry() {
+}
+
+protected WorkloadGroupPrivEntry(PatternMatcher workloadGroupPattern,
+String origWorkloadGroupName, PrivBitSet privSet) {
+super(privSet);
+this.workloadGroupPattern = workloadGroupPattern;
+this.origWorkloadGroupName = origWorkloadGroupName;
+if (origWorkloadGroupName.equals(ANY_WORKLOAD_GROUP)) {
+isAnyWorkloadGroup = true;
+}
+}
+
+public static WorkloadGroupPrivEntry create(String workloadGroupName,
PrivBitSet privs)
+throws AnalysisException, PatternMatcherException {
+PatternMatcher workloadGroupPattern =
PatternMatcher.createMysqlPattern(
+workloadGroupName.equals(ANY_WORKLOAD_GROUP) ? "%" :
workloadGroupName,
+CaseSensibility.WORKLOAD_GROUP.getCaseSensibility());
+if (privs.containsNodePriv() || privs.containsDbTablePriv()) {
Review Comment:
```suggestion
if (privs.containsNodePriv() || privs.containsDbTablePriv() ||
privs.containsResourcePriv()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20233: [Bug](memtable) fix a bug occurred when we were inserting data into duplicate table without keys
github-actions[bot] commented on PR #20233: URL: https://github.com/apache/doris/pull/20233#issuecomment-1569683266 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] airborne12 opened a new pull request, #20266: [Docs](inverted index) update docs for inverted index parser_mode and match_phrase support
airborne12 opened a new pull request, #20266: URL: https://github.com/apache/doris/pull/20266 ## Proposed changes Issue Number: close #xxx <--Describe your changes.--> ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] airborne12 commented on pull request #20266: [Docs](inverted index) update docs for inverted index parser_mode and match_phrase support
airborne12 commented on PR #20266: URL: https://github.com/apache/doris/pull/20266#issuecomment-1569703495 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wzymumon opened a new issue, #20267: [Feature] Support Wasm UDF
wzymumon opened a new issue, #20267: URL: https://github.com/apache/doris/issues/20267 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description ## Backgroup Doris currently supports both [Native UDF](https://doris.apache.org/zh-CN/docs/dev/ecosystem/udf/native-user-defined-function), [Remote UDF](https://doris.apache.org/zh-CN/docs/dev/ecosystem/udf/remote-user-defined-function) and [Java UDF](https://doris.apache.org/zh-CN/docs/dev/ecosystem/udf/java-user-defined-function) for user-defined functions. Native UDF is written in C++ and has the best performance, but it is more difficult to write and debug, and may be limited by some system library versions(such as libc) that may not be compatible after upgrade. Remote UDF solves the language problem very well, and in theory, UDF logic can be written in any language. But the disadvantage is that users need to implement their own high-performance UDF Service, and the efficiency is not good because of the RPC problem. Java UDF is the main user-defined function solution in Doris. which reduces the migration cost of big data ecological users(Some big data ecologies such as Hive, Spark, etc. already exist a large number of ready-made UDF). The implementation of Java UDF is to start the JVM and call the relevant UDF logic from the BE side through JNI. ## Motivation Support for Wasm UDF is motivated by the following points: 1. Embeddable in [multiple programming languages. Users can write functional logic in multiple programming languages (Rust, C/C++, Golang, Java, TypeScript, Haskell) 2. Secure by default. No file, network, or environment access, unless explicitly enabled. 3. High-performance. WebAssembly engine compiles bytecode into machine-native machine code for execution instead of interpreting it, which greatly improves execution efficiency and can achieve an efficiency close to native execution. ## How to implement Need to start the Wasm Runtime and call the relevant UDF logic from the BE side by way of wasmtime-c-api. ## Scheduling I have a preliminary plan to support Wasm UDF in Doris. Phase I, I'll complete Create-Function-Statement for Wasm UDF. Phase II, I'll add wasmtime lib and wasmtime-c-api to BE, and implement basic data type based on Wasm Basic ABI Type. ### Use case _No response_ ### Related issues Remote UDF: https://github.com/apache/doris/pull/7519 Lua UDF: https://github.com/apache/doris/pull/5979 Java UDF: https://github.com/apache/doris/issues/8389 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Mryange commented on pull request #20247: [feature](datetime) "timediff" supports calculating microseconds.
Mryange commented on PR #20247: URL: https://github.com/apache/doris/pull/20247#issuecomment-1569706931 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
yiguolei merged PR #19973: URL: https://github.com/apache/doris/pull/19973 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](bitmap) Using set to store a small number of elements to improve performance (#19973)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c03a19ea23 [improvement](bitmap) Using set to store a small number of
elements to improve performance (#19973)
c03a19ea23 is described below
commit c03a19ea23cd5ac1b75a0d130709a7bc6303073c
Author: Jerry Hu
AuthorDate: Wed May 31 16:13:42 2023 +0800
[improvement](bitmap) Using set to store a small number of elements to
improve performance (#19973)
Test on SSB 100g:
select lo_suppkey, count(distinct lo_linenumber) from lineorder group by
lo_suppkey;
exec time: 4.388s
create materialized view:
create materialized view customer_uv as select lo_suppkey,
bitmap_union(to_bitmap(lo_linenumber)) from lineorder group by lo_suppkey;
select lo_suppkey, count(distinct lo_linenumber) from lineorder group by
lo_suppkey;
exec time: 12.908s
test with the patch, exec time: 5.790s
---
be/src/common/config.cpp | 3 +
be/src/common/config.h | 3 +
be/src/util/bitmap_value.h | 738 ---
be/src/vec/functions/function_bitmap.cpp | 10 +-
be/test/util/bitmap_value_test.cpp | 22 -
be/test/vec/core/column_complex_test.cpp | 5 +-
regression-test/pipeline/p0/conf/be.conf | 1 +
regression-test/pipeline/p1/conf/be.conf | 1 +
8 files changed, 688 insertions(+), 95 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 90478001dc..04c29e8dd7 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1010,6 +1010,9 @@ DEFINE_mInt32(schema_cache_sweep_time_sec, "100");
// enable feature binlog, default false
DEFINE_Bool(enable_feature_binlog, "false");
+// enable set in BitmapValue
+DEFINE_Bool(enable_set_in_bitmap_value, "false");
+
#ifdef BE_TEST
// test s3
DEFINE_String(test_s3_resource, "resource");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index e06c40afbe..c1d51cfb3b 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1025,6 +1025,9 @@ DECLARE_mInt32(schema_cache_sweep_time_sec);
// enable binlog
DECLARE_Bool(enable_feature_binlog);
+// enable set in BitmapValue
+DECLARE_Bool(enable_set_in_bitmap_value);
+
#ifdef BE_TEST
// test s3
DECLARE_String(test_s3_resource);
diff --git a/be/src/util/bitmap_value.h b/be/src/util/bitmap_value.h
index 6b660b8caf..5ada50fb59 100644
--- a/be/src/util/bitmap_value.h
+++ b/be/src/util/bitmap_value.h
@@ -18,6 +18,7 @@
#pragma once
#include
+#include
#include
#include
@@ -28,10 +29,12 @@
#include
#include
#include
+#include
#include
#include
#include
+#include "common/config.h"
#include "common/logging.h"
#include "gutil/integral_types.h"
#include "udf/udf.h"
@@ -72,10 +75,11 @@ struct BitmapTypeCode {
// - MapValue := the standard RoaringBitmap format
//
// added in 0.12
-BITMAP64 = 4
+BITMAP64 = 4,
+SET = 5
};
Status static inline validate(int bitmap_type) {
-if (UNLIKELY(bitmap_type < type::EMPTY || bitmap_type >
type::BITMAP64)) {
+if (UNLIKELY(bitmap_type < type::EMPTY || bitmap_type > type::SET)) {
std::string err_msg =
fmt::format("BitmapTypeCode invalid, should between: {}
and {} actrual is {}",
BitmapTypeCode::EMPTY,
BitmapTypeCode::BITMAP64, bitmap_type);
@@ -1143,6 +1147,9 @@ inline Roaring64MapSetBitForwardIterator
Roaring64Map::end() const {
class BitmapValueIterator;
class BitmapValue {
public:
+template
+using SetContainer = phmap::flat_hash_set;
+
// Construct an empty bitmap.
BitmapValue() : _type(EMPTY), _is_shared(false) {}
@@ -1150,7 +1157,7 @@ public:
explicit BitmapValue(uint64_t value) : _sv(value), _type(SINGLE),
_is_shared(false) {}
// Construct a bitmap from serialized data.
-explicit BitmapValue(const char* src) {
+explicit BitmapValue(const char* src) : _is_shared(false) {
bool res = deserialize(src);
DCHECK(res);
}
@@ -1159,6 +1166,7 @@ public:
_type = other._type;
_sv = other._sv;
_bitmap = other._bitmap;
+_set = other._set;
_is_shared = true;
// should also set other's state to shared, so that other bitmap value
will
// create a new bitmap when it wants to modify it.
@@ -1170,6 +1178,7 @@ public:
_sv = other._sv;
_is_shared = other._is_shared;
_bitmap = std::move(other._bitmap);
+_set = std::move(other._set);
}
BitmapValue& operator=(const BitmapValue& other) {
@@ -1177,6 +1186,7 @@ public:
_sv = other._sv;
_bitmap = other._bitmap;
_is_shared = true;
+_set = other._set;
// sh
[GitHub] [doris] yuxuan-luo commented on pull request #20205: [fix](http) Let the sdk find the httpclient package determined
yuxuan-luo commented on PR #20205: URL: https://github.com/apache/doris/pull/20205#issuecomment-1569714669 run all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #20153: [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments
zhannngchen commented on code in PR #20153:
URL: https://github.com/apache/doris/pull/20153#discussion_r1211273429
##
be/src/olap/delete_bitmap_calculator.cpp:
##
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/delete_bitmap_calculator.h"
+
+#include "olap/primary_key_index.h"
+#include "vec/data_types/data_type_factory.hpp"
+
+namespace doris {
+
+Status MergeIndexDeleteBitmapCalculatorContext::get_current_key(Slice& slice) {
+if (_cur_row_id >= _num_rows) {
+return Status::EndOfFile("Reach the end of file");
+}
+if (_cur_pos >= _block_size) {
+RETURN_IF_ERROR(_iter->seek_to_ordinal(_cur_row_id));
+RETURN_IF_ERROR(_next_batch(_cur_row_id));
+}
+slice = Slice(_index_column->get_data_at(_cur_pos).data,
+ _index_column->get_data_at(_cur_pos).size);
+return Status::OK();
+}
+
+Status MergeIndexDeleteBitmapCalculatorContext::advance() {
+++_cur_pos;
+++_cur_row_id;
+if (_cur_row_id >= _num_rows) {
+return Status::EndOfFile("Reach the end of file");
+}
+return Status::OK();
+}
+
+Status MergeIndexDeleteBitmapCalculatorContext::seek_at_or_after(Slice const&
key) {
+auto st = _iter->seek_at_or_after(&key, &_excat_match);
+if (st.is()) {
+return Status::EndOfFile("Reach the end of file");
+}
+RETURN_IF_ERROR(st);
+auto current_ordinal = _iter->get_current_ordinal();
+DCHECK(current_ordinal > _cur_row_id)
+<< fmt::format("current_ordinal: {} should be greater than
_cur_row_id: {}",
+ current_ordinal, _cur_row_id);
+// if key is still in the block read before,
+// in other words, if `_cur_pos + current_ordinal - _cur_row_id <
_block_size` holds
+// we can seek simply by moving the pointers, aka. _cur_pos and _cur_row_id
+if (_cur_pos + current_ordinal - _cur_row_id < _block_size) {
+_cur_pos = _cur_pos + current_ordinal - _cur_row_id;
+_cur_row_id = current_ordinal;
+return Status::OK();
+}
+// otherwise, we have to read the data starts from `current_ordinal`
+return _next_batch(current_ordinal);
+}
+
+Status MergeIndexDeleteBitmapCalculatorContext::_next_batch(size_t row_id) {
+// _iter should be seeked before calling this function
+DCHECK(row_id < _num_rows) << fmt::format("row_id: {} should be less than
_num_rows: {}",
+ row_id, _num_rows);
+_index_column = _index_type->create_column();
+auto remaining = _num_rows - row_id;
+size_t num_to_read = std::min(_max_batch_size, remaining);
+size_t num_read = num_to_read;
+RETURN_IF_ERROR(_iter->next_batch(&num_read, _index_column));
+DCHECK(num_to_read == num_read) << fmt::format(
+"num_to_read: {} should be equal to num_to_read: {}", num_to_read,
num_read);
+_block_size = num_read;
+_cur_pos = 0;
+_cur_row_id = row_id;
+return Status::OK();
+}
+
+bool MergeIndexDeleteBitmapCalculatorContext::Comparator::operator()(
+MergeIndexDeleteBitmapCalculatorContext* lhs,
+MergeIndexDeleteBitmapCalculatorContext* rhs) const {
+// std::proiroty_queue is a max heap, and function should return the
result of `lhs < rhs`
+// so if the result of the function is true, rhs will be popped before lhs
+Slice key1, key2;
+RETURN_IF_ERROR(lhs->get_current_key(key1));
+RETURN_IF_ERROR(rhs->get_current_key(key2));
+if (_sequence_length == 0) {
+auto cmp_result = key1.compare(key2);
+// when key1 is the same as key2,
+// we want the one with greater segment id to be popped first
+return cmp_result ? (cmp_result > 0) : (lhs->segment_id() <
rhs->segment_id());
+}
+// smaller key popped first
+auto key1_without_seq = Slice(key1.get_data(), key1.get_size() -
_sequence_length);
+auto key2_without_seq = Slice(key2.get_data(), key2.get_size() -
_sequence_length);
+auto cmp_result = key1_without_seq.compare(key2_without_seq);
+if (cmp_result != 0) {
+return cmp_result > 0;
+}
+// greater sequence value popped first
+auto key1_sequence_val =
+Slic
[GitHub] [doris] zhannngchen commented on a diff in pull request #20153: [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments
zhannngchen commented on code in PR #20153:
URL: https://github.com/apache/doris/pull/20153#discussion_r1211273767
##
be/src/olap/delete_bitmap_calculator.cpp:
##
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/delete_bitmap_calculator.h"
+
+#include "olap/primary_key_index.h"
+#include "vec/data_types/data_type_factory.hpp"
+
+namespace doris {
+
+Status MergeIndexDeleteBitmapCalculatorContext::get_current_key(Slice& slice) {
+if (_cur_row_id >= _num_rows) {
+return Status::EndOfFile("Reach the end of file");
+}
+if (_cur_pos >= _block_size) {
+RETURN_IF_ERROR(_iter->seek_to_ordinal(_cur_row_id));
+RETURN_IF_ERROR(_next_batch(_cur_row_id));
+}
+slice = Slice(_index_column->get_data_at(_cur_pos).data,
+ _index_column->get_data_at(_cur_pos).size);
+return Status::OK();
+}
+
+Status MergeIndexDeleteBitmapCalculatorContext::advance() {
+++_cur_pos;
+++_cur_row_id;
+if (_cur_row_id >= _num_rows) {
+return Status::EndOfFile("Reach the end of file");
+}
+return Status::OK();
+}
+
+Status MergeIndexDeleteBitmapCalculatorContext::seek_at_or_after(Slice const&
key) {
+auto st = _iter->seek_at_or_after(&key, &_excat_match);
+if (st.is()) {
+return Status::EndOfFile("Reach the end of file");
+}
+RETURN_IF_ERROR(st);
+auto current_ordinal = _iter->get_current_ordinal();
+DCHECK(current_ordinal > _cur_row_id)
+<< fmt::format("current_ordinal: {} should be greater than
_cur_row_id: {}",
+ current_ordinal, _cur_row_id);
+// if key is still in the block read before,
+// in other words, if `_cur_pos + current_ordinal - _cur_row_id <
_block_size` holds
+// we can seek simply by moving the pointers, aka. _cur_pos and _cur_row_id
+if (_cur_pos + current_ordinal - _cur_row_id < _block_size) {
+_cur_pos = _cur_pos + current_ordinal - _cur_row_id;
+_cur_row_id = current_ordinal;
+return Status::OK();
+}
+// otherwise, we have to read the data starts from `current_ordinal`
+return _next_batch(current_ordinal);
+}
+
+Status MergeIndexDeleteBitmapCalculatorContext::_next_batch(size_t row_id) {
+// _iter should be seeked before calling this function
+DCHECK(row_id < _num_rows) << fmt::format("row_id: {} should be less than
_num_rows: {}",
+ row_id, _num_rows);
+_index_column = _index_type->create_column();
+auto remaining = _num_rows - row_id;
+size_t num_to_read = std::min(_max_batch_size, remaining);
+size_t num_read = num_to_read;
+RETURN_IF_ERROR(_iter->next_batch(&num_read, _index_column));
+DCHECK(num_to_read == num_read) << fmt::format(
+"num_to_read: {} should be equal to num_to_read: {}", num_to_read,
num_read);
+_block_size = num_read;
+_cur_pos = 0;
+_cur_row_id = row_id;
+return Status::OK();
+}
+
+bool MergeIndexDeleteBitmapCalculatorContext::Comparator::operator()(
+MergeIndexDeleteBitmapCalculatorContext* lhs,
+MergeIndexDeleteBitmapCalculatorContext* rhs) const {
+// std::proiroty_queue is a max heap, and function should return the
result of `lhs < rhs`
+// so if the result of the function is true, rhs will be popped before lhs
+Slice key1, key2;
+RETURN_IF_ERROR(lhs->get_current_key(key1));
+RETURN_IF_ERROR(rhs->get_current_key(key2));
+if (_sequence_length == 0) {
+auto cmp_result = key1.compare(key2);
+// when key1 is the same as key2,
+// we want the one with greater segment id to be popped first
+return cmp_result ? (cmp_result > 0) : (lhs->segment_id() <
rhs->segment_id());
+}
+// smaller key popped first
+auto key1_without_seq = Slice(key1.get_data(), key1.get_size() -
_sequence_length);
+auto key2_without_seq = Slice(key2.get_data(), key2.get_size() -
_sequence_length);
+auto cmp_result = key1_without_seq.compare(key2_without_seq);
+if (cmp_result != 0) {
+return cmp_result > 0;
+}
+// greater sequence value popped first
+auto key1_sequence_val =
+Slic
[GitHub] [doris] github-actions[bot] commented on pull request #20247: [feature](datetime) "timediff" supports calculating microseconds.
github-actions[bot] commented on PR #20247: URL: https://github.com/apache/doris/pull/20247#issuecomment-1569721133 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on pull request #20239: [fix](dynamic partition) partition create failed after modify column comment
cambyzju commented on PR #20239: URL: https://github.com/apache/doris/pull/20239#issuecomment-1569725003 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #20153: [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments
zhannngchen commented on code in PR #20153:
URL: https://github.com/apache/doris/pull/20153#discussion_r1211279533
##
be/src/olap/tablet.cpp:
##
@@ -3264,4 +3241,33 @@ bool Tablet::is_enable_binlog() {
void Tablet::set_binlog_config(BinlogConfig binlog_config) {
tablet_meta()->set_binlog_config(std::move(binlog_config));
}
+
+// caller should hold meta_lock
+Status Tablet::calc_delete_bitmap_between_segments(
+RowsetSharedPtr rowset, const
std::vector& segments,
+DeleteBitmapPtr delete_bitmap) {
+size_t const num_segments = segments.size();
+if (num_segments < 2) {
+return Status::OK();
+}
+
+OlapStopWatch watch;
+auto const rowset_id = rowset->rowset_id();
+size_t seq_col_length = 0;
+if (_schema->has_sequence_col()) {
+auto seq_col_idx = _schema->sequence_col_idx();
+seq_col_length = _schema->column(seq_col_idx).length();
+}
+
+MergeIndexDeleteBitmapCalculator calculator;
+RETURN_IF_ERROR(calculator.init(rowset_id, segments, seq_col_length));
+
+RETURN_IF_ERROR(calculator.calculate_all(delete_bitmap));
+
+LOG(INFO) << fmt::format(
Review Comment:
Also print the tablet id and rowset id, which is friendly for debugging.
Pls refer to the log at the end of `Tablet::calc_delete_bitmap`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20153: [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments
github-actions[bot] commented on PR #20153: URL: https://github.com/apache/doris/pull/20153#issuecomment-1569728815 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20153: [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments
github-actions[bot] commented on PR #20153: URL: https://github.com/apache/doris/pull/20153#issuecomment-1569728972 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20164: [Feature](agg_state) support agg_state combinator on nereids
github-actions[bot] commented on PR #20164: URL: https://github.com/apache/doris/pull/20164#issuecomment-1569732739 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20248: [Improve](json-array)Support json array with nereids bool
github-actions[bot] commented on PR #20248: URL: https://github.com/apache/doris/pull/20248#issuecomment-1569739420 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20248: [Improve](json-array)Support json array with nereids bool
github-actions[bot] commented on PR #20248: URL: https://github.com/apache/doris/pull/20248#issuecomment-1569739475 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #19807: [Fix](Nereids) Fix bitmap type in compare operator bug
morrySnow commented on code in PR #19807: URL: https://github.com/apache/doris/pull/19807#discussion_r1211291759 ## fe/fe-core/src/main/java/org/apache/doris/nereids/fe.pid: ## Review Comment: remove this file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #19807: [Fix](Nereids) Fix bitmap type in compare operator bug
LiBinfeng-01 commented on code in PR #19807: URL: https://github.com/apache/doris/pull/19807#discussion_r1211300285 ## fe/fe-core/src/main/java/org/apache/doris/nereids/fe.pid: ## Review Comment: get -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #20258: [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property
Tanya-W commented on PR #20258: URL: https://github.com/apache/doris/pull/20258#issuecomment-1569756974 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #19807: [Fix](Nereids) Fix bitmap type in compare operator bug
LiBinfeng-01 commented on PR #19807: URL: https://github.com/apache/doris/pull/19807#issuecomment-1569757104 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #20258: [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property
Tanya-W commented on PR #20258: URL: https://github.com/apache/doris/pull/20258#issuecomment-1569757402 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin commented on pull request #20242: [feature-wip](workload-group) Support for workload group Authentication
luozenglin commented on PR #20242: URL: https://github.com/apache/doris/pull/20242#issuecomment-1569770534 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] kaijchen commented on pull request #20178: [performance](load) support parallel memtable flush for unique key tables
kaijchen commented on PR #20178: URL: https://github.com/apache/doris/pull/20178#issuecomment-1569774239 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wolfboys commented on a diff in pull request #20109: [refactor](jdbc catalog) Refactored JDBC Client into Abstract Base Class and Database-specific Subclasses
wolfboys commented on code in PR #20109:
URL: https://github.com/apache/doris/pull/20109#discussion_r1211313763
##
fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcMySQLClient.java:
##
@@ -0,0 +1,177 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.external.jdbc;
+
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Type;
+
+import avro.shaded.com.google.common.collect.Lists;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.function.Consumer;
+
+public class JdbcMySQLClient extends JdbcClient {
+protected JdbcMySQLClient(JdbcClientConfig jdbcClientConfig) {
+super(jdbcClientConfig);
+}
+
+@Override
+protected String getDatabaseQuery() {
+return "SHOW DATABASES";
+}
+
+@Override
+protected List getSpecifiedDatabase(Connection conn) {
+List databaseNames = Lists.newArrayList();
+try {
+databaseNames.add(conn.getCatalog());
+} catch (SQLException e) {
+throw new JdbcClientException("failed to get specified database
name from jdbc", e);
+} finally {
+close(conn);
+}
+return databaseNames;
+}
+
+@Override
+protected void processTable(String dbName, String tableName, String[]
tableTypes,
+Consumer resultSetConsumer) {
+Connection conn = getConnection();
+ResultSet rs = null;
+try {
+DatabaseMetaData databaseMetaData = conn.getMetaData();
+rs = databaseMetaData.getTables(dbName, null, tableName,
tableTypes);
+resultSetConsumer.accept(rs);
+} catch (SQLException e) {
+throw new JdbcClientException("Failed to process table", e);
+} finally {
+close(rs, conn);
+}
+}
+
+@Override
+protected ResultSet getColumns(DatabaseMetaData databaseMetaData, String
catalogName, String schemaName,
+ String tableName) throws SQLException {
+return databaseMetaData.getColumns(schemaName, null, tableName, null);
+}
+
+@Override
+protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) {
+// For mysql type: "INT UNSIGNED":
+// fieldSchema.getDataTypeName().split(" ")[0] == "INT"
+// fieldSchema.getDataTypeName().split(" ")[1] == "UNSIGNED"
+String[] typeFields = fieldSchema.getDataTypeName().split(" ");
Review Comment:
We can improve the code:
`fieldSchema.getDataTypeName().split("\\s+")`
##
fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcOceanBaseClient.java:
##
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.external.jdbc;
+
+import org.apache.doris.catalog.JdbcResource;
+import org.apache.doris.catalog.Type;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+
+public class JdbcOceanBaseClient extends JdbcClient {
+private JdbcClient currentClient;
+
+public JdbcOceanBaseClient(JdbcClientConfig jdbcClientConfig) {
+super(jdbcClientConfig);
+
+Connection conn = super.getConnection();
+
+try {
+Stateme
[GitHub] [doris] github-actions[bot] commented on pull request #20178: [performance](load) support parallel memtable flush for unique key tables
github-actions[bot] commented on PR #20178: URL: https://github.com/apache/doris/pull/20178#issuecomment-1569778961 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin commented on a diff in pull request #20242: [feature-wip](workload-group) Support for workload group Authentication
luozenglin commented on code in PR #20242:
URL: https://github.com/apache/doris/pull/20242#discussion_r1211322366
##
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/WorkloadGroupPrivEntry.java:
##
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mysql.privilege;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.PatternMatcherException;
+import org.apache.doris.common.io.Text;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+public class WorkloadGroupPrivEntry extends PrivEntry {
+protected static final String ANY_WORKLOAD_GROUP = "*";
+
+protected PatternMatcher workloadGroupPattern;
+protected String origWorkloadGroupName;
+protected boolean isAnyWorkloadGroup;
+
+protected WorkloadGroupPrivEntry() {
+}
+
+protected WorkloadGroupPrivEntry(PatternMatcher workloadGroupPattern,
+String origWorkloadGroupName, PrivBitSet privSet) {
+super(privSet);
+this.workloadGroupPattern = workloadGroupPattern;
+this.origWorkloadGroupName = origWorkloadGroupName;
+if (origWorkloadGroupName.equals(ANY_WORKLOAD_GROUP)) {
+isAnyWorkloadGroup = true;
+}
+}
+
+public static WorkloadGroupPrivEntry create(String workloadGroupName,
PrivBitSet privs)
+throws AnalysisException, PatternMatcherException {
+PatternMatcher workloadGroupPattern =
PatternMatcher.createMysqlPattern(
+workloadGroupName.equals(ANY_WORKLOAD_GROUP) ? "%" :
workloadGroupName,
+CaseSensibility.WORKLOAD_GROUP.getCaseSensibility());
+if (privs.containsNodePriv() || privs.containsDbTablePriv()) {
+throw new AnalysisException(
+"Workload group privilege can not contains node or db
table privileges: " + privs);
+}
+return new WorkloadGroupPrivEntry(workloadGroupPattern,
+workloadGroupName, privs);
+}
+
+public PatternMatcher getWorkloadGroupPattern() {
+return workloadGroupPattern;
Review Comment:
This is consistent with other PrivEntry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19924: [improvement](pipeline) task group scan entity
github-actions[bot] commented on PR #19924: URL: https://github.com/apache/doris/pull/19924#issuecomment-1569779105 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiongjx751 commented on pull request #20153: [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments
xiongjx751 commented on PR #20153: URL: https://github.com/apache/doris/pull/20153#issuecomment-1569781158 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #20052: [Fix](Nereids)fix insert-into-table bugs.
sohardforaname commented on PR #20052: URL: https://github.com/apache/doris/pull/20052#issuecomment-1569789951 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
github-actions[bot] commented on PR #19681: URL: https://github.com/apache/doris/pull/19681#issuecomment-1569793220 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LiBinfeng-01 opened a new pull request, #20268: [Fix](Nereids) Fix minidump using put all of hashmap
LiBinfeng-01 opened a new pull request, #20268: URL: https://github.com/apache/doris/pull/20268 ## Proposed changes Minidump file wants to get information as much as possible, but when close the switch, these methods should not be called after refactor pr: https://github.com/apache/doris/pull/20049. Other place of doing more jobs after add Minidump feature also be checked. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LiBinfeng-01 closed pull request #19709: [Enhence](Nereids) Add minidump result detail compare and output
LiBinfeng-01 closed pull request #19709: [Enhence](Nereids) Add minidump result detail compare and output URL: https://github.com/apache/doris/pull/19709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr commented on pull request #20108: [fix](mtmv)all mv add manual job
zddr commented on PR #20108: URL: https://github.com/apache/doris/pull/20108#issuecomment-1569805083 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20112: [runtimeFilter](nereids)generate in-bloom filter if target is local for pipeline mode
github-actions[bot] commented on PR #20112: URL: https://github.com/apache/doris/pull/20112#issuecomment-1569810612 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20112: [runtimeFilter](nereids)generate in-bloom filter if target is local for pipeline mode
github-actions[bot] commented on PR #20112: URL: https://github.com/apache/doris/pull/20112#issuecomment-1569810692 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wolfboys commented on a diff in pull request #20109: [refactor](jdbc catalog) Refactored JDBC Client into Abstract Base Class and Database-specific Subclasses
wolfboys commented on code in PR #20109:
URL: https://github.com/apache/doris/pull/20109#discussion_r1211352554
##
fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcMySQLClient.java:
##
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.external.jdbc;
+
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Type;
+
+import avro.shaded.com.google.common.collect.Lists;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.function.Consumer;
+
+public class JdbcMySQLClient extends JdbcClient {
+protected JdbcMySQLClient(JdbcClientConfig jdbcClientConfig) {
+super(jdbcClientConfig);
+}
+
+@Override
+protected String getDatabaseQuery() {
+return "SHOW DATABASES";
+}
+
+@Override
+protected List getSpecifiedDatabase(Connection conn) {
+List databaseNames = Lists.newArrayList();
+try {
+databaseNames.add(conn.getCatalog());
+} catch (SQLException e) {
+throw new JdbcClientException("failed to get specified database
name from jdbc", e);
+} finally {
+close(conn);
+}
+return databaseNames;
+}
+
+@Override
+protected void processTable(String dbName, String tableName, String[]
tableTypes,
+Consumer resultSetConsumer) {
+Connection conn = getConnection();
Review Comment:
Connection conn = null;
##
fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcMySQLClient.java:
##
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.external.jdbc;
+
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Type;
+
+import avro.shaded.com.google.common.collect.Lists;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.function.Consumer;
+
+public class JdbcMySQLClient extends JdbcClient {
+protected JdbcMySQLClient(JdbcClientConfig jdbcClientConfig) {
+super(jdbcClientConfig);
+}
+
+@Override
+protected String getDatabaseQuery() {
+return "SHOW DATABASES";
+}
+
+@Override
+protected List getSpecifiedDatabase(Connection conn) {
+List databaseNames = Lists.newArrayList();
+try {
+databaseNames.add(conn.getCatalog());
+} catch (SQLException e) {
+throw new JdbcClientException("failed to get specified database
name from jdbc", e);
+} finally {
+close(conn);
+}
+return databaseNames;
+}
+
+@Override
+protected void processTable(String dbName, String tableName, String[]
tableTypes,
+Consumer resultSetConsumer) {
+Connection conn = getConnection();
+ResultSet rs = null;
+try {
+DatabaseMetaData databaseMetaData = conn.getMetaData();
Review Comment:
before 62 line:
`conn = getConnection();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th
[GitHub] [doris] morrySnow merged pull request #20112: [runtimeFilter](nereids)generate in-bloom filter if target is local for pipeline mode
morrySnow merged PR #20112: URL: https://github.com/apache/doris/pull/20112 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [opt](nereids) generate in-bloom filter if target is local for pipeline mode (#20112)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5f591a6d12 [opt](nereids) generate in-bloom filter if target is local
for pipeline mode (#20112)
5f591a6d12 is described below
commit 5f591a6d1234a71b4de4a2487737a86f949236a9
Author: minghong
AuthorDate: Wed May 31 17:24:38 2023 +0800
[opt](nereids) generate in-bloom filter if target is local for pipeline
mode (#20112)
update in-filter usage in pipeline mode:
1. if the target is local, we use in-bloom filter. Let BE choose in or
bloom according to actual distinctive number
2. set default runtime_filter_max_in_num to 1024
---
.../nereids/processor/post/FragmentProcessor.java | 52 ++
.../nereids/processor/post/PlanPostProcessors.java | 1 +
.../processor/post/RuntimeFilterContext.java | 5 ++-
.../processor/post/RuntimeFilterGenerator.java | 36 ++-
.../doris/nereids/trees/plans/AbstractPlan.java| 1 +
.../trees/plans/physical/PhysicalHashJoin.java | 4 +-
.../trees/plans/physical/PhysicalOlapScan.java | 3 +-
.../java/org/apache/doris/qe/SessionVariable.java | 2 +-
8 files changed, 88 insertions(+), 16 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/FragmentProcessor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/FragmentProcessor.java
new file mode 100644
index 00..ddc0e76067
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/FragmentProcessor.java
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.processor.post;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.trees.plans.AbstractPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan;
+
+/**
+ * generate fragment id for nereids physical plan
+ */
+public class FragmentProcessor extends PlanPostProcessor {
+private int frId = 0;
+
+public PhysicalDistribute visitPhysicalDistribute(PhysicalDistribute distribute,
+CascadesContext ctx) {
+frId++;
+distribute.child().accept(this, ctx);
+return distribute;
+}
+
+public PhysicalHashJoin visitPhysicalHashJoin(PhysicalHashJoin join,
+CascadesContext ctx) {
+join.setMutableState(AbstractPlan.FRAGMENT_ID, frId);
+join.left().accept(this, ctx);
+join.right().accept(this, ctx);
+return join;
+}
+
+public PhysicalOlapScan visitPhysicalOlapScan(PhysicalOlapScan scan,
CascadesContext ctx) {
+scan.setMutableState(AbstractPlan.FRAGMENT_ID, frId);
+return scan;
+}
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
index b53d316f1a..4e72b8738d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
@@ -60,6 +60,7 @@ public class PlanPostProcessors {
Builder builder = ImmutableList.builder();
builder.add(new MergeProjectPostProcessor());
builder.add(new PushdownFilterThroughProject());
+builder.add(new FragmentProcessor());
if
(!cascadesContext.getConnectContext().getSessionVariable().getRuntimeFilterMode()
.toUpperCase().equals(TRuntimeFilterMode.OFF.name())) {
builder.add(new RuntimeFilterGenerator());
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java
index 70b32d50ee..69986cde67 10
[GitHub] [doris] xiongjx751 commented on pull request #20153: [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments
xiongjx751 commented on PR #20153: URL: https://github.com/apache/doris/pull/20153#issuecomment-1569832227 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 opened a new pull request, #20269: [fix](nereids) Fix select partition
Kikyou1997 opened a new pull request, #20269: URL: https://github.com/apache/doris/pull/20269 ## Proposed changes This PR is to fix the select specific partition issue, certain codes related to this feature were accidentally deleted. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #20269: [fix](nereids) Fix select partition
Kikyou1997 commented on PR #20269: URL: https://github.com/apache/doris/pull/20269#issuecomment-1569834768 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20006: [feature](load) support single-stream-multi-table
github-actions[bot] commented on PR #20006: URL: https://github.com/apache/doris/pull/20006#issuecomment-1569835550 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #20269: [fix](nereids) Fix select partition
Kikyou1997 commented on PR #20269: URL: https://github.com/apache/doris/pull/20269#issuecomment-1569839710 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20006: [feature](load) support single-stream-multi-table
github-actions[bot] commented on PR #20006: URL: https://github.com/apache/doris/pull/20006#issuecomment-1569844896 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liutang123 commented on pull request #19924: [improvement](pipeline) task group scan entity
liutang123 commented on PR #19924: URL: https://github.com/apache/doris/pull/19924#issuecomment-1569860784 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on pull request #20248: [Improve](json-array)Support json array with nereids bool
amorynan commented on PR #20248: URL: https://github.com/apache/doris/pull/20248#issuecomment-1569878684 run arm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname opened a new pull request, #20270: [Enhancement](planner)add switch to determine if some unfixed functions will be folded on fe.
sohardforaname opened a new pull request, #20270: URL: https://github.com/apache/doris/pull/20270 ## Proposed changes Issue Number: close #xxx <--Describe your changes.--> ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qidaye merged pull request #19855: [improvement](stacktrace) do not captute stack trace for txn
qidaye merged PR #19855: URL: https://github.com/apache/doris/pull/19855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [improvement](stacktrace) do not captute stack trace for txn (#19855)
This is an automated email from the ASF dual-hosted git repository.
jianliangqi pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new d7574f59f3 [improvement](stacktrace) do not captute stack trace for
txn (#19855)
d7574f59f3 is described below
commit d7574f59f300ec30b9b2cb9fa842e065f164c08c
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Wed May 31 18:05:22 2023 +0800
[improvement](stacktrace) do not captute stack trace for txn (#19855)
---
be/src/common/status.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/be/src/common/status.h b/be/src/common/status.h
index fdaa0ca0a7..cefc8d6402 100644
--- a/be/src/common/status.h
+++ b/be/src/common/status.h
@@ -269,6 +269,7 @@ static constexpr bool capture_stacktrace() {
&& code != ErrorCode::SEGCOMPACTION_INIT_READER
&& code != ErrorCode::SEGCOMPACTION_INIT_WRITER
&& code != ErrorCode::SEGCOMPACTION_FAILED
+&& code != ErrorCode::META_KEY_NOT_FOUND
&& code != ErrorCode::PUSH_VERSION_ALREADY_EXIST
&& code != ErrorCode::TRANSACTION_NOT_EXIST
&& code != ErrorCode::TRANSACTION_ALREADY_VISIBLE
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qidaye merged pull request #20258: [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property
qidaye merged PR #20258: URL: https://github.com/apache/doris/pull/20258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: Revert "[improvement](libhdfs) Use keytab and principal to login kerberos (#19841)"
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new efc544e1d2 Revert "[improvement](libhdfs) Use keytab and principal to
login kerberos (#19841)"
efc544e1d2 is described below
commit efc544e1d287e4d0c1001c57aff382ad37b766d1
Author: morningman
AuthorDate: Wed May 31 18:09:33 2023 +0800
Revert "[improvement](libhdfs) Use keytab and principal to login kerberos
(#19841)"
This reverts commit 4db6aa67b38c75ecac9e10bedb662bf60ff9aa30.
---
be/CMakeLists.txt | 2 +-
be/src/io/hdfs_builder.cpp | 34 ++
2 files changed, 7 insertions(+), 29 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 12621403fb..aa40e61040 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -739,7 +739,7 @@ set(COMMON_THIRDPARTY
simdjson
)
-if ((ARCH_AMD64 OR ARCH_AARCH64) AND OS_LINUX)
+if (ARCH_AMD64 AND OS_LINUX)
add_library(hadoop_hdfs STATIC IMPORTED)
set_target_properties(hadoop_hdfs PROPERTIES IMPORTED_LOCATION
${THIRDPARTY_DIR}/lib/hadoop_hdfs/native/libhdfs.a)
diff --git a/be/src/io/hdfs_builder.cpp b/be/src/io/hdfs_builder.cpp
index ebed8b8c5b..51a7e539b6 100644
--- a/be/src/io/hdfs_builder.cpp
+++ b/be/src/io/hdfs_builder.cpp
@@ -47,17 +47,9 @@ Status HDFSCommonBuilder::run_kinit() {
return Status::InvalidArgument("Invalid hdfs_kerberos_principal or
hdfs_kerberos_keytab");
}
std::string ticket_path = TICKET_CACHE_PATH + generate_uuid_string();
-const char* krb_home = getenv("KRB_HOME");
-std::string krb_home_str(krb_home ? krb_home : "");
fmt::memory_buffer kinit_command;
-if (krb_home_str.empty()) {
-fmt::format_to(kinit_command, "kinit -c {} -R -t {} -k {}",
ticket_path,
- hdfs_kerberos_keytab, hdfs_kerberos_principal);
-} else {
-// Assign kerberos home in env, get kinit in kerberos home
-fmt::format_to(kinit_command, krb_home_str + "/bin/kinit -c {} -R -t
{} -k {}", ticket_path,
- hdfs_kerberos_keytab, hdfs_kerberos_principal);
-}
+fmt::format_to(kinit_command, "kinit -c {} -R -t {} -k {}", ticket_path,
hdfs_kerberos_keytab,
+ hdfs_kerberos_principal);
VLOG_NOTICE << "kinit command: " << fmt::to_string(kinit_command);
std::string msg;
AgentUtils util;
@@ -67,9 +59,8 @@ Status HDFSCommonBuilder::run_kinit() {
}
#ifdef USE_LIBHDFS3
hdfsBuilderSetPrincipal(hdfs_builder, hdfs_kerberos_principal.c_str());
+hdfsBuilderSetKerbTicketCachePath(hdfs_builder, ticket_path.c_str());
#endif
-hdfsBuilderConfSetStr(hdfs_builder,
"hadoop.security.kerberos.ticket.cache.path",
- ticket_path.c_str());
return Status::OK();
}
@@ -112,37 +103,24 @@ Status createHDFSBuilder(const THdfsParams& hdfsParams,
HDFSCommonBuilder* build
if (hdfsParams.__isset.hdfs_kerberos_principal) {
builder->need_kinit = true;
builder->hdfs_kerberos_principal = hdfsParams.hdfs_kerberos_principal;
-hdfsBuilderSetUserName(builder->get(),
hdfsParams.hdfs_kerberos_principal.c_str());
-} else if (hdfsParams.__isset.user) {
-hdfsBuilderSetUserName(builder->get(), hdfsParams.user.c_str());
-#ifdef USE_HADOOP_HDFS
-hdfsBuilderSetKerb5Conf(builder->get(), nullptr);
-hdfsBuilderSetKeyTabFile(builder->get(), nullptr);
-#endif
}
if (hdfsParams.__isset.hdfs_kerberos_keytab) {
builder->need_kinit = true;
builder->hdfs_kerberos_keytab = hdfsParams.hdfs_kerberos_keytab;
-#ifdef USE_HADOOP_HDFS
-hdfsBuilderSetKeyTabFile(builder->get(),
hdfsParams.hdfs_kerberos_keytab.c_str());
-#endif
}
// set other conf
if (hdfsParams.__isset.hdfs_conf) {
for (const THdfsConf& conf : hdfsParams.hdfs_conf) {
hdfsBuilderConfSetStr(builder->get(), conf.key.c_str(),
conf.value.c_str());
-#ifdef USE_HADOOP_HDFS
-// Set krb5.conf, we should define java.security.krb5.conf in
catalog properties
-if (strcmp(conf.key.c_str(), "java.security.krb5.conf") == 0) {
-hdfsBuilderSetKerb5Conf(builder->get(), conf.value.c_str());
-}
-#endif
}
}
hdfsBuilderConfSetStr(builder->get(),
"ipc.client.fallback-to-simple-auth-allowed", "true");
if (builder->is_need_kinit()) {
RETURN_IF_ERROR(builder->run_kinit());
+} else if (hdfsParams.__isset.user) {
+// set hdfs user
+hdfsBuilderSetUserName(builder->get(), hdfsParams.user.c_str());
}
return Status::OK();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property (#20258)
This is an automated email from the ASF dual-hosted git repository.
jianliangqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6adb3fdf11 [fix](match_phrase) Fix the inconsistent query result for
'match_phrase' after creating index without support_phrase property (#20258)
6adb3fdf11 is described below
commit 6adb3fdf11fedef90f649a90a892e4787bc671c3
Author: YueW <45946325+tany...@users.noreply.github.com>
AuthorDate: Wed May 31 18:09:50 2023 +0800
[fix](match_phrase) Fix the inconsistent query result for 'match_phrase'
after creating index without support_phrase property (#20258)
if create inverted index without support_phrase property, remaining the
match_phrase condition to filter by match function.
---
be/src/common/status.h | 2 +
be/src/olap/match_predicate.cpp| 15 +++
be/src/olap/match_predicate.h | 1 +
.../rowset/segment_v2/inverted_index_reader.cpp| 4 +
.../olap/rowset/segment_v2/inverted_index_reader.h | 5 +
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 1 +
be/src/vec/functions/match.cpp | 17 ++-
.../inverted_index_p0/test_index_match_phrase.out | 129 ++
...rase_select.out => test_index_match_select.out} | 0
...elect.groovy => test_index_match_phrase.groovy} | 145 -
...elect.groovy => test_index_match_select.groovy} | 4 +-
11 files changed, 231 insertions(+), 92 deletions(-)
diff --git a/be/src/common/status.h b/be/src/common/status.h
index 5b3acfae10..ba634df13c 100644
--- a/be/src/common/status.h
+++ b/be/src/common/status.h
@@ -265,6 +265,7 @@ E(INVERTED_INDEX_FILE_NOT_FOUND, -6003);
E(INVERTED_INDEX_FILE_HIT_LIMIT, -6004);
E(INVERTED_INDEX_NO_TERMS, -6005);
E(INVERTED_INDEX_RENAME_FILE_FAILED, -6006);
+E(INVERTED_INDEX_EVALUATE_SKIPPED, -6007);
#undef E
} // namespace ErrorCode
@@ -294,6 +295,7 @@ constexpr bool capture_stacktrace() {
&& code != ErrorCode::INVERTED_INDEX_FILE_NOT_FOUND
&& code != ErrorCode::INVERTED_INDEX_FILE_HIT_LIMIT
&& code != ErrorCode::INVERTED_INDEX_NO_TERMS
+&& code != ErrorCode::INVERTED_INDEX_EVALUATE_SKIPPED
&& code != ErrorCode::META_KEY_NOT_FOUND
&& code != ErrorCode::PUSH_VERSION_ALREADY_EXIST
&& code != ErrorCode::TRANSACTION_NOT_EXIST
diff --git a/be/src/olap/match_predicate.cpp b/be/src/olap/match_predicate.cpp
index 9a1a782db5..b5b4882713 100644
--- a/be/src/olap/match_predicate.cpp
+++ b/be/src/olap/match_predicate.cpp
@@ -21,6 +21,7 @@
#include "exec/olap_utils.h"
#include "olap/field.h"
+#include "olap/inverted_index_parser.h"
#include "olap/olap_common.h"
#include "olap/rowset/segment_v2/inverted_index_cache.h"
#include "olap/rowset/segment_v2/inverted_index_reader.h"
@@ -43,6 +44,10 @@ Status MatchPredicate::evaluate(const Schema& schema,
InvertedIndexIterator* ite
if (iterator == nullptr) {
return Status::OK();
}
+if (_skip_evaluate(iterator)) {
+LOG(INFO) << "match predicate evaluate skipped.";
+return Status::Error();
+}
auto column_desc = schema.column(_column_id);
roaring::Roaring roaring;
auto inverted_index_query_type =
_to_inverted_index_query_type(_match_type);
@@ -111,4 +116,14 @@ InvertedIndexQueryType
MatchPredicate::_to_inverted_index_query_type(MatchType m
return ret;
}
+bool MatchPredicate::_skip_evaluate(InvertedIndexIterator* iterator) const {
+if (_match_type == MatchType::MATCH_PHRASE &&
+iterator->get_inverted_index_reader_type() ==
InvertedIndexReaderType::FULLTEXT &&
+
get_parser_phrase_support_string_from_properties(iterator->get_index_properties())
==
+INVERTED_INDEX_PARSER_PHRASE_SUPPORT_NO) {
+return true;
+}
+return false;
+}
+
} // namespace doris
\ No newline at end of file
diff --git a/be/src/olap/match_predicate.h b/be/src/olap/match_predicate.h
index ad793b90f8..e3e2ca45bd 100644
--- a/be/src/olap/match_predicate.h
+++ b/be/src/olap/match_predicate.h
@@ -64,6 +64,7 @@ private:
std::string info = "MatchPredicate";
return info;
}
+bool _skip_evaluate(InvertedIndexIterator* iterator) const;
private:
std::string _value;
diff --git a/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
b/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
index 42f9a3362d..a97eae836f 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
@@ -902,5 +902,9 @@ InvertedIndexReaderType
InvertedIndexIterator::get_inverted_index_reader_type()
return _reader->type();
}
+const std::map& InvertedIndexIterator::get_index_properties()
const {
+return _reader->get_index_properties();
+}
+
} // namespace segment_v2
} // names
[GitHub] [doris] eldenmoon commented on pull request #20219: [Improve](BloomFilter) Suport caching bloom filters and filter segmen…
eldenmoon commented on PR #20219: URL: https://github.com/apache/doris/pull/20219#issuecomment-1569902862 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #20271: [branch1.2](deps)(aws) patch aws sdk cpp 1.9.211
morningman opened a new pull request, #20271: URL: https://github.com/apache/doris/pull/20271 ## Proposed changes cherry-pick #16625 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #20271: [branch1.2](deps)(aws) patch aws sdk cpp 1.9.211
morningman merged PR #20271: URL: https://github.com/apache/doris/pull/20271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [branch1.2](deps)(aws) patch aws sdk cpp 1.9.211 (#20271)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 3d3cbce229 [branch1.2](deps)(aws) patch aws sdk cpp 1.9.211 (#20271)
3d3cbce229 is described below
commit 3d3cbce229382bbfa904b262f69fc0e825d0bf31
Author: Mingyu Chen
AuthorDate: Wed May 31 18:14:10 2023 +0800
[branch1.2](deps)(aws) patch aws sdk cpp 1.9.211 (#20271)
cherry-pick #16625
---
thirdparty/download-thirdparty.sh| 1 +
thirdparty/patches/aws-sdk-cpp-1.9.211.patch | 169 +++
2 files changed, 170 insertions(+)
diff --git a/thirdparty/download-thirdparty.sh
b/thirdparty/download-thirdparty.sh
index 6cae1cc07d..1dbbfbb0f0 100755
--- a/thirdparty/download-thirdparty.sh
+++ b/thirdparty/download-thirdparty.sh
@@ -368,6 +368,7 @@ echo "Finished patching ${HYPERSCAN_SOURCE}"
cd "${TP_SOURCE_DIR}/${AWS_SDK_SOURCE}"
if [[ ! -f "${PATCHED_MARK}" ]]; then
if [[ "${AWS_SDK_SOURCE}" == "aws-sdk-cpp-1.9.211" ]]; then
+patch -p1 <"${TP_PATCH_DIR}/aws-sdk-cpp-1.9.211.patch"
if wget --no-check-certificate -q
https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/aws-crt-cpp-1.9.211.tar.gz
-O aws-crt-cpp-1.9.211.tar.gz; then
tar xzf aws-crt-cpp-1.9.211.tar.gz
else
diff --git a/thirdparty/patches/aws-sdk-cpp-1.9.211.patch
b/thirdparty/patches/aws-sdk-cpp-1.9.211.patch
new file mode 100644
index 00..d9e2cda3ee
--- /dev/null
+++ b/thirdparty/patches/aws-sdk-cpp-1.9.211.patch
@@ -0,0 +1,169 @@
+diff --git a/aws-cpp-sdk-core/include/aws/core/Aws.h
b/aws-cpp-sdk-core/include/aws/core/Aws.h
+index 419f5e14f6..dfbbb986a6 100644
+--- a/aws-cpp-sdk-core/include/aws/core/Aws.h
b/aws-cpp-sdk-core/include/aws/core/Aws.h
+@@ -80,7 +80,7 @@ namespace Aws
+ */
+ struct HttpOptions
+ {
+-HttpOptions() : initAndCleanupCurl(true), installSigPipeHandler(false)
++HttpOptions() : initAndCleanupCurl(true),
installSigPipeHandler(false), compliantRfc3986Encoding(false)
+ { }
+
+ /**
+@@ -100,6 +100,10 @@ namespace Aws
+ * NOTE: CURLOPT_NOSIGNAL is already being set.
+ */
+ bool installSigPipeHandler;
++/**
++ * Disable legacy URL encoding that leaves `$&,:@=` unescaped for
legacy purposes.
++ */
++bool compliantRfc3986Encoding;
+ };
+
+ /**
+diff --git a/aws-cpp-sdk-core/include/aws/core/http/URI.h
b/aws-cpp-sdk-core/include/aws/core/http/URI.h
+index 4cff72d4a8..b536926cb0 100644
+--- a/aws-cpp-sdk-core/include/aws/core/http/URI.h
b/aws-cpp-sdk-core/include/aws/core/http/URI.h
+@@ -21,6 +21,9 @@ namespace Aws
+ static const uint16_t HTTP_DEFAULT_PORT = 80;
+ static const uint16_t HTTPS_DEFAULT_PORT = 443;
+
++extern bool s_compliantRfc3986Encoding;
++AWS_CORE_API void SetCompliantRfc3986Encoding(bool compliant);
++
+ //per https://tools.ietf.org/html/rfc3986#section-3.4 there is
nothing preventing servers from allowing
+ //multiple values for the same key. So use a multimap instead of a
map.
+ typedef Aws::MultiMap
QueryStringParameterCollection;
+diff --git a/aws-cpp-sdk-core/source/Aws.cpp b/aws-cpp-sdk-core/source/Aws.cpp
+index 6049b12264..e9aca9c580 100644
+--- a/aws-cpp-sdk-core/source/Aws.cpp
b/aws-cpp-sdk-core/source/Aws.cpp
+@@ -136,6 +136,7 @@ namespace Aws
+
+
Aws::Http::SetInitCleanupCurlFlag(options.httpOptions.initAndCleanupCurl);
+
Aws::Http::SetInstallSigPipeHandlerFlag(options.httpOptions.installSigPipeHandler);
++
Aws::Http::SetCompliantRfc3986Encoding(options.httpOptions.compliantRfc3986Encoding);
+ Aws::Http::InitHttp();
+ Aws::InitializeEnumOverflowContainer();
+ cJSON_AS4CPP_Hooks hooks;
+diff --git a/aws-cpp-sdk-core/source/http/URI.cpp
b/aws-cpp-sdk-core/source/http/URI.cpp
+index ce9ec064cb..9dfc10a875 100644
+--- a/aws-cpp-sdk-core/source/http/URI.cpp
b/aws-cpp-sdk-core/source/http/URI.cpp
+@@ -24,6 +24,48 @@ namespace Http
+
+ const char* SEPARATOR = "://";
+
++bool s_compliantRfc3986Encoding = false;
++void SetCompliantRfc3986Encoding(bool compliant) { s_compliantRfc3986Encoding
= compliant; }
++
++Aws::String urlEncodeSegment(const Aws::String& segment)
++{
++// consolidates legacy escaping logic into one local method
++if (s_compliantRfc3986Encoding)
++{
++return StringUtils::URLEncode(segment.c_str());
++}
++else
++{
++Aws::StringStream ss;
++ss << std::hex << std::uppercase;
++for(unsigned char c : segment) // alnum results in UB if the value of
c is not unsigned char & is not EOF
++{
++// RFC 3986 §2.3 unreserved characters
++if (StringUtils::IsAlnum(c))
++{
++ss << c
[GitHub] [doris] dataroaring merged pull request #20252: [deps](aws) upgrade to 1.9.272 to fix non-compliant RFC3986 encoding
dataroaring merged PR #20252: URL: https://github.com/apache/doris/pull/20252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring closed issue #19275: [Bug] S3 Load fail when the path include special characters
dataroaring closed issue #19275: [Bug] S3 Load fail when the path include special characters URL: https://github.com/apache/doris/issues/19275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring closed pull request #16625: [Fix](multi-catalog) fix oss access issue with aws s3 sdk
dataroaring closed pull request #16625: [Fix](multi-catalog) fix oss access issue with aws s3 sdk URL: https://github.com/apache/doris/pull/16625 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [deps](aws) upgrade to 1.9.272 to fix non-compliant RFC3986 encoding (#20252)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d963bf8d79 [deps](aws) upgrade to 1.9.272 to fix non-compliant RFC3986
encoding (#20252)
d963bf8d79 is described below
commit d963bf8d794f6ee23a6a26a7da7257a3a1badff0
Author: Mingyu Chen
AuthorDate: Wed May 31 18:19:06 2023 +0800
[deps](aws) upgrade to 1.9.272 to fix non-compliant RFC3986 encoding
(#20252)
---
thirdparty/CHANGELOG.md | 5 +
thirdparty/vars.sh | 8
2 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/thirdparty/CHANGELOG.md b/thirdparty/CHANGELOG.md
index b79daee5ad..60ec92e78e 100644
--- a/thirdparty/CHANGELOG.md
+++ b/thirdparty/CHANGELOG.md
@@ -2,6 +2,11 @@
This file contains version of the third-party dependency libraries in the
build-env image. The docker build-env image is apache/doris, and the tag is
`build-env-${version}`
+## v20230531
+
+- Modified hadoop libhdfs 3.3.4.2 -> 3.3.4.3
+- Modified aws sdk 1.9.211 -> 1.9.272
+
## v20230525
- Removed: clucene changed to submodule
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index f7888d1144..6727cd82c8 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -326,10 +326,10 @@ BOOTSTRAP_TABLE_CSS_FILE="bootstrap-table.min.css"
BOOTSTRAP_TABLE_CSS_MD5SUM="23389d4456da412e36bae30c469a766a"
# aws sdk
-AWS_SDK_DOWNLOAD="https://github.com/aws/aws-sdk-cpp/archive/refs/tags/1.9.211.tar.gz";
-AWS_SDK_NAME="aws-sdk-cpp-1.9.211.tar.gz"
-AWS_SDK_SOURCE="aws-sdk-cpp-1.9.211"
-AWS_SDK_MD5SUM="667b8e08baf0b9967c19224198e33160"
+AWS_SDK_DOWNLOAD="https://github.com/aws/aws-sdk-cpp/archive/refs/tags/1.9.272.tar.gz";
+AWS_SDK_NAME="aws-sdk-cpp-1.9.272.tar.gz"
+AWS_SDK_SOURCE="aws-sdk-cpp-1.9.272"
+AWS_SDK_MD5SUM="3a9190e3d14b81bfdd9c41a89a419202"
# tsan_header
TSAN_HEADER_DOWNLOAD="https://gcc.gnu.org/git/?p=gcc.git;a=blob_plain;f=libsanitizer/include/sanitizer/tsan_interface_atomic.h;hb=refs/heads/releases/gcc-7";
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
