[GitHub] [doris] github-actions[bot] commented on pull request #15360: [fix](string) fix offsets over flow for extreme large String column
github-actions[bot] commented on PR #15360: URL: https://github.com/apache/doris/pull/15360#issuecomment-1364975100 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname opened a new pull request, #15361: [Fix](Nereids)fix bug: expressions is not be extracted bug when binding function
sohardforaname opened a new pull request, #15361: URL: https://github.com/apache/doris/pull/15361 # Proposed changes Issue Number: close #xxx ## Problem summary fix bug: expressions is not be extracted bug when binding function by: add binding function accept a set and return a set. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15360: [fix](string) fix offsets over flow for extreme large String column
github-actions[bot] commented on PR #15360: URL: https://github.com/apache/doris/pull/15360#issuecomment-1364986763 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang opened a new pull request, #15362: [typo](docs) fix version tag for docs of s3 token
Yulei-Yang opened a new pull request, #15362: URL: https://github.com/apache/doris/pull/15362 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on a diff in pull request #15340: [Improvement](S3) support access s3 via temporary security credentials
Yulei-Yang commented on code in PR #15340: URL: https://github.com/apache/doris/pull/15340#discussion_r1057127378 ## docs/zh-CN/docs/data-operate/import/import-scenes/external-storage-load.md: ## @@ -180,3 +180,17 @@ S3 SDK 默认使用 `virtual-hosted style` 方式。但某些对象存储系统 "use_path_style" = "true" ) ``` + + Review Comment: fix it in https://github.com/apache/doris/pull/15362 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg merged pull request #15352: [bugfix](from_unixtime) fix timezone not work for from_unixtime
yangzhg merged PR #15352: URL: https://github.com/apache/doris/pull/15352 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [bugfix](from_unixtime) fix timezone not work for from_unixtime (#15298) (#15352)
This is an automated email from the ASF dual-hosted git repository.
yangzhg pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new c45f30023e [bugfix](from_unixtime) fix timezone not work for
from_unixtime (#15298) (#15352)
c45f30023e is described below
commit c45f30023ebae7fe26a267992444e0aca481fe14
Author: Zhengguo Yang
AuthorDate: Mon Dec 26 16:17:52 2022 +0800
[bugfix](from_unixtime) fix timezone not work for from_unixtime (#15298)
(#15352)
* [bugfix](from_unixtime) fix timezone not work for from_unixtime
---
be/src/gutil/casts.h | 7 +-
be/src/gutil/charmap.h | 5 -
be/src/gutil/gscoped_ptr.h | 12 +-
be/src/gutil/once.h| 7 +-
be/src/gutil/strings/stringpiece.h | 5 -
be/src/gutil/type_traits.h | 468 -
be/src/http/action/tablet_migration_action.cpp | 2 +-
be/src/util/thread.cpp | 2 +-
.../function_utils.h => src/util/type_traits.h}| 40 +-
be/src/vec/functions/date_time_transforms.h| 29 +-
.../function_date_or_datetime_to_string.h | 2 +-
.../function_datetime_string_to_string.cpp | 7 +-
.../functions/function_datetime_string_to_string.h | 38 +-
be/test/exprs/timestamp_functions_test.cpp | 25 +-
be/test/testutil/function_utils.cpp| 22 +-
be/test/testutil/function_utils.h | 1 -
.../data/datatype_p0/date/test_from_unixtime.out | 7 +
.../datatype_p0/date/test_from_unixtime.groovy | 34 +-
18 files changed, 94 insertions(+), 619 deletions(-)
diff --git a/be/src/gutil/casts.h b/be/src/gutil/casts.h
index 71a147cfaf..8e2a1a5cad 100644
--- a/be/src/gutil/casts.h
+++ b/be/src/gutil/casts.h
@@ -14,9 +14,10 @@
#include // for enumeration casts and tests
#include // for memcpy
+#include
+
#include "gutil/macros.h"
#include "gutil/template_util.h"
-#include "gutil/type_traits.h"
// Use implicit_cast as a safe version of static_cast or const_cast
// for implicit conversions. For example:
@@ -89,8 +90,8 @@ inline To down_cast(From* f) {// so we only accept
pointers
// compiler will just bind From to const T.
template
inline To down_cast(From& f) {
-COMPILE_ASSERT(base::is_reference::value, target_type_not_a_reference);
-typedef typename base::remove_reference::type* ToAsPointer;
+COMPILE_ASSERT(std::is_reference::value, target_type_not_a_reference);
+using ToAsPointer = typename std::remove_reference::type*;
if (false) {
// Compile-time check that To inherits from From. See above for
details.
::implicit_cast(NULL);
diff --git a/be/src/gutil/charmap.h b/be/src/gutil/charmap.h
index 213d44f726..797abcce5c 100644
--- a/be/src/gutil/charmap.h
+++ b/be/src/gutil/charmap.h
@@ -17,10 +17,6 @@
#include
-#include "gutil/basictypes.h"
-#include "gutil/integral_types.h"
-#include "gutil/type_traits.h"
-
class Charmap {
public:
// Initializes with given uint32 values. For instance, the first
@@ -73,4 +69,3 @@ protected:
}
}
};
-DECLARE_POD(Charmap);
diff --git a/be/src/gutil/gscoped_ptr.h b/be/src/gutil/gscoped_ptr.h
index d1d88d1da7..01305b9294 100644
--- a/be/src/gutil/gscoped_ptr.h
+++ b/be/src/gutil/gscoped_ptr.h
@@ -102,11 +102,11 @@
#include
#include // For std::swap().
+#include
#include "gutil/basictypes.h"
#include "gutil/move.h"
#include "gutil/template_util.h"
-#include "gutil/type_traits.h"
namespace doris {
@@ -137,7 +137,7 @@ struct DefaultDeleter {
// cannot convert to T*.
enum { T_must_be_complete = sizeof(T) };
enum { U_must_be_complete = sizeof(U) };
-COMPILE_ASSERT((base::is_convertible::value),
+COMPILE_ASSERT((std::is_convertible::value),
U_ptr_must_implicitly_convert_to_T_ptr);
}
inline void operator()(T* ptr) const {
@@ -186,8 +186,8 @@ namespace internal {
template
struct IsNotRefCounted {
enum {
-value = !base::is_convertible::value &&
-!base::is_convertible::value
+value = !std::is_convertible::value &&
+!std::is_convertible::value
};
};
@@ -345,7 +345,7 @@ public:
// implementation of gscoped_ptr.
template
gscoped_ptr(gscoped_ptr other) : impl_(&other.impl_) {
-COMPILE_ASSERT(!base::is_array::value, U_cannot_be_an_array);
+COMPILE_ASSERT(!std::is_array::value, U_cannot_be_an_array);
}
// Constructor. Move constructor for C++03 move emulation of this type.
@@ -363,7 +363,7 @@ public:
// gscoped_ptr.
template
gscoped_ptr& operator=(gscoped_ptr rhs) {
-COMPILE_ASSERT(!base::is_array::value, U_cannot_be_an_array);
+
[GitHub] [doris] lsy3993 opened a new pull request, #15363: [typo](docs) fix wrong date format
lsy3993 opened a new pull request, #15363: URL: https://github.com/apache/doris/pull/15363 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu opened a new pull request, #15364: Add drop storage policy
pengxiangyu opened a new pull request, #15364: URL: https://github.com/apache/doris/pull/15364 # Proposed changes Issue Number: close #14331 ## Problem summary Our current cold and hot separation creation strategy does not support deletion. After creating an error, there is no way to delete it. After the user deletes the table, it keeps reporting this hot and cold separation creation strategy check error The s3_region attribute was wrongly written when creating the s3 resource. This attribute cannot be changed, and the policy cannot be deleted. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #14722: [fix](vec) VMergeIterator add key same label for agg table
freemandealer commented on code in PR #14722:
URL: https://github.com/apache/doris/pull/14722#discussion_r1057121438
##
be/test/olap/segcompaction_test.cpp:
##
@@ -419,6 +419,7 @@ TEST_F(SegCompactionTest,
SegCompactionInterleaveWithBig_OOoOO) {
}
s = rowset_writer->flush();
EXPECT_EQ(Status::OK(), s);
+sleep(1);
Review Comment:
why we need sleep here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14722: [fix](vec) VMergeIterator add key same label for agg table
github-actions[bot] commented on PR #14722: URL: https://github.com/apache/doris/pull/14722#issuecomment-1364995077 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15357: [chore](gutil) remove some gutil files and use c++ stl instead
github-actions[bot] commented on PR #15357: URL: https://github.com/apache/doris/pull/15357#issuecomment-1364996005 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15357: [chore](gutil) remove some gutil files and use c++ stl instead
github-actions[bot] commented on PR #15357: URL: https://github.com/apache/doris/pull/15357#issuecomment-1364997739 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15212: [Feature](Materialized-View) support advanced Materialized-View
github-actions[bot] commented on PR #15212: URL: https://github.com/apache/doris/pull/15212#issuecomment-1365004412 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #15311: [feature-wip](nereids) Implement using join
morrySnow commented on code in PR #15311:
URL: https://github.com/apache/doris/pull/15311#discussion_r1057142886
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##
@@ -122,41 +124,60 @@ public List buildRules() {
return new LogicalFilter<>(boundPredicates,
filter.child());
})
),
-RuleType.BINDING_JOIN_SLOT.build(
-logicalJoin().when(Plan::canBind)
-.whenNot(j ->
j.getJoinType().equals(JoinType.USING_JOIN)).thenApply(ctx -> {
-LogicalJoin join =
ctx.root;
-List cond =
join.getOtherJoinConjuncts().stream()
-.map(expr -> bind(expr,
join.children(), join, ctx.cascadesContext))
-.collect(Collectors.toList());
-List hashJoinConjuncts =
join.getHashJoinConjuncts().stream()
-.map(expr -> bind(expr,
join.children(), join, ctx.cascadesContext))
-.collect(Collectors.toList());
-return new LogicalJoin<>(join.getJoinType(),
-hashJoinConjuncts, cond,
join.getHint(), join.left(), join.right());
-})
-),
+
RuleType.BINDING_USING_JOIN_SLOT.build(
-logicalJoin().when(j ->
j.getJoinType().equals(JoinType.USING_JOIN)).thenApply(ctx -> {
-LogicalJoin join = ctx.root;
-List unboundSlots =
join.getHashJoinConjuncts();
-List leftSlots = unboundSlots.stream()
-.map(expr -> bind(expr,
Collections.singletonList(join.left()),
-join, ctx.cascadesContext))
-.collect(Collectors.toList());
-List rightSlots = unboundSlots.stream()
-.map(expr -> bind(expr,
Collections.singletonList(join.right()),
-join, ctx.cascadesContext))
-.collect(Collectors.toList());
+usingJoin().thenApply(ctx -> {
+UsingJoin using = ctx.root;
+LogicalJoin lj = new LogicalJoin(using.getJoinType()
== JoinType.CROSS_JOIN
+? JoinType.INNER_JOIN : using.getJoinType(),
+using.getHashJoinConjuncts(),
+using.getOtherJoinConjuncts(),
using.getHint(), (Plan) using.left(),
+(Plan) using.right());
+List unboundSlots =
lj.getHashJoinConjuncts();
+Set slotNames = new HashSet<>();
+List leftOutput = new
ArrayList<>(lj.left().getOutput());
+Collections.reverse(leftOutput);
Review Comment:
why need reverse?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##
@@ -122,41 +124,60 @@ public List buildRules() {
return new LogicalFilter<>(boundPredicates,
filter.child());
})
),
-RuleType.BINDING_JOIN_SLOT.build(
-logicalJoin().when(Plan::canBind)
-.whenNot(j ->
j.getJoinType().equals(JoinType.USING_JOIN)).thenApply(ctx -> {
-LogicalJoin join =
ctx.root;
-List cond =
join.getOtherJoinConjuncts().stream()
-.map(expr -> bind(expr,
join.children(), join, ctx.cascadesContext))
-.collect(Collectors.toList());
-List hashJoinConjuncts =
join.getHashJoinConjuncts().stream()
-.map(expr -> bind(expr,
join.children(), join, ctx.cascadesContext))
-.collect(Collectors.toList());
-return new LogicalJoin<>(join.getJoinType(),
-hashJoinConjuncts, cond,
join.getHint(), join.left(), join.right());
-})
-),
+
RuleType.BINDING_USING_JOIN_SLOT.build(
-logicalJoin().when(j ->
j.getJoinType().equals(JoinType.USING_JOIN)).thenApply(ctx -> {
-LogicalJoin join = ctx.root;
-List unboundSlots =
join.getHashJoinConjuncts();
-List leftSlots = unboundSlots.stream()
-.map(expr -> bind(expr,
Collections.singletonList(join.left()),
-join, ct
[GitHub] [doris] github-actions[bot] commented on pull request #15358: [fix](multi-catalog) throw NPE when reading data after EOF
github-actions[bot] commented on PR #15358: URL: https://github.com/apache/doris/pull/15358#issuecomment-1365010904 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] JNSimba opened a new pull request, #15365: [Doc][Flink] update flink connector doc add new version
JNSimba opened a new pull request, #15365: URL: https://github.com/apache/doris/pull/15365 ## Problem summary update flink connector doc add new version ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 merged pull request #15242: [Feature](Nereids) add simplify arithmetic rule
924060929 merged PR #15242: URL: https://github.com/apache/doris/pull/15242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 closed issue #14104: [Feature] (Nereids) Simplify arithmetic operation of expression
924060929 closed issue #14104: [Feature] (Nereids) Simplify arithmetic operation of expression URL: https://github.com/apache/doris/issues/14104 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Feature](Nereids) add simplify arithmetic rule (#15242)
This is an automated email from the ASF dual-hosted git repository.
huajianlan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ae87415174 [Feature](Nereids) add simplify arithmetic rule (#15242)
ae87415174 is described below
commit ae874151746157528386674571abf264fbfce8d4
Author: shee <13843187+qz...@users.noreply.github.com>
AuthorDate: Mon Dec 26 16:57:59 2022 +0800
[Feature](Nereids) add simplify arithmetic rule (#15242)
support simplify arithmetic rule
for example ：
a + 1 > 1
=> a > 0
---
.../rewrite/ExpressionNormalization.java | 6 +-
.../rules/SimplifyArithmeticComparisonRule.java| 103 +++
.../rewrite/rules/SimplifyArithmeticRule.java | 194 +
.../org/apache/doris/nereids/util/TypeUtils.java | 55 ++
.../rewrite/ExpressionRewriteTestHelper.java | 2 +-
.../rewrite/SimplifyArithmeticRuleTest.java| 93 ++
6 files changed, 451 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionNormalization.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionNormalization.java
index d79a9bb62e..484eac94bf 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionNormalization.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionNormalization.java
@@ -23,6 +23,8 @@ import
org.apache.doris.nereids.rules.expression.rewrite.rules.DigitalMaskingCon
import
org.apache.doris.nereids.rules.expression.rewrite.rules.FoldConstantRule;
import
org.apache.doris.nereids.rules.expression.rewrite.rules.InPredicateToEqualToRule;
import
org.apache.doris.nereids.rules.expression.rewrite.rules.NormalizeBinaryPredicatesRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.rules.SimplifyArithmeticComparisonRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.rules.SimplifyArithmeticRule;
import
org.apache.doris.nereids.rules.expression.rewrite.rules.SimplifyCastRule;
import
org.apache.doris.nereids.rules.expression.rewrite.rules.SimplifyNotExprRule;
import org.apache.doris.nereids.rules.expression.rewrite.rules.TypeCoercion;
@@ -43,10 +45,12 @@ public class ExpressionNormalization extends
ExpressionRewrite {
InPredicateToEqualToRule.INSTANCE,
SimplifyNotExprRule.INSTANCE,
CharacterLiteralTypeCoercion.INSTANCE,
+SimplifyArithmeticRule.INSTANCE,
TypeCoercion.INSTANCE,
FoldConstantRule.INSTANCE,
SimplifyCastRule.INSTANCE,
-DigitalMaskingConvert.INSTANCE
+DigitalMaskingConvert.INSTANCE,
+SimplifyArithmeticComparisonRule.INSTANCE
);
public ExpressionNormalization(ConnectContext context) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/SimplifyArithmeticComparisonRule.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/SimplifyArithmeticComparisonRule.java
new file mode 100644
index 00..16aee8d2ad
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/SimplifyArithmeticComparisonRule.java
@@ -0,0 +1,103 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.expressions.Add;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apac
[GitHub] [doris] github-actions[bot] commented on pull request #15360: [fix](string) fix offsets over flow for extreme large String column
github-actions[bot] commented on PR #15360: URL: https://github.com/apache/doris/pull/15360#issuecomment-1365014786 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15360: [fix](string) fix offsets over flow for extreme large String column
github-actions[bot] commented on PR #15360: URL: https://github.com/apache/doris/pull/15360#issuecomment-1365014752 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15365: [Doc](Flink) update flink connector doc add new version
github-actions[bot] commented on PR #15365: URL: https://github.com/apache/doris/pull/15365#issuecomment-1365014957 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15365: [Doc](Flink) update flink connector doc add new version
github-actions[bot] commented on PR #15365: URL: https://github.com/apache/doris/pull/15365#issuecomment-1365014928 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15361: [Fix](Nereids)fix bug: expressions is not be extracted bug when binding function
hello-stephen commented on PR #15361: URL: https://github.com/apache/doris/pull/15361#issuecomment-1365025828 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.79 seconds load time: 637 seconds storage size: 17123571239 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226091544_clickbench_pr_68740.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #15357: [chore](gutil) remove some gutil files and use c++ stl instead
yiguolei commented on PR #15357: URL: https://github.com/apache/doris/pull/15357#issuecomment-1365028437 Greate!!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15360: [fix](string) fix offsets over flow for extreme large String column
hello-stephen commented on PR #15360: URL: https://github.com/apache/doris/pull/15360#issuecomment-1365037861 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.89 seconds load time: 635 seconds storage size: 17123249058 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226093715_clickbench_pr_68746.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [fix](compile) fix compile failed in some case due to std::pair
This is an automated email from the ASF dual-hosted git repository.
yangzhg pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new f51cfce6c3 [fix](compile) fix compile failed in some case due to
std::pair
f51cfce6c3 is described below
commit f51cfce6c3484f9f8ddb1317935d81b22bf97c8d
Author: yangzhg
AuthorDate: Mon Dec 26 17:41:39 2022 +0800
[fix](compile) fix compile failed in some case due to std::pair
---
be/src/runtime/memory/mem_tracker_limiter.cpp | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/be/src/runtime/memory/mem_tracker_limiter.cpp
b/be/src/runtime/memory/mem_tracker_limiter.cpp
index a37a0be0f9..aea4b5f806 100644
--- a/be/src/runtime/memory/mem_tracker_limiter.cpp
+++ b/be/src/runtime/memory/mem_tracker_limiter.cpp
@@ -285,17 +285,17 @@ int64_t MemTrackerLimiter::free_top_memory_query(int64_t
min_free_mem) {
std::greater>>
min_pq_null;
std::swap(min_pq, min_pq_null);
-min_pq.push(
-pair(tracker->consumption(),
tracker->label()));
+min_pq.push(std::pair(tracker->consumption(),
+
tracker->label()));
return cancel_top_query(min_pq);
} else if (tracker->consumption() + prepare_free_mem <
min_free_mem) {
-min_pq.push(
-pair(tracker->consumption(),
tracker->label()));
+min_pq.push(std::pair(tracker->consumption(),
+
tracker->label()));
prepare_free_mem += tracker->consumption();
} else if (tracker->consumption() > min_pq.top().first) {
// No need to modify prepare_free_mem, prepare_free_mem
will always be greater than min_free_mem.
-min_pq.push(
-pair(tracker->consumption(),
tracker->label()));
+min_pq.push(std::pair(tracker->consumption(),
+
tracker->label()));
min_pq.pop();
}
}
@@ -320,7 +320,7 @@ int64_t
MemTrackerLimiter::free_top_overcommit_query(int64_t min_free_mem) {
if (overcommit_ratio == 0) { // Small query does not cancel
continue;
}
-min_pq.push(pair(overcommit_ratio,
tracker->label()));
+min_pq.push(std::pair(overcommit_ratio,
tracker->label()));
query_consumption[tracker->label()] = tracker->consumption();
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15212: [Feature](Materialized-View) support advanced Materialized-View
github-actions[bot] commented on PR #15212: URL: https://github.com/apache/doris/pull/15212#issuecomment-1365044087 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #15366: [fix](pipeline) query blocked by waiting for shared hash table
mrhhsg opened a new pull request, #15366: URL: https://github.com/apache/doris/pull/15366 # Proposed changes `SharedHashTableController::wait_for_signal` will block the `HashJoinBuildSink` operator and the query will be blocked if there are too many instances waiting for shared hash table. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15366: [fix](pipeline) query blocked by waiting for shared hash table
github-actions[bot] commented on PR #15366: URL: https://github.com/apache/doris/pull/15366#issuecomment-1365046792 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #15367: [fix](planner) set limit to nagtive value when has offset in limit
morrySnow opened a new pull request, #15367: URL: https://github.com/apache/doris/pull/15367 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #15368: [fix](fe)fix bug of the bucket shuffle join is not recognized
starocean999 opened a new pull request, #15368: URL: https://github.com/apache/doris/pull/15368 # Proposed changes Issue Number: close #xxx ## Problem summary pick from master https://github.com/apache/doris/pull/15255 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15364: [feature](remote)Add drop storage policy
hello-stephen commented on PR #15364: URL: https://github.com/apache/doris/pull/15364#issuecomment-1365050203 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.23 seconds load time: 632 seconds storage size: 17123650448 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226100136_clickbench_pr_68816.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15357: [chore](gutil) remove some gutil files and use c++ stl instead
hello-stephen commented on PR #15357: URL: https://github.com/apache/doris/pull/15357#issuecomment-1365050374 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.84 seconds load time: 636 seconds storage size: 17123331261 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226100146_clickbench_pr_68790.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057171538
##
gensrc/thrift/PaloInternalService.thrift:
##
@@ -513,13 +513,21 @@ struct TFetchDataResult {
4: optional Status.TStatus status
}
+enum TCompoundType {
+UNKNOWN = 0,
+AND = 1,
+OR = 2,
+NOT = 3,
+}
+
struct TCondition {
1: required string column_name
2: required string condition_op
3: required list condition_values
// In delete condition, the different column may have same column name,
need
// using unique id to distinguish them
4: optional i32 column_unique_id
+5: optional TCompoundType compound_type = TCompoundType.UNKNOWN
Review Comment:
condition_op is condition operator, such as '=' '>' '<' and so on,
compound_type is combination type between multiple conditions, such as 'and'
'or' 'not', but compound_type is unused, has been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057171803
##
be/src/vec/exec/scan/vscan_node.h:
##
@@ -197,7 +197,8 @@ class VScanNode : public ExecNode {
_slot_id_to_value_range;
// column -> ColumnValueRange
std::unordered_map
_colname_to_value_range;
-// We use _colname_to_value_range to store a column and its corresponding
value ranges.
+std::vector> _compound_value_ranges;
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057172588
##
be/src/olap/rowset/segment_v2/segment_iterator.cpp:
##
@@ -352,6 +362,15 @@ Status
SegmentIterator::_get_row_ranges_from_conditions(RowRanges* condition_row
Status SegmentIterator::_apply_bitmap_index() {
SCOPED_RAW_TIMER(&_opts.stats->bitmap_index_filter_timer);
size_t input_rows = _row_bitmap.cardinality();
+if (config::enable_index_apply_compound_predicates) {
+RETURN_IF_ERROR(_apply_index_in_compound());
+if (_is_index_for_compound_predicate()) {
+_execute_all_compound_predicates(_remaining_vconjunct_root);
+DCHECK(_compound_predicate_execute_result.size() == 1);
+_row_bitmap &= _compound_predicate_execute_result[0];
Review Comment:
the latest code implementation takes this into account
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15366: [fix](pipeline) query blocked by waiting for shared hash table
github-actions[bot] commented on PR #15366: URL: https://github.com/apache/doris/pull/15366#issuecomment-1365058754 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 opened a new pull request, #15369: [fix](nereids) fix alias
Kikyou1997 opened a new pull request, #15369: URL: https://github.com/apache/doris/pull/15369 # Proposed changes ## Problem summary Support such syntax: ```sql select '' as 'b', col1 from select_with_const ``` ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057175851
##
be/src/olap/rowset/segment_v2/segment_iterator.h:
##
@@ -226,6 +250,10 @@ class SegmentIterator : public RowwiseIterator {
StorageReadOptions _opts;
// make a copy of `_opts.column_predicates` in order to make local changes
std::vector _col_predicates;
+std::vector _all_compound_col_predicates;
Review Comment:
rename to `_col_preds_except_leafnode_of_andnode`, the difference with
`_col_predicates` is those predicates are not leaf node of AND node in
`conjunct_expr_root`, _col_preds_except_leafnode_of_andnode's predicates
although pushed down, still remain in the `conjunct_expr_root`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057176196
##
be/src/olap/rowset/segment_v2/segment_iterator.cpp:
##
@@ -373,6 +392,145 @@ Status SegmentIterator::_apply_bitmap_index() {
return Status::OK();
}
+bool SegmentIterator::_is_literal_node(const TExprNodeType::type& node_type) {
+switch (node_type) {
+case TExprNodeType::BOOL_LITERAL:
+case TExprNodeType::INT_LITERAL:
+case TExprNodeType::LARGE_INT_LITERAL:
+case TExprNodeType::FLOAT_LITERAL:
+case TExprNodeType::DECIMAL_LITERAL:
+case TExprNodeType::STRING_LITERAL:
+case TExprNodeType::DATE_LITERAL:
+return true;
+default:
+return false;
+}
+}
+
+Status SegmentIterator::_execute_all_compound_predicates(vectorized::VExpr*
expr) {
+if (expr == nullptr) {
+return Status::OK();
+}
+
+auto children = expr->children();
+for (int i = 0; i < children.size(); ++i) {
+RETURN_IF_ERROR(_execute_all_compound_predicates(children[i]));
+}
+
+auto node_type = expr->node_type();
+if (node_type == TExprNodeType::SLOT_REF) {
+_column_predicate_info->column_name = expr->expr_name();
+} else if (_is_literal_node(node_type)) {
+auto v_literal_expr =
dynamic_cast(expr);
+_column_predicate_info->query_value = v_literal_expr->value();
+} else if (node_type == TExprNodeType::BINARY_PRED) {
+_column_predicate_info->query_op = expr->fn().name.function_name;
+// get child condition result in compound condtions
+auto column_sign = _gen_predicate_sign(_column_predicate_info.get());
+_column_predicate_info.reset(new ColumnPredicateInfo());
+if (_rowid_result_for_index.count(column_sign) > 0
+&& _rowid_result_for_index[column_sign].first) {
+auto apply_reuslt = _rowid_result_for_index[column_sign].second;
+_compound_predicate_execute_result.push_back(apply_reuslt);
+}
+} else if (node_type == TExprNodeType::COMPOUND_PRED) {
+auto function_name = expr->fn().name.function_name;
+// execute logic function
+RETURN_IF_ERROR(_execute_compound_fn(function_name));
+}
+
+return Status::OK();
+}
+
+Status SegmentIterator::_execute_compound_fn(const std::string& function_name)
{
+auto and_execute_result = [&]() {
Review Comment:
you are right, has changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #15311: [feature-wip](nereids) Implement using join
Kikyou1997 commented on code in PR #15311:
URL: https://github.com/apache/doris/pull/15311#discussion_r1057177097
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/UsingJoin.java:
##
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.JoinHint;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableList.Builder;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * select col1 from t1 join t2 using(col1);
+ */
+public class UsingJoin
+extends LogicalBinary {
Review Comment:
Using Join is actually not any type of join.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15264: [feature](Nereids): generate phyiscal plan in DPhyp
hello-stephen commented on PR #15264: URL: https://github.com/apache/doris/pull/15264#issuecomment-1365065513 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.07 seconds load time: 633 seconds storage size: 17123581386 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226102432_clickbench_pr_68797.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057185549
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -101,7 +110,8 @@ Status Segment::new_iterator(const Schema& schema, const
StorageReadOptions& rea
continue;
}
int32_t uid =
read_options.tablet_schema->column(column_id).unique_id();
-if (_column_readers.count(uid) < 1 ||
!_column_readers.at(uid)->has_zone_map()) {
+if (_column_readers.count(uid) < 1 ||
!_column_readers.at(uid)->has_zone_map() ||
+cond_in_compound_query(column_id)) {
Review Comment:
It is unnecessary in the latest code implementation and has been removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15358: [fix](multi-catalog) throw NPE when reading data after EOF
hello-stephen commented on PR #15358: URL: https://github.com/apache/doris/pull/15358#issuecomment-1365080195 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.7 seconds load time: 694 seconds storage size: 17123400441 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226105030_clickbench_pr_68829.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15224: [Feature](bitmap)Support return bitmap/hll data in select statement in vectoriza…
github-actions[bot] commented on PR #15224: URL: https://github.com/apache/doris/pull/15224#issuecomment-1365083564 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15224: [Feature](bitmap)Support return bitmap/hll data in select statement in vectoriza…
github-actions[bot] commented on PR #15224: URL: https://github.com/apache/doris/pull/15224#issuecomment-1365083582 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15350: [fix](nerieds)get datatype for binary arithmetic
hello-stephen commented on PR #15350: URL: https://github.com/apache/doris/pull/15350#issuecomment-1365096165 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.72 seconds load time: 633 seconds storage size: 17123649984 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226111020_clickbench_pr_68843.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #15342: [enhancement](regression) use sf0.1 data in datev2 and decimalv3 cases
dataroaring merged PR #15342: URL: https://github.com/apache/doris/pull/15342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #15362: [typo](docs) fix version tag for docs of s3 token
morningman merged PR #15362: URL: https://github.com/apache/doris/pull/15362 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (72f0003753 -> c3d0e2931a)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 72f0003753 [enhancement](regression) use sf0.1 data in datev2 and decimalv3 cases (#15342) add c3d0e2931a [typo](docs) fix version tag for docs of s3 token (#15362) No new revisions were added by this update. Summary of changes: .../docs/data-operate/import/import-scenes/external-storage-load.md | 5 - docs/en/docs/data-operate/import/import-way/s3-load-manual.md| 5 - .../Backup-and-Restore/CREATE-REPOSITORY.md | 5 - .../docs/data-operate/import/import-scenes/external-storage-load.md | 5 - docs/zh-CN/docs/data-operate/import/import-way/s3-load-manual.md | 4 +++- .../Backup-and-Restore/CREATE-REPOSITORY.md | 4 +++- 6 files changed, 22 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15358: [fix](multi-catalog) throw NPE when reading data after EOF
github-actions[bot] commented on PR #15358: URL: https://github.com/apache/doris/pull/15358#issuecomment-1365106792 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15358: [fix](multi-catalog) throw NPE when reading data after EOF
github-actions[bot] commented on PR #15358: URL: https://github.com/apache/doris/pull/15358#issuecomment-1365106872 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15366: [fix](pipeline) query blocked by waiting for shared hash table
hello-stephen commented on PR #15366: URL: https://github.com/apache/doris/pull/15366#issuecomment-1365111806 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.8 seconds load time: 643 seconds storage size: 17123673079 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226113733_clickbench_pr_68879.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15367: [fix](planner) set limit to nagtive value when has offset in limit
hello-stephen commented on PR #15367: URL: https://github.com/apache/doris/pull/15367#issuecomment-1365112345 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.84 seconds load time: 696 seconds storage size: 17123302880 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226113909_clickbench_pr_68870.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15368: [fix](fe)fix bug of the bucket shuffle join is not recognized
yiguolei merged PR #15368: URL: https://github.com/apache/doris/pull/15368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [fix](fe)fix bug of the bucket shuffle join is not recognized (#15368)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new facf76ea97 [fix](fe)fix bug of the bucket shuffle join is not
recognized (#15368)
facf76ea97 is described below
commit facf76ea9771384a0b2392ab81a4107f7330e173
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Mon Dec 26 19:41:19 2022 +0800
[fix](fe)fix bug of the bucket shuffle join is not recognized (#15368)
---
.../org/apache/doris/planner/DistributedPlanner.java| 2 +-
.../main/java/org/apache/doris/planner/PlanNode.java| 17 +
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java
index 840f28fb9d..682ad03553 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java
@@ -639,7 +639,7 @@ public class DistributedPlanner {
continue;
}
-SlotRef leftSlot = lhsJoinExpr.unwrapSlotRef();
+SlotRef leftSlot =
node.getChild(0).findSrcSlotRef(lhsJoinExpr.getSrcSlotRef());
if (leftSlot.getTable() instanceof OlapTable
&&
leftScanNode.desc.getSlots().contains(leftSlot.getDesc())) {
// table name in SlotRef is not the really name. `select *
from test as t`
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
index 63f28db103..13bf4921c3 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
@@ -28,6 +28,7 @@ import org.apache.doris.analysis.SlotRef;
import org.apache.doris.analysis.TupleDescriptor;
import org.apache.doris.analysis.TupleId;
import org.apache.doris.catalog.Function;
+import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.NotImplementedException;
@@ -861,6 +862,22 @@ abstract public class PlanNode extends TreeNode {
return null;
}
+public SlotRef findSrcSlotRef(SlotRef slotRef) {
+if (slotRef.getTable() instanceof OlapTable) {
+return slotRef;
+}
+if (this instanceof HashJoinNode) {
+HashJoinNode hashJoinNode = (HashJoinNode) this;
+SlotRef inputSlotRef = hashJoinNode.getMappedInputSlotRef(slotRef);
+if (inputSlotRef != null) {
+return hashJoinNode.getChild(0).findSrcSlotRef(inputSlotRef);
+} else {
+return slotRef;
+}
+}
+return slotRef;
+}
+
protected void addRuntimeFilter(RuntimeFilter filter) {
runtimeFilters.add(filter); }
protected Collection getRuntimeFilters() { return
runtimeFilters; }
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057206727
##
be/src/olap/rowset/segment_v2/segment_iterator.cpp:
##
@@ -1240,6 +1441,44 @@ Status SegmentIterator::next_batch(vectorized::Block*
block) {
return Status::OK();
}
+void SegmentIterator::_output_index_return_column(vectorized::Block* block) {
+if (block->rows() == 0) {
+return;
+}
+
+for (auto column_sign : _rowid_result_for_index) {
+block->insert({vectorized::ColumnUInt8::create(),
+ std::make_shared(),
column_sign.first});
+if (!column_sign.second.first) {
+// predicate not in compound query
+continue;
+}
+_build_index_return_column(block, column_sign.first,
column_sign.second.second);
+}
+}
+
+void SegmentIterator::_build_index_return_column(vectorized::Block* block,
Review Comment:
got it, has been updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057206996
##
be/src/olap/iterators.h:
##
@@ -82,6 +83,7 @@ class StorageReadOptions {
// reader's column predicate, nullptr if not existed
// used to fiter rows in row block
std::vector column_predicates;
+std::vector all_compound_column_predicates;
Review Comment:
rename to `column_predicates_except_leafnode_of_andnode`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057207271
##
be/src/olap/reader.h:
##
@@ -204,6 +209,7 @@ class TabletReader {
std::vector _is_lower_keys_included;
std::vector _is_upper_keys_included;
std::vector _col_predicates;
+std::vector _all_compound_col_predicates;
Review Comment:
rename to _col_preds_except_leafnode_of_andnode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly opened a new pull request, #15370: [feature](nereids)add variance function for nereids
englefly opened a new pull request, #15370: URL: https://github.com/apache/doris/pull/15370 # Proposed changes support `variance` function currently, it dose not support decimalV3 type Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057208267
##
be/src/olap/rowset/rowset_reader_context.h:
##
@@ -45,6 +46,7 @@ struct RowsetReaderContext {
// column name -> column predicate
// adding column_name for predicate to make use of column selectivity
const std::vector* predicates = nullptr;
+const std::vector* all_compound_predicates = nullptr;
Review Comment:
rename to `predicates_except_leafnode_of_andnode`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035: URL: https://github.com/apache/doris/pull/13035#discussion_r1057209350 ## be/src/vec/exec/scan/vscan_node.cpp: ## @@ -425,6 +425,7 @@ Status VScanNode::_normalize_predicate(VExpr* conjunct_expr_root, VExpr** output auto impl = conjunct_expr_root->get_impl(); // If impl is not null, which means this a conjuncts from runtime filter. VExpr* cur_expr = impl ? const_cast(impl) : conjunct_expr_root; +bool is_compound_predicate = TExprNodeType::COMPOUND_PRED == cur_expr->node_type(); Review Comment: you are right, changed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057209928
##
be/src/vec/exec/scan/vscan_node.cpp:
##
@@ -868,6 +880,193 @@ Status
VScanNode::_normalize_noneq_binary_predicate(VExpr* expr, VExprContext* e
return Status::OK();
}
+Status VScanNode::_normalize_compound_predicate(
+vectorized::VExpr* expr, VExprContext* expr_ctx, PushDownType* pdt,
+std::vector* column_value_rangs,
+const std::function&, const VSlotRef**,
VExpr**)>&
+in_predicate_checker,
+const std::function&, const VSlotRef**,
VExpr**)>&
+eq_predicate_checker) {
+if (TExprNodeType::COMPOUND_PRED == expr->node_type()) {
+DCHECK(expr->children().size() == 2);
+auto compound_fn_name = expr->fn().name.function_name;
+auto children_num = expr->children().size();
+for (auto i = 0; i < children_num; ++i) {
+VExpr* child_expr = expr->children()[i];
+if (TExprNodeType::BINARY_PRED == child_expr->node_type()) {
+SlotDescriptor* slot = nullptr;
+ColumnValueRangeType* range_on_slot = nullptr;
+if (_is_predicate_acting_on_slot(child_expr,
in_predicate_checker, &slot,
+ &range_on_slot) ||
+_is_predicate_acting_on_slot(child_expr,
eq_predicate_checker, &slot,
+ &range_on_slot)) {
+ColumnValueRangeType active_range =
+*range_on_slot; // copy, in order not to affect
the range in the _colname_to_value_range
+std::visit(
Review Comment:
active_range is a std::variant, should use std::visit to operate it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057210161
##
be/src/exec/olap_common.h:
##
@@ -239,6 +244,28 @@ class ColumnValueRange {
}
}
+void to_boundary_condition(std::vector& filters) {
+for (const auto& value : _boundary_values) {
+TCondition condition;
+condition.__set_column_name(_column_name);
+condition.__set_compound_type(_compound_type);
Review Comment:
You are right, I've removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15369: [fix](nereids) fix alias
hello-stephen commented on PR #15369: URL: https://github.com/apache/doris/pull/15369#issuecomment-1365119964 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.63 seconds load time: 631 seconds storage size: 17123712365 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226120033_clickbench_pr_68886.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15259: [Pipeline] Fix PipScannerContext::can_finish return wrong status
github-actions[bot] commented on PR #15259: URL: https://github.com/apache/doris/pull/15259#issuecomment-1365134000 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] huanccwang opened a new issue, #15371: [stream laod result stattus is success,but when I query data immediately,sometimes not find]
huanccwang opened a new issue, #15371: URL: https://github.com/apache/doris/issues/15371 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.4 ### What's Wrong? stream laod result stattus is success,but when I query data immediately,sometimes not find ### What You Expected? please tell me this is doris bug or my code bug?Anybody had met this question? ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin opened a new pull request, #15372: [fix](docs) fix some docs about stream load and select.
luozenglin opened a new pull request, #15372: URL: https://github.com/apache/doris/pull/15372 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057228269
##
be/src/vec/exec/scan/vscan_node.cpp:
##
@@ -447,6 +448,15 @@ VExpr* VScanNode::_normalize_predicate(VExpr*
conjunct_expr_root) {
},
*range);
}
+
+if (pdt == PushDownType::UNACCEPTABLE && is_compound_predicate) {
+std::vector column_value_rangs;
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057228880
##
be/src/vec/exec/scan/vscan_node.cpp:
##
@@ -822,6 +832,186 @@ Status
VScanNode::_normalize_noneq_binary_predicate(VExpr* expr, VExprContext* e
return Status::OK();
}
+Status VScanNode::_normalize_compound_predicate(vectorized::VExpr* expr,
+VExprContext* expr_ctx,
+PushDownType* pdt,
+std::vector* column_value_rangs,
+const std::function&, const
VSlotRef**, VExpr**)>& in_predicate_checker,
+const std::function&, const
VSlotRef**, VExpr**)>& eq_predicate_checker) {
+if (TExprNodeType::COMPOUND_PRED == expr->node_type()) {
+DCHECK(expr->children().size() == 2);
+auto compound_fn_name = expr->fn().name.function_name;
+auto children_num = expr->children().size();
+for (auto i = 0; i < children_num; ++i) {
+VExpr* child_expr = expr->children()[i];
+if (TExprNodeType::BINARY_PRED == child_expr->node_type()) {
+SlotDescriptor* slot = nullptr;
+ColumnValueRangeType* range_on_slot = nullptr;
+if (_is_predicate_acting_on_slot(child_expr,
in_predicate_checker, &slot, &range_on_slot) ||
+_is_predicate_acting_on_slot(child_expr,
eq_predicate_checker, &slot, &range_on_slot)) {
+ColumnValueRangeType active_range = *range_on_slot; //
copy, in order not to affect the range in the _colname_to_value_range
+std::visit(
+[&](auto& value_range) {
+ _normalize_binary_in_compound_predicate(
+child_expr, expr_ctx, slot, value_range,
pdt,
+
_get_compound_type_by_fn_name(compound_fn_name));
+},
+active_range);
+
+column_value_rangs->emplace_back(active_range);
+}
Review Comment:
no need else branch, can use _compound_value_ranges size to decide
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057229448
##
be/src/vec/exec/scan/vscan_node.cpp:
##
@@ -822,6 +832,186 @@ Status
VScanNode::_normalize_noneq_binary_predicate(VExpr* expr, VExprContext* e
return Status::OK();
}
+Status VScanNode::_normalize_compound_predicate(vectorized::VExpr* expr,
+VExprContext* expr_ctx,
+PushDownType* pdt,
+std::vector* column_value_rangs,
+const std::function&, const
VSlotRef**, VExpr**)>& in_predicate_checker,
+const std::function&, const
VSlotRef**, VExpr**)>& eq_predicate_checker) {
+if (TExprNodeType::COMPOUND_PRED == expr->node_type()) {
+DCHECK(expr->children().size() == 2);
+auto compound_fn_name = expr->fn().name.function_name;
+auto children_num = expr->children().size();
+for (auto i = 0; i < children_num; ++i) {
+VExpr* child_expr = expr->children()[i];
+if (TExprNodeType::BINARY_PRED == child_expr->node_type()) {
+SlotDescriptor* slot = nullptr;
+ColumnValueRangeType* range_on_slot = nullptr;
+if (_is_predicate_acting_on_slot(child_expr,
in_predicate_checker, &slot, &range_on_slot) ||
+_is_predicate_acting_on_slot(child_expr,
eq_predicate_checker, &slot, &range_on_slot)) {
+ColumnValueRangeType active_range = *range_on_slot; //
copy, in order not to affect the range in the _colname_to_value_range
+std::visit(
+[&](auto& value_range) {
+ _normalize_binary_in_compound_predicate(
+child_expr, expr_ctx, slot, value_range,
pdt,
+
_get_compound_type_by_fn_name(compound_fn_name));
+},
+active_range);
+
+column_value_rangs->emplace_back(active_range);
+}
+} else if (TExprNodeType::COMPOUND_PRED ==
child_expr->node_type()) {
+_normalize_compound_predicate(
+child_expr, expr_ctx,
+pdt, column_value_rangs,
+in_predicate_checker, eq_predicate_checker);
+}
+}
+}
+
+return Status::OK();
+}
+
+template
+Status VScanNode::_normalize_binary_in_compound_predicate(
+vectorized::VExpr* expr, VExprContext* expr_ctx,
+SlotDescriptor* slot, ColumnValueRange& range,
+PushDownType* pdt, const TCompoundType::type& compound_type) {
+DCHECK(expr->children().size() == 2);
+if (TExprNodeType::BINARY_PRED == expr->node_type()) {
+auto eq_checker = [](const std::string& fn_name) { return fn_name ==
"eq"; };
+auto ne_checker = [](const std::string& fn_name) { return fn_name ==
"ne"; };
+auto noneq_checker = [](const std::string& fn_name) {
+return fn_name != "ne" && fn_name != "eq";
+};
+
+StringRef value;
+int slot_ref_child = -1;
+PushDownType eq_pdt =
+
_should_push_down_binary_predicate(reinterpret_cast(expr),
+ expr_ctx, &value,
&slot_ref_child, eq_checker);
+PushDownType ne_pdt =
+
_should_push_down_binary_predicate(reinterpret_cast(expr),
+ expr_ctx, &value,
&slot_ref_child, ne_checker);
+PushDownType noneq_pdt =
_should_push_down_binary_predicate(reinterpret_cast(expr),
+expr_ctx, &value,
&slot_ref_child, noneq_checker);
+
+if (eq_pdt == PushDownType::UNACCEPTABLE
+&& ne_pdt == PushDownType::UNACCEPTABLE
+&& noneq_pdt == PushDownType::UNACCEPTABLE) {
+return Status::OK();
+}
+DCHECK(slot_ref_child >= 0);
+
+if (eq_pdt == PushDownType::ACCEPTABLE) {
+auto temp_range =
ColumnValueRange::create_empty_column_value_range(slot->type().precision,
+
slot->type().scale);
+auto fn_name = std::string("");
Review Comment:
_change_value_range with fixed value not use fn_name, so set empty is ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057229781
##
be/src/olap/rowset/segment_v2/segment_iterator.cpp:
##
@@ -373,6 +392,145 @@ Status SegmentIterator::_apply_bitmap_index() {
return Status::OK();
}
+bool SegmentIterator::_is_literal_node(const TExprNodeType::type& node_type) {
+switch (node_type) {
+case TExprNodeType::BOOL_LITERAL:
+case TExprNodeType::INT_LITERAL:
+case TExprNodeType::LARGE_INT_LITERAL:
+case TExprNodeType::FLOAT_LITERAL:
+case TExprNodeType::DECIMAL_LITERAL:
+case TExprNodeType::STRING_LITERAL:
+case TExprNodeType::DATE_LITERAL:
+return true;
+default:
+return false;
+}
+}
+
+Status SegmentIterator::_execute_all_compound_predicates(vectorized::VExpr*
expr) {
+if (expr == nullptr) {
+return Status::OK();
+}
+
+auto children = expr->children();
+for (int i = 0; i < children.size(); ++i) {
+RETURN_IF_ERROR(_execute_all_compound_predicates(children[i]));
+}
+
+auto node_type = expr->node_type();
+if (node_type == TExprNodeType::SLOT_REF) {
+_column_predicate_info->column_name = expr->expr_name();
+} else if (_is_literal_node(node_type)) {
+auto v_literal_expr =
dynamic_cast(expr);
+_column_predicate_info->query_value = v_literal_expr->value();
+} else if (node_type == TExprNodeType::BINARY_PRED) {
+_column_predicate_info->query_op = expr->fn().name.function_name;
+// get child condition result in compound condtions
+auto column_sign = _gen_predicate_sign(_column_predicate_info.get());
+_column_predicate_info.reset(new ColumnPredicateInfo());
+if (_rowid_result_for_index.count(column_sign) > 0
+&& _rowid_result_for_index[column_sign].first) {
+auto apply_reuslt = _rowid_result_for_index[column_sign].second;
+_compound_predicate_execute_result.push_back(apply_reuslt);
+}
Review Comment:
no need else branch, can use _compound_predicate_execute_result size to
decide
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15333: [fix](inbitmap) fix core dump caused by bitmap filter with union
github-actions[bot] commented on PR #15333: URL: https://github.com/apache/doris/pull/15333#issuecomment-1365150871 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15333: [fix](inbitmap) fix core dump caused by bitmap filter with union
github-actions[bot] commented on PR #15333: URL: https://github.com/apache/doris/pull/15333#issuecomment-1365150840 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057231477
##
be/src/vec/exec/scan/vscan_node.cpp:
##
@@ -868,6 +880,193 @@ Status
VScanNode::_normalize_noneq_binary_predicate(VExpr* expr, VExprContext* e
return Status::OK();
}
+Status VScanNode::_normalize_compound_predicate(
+vectorized::VExpr* expr, VExprContext* expr_ctx, PushDownType* pdt,
+std::vector* column_value_rangs,
+const std::function&, const VSlotRef**,
VExpr**)>&
+in_predicate_checker,
+const std::function&, const VSlotRef**,
VExpr**)>&
+eq_predicate_checker) {
+if (TExprNodeType::COMPOUND_PRED == expr->node_type()) {
+DCHECK(expr->children().size() == 2);
+auto compound_fn_name = expr->fn().name.function_name;
+auto children_num = expr->children().size();
+for (auto i = 0; i < children_num; ++i) {
+VExpr* child_expr = expr->children()[i];
+if (TExprNodeType::BINARY_PRED == child_expr->node_type()) {
+SlotDescriptor* slot = nullptr;
+ColumnValueRangeType* range_on_slot = nullptr;
+if (_is_predicate_acting_on_slot(child_expr,
in_predicate_checker, &slot,
+ &range_on_slot) ||
+_is_predicate_acting_on_slot(child_expr,
eq_predicate_checker, &slot,
+ &range_on_slot)) {
+ColumnValueRangeType active_range =
+*range_on_slot; // copy, in order not to affect
the range in the _colname_to_value_range
+std::visit(
+[&](auto& value_range) {
+_normalize_binary_in_compound_predicate(
+child_expr, expr_ctx, slot,
value_range, pdt,
+
_get_compound_type_by_fn_name(compound_fn_name));
+},
+active_range);
+
+column_value_rangs->emplace_back(active_range);
+}
+} else if (TExprNodeType::COMPOUND_PRED ==
child_expr->node_type()) {
+_normalize_compound_predicate(child_expr, expr_ctx, pdt,
column_value_rangs,
+ in_predicate_checker,
eq_predicate_checker);
+}
Review Comment:
Not yet supported `IN` predicate, I will support IN predicate in another pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15370: [feature](nereids)add variance function for nereids
hello-stephen commented on PR #15370: URL: https://github.com/apache/doris/pull/15370#issuecomment-1365156015 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.03 seconds load time: 632 seconds storage size: 17123617731 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226130927_clickbench_pr_68939.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] etcZYP commented on pull request #15356: [Enhancement](fe): replace assertTrue(X.equals(X)) with assertEquals
etcZYP commented on PR #15356: URL: https://github.com/apache/doris/pull/15356#issuecomment-1365157448 apache master exit bug or mine? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #15356: [Enhancement](fe): replace assertTrue(X.equals(X)) with assertEquals
jackwener commented on PR #15356: URL: https://github.com/apache/doris/pull/15356#issuecomment-1365163450 You need rebase the master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15360: [fix](string) fix offsets over flow for extreme large String column
yiguolei merged PR #15360: URL: https://github.com/apache/doris/pull/15360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #15359: [Bug] BE coredump for extreme large string columns
yiguolei closed issue #15359: [Bug] BE coredump for extreme large string columns URL: https://github.com/apache/doris/issues/15359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (c3d0e2931a -> 301640d3c0)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from c3d0e2931a [typo](docs) fix version tag for docs of s3 token (#15362) add 301640d3c0 [fix](string) fix offsets over flow for extreme large String column (#15360) No new revisions were added by this update. Summary of changes: be/src/vec/columns/column_string.cpp | 6 ++ be/src/vec/columns/column_string.h | 24 +- .../main/java/org/apache/doris/common/Config.java | 2 +- 3 files changed, 30 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15357: [chore](gutil) remove some gutil files and use c++ stl instead
yiguolei merged PR #15357: URL: https://github.com/apache/doris/pull/15357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (301640d3c0 -> aa0f38f864)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 301640d3c0 [fix](string) fix offsets over flow for extreme large String column (#15360) add aa0f38f864 [chore](gutil) remove some gutil files and use c++ stl instead (#15357) No new revisions were added by this update. Summary of changes: be/src/gutil/CMakeLists.txt | 2 - be/src/gutil/cycleclock-inl.h | 209 be/src/gutil/once.cc | 50 --- be/src/gutil/once.h | 115 --- be/src/gutil/sysinfo-test.cc | 71 be/src/gutil/sysinfo.cc | 465 -- be/src/gutil/sysinfo.h| 67 be/src/gutil/walltime.h | 206 be/src/runtime/load_channel_mgr.cpp | 1 + be/src/runtime/load_channel_mgr.h | 1 - be/src/runtime/memory/mem_tracker_limiter.cpp | 1 - be/src/runtime/threadlocal.cc | 5 +- be/src/util/jni-util.cpp | 6 +- be/src/util/thread.cpp| 5 +- be/src/util/threadpool.cpp| 4 +- be/src/util/trace.cpp | 4 +- be/src/util/trace.h | 5 +- be/test/util/threadpool_test.cpp | 5 +- be/test/util/trace_test.cpp | 1 - 19 files changed, 16 insertions(+), 1207 deletions(-) delete mode 100644 be/src/gutil/cycleclock-inl.h delete mode 100644 be/src/gutil/once.cc delete mode 100644 be/src/gutil/once.h delete mode 100644 be/src/gutil/sysinfo-test.cc delete mode 100644 be/src/gutil/sysinfo.cc delete mode 100644 be/src/gutil/sysinfo.h delete mode 100644 be/src/gutil/walltime.h - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang opened a new pull request, #15373: [improvment](function) add new function substring_index
Yulei-Yang opened a new pull request, #15373: URL: https://github.com/apache/doris/pull/15373 pr https://github.com/apache/doris/pull/15024 for branch-1.2-lt # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057242836
##
be/src/olap/column_predicate.h:
##
@@ -32,6 +32,10 @@ namespace doris {
class Schema;
class RowBlockV2;
+struct PredicateParams {
Review Comment:
`PredicateParams` also can add other options which would push down to
storage level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057245023
##
be/src/vec/exec/scan/vscan_node.cpp:
##
@@ -868,6 +880,193 @@ Status
VScanNode::_normalize_noneq_binary_predicate(VExpr* expr, VExprContext* e
return Status::OK();
}
+Status VScanNode::_normalize_compound_predicate(
+vectorized::VExpr* expr, VExprContext* expr_ctx, PushDownType* pdt,
+std::vector* column_value_rangs,
+const std::function&, const VSlotRef**,
VExpr**)>&
+in_predicate_checker,
+const std::function&, const VSlotRef**,
VExpr**)>&
+eq_predicate_checker) {
+if (TExprNodeType::COMPOUND_PRED == expr->node_type()) {
+DCHECK(expr->children().size() == 2);
+auto compound_fn_name = expr->fn().name.function_name;
+auto children_num = expr->children().size();
+for (auto i = 0; i < children_num; ++i) {
+VExpr* child_expr = expr->children()[i];
+if (TExprNodeType::BINARY_PRED == child_expr->node_type()) {
+SlotDescriptor* slot = nullptr;
+ColumnValueRangeType* range_on_slot = nullptr;
+if (_is_predicate_acting_on_slot(child_expr,
in_predicate_checker, &slot,
+ &range_on_slot) ||
+_is_predicate_acting_on_slot(child_expr,
eq_predicate_checker, &slot,
+ &range_on_slot)) {
+ColumnValueRangeType active_range =
+*range_on_slot; // copy, in order not to affect
the range in the _colname_to_value_range
+std::visit(
+[&](auto& value_range) {
+_normalize_binary_in_compound_predicate(
+child_expr, expr_ctx, slot,
value_range, pdt,
+
_get_compound_type_by_fn_name(compound_fn_name));
+},
+active_range);
+
+column_value_rangs->emplace_back(active_range);
+}
+} else if (TExprNodeType::COMPOUND_PRED ==
child_expr->node_type()) {
+_normalize_compound_predicate(child_expr, expr_ctx, pdt,
column_value_rangs,
+ in_predicate_checker,
eq_predicate_checker);
+}
+}
+}
+
+return Status::OK();
+}
+
+template
+Status VScanNode::_normalize_binary_in_compound_predicate(
+vectorized::VExpr* expr, VExprContext* expr_ctx, SlotDescriptor* slot,
+ColumnValueRange& range, PushDownType* pdt, const
TCompoundType::type& compound_type) {
+DCHECK(expr->children().size() == 2);
+if (TExprNodeType::BINARY_PRED == expr->node_type()) {
+auto eq_checker = [](const std::string& fn_name) { return fn_name ==
"eq"; };
+auto ne_checker = [](const std::string& fn_name) { return fn_name ==
"ne"; };
+auto noneq_checker = [](const std::string& fn_name) {
+return fn_name != "ne" && fn_name != "eq";
+};
+
+StringRef value;
+int slot_ref_child = -1;
+PushDownType eq_pdt;
+PushDownType ne_pdt;
+PushDownType noneq_pdt;
+RETURN_IF_ERROR(_should_push_down_binary_predicate(
+reinterpret_cast(expr), expr_ctx, &value,
&slot_ref_child,
+eq_checker, eq_pdt));
+RETURN_IF_ERROR(_should_push_down_binary_predicate(
+reinterpret_cast(expr), expr_ctx, &value,
&slot_ref_child,
+ne_checker, ne_pdt));
+RETURN_IF_ERROR(_should_push_down_binary_predicate(
+reinterpret_cast(expr), expr_ctx, &value,
&slot_ref_child,
+noneq_checker, noneq_pdt));
+if (eq_pdt == PushDownType::UNACCEPTABLE && ne_pdt ==
PushDownType::UNACCEPTABLE &&
+noneq_pdt == PushDownType::UNACCEPTABLE) {
+return Status::OK();
+}
+DCHECK(slot_ref_child >= 0);
+
+if (eq_pdt == PushDownType::ACCEPTABLE) {
+auto temp_range =
ColumnValueRange::create_empty_column_value_range(
+slot->type().precision, slot->type().scale);
+auto fn_name = std::string("");
+if (value.data != nullptr) {
+if constexpr (T == TYPE_CHAR || T == TYPE_VARCHAR || T ==
TYPE_STRING ||
+ T == TYPE_HLL) {
+auto val = StringValue(value.data, value.size);
+RETURN_IF_ERROR(_change_value_range(
+temp_range, reinterpret_cast(&val),
+ColumnValueRange::add_fixed_value_range,
fn_name));
+} else {
+RETURN_IF_ERROR(_change_value_range(
+temp_range,
reinterpret_cast(const_cast(value.data)),
+
[GitHub] [doris] Tanya-W commented on a diff in pull request #13035: [improvement](index) Support bitmap index can be applied with compound predicate when enable vectorized engine query
Tanya-W commented on code in PR #13035:
URL: https://github.com/apache/doris/pull/13035#discussion_r1057245888
##
be/src/vec/exec/scan/new_olap_scan_node.cpp:
##
@@ -207,6 +207,29 @@ Status NewOlapScanNode::_build_key_ranges_and_filters() {
}
}
+for (auto i = 0; i < _compound_value_ranges.size(); ++i) {
+std::vector conditions;
+for (auto& iter : _compound_value_ranges[i]) {
+std::vector filters;
+std::visit([&](auto&& range) {
+if (range.is_boundary_value_range()) {
+range.to_boundary_condition(filters);
Review Comment:
Boundary value of a range, such as condition with operator `>` `<` `>=` `<=`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhengshengjun opened a new issue, #15374: [Bug] dereference to nullptr in Tablet::calc_delete_bitmap
zhengshengjun opened a new issue, #15374: URL: https://github.com/apache/doris/issues/15374 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.2-lts ### What's Wrong? *** Query id: 0-0 *** *** Aborted at 1671959691 (unix time) try "date -d @1671959691" if you are using GNU date *** *** Current BE git commitID: 2b61c39 *** *** SIGSEGV address not mapped to object (@0x18) received by PID 1682989 (TID 0x7f82b1bfb700) from PID 24; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at ../src/common/signal_handler.h:420 1# 0x7F846C128060 in /lib/x86_64-linux-gnu/libc.so.6 2# doris::Tablet::calc_delete_bitmap(doris::RowsetId, std::vector, std::allocator > > const&, std::unordered_set, std::allocator > const*, std::shared_ptr, long, bool) at ../src/olap/tablet.cpp:2049 3# doris::Tablet::update_delete_bitmap_without_lock(std::shared_ptr const&) at /usr/local/include/c++/11.3.0/bits/shared_ptr.h:122 4# doris::Tablet::revise_tablet_meta(std::vector, std::allocator > > const&, std::vector > const&) at ../src/olap/tablet.cpp:212 5# doris::EngineCloneTask::_finish_full_clone(doris::Tablet*, doris::TabletMeta*) at ../src/olap/task/engine_clone_task.cpp:654 6# doris::EngineCloneTask::_finish_clone(doris::Tablet*, std::__cxx11::basic_string, std::allocator > const&, long, bool) at ../src/olap/task/engine_clone_task.cpp:521 7# doris::EngineCloneTask::_do_clone() at ../src/olap/task/engine_clone_task.cpp:117 8# doris::EngineCloneTask::execute() at ../src/olap/task/engine_clone_task.cpp:69 9# doris::StorageEngine::execute_task(doris::EngineTask*) at ../src/olap/storage_engine.cpp:1025 10# doris::TaskWorkerPool::_clone_worker_thread_callback() at ../src/agent/task_worker_pool.cpp:917 11# doris::ThreadPool::dispatch_thread() at ../src/util/threadpool.cpp:542 12# doris::Thread::supervise_thread(void*) at ../src/util/thread.cpp:455 13# start_thread at /build/glibc-77giwP/glibc-2.24/nptl/pthread_create.c:456 14# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:99 ### What You Expected? check null ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhengshengjun opened a new pull request, #15375: [fix] fix dereference to nullptr in Tablet::calc_delete_bitmap
zhengshengjun opened a new pull request, #15375: URL: https://github.com/apache/doris/pull/15375 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhengshengjun commented on pull request #15375: [fix] fix dereference to nullptr in Tablet::calc_delete_bitmap
zhengshengjun commented on PR #15375: URL: https://github.com/apache/doris/pull/15375#issuecomment-1365204722 *** Query id: 0-0 *** *** Aborted at 1671959691 (unix time) try "date -d @1671959691" if you are using GNU date *** *** Current BE git commitID: 2b61c39 *** *** SIGSEGV address not mapped to object (@0x18) received by PID 1682989 (TID 0x7f82b1bfb700) from PID 24; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at ../src/common/signal_handler.h:420 1# 0x7F846C128060 in /lib/x86_64-linux-gnu/libc.so.6 2# doris::Tablet::calc_delete_bitmap(doris::RowsetId, std::vector > const&, std::unordered_set const*, std::shared_ptrdoris::DeleteBitmap, long, bool) at ../src/olap/tablet.cpp:2049 3# doris::Tablet::update_delete_bitmap_without_lock(std::shared_ptrdoris::Rowset const&) at /usr/local/include/c++/11.3.0/bits/shared_ptr.h:122 4# doris::Tablet::revise_tablet_meta(std::vector > const&, std::vector const&) at ../src/olap/tablet.cpp:212 5# doris::EngineCloneTask::_finish_full_clone(doris::Tablet*, doris::TabletMeta*) at ../src/olap/task/engine_clone_task.cpp:654 6# doris::EngineCloneTask::_finish_clone(doris::Tablet*, std::__cxx11::basic_string const&, long, bool) at ../src/olap/task/engine_clone_task.cpp:521 7# doris::EngineCloneTask::_do_clone() at ../src/olap/task/engine_clone_task.cpp:117 8# doris::EngineCloneTask::execute() at ../src/olap/task/engine_clone_task.cpp:69 9# doris::StorageEngine::execute_task(doris::EngineTask*) at ../src/olap/storage_engine.cpp:1025 10# doris::TaskWorkerPool::_clone_worker_thread_callback() at ../src/agent/task_worker_pool.cpp:917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15375: [fix] fix dereference to nullptr in Tablet::calc_delete_bitmap
github-actions[bot] commented on PR #15375: URL: https://github.com/apache/doris/pull/15375#issuecomment-1365206024 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15375: [fix] fix dereference to nullptr in Tablet::calc_delete_bitmap
hello-stephen commented on PR #15375: URL: https://github.com/apache/doris/pull/15375#issuecomment-1365218877 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.81 seconds load time: 641 seconds storage size: 17123372435 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226142026_clickbench_pr_69001.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #15358: [fix](multi-catalog) throw NPE when reading data after EOF
morningman merged PR #15358: URL: https://github.com/apache/doris/pull/15358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](multi-catalog) throw NPE when reading data after EOF (#15358)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fc8f6a0715 [fix](multi-catalog) throw NPE when reading data after EOF
(#15358)
fc8f6a0715 is described below
commit fc8f6a0715aae0478be2aa1b80338204117199b4
Author: Ashin Gau
AuthorDate: Mon Dec 26 22:49:35 2022 +0800
[fix](multi-catalog) throw NPE when reading data after EOF (#15358)
1. Fix 1 bug:
Throw null pointer exception when reading data after the reader reaches the
end of file, so should return directly when `_do_lazy_read` read no data.
2. Optimize code:
Remove unused parameters.
3. Fix regression test
---
.../exec/format/parquet/vparquet_column_reader.cpp | 1 -
.../exec/format/parquet/vparquet_column_reader.h | 2 +-
.../exec/format/parquet/vparquet_group_reader.cpp | 3 +-
.../hive/test_external_brown.out | 62 +++---
.../hive/test_external_brown.groovy| 11 ++--
.../hive/test_external_catalog_hive.groovy | 3 +-
6 files changed, 43 insertions(+), 39 deletions(-)
diff --git a/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
b/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
index a7d95225b6..da08d1299e 100644
--- a/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
+++ b/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
@@ -27,7 +27,6 @@
namespace doris::vectorized {
Status ParquetColumnReader::create(io::FileReaderSPtr file, FieldSchema* field,
- const ParquetReadColumn& column,
const tparquet::RowGroup& row_group,
const std::vector& row_ranges,
cctz::time_zone* ctz,
std::unique_ptr&
reader,
diff --git a/be/src/vec/exec/format/parquet/vparquet_column_reader.h
b/be/src/vec/exec/format/parquet/vparquet_column_reader.h
index 0f61c5742c..8dff684730 100644
--- a/be/src/vec/exec/format/parquet/vparquet_column_reader.h
+++ b/be/src/vec/exec/format/parquet/vparquet_column_reader.h
@@ -108,7 +108,7 @@ public:
ColumnSelectVector& select_vector, size_t
batch_size,
size_t* read_rows, bool* eof) = 0;
static Status create(io::FileReaderSPtr file, FieldSchema* field,
- const ParquetReadColumn& column, const
tparquet::RowGroup& row_group,
+ const tparquet::RowGroup& row_group,
const std::vector& row_ranges,
cctz::time_zone* ctz,
std::unique_ptr& reader, size_t
max_buf_size);
void init_column_metadata(const tparquet::ColumnChunk& chunk);
diff --git a/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp
b/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp
index b406919b4c..b88f6981b3 100644
--- a/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp
+++ b/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp
@@ -57,7 +57,7 @@ Status RowGroupReader::init(const FieldDescriptor& schema,
std::vector
for (auto& read_col : _read_columns) {
auto field =
const_cast(schema.get_column(read_col._file_slot_name));
std::unique_ptr reader;
-RETURN_IF_ERROR(ParquetColumnReader::create(_file_reader, field,
read_col, _row_group_meta,
+RETURN_IF_ERROR(ParquetColumnReader::create(_file_reader, field,
_row_group_meta,
_read_ranges, _ctz,
reader, max_buf_size));
auto col_iter = col_offsets.find(read_col._parquet_col_id);
if (col_iter != col_offsets.end()) {
@@ -233,6 +233,7 @@ Status RowGroupReader::_do_lazy_read(Block* block, size_t
batch_size, size_t* re
DCHECK_EQ(pre_read_rows + _cached_filtered_rows, 0);
*read_rows = 0;
*batch_eof = true;
+return Status::OK();
}
ColumnSelectVector& select_vector = *select_vector_ptr;
diff --git
a/regression-test/data/external_table_emr_p2/hive/test_external_brown.out
b/regression-test/data/external_table_emr_p2/hive/test_external_brown.out
index c4dca4f351..58936b7593 100644
--- a/regression-test/data/external_table_emr_p2/hive/test_external_brown.out
+++ b/regression-test/data/external_table_emr_p2/hive/test_external_brown.out
@@ -769,17 +769,17 @@ cslab9e 2017-01-10T14:19:45
2019-12-01T09:59:5816C448031108ABA balcony_5 door5
temperature F 32.0
-- !15 --
+balcony_3 door3 SUMMER
balcony_3 door3 WINTER
+balcony_5 door5 SUMMER
balcony_5 door5 WINTER
+front_center_1 door1 SUMMER
front_center_1 door1 WINTER
+front_left_1 door1 SUMMER
front_left_1 door1 WIN
[GitHub] [doris] hello-stephen commented on pull request #15356: [Enhancement](fe): replace assertTrue(X.equals(X)) with assertEquals
hello-stephen commented on PR #15356: URL: https://github.com/apache/doris/pull/15356#issuecomment-1365230927 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35 seconds load time: 633 seconds storage size: 17122870560 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221226145145_clickbench_pr_69015.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15367: [fix](planner) set limit to nagtive value when has offset in limit
github-actions[bot] commented on PR #15367: URL: https://github.com/apache/doris/pull/15367#issuecomment-1365242112 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15367: [fix](planner) set limit to nagtive value when has offset in limit
github-actions[bot] commented on PR #15367: URL: https://github.com/apache/doris/pull/15367#issuecomment-1365242129 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed issue #15331: [Bug] be core dump caused by bitmap filter with union
morningman closed issue #15331: [Bug] be core dump caused by bitmap filter with union URL: https://github.com/apache/doris/issues/15331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
