[doris] branch master updated (90f11ed7c1 -> d73e437718)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 90f11ed7c1 [enhancement](Nereids) remove unnecessary exchange between global and distinct local aggregate node (#13057) add d73e437718 [fix](array-type) fix the be core dump when use string to insert array (#12728) No new revisions were added by this update. Summary of changes: be/src/vec/data_types/data_type_number_base.cpp | 8 be/src/vec/functions/function_cast.h | 19 +-- .../data/load/insert/test_array_insert.out| 4 .../suites/load/insert/test_array_insert.groovy | 17 + 4 files changed, 38 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug](array-type) Fix memory buffer overflow (#13074)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 20b583c91e [Bug](array-type) Fix memory buffer overflow (#13074)
20b583c91e is described below
commit 20b583c91ed3e75e3893b0fb02e60ab4c91e7647
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Mon Oct 10 11:42:13 2022 +0800
[Bug](array-type) Fix memory buffer overflow (#13074)
---
be/src/olap/rowset/segment_v2/column_reader.cpp | 1 +
be/src/olap/rowset/segment_v2/column_reader.h| 2 +-
be/src/olap/rowset/segment_v2/column_writer.cpp | 4 ++--
.../sql_functions/array_functions/sql/q03.out| 20
.../sql_functions/array_functions/sql/q03.sql| 6 ++
5 files changed, 30 insertions(+), 3 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/column_reader.cpp
b/be/src/olap/rowset/segment_v2/column_reader.cpp
index 1d0345ebb8..6ee44e6e0a 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/column_reader.cpp
@@ -755,6 +755,7 @@ Status FileColumnIterator::next_batch_of_zone_map(size_t*
n, vectorized::Mutable
Status FileColumnIterator::next_batch(size_t* n, vectorized::MutableColumnPtr&
dst,
bool* has_null) {
size_t curr_size = dst->byte_size();
+dst->reserve(*n);
size_t remaining = *n;
*has_null = false;
while (remaining > 0) {
diff --git a/be/src/olap/rowset/segment_v2/column_reader.h
b/be/src/olap/rowset/segment_v2/column_reader.h
index d165dff067..ca6e08e229 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.h
+++ b/be/src/olap/rowset/segment_v2/column_reader.h
@@ -375,7 +375,7 @@ public:
class ArrayFileColumnIterator final : public ColumnIterator {
public:
-explicit ArrayFileColumnIterator(ColumnReader* reader, FileColumnIterator*
length_reader,
+explicit ArrayFileColumnIterator(ColumnReader* reader, FileColumnIterator*
offset_reader,
ColumnIterator* item_iterator,
ColumnIterator* null_iterator);
~ArrayFileColumnIterator() override = default;
diff --git a/be/src/olap/rowset/segment_v2/column_writer.cpp
b/be/src/olap/rowset/segment_v2/column_writer.cpp
index caf9dc0629..2f1696a435 100644
--- a/be/src/olap/rowset/segment_v2/column_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/column_writer.cpp
@@ -513,13 +513,13 @@ Status ScalarColumnWriter::finish_current_page() {
ArrayColumnWriter::ArrayColumnWriter(const ColumnWriterOptions& opts,
std::unique_ptr field,
- ScalarColumnWriter* length_writer,
+ ScalarColumnWriter* offset_writer,
ScalarColumnWriter* null_writer,
std::unique_ptr item_writer)
: ColumnWriter(std::move(field), opts.meta->is_nullable()),
_item_writer(std::move(item_writer)),
_opts(opts) {
-_offset_writer.reset(length_writer);
+_offset_writer.reset(offset_writer);
if (is_nullable()) {
_null_writer.reset(null_writer);
}
diff --git
a/regression-test/data/query_p0/sql_functions/array_functions/sql/q03.out
b/regression-test/data/query_p0/sql_functions/array_functions/sql/q03.out
new file mode 100644
index 00..fb16bef5ae
--- /dev/null
+++ b/regression-test/data/query_p0/sql_functions/array_functions/sql/q03.out
@@ -0,0 +1,20 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !q03 --
+0
+
+-- !q03_2 --
+0
+
+-- !q03_3 --
+0
+
+-- !q03_4 --
+0
+
+-- !q03_5 --
+3
+
+-- !q03_6 --
+[1001]
+[1001]
+[1002]
diff --git
a/regression-test/suites/query_p0/sql_functions/array_functions/sql/q03.sql
b/regression-test/suites/query_p0/sql_functions/array_functions/sql/q03.sql
new file mode 100644
index 00..f87f346cfe
--- /dev/null
+++ b/regression-test/suites/query_p0/sql_functions/array_functions/sql/q03.sql
@@ -0,0 +1,6 @@
+ADMIN SET FRONTEND CONFIG ('enable_array_type' = 'true');
+set enable_vectorized_engine=true;
+DROP TABLE IF EXISTS array_insert_select_test;
+CREATE TABLE array_insert_select_test (id int, c_array array) ENGINE
= Olap DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES('replication_num' = '1');
+insert into array_insert_select_test select k1, collect_list(k3) from
test_query_db.test group by k1;
+select c_array from array_insert_select_test;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [ResourceTag](tag) Unified tag format verification (#13312)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a2a2be22a5 [ResourceTag](tag) Unified tag format verification (#13312)
a2a2be22a5 is described below
commit a2a2be22a502980b3a9bec7860d708989b831800
Author: Zhengguo Yang
AuthorDate: Fri Oct 14 14:21:55 2022 +0800
[ResourceTag](tag) Unified tag format verification (#13312)
---
.../org/apache/doris/common/util/PropertyAnalyzer.java| 3 ++-
.../src/main/java/org/apache/doris/resource/Tag.java | 15 ++-
.../org/apache/doris/catalog/ReplicaAllocationTest.java | 2 +-
3 files changed, 13 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
index 8b72c9cd33..b7e604f5aa 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
@@ -653,7 +653,8 @@ public class PropertyAnalyzer {
continue;
}
String val = entry.getValue().replaceAll(" ", "");
-tagMap.put(keyParts[1], val);
+Tag tag = Tag.create(keyParts[1], val);
+tagMap.put(tag.type, tag.value);
iter.remove();
}
if (tagMap.isEmpty() && defaultValue != null) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/Tag.java
b/fe/fe-core/src/main/java/org/apache/doris/resource/Tag.java
index d713e2c61e..6b697d15ba 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/Tag.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/Tag.java
@@ -71,7 +71,9 @@ public class Tag implements Writable {
public static final ImmutableSet RESERVED_TAG_VALUES =
ImmutableSet.of(
VALUE_FRONTEND, VALUE_BACKEND, VALUE_BROKER, VALUE_REMOTE_STORAGE,
VALUE_STORE, VALUE_COMPUTATION,
VALUE_DEFAULT_CLUSTER);
-private static final String TAG_REGEX = "^[a-zA-Z][a-zA-Z0-9_]{0,32}$";
+private static final String TAG_TYPE_REGEX = "^[a-z][a-z0-9_]{0,32}$";
+private static final String TAG_VALUE_REGEX =
"^[a-zA-Z][a-zA-Z0-9_]{0,32}$";
+
public static final Tag DEFAULT_BACKEND_TAG;
public static final Tag INVALID_TAG;
@@ -87,13 +89,16 @@ public class Tag implements Writable {
public String value;
private Tag(String type, String val) {
-this.type = type.toLowerCase();
-this.value = val.toLowerCase();
+this.type = type;
+this.value = val;
}
public static Tag create(String type, String value) throws
AnalysisException {
-if (!type.matches(TAG_REGEX) || !value.matches(TAG_REGEX)) {
-throw new AnalysisException("Invalid tag format: " + type + ":" +
value);
+if (!type.matches(TAG_TYPE_REGEX)) {
+throw new AnalysisException("Invalid tag type format: " + type);
+}
+if (!value.matches(TAG_VALUE_REGEX)) {
+throw new AnalysisException("Invalid tag value format: " + value);
}
return new Tag(type, value);
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/ReplicaAllocationTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/ReplicaAllocationTest.java
index e751a39761..00dbf1443c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/catalog/ReplicaAllocationTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/catalog/ReplicaAllocationTest.java
@@ -123,7 +123,7 @@ public class ReplicaAllocationTest {
properties.clear();
properties.put(PropertyAnalyzer.PROPERTIES_REPLICATION_ALLOCATION,
"tag.location.12321:1");
-ExceptionChecker.expectThrowsWithMsg(AnalysisException.class, "Invalid
tag format: location:12321",
+ExceptionChecker.expectThrowsWithMsg(AnalysisException.class, "Invalid
tag value format: 12321",
() -> PropertyAnalyzer.analyzeReplicaAllocation(properties,
""));
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bugfix](MV) Fix insert negative value to table with bitmap_union MV will cause count distinct result incorrect (#13507)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3e92f742bf [Bugfix](MV) Fix insert negative value to table with
bitmap_union MV will cause count distinct result incorrect (#13507)
3e92f742bf is described below
commit 3e92f742bfd7b0c293a6279cd597535eb26d27a5
Author: Zhengguo Yang
AuthorDate: Fri Oct 21 16:07:31 2022 +0800
[Bugfix](MV) Fix insert negative value to table with bitmap_union MV will
cause count distinct result incorrect (#13507)
---
be/src/exprs/bitmap_function.cpp | 24 +++
be/src/exprs/bitmap_function.h | 1 +
be/src/olap/schema_change.cpp | 35 ++
be/src/vec/functions/function_bitmap.cpp | 13 ++--
gensrc/script/doris_builtins_functions.py | 2 +-
.../rollup/test_materialized_view_bitmap.groovy| 77 ++
6 files changed, 133 insertions(+), 19 deletions(-)
diff --git a/be/src/exprs/bitmap_function.cpp b/be/src/exprs/bitmap_function.cpp
index 834d283a29..1d4fc4df47 100644
--- a/be/src/exprs/bitmap_function.cpp
+++ b/be/src/exprs/bitmap_function.cpp
@@ -173,6 +173,30 @@ StringVal
BitmapFunctions::to_bitmap(doris_udf::FunctionContext* ctx,
return serialize(ctx, &bitmap);
}
+StringVal BitmapFunctions::to_bitmap_with_check(doris_udf::FunctionContext*
ctx,
+const doris_udf::StringVal&
src) {
+BitmapValue bitmap;
+
+if (!src.is_null) {
+StringParser::ParseResult parse_result = StringParser::PARSE_SUCCESS;
+uint64_t int_value = StringParser::string_to_unsigned_int(
+reinterpret_cast(src.ptr), src.len, &parse_result);
+if (parse_result == StringParser::PARSE_SUCCESS) {
+bitmap.add(int_value);
+} else {
+std::stringstream ss;
+ss << "The input: " << src.to_string()
+ << " is not valid, to_bitmap only support bigint value from 0
to "
+ "18446744073709551615 currently, cannot load negative values
to column with"
+ " to_bitmap MV on it.";
+ctx->set_error(ss.str().c_str());
+return StringVal::null();
+}
+}
+
+return serialize(ctx, &bitmap);
+}
+
StringVal BitmapFunctions::bitmap_hash(doris_udf::FunctionContext* ctx,
const doris_udf::StringVal& src) {
BitmapValue bitmap;
diff --git a/be/src/exprs/bitmap_function.h b/be/src/exprs/bitmap_function.h
index 15e164eef1..7de2363d3a 100644
--- a/be/src/exprs/bitmap_function.h
+++ b/be/src/exprs/bitmap_function.h
@@ -68,6 +68,7 @@ public:
static StringVal bitmap_serialize(FunctionContext* ctx, const StringVal&
src);
static StringVal to_bitmap(FunctionContext* ctx, const StringVal& src);
+static StringVal to_bitmap_with_check(FunctionContext* ctx, const
StringVal& src);
static StringVal bitmap_hash(FunctionContext* ctx, const StringVal& src);
static StringVal bitmap_hash64(FunctionContext* ctx, const StringVal& src);
static StringVal bitmap_or(FunctionContext* ctx, const StringVal& src,
const StringVal& dst);
diff --git a/be/src/olap/schema_change.cpp b/be/src/olap/schema_change.cpp
index 3295035f4b..45a8e4268b 100644
--- a/be/src/olap/schema_change.cpp
+++ b/be/src/olap/schema_change.cpp
@@ -437,9 +437,11 @@ bool to_bitmap(RowCursor* read_helper, RowCursor*
write_helper, const TabletColu
switch (ref_column.type()) {
case OLAP_FIELD_TYPE_TINYINT:
if (*(int8_t*)src < 0) {
-LOG(WARNING) << "The input: " << *(int8_t*)src
- << " is not valid, to_bitmap only support bigint
value from 0 to "
-"18446744073709551615 currently";
+LOG(WARNING)
+<< "The input: " << *(int8_t*)src
+<< " is not valid, to_bitmap only support bigint value
from 0 to "
+ "18446744073709551615 currently, cannot create MV
with to_bitmap on "
+ "column with negative values.";
return false;
}
origin_value = *(int8_t*)src;
@@ -449,9 +451,11 @@ bool to_bitmap(RowCursor* read_helper, RowCursor*
write_helper, const TabletColu
break;
case OLAP_FIELD_TYPE_SMALLINT:
if (*(int16_t*)src < 0) {
-LOG(WARNING) << "The input: " << *(int16_t*)src
- << " is not valid, to_bitmap only sup
[doris] branch struct-type updated: [WIP](struct-type) support struct-type in vectorize engine (#15665)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/struct-type by this push:
new 2578bad186 [WIP](struct-type) support struct-type in vectorize engine
(#15665)
2578bad186 is described below
commit 2578bad186bc234be6922a6ef9f2175d8819b14a
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Thu Jan 12 14:40:08 2023 +0800
[WIP](struct-type) support struct-type in vectorize engine (#15665)
---
be/src/olap/aggregate_func.cpp | 2 +
be/src/olap/aggregate_func.h | 15 ++
be/src/olap/field.h| 57 -
be/src/olap/olap_define.h | 3 +
be/src/olap/rowset/segment_v2/column_reader.cpp| 112 ++
be/src/olap/rowset/segment_v2/column_reader.h | 38
be/src/olap/rowset/segment_v2/column_writer.cpp| 174 +++
be/src/olap/rowset/segment_v2/column_writer.h | 51 +
be/src/olap/rowset/segment_v2/segment_writer.cpp | 9 +
be/src/olap/tablet_meta.cpp| 8 +-
be/src/olap/tablet_schema.cpp | 20 +-
be/src/olap/types.cpp | 52 -
be/src/olap/types.h| 245 -
be/src/runtime/CMakeLists.txt | 1 +
be/src/runtime/primitive_type.cpp | 15 ++
be/src/runtime/primitive_type.h| 1 +
be/src/runtime/struct_value.cpp| 27 +++
be/src/runtime/struct_value.h | 67 ++
be/src/runtime/types.cpp | 48 +++-
be/src/runtime/types.h | 2 +-
be/src/udf/udf.h | 1 +
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/columns/column_struct.cpp | 34 +--
be/src/vec/columns/column_struct.h | 15 +-
be/src/vec/data_types/data_type_factory.cpp| 41 +++-
be/src/vec/data_types/data_type_factory.hpp| 4 +
be/src/vec/data_types/data_type_struct.cpp | 33 ++-
be/src/vec/data_types/data_type_struct.h | 12 +-
be/src/vec/exprs/vcast_expr.cpp| 1 +
be/src/vec/exprs/vexpr.cpp | 5 +
be/src/vec/exprs/vstruct_literal.cpp | 37
be/src/vec/exprs/vstruct_literal.h | 34 +++
be/src/vec/functions/function_cast.h | 1 +
be/src/vec/olap/olap_data_convertor.cpp| 62 ++
be/src/vec/olap/olap_data_convertor.h | 21 ++
be/src/vec/sink/vmysql_result_writer.cpp | 100 +++--
be/src/vec/sink/vmysql_result_writer.h | 2 +-
be/src/vec/sink/vtablet_sink.cpp | 3 +-
fe/fe-core/src/main/cup/sql_parser.cup | 33 ++-
.../java/org/apache/doris/analysis/ColumnDef.java | 4 +
.../org/apache/doris/analysis/CreateTableStmt.java | 12 +-
.../main/java/org/apache/doris/analysis/Expr.java | 7 +-
.../org/apache/doris/analysis/StructLiteral.java | 150 +
.../java/org/apache/doris/analysis/Subquery.java | 2 +-
.../java/org/apache/doris/analysis/TypeDef.java| 47 ++--
.../java/org/apache/doris/catalog/ArrayType.java | 10 +
.../main/java/org/apache/doris/catalog/Column.java | 66 +++---
.../java/org/apache/doris/catalog/MapType.java | 5 +
.../org/apache/doris/catalog/PrimitiveType.java| 5 +-
.../java/org/apache/doris/catalog/StructField.java | 55 -
.../java/org/apache/doris/catalog/StructType.java | 94 +++-
.../main/java/org/apache/doris/catalog/Type.java | 24 +-
fe/fe-core/src/main/jflex/sql_scanner.flex | 4 +
gensrc/proto/types.proto | 1 +
gensrc/thrift/Exprs.thrift | 1 +
gensrc/thrift/Types.thrift | 1 +
56 files changed, 1702 insertions(+), 173 deletions(-)
diff --git a/be/src/olap/aggregate_func.cpp b/be/src/olap/aggregate_func.cpp
index 027105952c..7236488f78 100644
--- a/be/src/olap/aggregate_func.cpp
+++ b/be/src/olap/aggregate_func.cpp
@@ -147,6 +147,8 @@ AggregateFuncResolver::AggregateFuncResolver() {
OLAP_FIELD_TYPE_DECIMAL128I>();
add_aggregate_mapping();
+// struct types
+add_aggregate_mapping();
// Min Aggregate Function
add_aggregate_mapping();
diff --git a/be/src/olap/aggregate_func.h b/be/src/olap/aggregate_func.h
index 5a397b658f..b72be9a7cd 100644
--- a/be/src/olap/aggregate_func.h
+++ b/be/src/olap/aggregate_func.h
@@ -107,6 +107,21 @@ struct BaseAggregateFuncs {
static void finalize(RowCursorCell* src, MemPool* mem_pool) {}
};
+// Define an empty trait for struct type because
+// we now only suppo
[doris] branch struct-type updated: [feature](struct-type) adjust the vexpr for struct type (#15669)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/struct-type by this push:
new 7f4de925d8 [feature](struct-type) adjust the vexpr for struct type
(#15669)
7f4de925d8 is described below
commit 7f4de925d81048ede17f760037531866f9eb6710
Author: carlvinhust2012
AuthorDate: Fri Jan 13 17:14:11 2023 +0800
[feature](struct-type) adjust the vexpr for struct type (#15669)
---
be/src/vec/columns/column_struct.cpp | 7 --
be/src/vec/columns/column_struct.h | 1 -
be/src/vec/data_types/data_type_factory.hpp| 8 ++
be/src/vec/data_types/data_type_struct.cpp | 99 +++---
be/src/vec/data_types/data_type_struct.h | 3 +
be/src/vec/functions/function_cast.h | 13 ++-
.../java/org/apache/doris/analysis/CastExpr.java | 8 ++
.../java/org/apache/doris/analysis/TypeDef.java| 2 +-
.../main/java/org/apache/doris/catalog/Column.java | 1 +
.../main/java/org/apache/doris/catalog/Type.java | 14 ++-
10 files changed, 134 insertions(+), 22 deletions(-)
diff --git a/be/src/vec/columns/column_struct.cpp
b/be/src/vec/columns/column_struct.cpp
index d51bebb536..afef00c314 100644
--- a/be/src/vec/columns/column_struct.cpp
+++ b/be/src/vec/columns/column_struct.cpp
@@ -22,13 +22,6 @@
namespace doris::vectorized {
-namespace ErrorCodes {
-extern const int ILLEGAL_COLUMN;
-extern const int NOT_IMPLEMENTED;
-extern const int CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE;
-extern const int LOGICAL_ERROR;
-} // namespace ErrorCodes
-
std::string ColumnStruct::get_name() const {
std::stringstream res;
res << "Struct(";
diff --git a/be/src/vec/columns/column_struct.h
b/be/src/vec/columns/column_struct.h
index a66d91f3e0..393b2c275f 100644
--- a/be/src/vec/columns/column_struct.h
+++ b/be/src/vec/columns/column_struct.h
@@ -103,7 +103,6 @@ public:
bool can_be_inside_nullable() const override { return true; }
MutableColumnPtr clone_empty() const override;
MutableColumnPtr clone_resized(size_t size) const override;
-
size_t size() const override { return columns.at(0)->size(); }
Field operator[](size_t n) const override;
diff --git a/be/src/vec/data_types/data_type_factory.hpp
b/be/src/vec/data_types/data_type_factory.hpp
index c5c6269288..be8b5b9a7b 100644
--- a/be/src/vec/data_types/data_type_factory.hpp
+++ b/be/src/vec/data_types/data_type_factory.hpp
@@ -112,6 +112,14 @@ public:
return entity.second;
}
}
+if (type_ptr->get_type_id() == TypeIndex::Struct) {
+
DataTypeFactory::instance().register_data_type(type_ptr->get_name(), type_ptr);
+for (const auto& entity : _invert_data_type_map) {
+if (entity.first->equals(*type_ptr)) {
+return entity.second;
+}
+}
+}
return _empty_string;
}
diff --git a/be/src/vec/data_types/data_type_struct.cpp
b/be/src/vec/data_types/data_type_struct.cpp
index 4364f5c34f..87b5c4f110 100644
--- a/be/src/vec/data_types/data_type_struct.cpp
+++ b/be/src/vec/data_types/data_type_struct.cpp
@@ -22,17 +22,6 @@
namespace doris::vectorized {
-namespace ErrorCodes {
-extern const int BAD_ARGUMENTS;
-extern const int DUPLICATE_COLUMN;
-extern const int EMPTY_DATA_PASSED;
-extern const int NOT_FOUND_COLUMN_IN_BLOCK;
-extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
-extern const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH;
-extern const int ILLEGAL_INDEX;
-extern const int LOGICAL_ERROR;
-} // namespace ErrorCodes
-
DataTypeStruct::DataTypeStruct(const DataTypes& elems_)
: elems(elems_), have_explicit_names(false) {
/// Automatically assigned names in form of '1', '2', ...
@@ -91,6 +80,94 @@ std::string DataTypeStruct::do_get_name() const {
return s.str();
}
+Status DataTypeStruct::from_string(ReadBuffer& rb, IColumn* column) const {
+DCHECK(!rb.eof());
+auto* struct_column = assert_cast(column);
+
+if (*rb.position() != '{') {
+return Status::InvalidArgument("Struct does not start with '{'
character, found '{}'",
+ *rb.position());
+}
+if (rb.count() < 2 || *(rb.end() - 1) != '}') {
+return Status::InvalidArgument("Struct does not end with '}'
character, found '{}'",
+ *(rb.end() - 1));
+}
+
+// here need handle the empty struct '{}'
+if (rb.count() == 2) {
+return Status::OK();
+}
+
+++rb.position();
+std::vector field_rbs;
+field_rbs.reserve(elems.size());
+
+// here get the value "jack" an
[doris] branch struct-type updated: save (#15906)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/struct-type by this push:
new d32456d603 save (#15906)
d32456d603 is described below
commit d32456d603908b56c3927c6cc30326994a6433f4
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Fri Jan 13 17:29:42 2023 +0800
save (#15906)
---
be/src/vec/data_types/data_type_factory.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/be/src/vec/data_types/data_type_factory.cpp
b/be/src/vec/data_types/data_type_factory.cpp
index 7100a679c0..09ddf70bbf 100644
--- a/be/src/vec/data_types/data_type_factory.cpp
+++ b/be/src/vec/data_types/data_type_factory.cpp
@@ -36,7 +36,7 @@ DataTypePtr DataTypeFactory::create_data_type(const
doris::Field& col_desc) {
names.reserve(field_size);
for (size_t i = 0; i < field_size; i++) {
dataTypes.push_back(create_data_type(*col_desc.get_sub_field(i)));
-names.push_back(col_desc.name());
+names.push_back(col_desc.get_sub_field(i).name());
}
nested = std::make_shared(dataTypes, names);
} else {
@@ -64,7 +64,7 @@ DataTypePtr DataTypeFactory::create_data_type(const
TabletColumn& col_desc, bool
names.reserve(col_size);
for (size_t i = 0; i < col_size; i++) {
dataTypes.push_back(create_data_type(col_desc.get_sub_column(i)));
-names.push_back(col_desc.name());
+names.push_back(col_desc.get_sub_field(i).name());
}
nested = std::make_shared(dataTypes, names);
} else {
@@ -343,7 +343,7 @@ DataTypePtr DataTypeFactory::create_data_type(const
PColumnMeta& pcolumn) {
names.reserve(col_size);
for (size_t i = 0; i < col_size; i++) {
dataTypes.push_back(create_data_type(pcolumn.children(i)));
-names.push_back(pcolumn.name());
+names.push_back(pcolumn.children(i).name());
}
nested = std::make_shared(dataTypes, names);
break;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch struct-type updated: save (#15908)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/struct-type by this push:
new 4ab824250a save (#15908)
4ab824250a is described below
commit 4ab824250a3a681bb1be85945cf9a1c91c45d1eb
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Fri Jan 13 17:46:49 2023 +0800
save (#15908)
---
be/src/vec/data_types/data_type_factory.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/data_types/data_type_factory.cpp
b/be/src/vec/data_types/data_type_factory.cpp
index 09ddf70bbf..530ccfa762 100644
--- a/be/src/vec/data_types/data_type_factory.cpp
+++ b/be/src/vec/data_types/data_type_factory.cpp
@@ -36,7 +36,7 @@ DataTypePtr DataTypeFactory::create_data_type(const
doris::Field& col_desc) {
names.reserve(field_size);
for (size_t i = 0; i < field_size; i++) {
dataTypes.push_back(create_data_type(*col_desc.get_sub_field(i)));
-names.push_back(col_desc.get_sub_field(i).name());
+names.push_back(col_desc.get_sub_field(i)->name());
}
nested = std::make_shared(dataTypes, names);
} else {
@@ -64,7 +64,7 @@ DataTypePtr DataTypeFactory::create_data_type(const
TabletColumn& col_desc, bool
names.reserve(col_size);
for (size_t i = 0; i < col_size; i++) {
dataTypes.push_back(create_data_type(col_desc.get_sub_column(i)));
-names.push_back(col_desc.get_sub_field(i).name());
+names.push_back(col_desc.get_sub_column(i).name());
}
nested = std::make_shared(dataTypes, names);
} else {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch struct-type updated: [fix](struct-type) support to remove double or single quote (#15977)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/struct-type by this push:
new b5a048b4ce [fix](struct-type) support to remove double or single quote
(#15977)
b5a048b4ce is described below
commit b5a048b4cefb61391f10640c79d74e505590c700
Author: carlvinhust2012
AuthorDate: Mon Jan 16 18:01:16 2023 +0800
[fix](struct-type) support to remove double or single quote (#15977)
---
be/src/vec/data_types/data_type_struct.cpp | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/be/src/vec/data_types/data_type_struct.cpp
b/be/src/vec/data_types/data_type_struct.cpp
index 87b5c4f110..a3a54966b0 100644
--- a/be/src/vec/data_types/data_type_struct.cpp
+++ b/be/src/vec/data_types/data_type_struct.cpp
@@ -114,6 +114,7 @@ Status DataTypeStruct::from_string(ReadBuffer& rb, IColumn*
column) const {
return Status::InvalidArgument("Invalid Length");
}
ReadBuffer field_rb(rb.position(), field_len);
+
size_t len = 0;
auto start_rb = field_rb.position();
while (!field_rb.eof() && *start_rb != ':') {
@@ -122,11 +123,12 @@ Status DataTypeStruct::from_string(ReadBuffer& rb,
IColumn* column) const {
}
ReadBuffer field(field_rb.position() + len + 1, field_rb.count() - len
- 1);
-if (field.count() < 2 || *field.position() != '"' || *field.end() !=
'"') {
-field_rbs.push_back(field);
+if (field.count() >= 2 && ((*field.position() == '"' && *(field.end()
- 1) == '"') ||
+ (*field.position() == '\'' && *(field.end()
- 1) == '\''))) {
+ReadBuffer field_no_quote(field.position() + 1, field.count() - 2);
+field_rbs.push_back(field_no_quote);
} else {
-ReadBuffer field_has_quote(field.position() + 1, field.count() -
2);
-field_rbs.push_back(field_has_quote);
+field_rbs.push_back(field);
}
rb.position() += field_len + 1;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch struct-type updated: [hotfix](struct-type) Fix type descriptor make array load core (#16020)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a commit to branch struct-type in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/struct-type by this push: new 92fcf9d931 [hotfix](struct-type) Fix type descriptor make array load core (#16020) 92fcf9d931 is described below commit 92fcf9d931a40d4772cc1d03b2694d4ca4652689 Author: xy720 <22125576+xy...@users.noreply.github.com> AuthorDate: Tue Jan 17 18:52:49 2023 +0800 [hotfix](struct-type) Fix type descriptor make array load core (#16020) --- be/src/vec/sink/vtablet_sink.cpp | 2 +- thirdparty/build-thirdparty.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/be/src/vec/sink/vtablet_sink.cpp b/be/src/vec/sink/vtablet_sink.cpp index 2f4cc66cc9..9dd772eee0 100644 --- a/be/src/vec/sink/vtablet_sink.cpp +++ b/be/src/vec/sink/vtablet_sink.cpp @@ -1403,7 +1403,7 @@ Status VOlapTableSink::_validate_column(RuntimeState* state, const TypeDescripto } fmt::format_to(error_prefix, "ARRAY type failed: "); RETURN_IF_ERROR(_validate_column( -state, nested_type, nested_type.contains_nulls[0], column_array->get_data_ptr(), +state, nested_type, type.contains_nulls[0], column_array->get_data_ptr(), slot_index, filter_bitmap, stop_processing, error_prefix, &permutation)); } break; diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh index 3798c54979..f7eb42fd22 100755 --- a/thirdparty/build-thirdparty.sh +++ b/thirdparty/build-thirdparty.sh @@ -1579,7 +1579,6 @@ if [[ "$(uname -s)" == 'Darwin' ]]; then build_gettext fi -build_clucene build_libunixodbc build_openssl build_libevent @@ -1636,5 +1635,6 @@ build_libbacktrace build_sse2neon build_xxhash build_concurrentqueue +build_clucene echo "Finished to build all thirdparties" - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (b73f345479 -> cb29d1a395)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from b73f345479 [fix](intersect) fix wrong result of intersect node (#24044) add cb29d1a395 fix compile error with gcc12 (#24049) No new revisions were added by this update. Summary of changes: be/src/olap/rowset/rowset_meta_manager.cpp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (6e28d878b5 -> 0c30fff811)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 6e28d878b5 [fix](hudi) compatible with hudi spark configuration and
support skip merge (#24067)
add 0c30fff811 add add navigator for vector distance functions (#24081)
No new revisions were added by this update.
Summary of changes:
.../{cosine_distance.md => cosine-distance.md} | 0
.../distance-functions/{inner_product.md => inner-product.md} | 0
.../distance-functions/{l1_distance.md => l1-distance.md} | 0
.../distance-functions/{l2_distance.md => l2-distance.md} | 0
docs/sidebars.json | 10 ++
.../{cosine_distance.md => cosine-distance.md} | 0
.../distance-functions/{inner_product.md => inner-product.md} | 0
.../distance-functions/{l1_distance.md => l1-distance.md} | 0
.../distance-functions/{l2_distance.md => l2-distance.md} | 0
9 files changed, 10 insertions(+)
rename
docs/en/docs/sql-manual/sql-functions/distance-functions/{cosine_distance.md =>
cosine-distance.md} (100%)
rename
docs/en/docs/sql-manual/sql-functions/distance-functions/{inner_product.md =>
inner-product.md} (100%)
rename
docs/en/docs/sql-manual/sql-functions/distance-functions/{l1_distance.md =>
l1-distance.md} (100%)
rename
docs/en/docs/sql-manual/sql-functions/distance-functions/{l2_distance.md =>
l2-distance.md} (100%)
rename
docs/zh-CN/docs/sql-manual/sql-functions/distance-functions/{cosine_distance.md
=> cosine-distance.md} (100%)
rename
docs/zh-CN/docs/sql-manual/sql-functions/distance-functions/{inner_product.md
=> inner-product.md} (100%)
rename
docs/zh-CN/docs/sql-manual/sql-functions/distance-functions/{l1_distance.md =>
l1-distance.md} (100%)
rename
docs/zh-CN/docs/sql-manual/sql-functions/distance-functions/{l2_distance.md =>
l2-distance.md} (100%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (e0834b2f46 -> 320f1e9bbf)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from e0834b2f46 [chore](explain) add annotation in explain string whether nereids is ON #24394 add 320f1e9bbf [improve](routineload) improve show routine load output (#24264) No new revisions were added by this update. Summary of changes: .../Show-Statements/SHOW-ROUTINE-LOAD-TASK.md | 2 +- .../Show-Statements/SHOW-ROUTINE-LOAD-TASK.md | 2 +- .../doris/analysis/CreateRoutineLoadStmt.java | 1 + .../doris/load/routineload/RoutineLoadJob.java | 29 +- 4 files changed, 20 insertions(+), 14 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (bcdb481374 -> 4a690f8b13)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from bcdb481374 [refactor](fragment) refactor non pipeline fragment executor (#23281) add 4a690f8b13 fix compile error with gcc12 (#23190) No new revisions were added by this update. Summary of changes: be/src/olap/rowset/rowset_meta_manager.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (2dda44d7b5 -> 22e373a799)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 2dda44d7b5 [fix](csv-reader)fix bug of multi-char delimiter in csv reader add 22e373a799 [feature](vector-search) add 4 distance functions to support vector search (#23129) No new revisions were added by this update. Summary of changes: be/src/vec/columns/column_nullable.h | 1 + .../functions/array/function_array_distance.cpp| 31 .../vec/functions/array/function_array_distance.h | 187 + .../functions/array/function_array_register.cpp| 2 + .../distance-functions/cosine_distance.md | 53 ++ .../distance-functions/inner_product.md| 53 ++ .../distance-functions/l1_distance.md | 53 ++ .../distance-functions/l2_distance.md | 53 ++ .../distance-functions/cosine_distance.md | 53 ++ .../distance-functions/inner_product.md| 53 ++ .../distance-functions/l1_distance.md | 53 ++ .../distance-functions/l2_distance.md | 53 ++ gensrc/script/doris_builtins_functions.py | 32 .../test_array_distance_functions.out | 19 +++ .../test_array_distance_functions.groovy | 39 + 15 files changed, 735 insertions(+) create mode 100644 be/src/vec/functions/array/function_array_distance.cpp create mode 100644 be/src/vec/functions/array/function_array_distance.h create mode 100644 docs/en/docs/sql-manual/sql-functions/distance-functions/cosine_distance.md create mode 100644 docs/en/docs/sql-manual/sql-functions/distance-functions/inner_product.md create mode 100644 docs/en/docs/sql-manual/sql-functions/distance-functions/l1_distance.md create mode 100644 docs/en/docs/sql-manual/sql-functions/distance-functions/l2_distance.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/distance-functions/cosine_distance.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/distance-functions/inner_product.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/distance-functions/l1_distance.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/distance-functions/l2_distance.md create mode 100644 regression-test/data/query_p0/sql_functions/array_functions/test_array_distance_functions.out create mode 100644 regression-test/suites/query_p0/sql_functions/array_functions/test_array_distance_functions.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch struct-type updated: [feature](struct-type) add children column when create table (#15089)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/struct-type by this push:
new a6c471bcdb [feature](struct-type) add children column when create
table (#15089)
a6c471bcdb is described below
commit a6c471bcdb1a88a32ab49fc58d83b1143b20b543
Author: carlvinhust2012
AuthorDate: Wed Dec 21 14:25:52 2022 +0800
[feature](struct-type) add children column when create table (#15089)
---
.../main/java/org/apache/doris/catalog/Column.java | 63 +-
.../java/org/apache/doris/catalog/StructType.java | 5 ++
.../main/java/org/apache/doris/catalog/Type.java | 4 +-
3 files changed, 47 insertions(+), 25 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
index 532a49b0eb..4b00ca9458 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
@@ -57,6 +57,7 @@ public class Column implements Writable {
public static final String DELETE_SIGN = "__DORIS_DELETE_SIGN__";
public static final String SEQUENCE_COL = "__DORIS_SEQUENCE_COL__";
private static final String COLUMN_ARRAY_CHILDREN = "item";
+private static final String COLUMN_STRUCT_CHILDREN = "field";
public static final int COLUMN_UNIQUE_ID_INIT_VALUE = -1;
@SerializedName(value = "name")
@@ -185,6 +186,13 @@ public class Column implements Writable {
Column c = new Column(COLUMN_ARRAY_CHILDREN, ((ArrayType)
type).getItemType());
c.setIsAllowNull(((ArrayType) type).getContainsNull());
column.addChildrenColumn(c);
+} else if (type.isStructType()) {
+ArrayList fileds = ((StructType) type).getFields();
+for (StructField field : fileds) {
+Column c = new Column(COLUMN_STRUCT_CHILDREN, field.getType());
+// c.setIsAllowNull(field.getContainsNull());
+column.addChildrenColumn(c);
+}
}
}
@@ -395,34 +403,41 @@ public class Column implements Writable {
return tColumn;
}
+private void setChildrenTColumn(Column children, TColumn tColumn) {
+TColumn childrenTColumn = new TColumn();
+childrenTColumn.setColumnName(children.name);
+
+TColumnType childrenTColumnType = new TColumnType();
+childrenTColumnType.setType(children.getDataType().toThrift());
+childrenTColumnType.setLen(children.getStrLen());
+childrenTColumnType.setPrecision(children.getPrecision());
+childrenTColumnType.setScale(children.getScale());
+childrenTColumnType.setIndexLen(children.getOlapColumnIndexSize());
+
+childrenTColumn.setColumnType(childrenTColumnType);
+childrenTColumn.setIsAllowNull(children.isAllowNull());
+// TODO: If we don't set the aggregate type for children, the type
will be
+// considered as TAggregationType::SUM after deserializing in BE.
+// For now, we make children inherit the aggregate type from their
parent.
+if (tColumn.getAggregationType() != null) {
+childrenTColumn.setAggregationType(tColumn.getAggregationType());
+}
+
+tColumn.children_column.add(childrenTColumn);
+toChildrenThrift(children, childrenTColumn);
+}
+
private void toChildrenThrift(Column column, TColumn tColumn) {
if (column.type.isArrayType()) {
Column children = column.getChildren().get(0);
-
-TColumn childrenTColumn = new TColumn();
-childrenTColumn.setColumnName(children.name);
-
-TColumnType childrenTColumnType = new TColumnType();
-childrenTColumnType.setType(children.getDataType().toThrift());
-childrenTColumnType.setType(children.getDataType().toThrift());
-childrenTColumnType.setLen(children.getStrLen());
-childrenTColumnType.setPrecision(children.getPrecision());
-childrenTColumnType.setScale(children.getScale());
-
-childrenTColumnType.setIndexLen(children.getOlapColumnIndexSize());
-childrenTColumn.setColumnType(childrenTColumnType);
-childrenTColumn.setIsAllowNull(children.isAllowNull());
-// TODO: If we don't set the aggregate type for children, the type
will be
-// considered as TAggregationType::SUM after deserializing in BE.
-// For now, we make children inherit the aggregate type from
their parent.
-if (tColumn.getAggregationType() != null) {
-
childrenTColumn.setAggregationType(tColumn.getAggregationType());
-}
-
tColumn.setChildrenColumn(new ArrayList<
[doris] branch master updated: [fix](InBitmap) Check whether the in bitmap contains correlated subqueries (#15184)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e65b577f90 [fix](InBitmap) Check whether the in bitmap contains
correlated subqueries (#15184)
e65b577f90 is described below
commit e65b577f905eb0f6d3f789d9f216aed0439f73b5
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Wed Dec 21 16:52:27 2022 +0800
[fix](InBitmap) Check whether the in bitmap contains correlated subqueries
(#15184)
---
.../src/main/java/org/apache/doris/analysis/StmtRewriter.java | 4
regression-test/suites/query_p0/join/test_bitmap_filter.groovy | 10 +++---
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
index 17f29404b8..f837bf6af5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
@@ -1189,6 +1189,10 @@ public class StmtRewriter {
Expr subquerySubstitute = slotRef;
if (exprWithSubquery instanceof InPredicate) {
if (slotRef.getType().isBitmapType()) {
+if (isCorrelated) {
+throw new AnalysisException(
+"In bitmap does not support correlated subquery: "
+ exprWithSubquery.toSql());
+}
Expr pred = new
BitmapFilterPredicate(exprWithSubquery.getChild(0), slotRef,
((InPredicate) exprWithSubquery).isNotIn());
pred.analyze(analyzer);
diff --git a/regression-test/suites/query_p0/join/test_bitmap_filter.groovy
b/regression-test/suites/query_p0/join/test_bitmap_filter.groovy
index d1102ae193..eaa55bc0eb 100644
--- a/regression-test/suites/query_p0/join/test_bitmap_filter.groovy
+++ b/regression-test/suites/query_p0/join/test_bitmap_filter.groovy
@@ -16,12 +16,11 @@
// under the License.
suite("test_bitmap_filter", "query_p0") {
-def tbl1 = "bigtable"
+def tbl1 = "test_query_db.bigtable"
def tbl2 = "bitmap_table"
-def tbl3 = "baseall"
+def tbl3 = "test_query_db.baseall"
sql "set runtime_filter_type = 16"
-sql "use test_query_db"
sql "DROP TABLE IF EXISTS ${tbl2}"
sql """
CREATE TABLE ${tbl2} (
@@ -50,4 +49,9 @@ suite("test_bitmap_filter", "query_p0") {
qt_sql5 "select k1, k2 from ${tbl1} where k1 in (select k2 from ${tbl2})
and k2 not in (select k3 from ${tbl2}) order by k1;"
qt_sql6 "select k2, count(k2) from ${tbl1} where k1 in (select k2 from
${tbl2}) group by k2 order by k2;"
+
+test {
+sql "select k1, k2 from ${tbl1} b1 where k1 in (select k2 from ${tbl2}
b2 where b1.k2 = b2.k1) order by k1;"
+exception "In bitmap does not support correlated subquery"
+}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](inbitmap) forbid having clause to include in bitmap. (#15494)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a4af1fbf90 [fix](inbitmap) forbid having clause to include in bitmap.
(#15494)
a4af1fbf90 is described below
commit a4af1fbf90e07ee79afacfb3b17182abc2d950c6
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Wed Jan 4 14:33:18 2023 +0800
[fix](inbitmap) forbid having clause to include in bitmap. (#15494)
---
.../src/main/java/org/apache/doris/analysis/SelectStmt.java | 11 +++
.../suites/query_p0/join/test_bitmap_filter.groovy| 5 +
2 files changed, 16 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index 1af417841a..0950b3d839 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -957,6 +957,13 @@ public class SelectStmt extends QueryStmt {
}
}
+private boolean isContainInBitmap(Expr expr) {
+List inPredicates = Lists.newArrayList();
+expr.collect(InPredicate.class, inPredicates);
+return inPredicates.stream().anyMatch(e -> e.getChild(1) instanceof
Subquery
+&& ((Subquery)
e.getChild(1)).getStatement().getResultExprs().get(0).getType().isBitmapType());
+}
+
/**
* Analyze aggregation-relevant components of the select block (Group By
clause,
* select list, Order By clause),
@@ -1017,6 +1024,10 @@ public class SelectStmt extends QueryStmt {
"HAVING clause must not contain analytic expressions: "
+ analyticExpr.toSql());
}
+if (isContainInBitmap(havingClauseAfterAnaylzed)) {
+throw new AnalysisException(
+"HAVING clause dose not support in bitmap syntax: " +
havingClauseAfterAnaylzed.toSql());
+}
}
if (groupByClause == null && !selectList.isDistinct()
diff --git a/regression-test/suites/query_p0/join/test_bitmap_filter.groovy
b/regression-test/suites/query_p0/join/test_bitmap_filter.groovy
index 788f26ec3a..7e70d5c0f1 100644
--- a/regression-test/suites/query_p0/join/test_bitmap_filter.groovy
+++ b/regression-test/suites/query_p0/join/test_bitmap_filter.groovy
@@ -69,4 +69,9 @@ suite("test_bitmap_filter", "query_p0") {
sql "select k1, k2 from ${tbl1} b1 where k1 in (select k2 from ${tbl2}
b2 where b1.k2 = b2.k1) order by k1;"
exception "In bitmap does not support correlated subquery"
}
+
+test {
+sql "select k1, count(*) from ${tbl1} b1 group by k1 having k1 in
(select k2 from ${tbl2} b2) order by k1;"
+exception "HAVING clause dose not support in bitmap"
+}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (a4af1fbf90 -> c42c61dcad)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a4af1fbf90 [fix](inbitmap) forbid having clause to include in bitmap. (#15494) add c42c61dcad [fix](bitmapfilter) fix bitmap filter not pushing down (#15532) No new revisions were added by this update. Summary of changes: be/src/exprs/runtime_filter.cpp| 7 +-- be/src/vec/exec/scan/vscan_node.cpp| 2 +- .../src/main/java/org/apache/doris/planner/RuntimeFilter.java | 4 3 files changed, 6 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (93f5e440eb -> 59f34be41f)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 93f5e440eb [Bug](execute) fix get next non stop for eos on streaming preagg (#15611) add 59f34be41f [fix](having-clause) having clause do not works correct with same alias name (#15143) No new revisions were added by this update. Summary of changes: .../apache/doris/analysis/ExprSubstitutionMap.java | 10 ++ .../java/org/apache/doris/analysis/SelectStmt.java | 38 -- .../correctness_p0/test_group_having_alias.out | 10 ++ .../correctness_p0/test_group_having_alias.groovy | 25 +- 4 files changed, 72 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (3ea6478ba8 -> 91fc9fae8e)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 3ea6478ba8 [feature](multi-catalog) parquet reader support nested
array column (#16961)
add 91fc9fae8e [Bug](complex-type) Fix is null predicate in delete stmt
for array/struct/map type (#17018)
No new revisions were added by this update.
Summary of changes:
be/src/olap/rowset/segment_v2/column_writer.cpp| 5
be/src/olap/schema.cpp | 30 +-
be/src/olap/schema.h | 3 ++-
be/src/vec/columns/column_map.cpp | 12 +
be/src/vec/columns/column_map.h| 5 ++--
be/src/vec/columns/column_struct.cpp | 10
be/src/vec/columns/column_struct.h | 5 ++--
be/src/vec/data_types/data_type_map.cpp| 8 --
be/src/vec/data_types/data_type_map.h | 7 +
.../test_map_column_delete.out}| 4 +--
.../test_struct_column_delete.out} | 4 +--
...delete.groovy => test_map_column_delete.groovy} | 11
...ete.groovy => test_struct_column_delete.groovy} | 11
13 files changed, 74 insertions(+), 41 deletions(-)
copy regression-test/data/{correctness/test_crossjoin_inlineview_slot.out =>
delete_p0/test_map_column_delete.out} (80%)
copy regression-test/data/{correctness/test_forbidden_infer_filter_rule.out =>
delete_p0/test_struct_column_delete.out} (90%)
copy regression-test/suites/delete_p0/{test_array_column_delete.groovy =>
test_map_column_delete.groovy} (64%)
copy regression-test/suites/delete_p0/{test_array_column_delete.groovy =>
test_struct_column_delete.groovy} (64%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (91fc9fae8e -> 52a731a2df)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 91fc9fae8e [Bug](complex-type) Fix is null predicate in delete stmt for array/struct/map type (#17018) add 52a731a2df fix compile error while use gcc12 (#17016) No new revisions were added by this update. Summary of changes: be/src/vec/core/field.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (8043418db4 -> d183199319)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8043418db4 [optimization](array-type) update the exception message when create table with array column (#13731) add d183199319 [Bug](array-type) Fix array product calculate decimal type return wrong result (#13794) No new revisions were added by this update. Summary of changes: .../aggregate_function_product.h | 27 ++ .../test_array_functions_by_literal.out| 18 +++ .../test_array_functions_by_literal.groovy | 6 + 3 files changed, 51 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [chore](build) add build param to version string (#12591)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c6c84a2784 [chore](build) add build param to version string (#12591)
c6c84a2784 is described below
commit c6c84a278459b05ddeabb714d647b12416f1faa1
Author: Zhengguo Yang
AuthorDate: Thu Sep 15 17:09:22 2022 +0800
[chore](build) add build param to version string (#12591)
---
be/CMakeLists.txt | 2 +-
be/src/util/debug_util.cpp | 13 +
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 9c610e1f2c..2fc57ecf3c 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -410,7 +410,7 @@ check_function_exists(sched_getcpu HAVE_SCHED_GETCPU)
# -pthread: enable multithreaded malloc
# -DBOOST_DATE_TIME_POSIX_TIME_STD_CONFIG: enable nanosecond precision for
boost
# -fno-omit-frame-pointers: Keep frame pointer for functions in register
-set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -Wall -Wno-sign-compare -pthread
-Werror")
+set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -frecord-gcc-switches -Wall
-Wno-sign-compare -pthread -Werror")
set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -fstrict-aliasing
-fno-omit-frame-pointer")
set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -std=gnu++17 -D__STDC_FORMAT_MACROS")
set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS}
-DBOOST_DATE_TIME_POSIX_TIME_STD_CONFIG")
diff --git a/be/src/util/debug_util.cpp b/be/src/util/debug_util.cpp
index 854d40f0ba..a917d6f6ab 100644
--- a/be/src/util/debug_util.cpp
+++ b/be/src/util/debug_util.cpp
@@ -59,6 +59,19 @@ std::string get_build_version(bool compact) {
<< " RELEASE"
#else
<< " DEBUG"
+#if defined(ADDRESS_SANITIZER)
+ << " with ASAN"
+#elif defined(LEAK_SANITIZER)
+ << " with LSAN"
+#elif defined(THREAD_SANITIZER)
+ << " with TSAN"
+#elif defined(UNDEFINED_BEHAVIOR_SANITIZER)
+ << " with UBSAN"
+#elif defined(MEMORY_SANITIZER)
+ << " with MSAN"
+#elif defined(BLACKLIST_SANITIZER)
+ << " with BLSAN"
+#endif
#endif
<< " (build " << DORIS_BUILD_HASH << ")";
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](plan) wrong result for random distributed agg table with all keys not null (#36270)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8f5688f8a5e [fix](plan) wrong result for random distributed agg table
with all keys not null (#36270)
8f5688f8a5e is described below
commit 8f5688f8a5e743eb06d224e49be93bb8d9c69633
Author: camby
AuthorDate: Mon Jun 17 09:25:22 2024 +0800
[fix](plan) wrong result for random distributed agg table with all keys not
null (#36270)
---
.../java/org/apache/doris/catalog/OlapTable.java | 4 +++
.../plans/logical/LogicalCatalogRelation.java | 23 +
.../aggregate/select_random_distributed_tbl.out| 12 +
.../aggregate/select_random_distributed_tbl.groovy | 29 +-
4 files changed, 57 insertions(+), 11 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 75a2fcd96c8..406c6d61793 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -936,6 +936,10 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf {
return distributionColumnNames;
}
+public boolean isRandomDistribution() {
+return defaultDistributionInfo instanceof RandomDistributionInfo;
+}
+
public void renamePartition(String partitionName, String newPartitionName)
{
if (partitionInfo.getType() == PartitionType.UNPARTITIONED) {
// bug fix
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java
index 3711ce31f98..49d0ed29688 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java
@@ -128,18 +128,21 @@ public abstract class LogicalCatalogRelation extends
LogicalRelation implements
@Override
public void computeUnique(DataTrait.Builder builder) {
Set outputSet = Utils.fastToImmutableSet(getOutputSet());
-if (table instanceof OlapTable && ((OlapTable)
table).getKeysType().isAggregationFamily()) {
-ImmutableSet.Builder uniqSlots =
ImmutableSet.builderWithExpectedSize(outputSet.size());
-for (Slot slot : outputSet) {
-if (!(slot instanceof SlotReference)) {
-continue;
-}
-SlotReference slotRef = (SlotReference) slot;
-if (slotRef.getColumn().isPresent() &&
slotRef.getColumn().get().isKey()) {
-uniqSlots.add(slot);
+if (table instanceof OlapTable) {
+OlapTable olapTable = (OlapTable) table;
+if (olapTable.getKeysType().isAggregationFamily() &&
!olapTable.isRandomDistribution()) {
+ImmutableSet.Builder uniqSlots =
ImmutableSet.builderWithExpectedSize(outputSet.size());
+for (Slot slot : outputSet) {
+if (!(slot instanceof SlotReference)) {
+continue;
+}
+SlotReference slotRef = (SlotReference) slot;
+if (slotRef.getColumn().isPresent() &&
slotRef.getColumn().get().isKey()) {
+uniqSlots.add(slot);
+}
}
+builder.addUniqueSlot(uniqSlots.build());
}
-builder.addUniqueSlot(uniqSlots.build());
}
for (PrimaryKeyConstraint c : table.getPrimaryKeyConstraints()) {
diff --git
a/regression-test/data/query_p0/aggregate/select_random_distributed_tbl.out
b/regression-test/data/query_p0/aggregate/select_random_distributed_tbl.out
index 1afb2a06762..c03e72c8f9e 100644
--- a/regression-test/data/query_p0/aggregate/select_random_distributed_tbl.out
+++ b/regression-test/data/query_p0/aggregate/select_random_distributed_tbl.out
@@ -215,3 +215,15 @@
-- !sql_16 --
2
+-- !sql_17 --
+1
+
+-- !sql_18 --
+1
+
+-- !sql_19 --
+1
+
+-- !sql_20 --
+1
+
diff --git
a/regression-test/suites/query_p0/aggregate/select_random_distributed_tbl.groovy
b/regression-test/suites/query_p0/aggregate/select_random_distributed_tbl.groovy
index ff0df74589a..439559aa5e0 100644
---
a/regression-test/suites/query_p0/aggregate/select_random_distributed_tbl.groovy
+++
b/regression-test/suites/query_p0/aggregate/select_random_distributed_tbl.groovy
@@ -131,4 +131,31 @@ suite("select_random_distributed_tbl") {
}
sql "drop table ${tableName};"
-}
\ No newline at end of file
+
+// test all keys are N
(doris) branch branch-2.1 updated: [fix](plan) fix wrong result for random distributed agg table with all keys not null (#36271)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new e2350403a60 [fix](plan) fix wrong result for random distributed agg
table with all keys not null (#36271)
e2350403a60 is described below
commit e2350403a60beff25db059a73bd857b757d985a5
Author: camby
AuthorDate: Tue Jun 18 11:25:31 2024 +0800
[fix](plan) fix wrong result for random distributed agg table with all keys
not null (#36271)
---
.../java/org/apache/doris/catalog/OlapTable.java | 4 +++
.../plans/logical/LogicalCatalogRelation.java | 23 +
.../aggregate/select_random_distributed_tbl.out| 12 +
.../aggregate/select_random_distributed_tbl.groovy | 29 +-
4 files changed, 57 insertions(+), 11 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index d901d2340fb..fa6425c72af 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -868,6 +868,10 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf {
return distributionColumnNames;
}
+public boolean isRandomDistribution() {
+return defaultDistributionInfo instanceof RandomDistributionInfo;
+}
+
public void renamePartition(String partitionName, String newPartitionName)
{
if (partitionInfo.getType() == PartitionType.UNPARTITIONED) {
// bug fix
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java
index 277695be079..4d8fba447b3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java
@@ -137,18 +137,21 @@ public abstract class LogicalCatalogRelation extends
LogicalRelation implements
@Override
public void computeUnique(FunctionalDependencies.Builder fdBuilder) {
Set outputSet = Utils.fastToImmutableSet(getOutputSet());
-if (table instanceof OlapTable && ((OlapTable)
table).getKeysType().isAggregationFamily()) {
-ImmutableSet.Builder uniqSlots =
ImmutableSet.builderWithExpectedSize(outputSet.size());
-for (Slot slot : outputSet) {
-if (!(slot instanceof SlotReference)) {
-continue;
-}
-SlotReference slotRef = (SlotReference) slot;
-if (slotRef.getColumn().isPresent() &&
slotRef.getColumn().get().isKey()) {
-uniqSlots.add(slot);
+if (table instanceof OlapTable) {
+OlapTable olapTable = (OlapTable) table;
+if (olapTable.getKeysType().isAggregationFamily() &&
!olapTable.isRandomDistribution()) {
+ImmutableSet.Builder uniqSlots =
ImmutableSet.builderWithExpectedSize(outputSet.size());
+for (Slot slot : outputSet) {
+if (!(slot instanceof SlotReference)) {
+continue;
+}
+SlotReference slotRef = (SlotReference) slot;
+if (slotRef.getColumn().isPresent() &&
slotRef.getColumn().get().isKey()) {
+uniqSlots.add(slot);
+}
}
+fdBuilder.addUniqueSlot(uniqSlots.build());
}
-fdBuilder.addUniqueSlot(uniqSlots.build());
}
for (PrimaryKeyConstraint c : table.getPrimaryKeyConstraints()) {
diff --git
a/regression-test/data/query_p0/aggregate/select_random_distributed_tbl.out
b/regression-test/data/query_p0/aggregate/select_random_distributed_tbl.out
index 1afb2a06762..c03e72c8f9e 100644
--- a/regression-test/data/query_p0/aggregate/select_random_distributed_tbl.out
+++ b/regression-test/data/query_p0/aggregate/select_random_distributed_tbl.out
@@ -215,3 +215,15 @@
-- !sql_16 --
2
+-- !sql_17 --
+1
+
+-- !sql_18 --
+1
+
+-- !sql_19 --
+1
+
+-- !sql_20 --
+1
+
diff --git
a/regression-test/suites/query_p0/aggregate/select_random_distributed_tbl.groovy
b/regression-test/suites/query_p0/aggregate/select_random_distributed_tbl.groovy
index ff0df74589a..439559aa5e0 100644
---
a/regression-test/suites/query_p0/aggregate/select_random_distributed_tbl.groovy
+++
b/regression-test/suites/query_p0/aggregate/select_random_distributed_tbl.groovy
@@ -131,4 +131,31 @@ suite("select_random_distributed_tbl") {
}
sql "drop table ${tableName};"
-}
\ No newline at end
(doris) branch branch-1.2-lts updated: [branch-1.2](complex type) forbid MAP type and STRUCT type in v1.2 (#36494)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new af3bf67c5b4 [branch-1.2](complex type) forbid MAP type and STRUCT type
in v1.2 (#36494)
af3bf67c5b4 is described below
commit af3bf67c5b48ba83ad4c4211b6df1a394e0175f1
Author: camby
AuthorDate: Wed Jun 19 14:47:59 2024 +0800
[branch-1.2](complex type) forbid MAP type and STRUCT type in v1.2 (#36494)
---
fe/fe-common/src/main/java/org/apache/doris/catalog/MapType.java| 5 +
fe/fe-common/src/main/java/org/apache/doris/catalog/StructType.java | 5 +
2 files changed, 10 insertions(+)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/catalog/MapType.java
b/fe/fe-common/src/main/java/org/apache/doris/catalog/MapType.java
index 1a746a23749..0534790d29f 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/catalog/MapType.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/catalog/MapType.java
@@ -56,6 +56,11 @@ public class MapType extends Type {
return valueType;
}
+@Override
+public boolean isSupported() {
+return false;
+}
+
@Override
public boolean equals(Object other) {
if (!(other instanceof MapType)) {
diff --git
a/fe/fe-common/src/main/java/org/apache/doris/catalog/StructType.java
b/fe/fe-common/src/main/java/org/apache/doris/catalog/StructType.java
index e3ffa3a5218..2e685416b58 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/catalog/StructType.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/catalog/StructType.java
@@ -51,6 +51,11 @@ public class StructType extends Type {
fields = Lists.newArrayList();
}
+@Override
+public boolean isSupported() {
+return false;
+}
+
@Override
public String toSql(int depth) {
if (depth >= MAX_NESTING_DEPTH) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (e6baf22bee5 -> 1b1029ff72e)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from e6baf22bee5 [fix](cloud) Fix loss some tag info in cloud (#36866) add 1b1029ff72e [opt](mow) reduce memory usage for mow table compaction (#36865) No new revisions were added by this update. Summary of changes: be/src/common/config.cpp | 2 + be/src/common/config.h | 2 + be/src/olap/base_tablet.cpp| 8 +++- be/src/olap/compaction.cpp | 53 +++--- be/src/olap/utils.h| 2 + regression-test/pipeline/external/conf/be.conf | 1 + regression-test/pipeline/p0/conf/be.conf | 1 + regression-test/pipeline/p1/conf/be.conf | 1 + 8 files changed, 46 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](round) fix round decimal128 overflow (#37733) (#37965)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new ce71a131c0b [fix](round) fix round decimal128 overflow (#37733)
(#37965)
ce71a131c0b is described below
commit ce71a131c0bd972bf0c80680fe8596453c150598
Author: camby
AuthorDate: Fri Jul 19 22:01:12 2024 +0800
[fix](round) fix round decimal128 overflow (#37733) (#37965)
---
be/src/vec/functions/round.h | 21 ++
.../math_functions/test_round_overflow.out | 10 +
.../math_functions/test_round_overflow.groovy | 25 ++
3 files changed, 47 insertions(+), 9 deletions(-)
diff --git a/be/src/vec/functions/round.h b/be/src/vec/functions/round.h
index 30c2b71e841..824fad08147 100644
--- a/be/src/vec/functions/round.h
+++ b/be/src/vec/functions/round.h
@@ -22,6 +22,7 @@
#include "vec/columns/column_const.h"
#include "vec/columns/columns_number.h"
+#include "vec/exec/format/format_common.h"
#include "vec/functions/function.h"
#if defined(__SSE4_1__) || defined(__aarch64__)
#include "util/sse_util.hpp"
@@ -29,6 +30,7 @@
#include
#endif
#include
+#include
#include "vec/columns/column.h"
#include "vec/columns/column_decimal.h"
@@ -64,7 +66,7 @@ enum class TieBreakingMode {
};
template
+ TieBreakingMode tie_breaking_mode, typename U>
struct IntegerRoundingComputation {
static const size_t data_count = 1;
@@ -116,7 +118,7 @@ struct IntegerRoundingComputation {
__builtin_unreachable();
}
-static ALWAYS_INLINE T compute(T x, T scale, size_t target_scale) {
+static ALWAYS_INLINE T compute(T x, T scale, T target_scale) {
switch (scale_mode) {
case ScaleMode::Zero:
case ScaleMode::Positive:
@@ -128,10 +130,10 @@ struct IntegerRoundingComputation {
__builtin_unreachable();
}
-static ALWAYS_INLINE void compute(const T* __restrict in, size_t scale, T*
__restrict out,
- size_t target_scale) {
+static ALWAYS_INLINE void compute(const T* __restrict in, U scale, T*
__restrict out,
+ U target_scale) {
if constexpr (sizeof(T) <= sizeof(scale) && scale_mode ==
ScaleMode::Negative) {
-if (scale > size_t(std::numeric_limits::max())) {
+if (scale >= std::numeric_limits::max()) {
*out = 0;
return;
}
@@ -145,7 +147,7 @@ class DecimalRoundingImpl {
private:
using NativeType = typename T::NativeType;
using Op = IntegerRoundingComputation;
+ tie_breaking_mode, NativeType>;
using Container = typename ColumnDecimal::Container;
public:
@@ -153,15 +155,16 @@ public:
Int16 out_scale) {
Int16 scale_arg = in_scale - out_scale;
if (scale_arg > 0) {
-size_t scale = int_exp10(scale_arg);
+auto scale =
DecimalScaleParams::get_scale_factor(scale_arg);
const NativeType* __restrict p_in = reinterpret_cast(in.data());
const NativeType* end_in = reinterpret_cast(in.data()) + in.size();
NativeType* __restrict p_out =
reinterpret_cast(out.data());
if (out_scale < 0) {
+auto negative_scale =
DecimalScaleParams::get_scale_factor(-out_scale);
while (p_in < end_in) {
-Op::compute(p_in, scale, p_out, int_exp10(-out_scale));
+Op::compute(p_in, scale, p_out, negative_scale);
++p_in;
++p_out;
}
@@ -320,7 +323,7 @@ template
struct IntegerRoundingImpl {
private:
-using Op = IntegerRoundingComputation;
+using Op = IntegerRoundingComputation;
using Container = typename ColumnVector::Container;
public:
diff --git
a/regression-test/data/nereids_p0/sql_functions/math_functions/test_round_overflow.out
b/regression-test/data/nereids_p0/sql_functions/math_functions/test_round_overflow.out
new file mode 100644
index 000..a18cc094872
--- /dev/null
+++
b/regression-test/data/nereids_p0/sql_functions/math_functions/test_round_overflow.out
@@ -0,0 +1,10 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select1 --
+186
+
+-- !select2 --
+0
+
+-- !select3 --
+200
+
diff --git
a/regression-test/suites/nereids_p0/sql_functions/math_functions/test_round_overflow.groovy
b/regression-test/suites/nereids_p0/sql_functions/math_functions/test_round_overflow.groovy
new file mode 100644
index 000..12311537c27
--- /dev/null
+++
b/regression-test/suites/n
(doris) branch master updated (10aa083397c -> 1d25dff690e)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 10aa083397c [chore](backup) Remove delete_if_exists properties for creating repository (#38190) add 1d25dff690e [fix](round) fix round decimal128 overflow (#38106) No new revisions were added by this update. Summary of changes: be/src/vec/functions/round.h | 21 +++-- 1 file changed, 11 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [Improvement](cooldown) add statment to show objects which use storage policy (#701)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new ccac349093 [Improvement](cooldown) add statment to show objects which
use storage policy (#701)
ccac349093 is described below
commit ccac349093815da0bdc1cb12ed1bb79ef5c223b5
Author: Yulei-Yang
AuthorDate: Tue Jun 4 10:55:36 2024 +0800
[Improvement](cooldown) add statment to show objects which use storage
policy (#701)
---
.../Show-Statements/SHOW-STORAGE-POLICY-USING.md | 74 +
docs/table-design/cold-hot-separation.md | 2 -
.../Show-Statements/SHOW-STORAGE-POLICY-USING.md | 74 +
.../current/table-design/cold-hot-separation.md| 2 -
.../Show-Statements/SHOW-STORAGE-POLICY-USING.md | 74 +
.../table-design/cold-hot-separation.md| 2 -
.../Show-Statements/SHOW-STORAGE-POLICY-USING.md | 75 ++
.../table-design/cold-hot-separation.md| 2 -
sidebars.json | 1 +
.../Show-Statements/SHOW-STORAGE-POLICY-USING.md | 74 +
.../table-design/cold-hot-separation.md| 2 -
.../Show-Statements/SHOW-STORAGE-POLICY-USING.md | 74 +
.../table-design/cold-hot-separation.md| 2 -
versioned_sidebars/version-2.0-sidebars.json | 1 +
versioned_sidebars/version-2.1-sidebars.json | 1 +
15 files changed, 448 insertions(+), 12 deletions(-)
diff --git
a/docs/sql-manual/sql-statements/Show-Statements/SHOW-STORAGE-POLICY-USING.md
b/docs/sql-manual/sql-statements/Show-Statements/SHOW-STORAGE-POLICY-USING.md
new file mode 100644
index 00..803c94cbc8
--- /dev/null
+++
b/docs/sql-manual/sql-statements/Show-Statements/SHOW-STORAGE-POLICY-USING.md
@@ -0,0 +1,74 @@
+---
+{
+"title": "SHOW-STORAGE-POLICY-USING",
+"language": "en"
+}
+---
+
+
+
+## SHOW-STORAGE-POLICY-USING
+
+### Name
+
+SHOW STORAGE POLICY USING
+
+### Description
+
+This command is used to show tables and partitions which is using storage
policy
+
+```sql
+SHOW STORAGE POLICY USING [FOR some_policy]
+```
+
+### Example
+
+1. get all objects which are using storage policy
+
+ ```sql
+ mysql> show storage policy using;
+
+---+-+++
+ | PolicyName| Database| Table
| Partitions |
+
+---+-+++
+ | test_storage_policy | regression_test_cold_heat_separation_p2 |
table_with_storage_policy_1| ALL|
+ | test_storage_policy | regression_test_cold_heat_separation_p2 |
partition_with_multiple_storage_policy | p201701|
+ | test_storage_policy_2 | regression_test_cold_heat_separation_p2 |
partition_with_multiple_storage_policy | p201702|
+ | test_storage_policy_2 | regression_test_cold_heat_separation_p2 |
table_with_storage_policy_2| ALL|
+ | test_policy | db2 |
db2_test_1 | ALL|
+
+---+-+++
+ ```
+
+2. get objects which are using the storage policy named test_storage_policy
+
+```sql
+mysql> show storage policy using for test_storage_policy;
+
+-+---+-++
+| PolicyName | Database | Table |
Partitions |
+
+-+---+-++
+| test_storage_policy | db_1 | partition_with_storage_policy_1 |
p201701|
+| test_storage_policy | db_1 | table_with_storage_policy_1 | ALL
|
+
+-+---+-++
+ ```
+
+### Keywords
+
+SHOW, STORAGE, POLICY, USING
+
+### Best Practice
diff --git a/docs/table-design/cold-hot-separation.md
b/docs/table-design/cold-hot-separation.md
index 29471dca15..736d2a7061 100644
--- a/docs/table-design/cold-hot-separation.md
+++ b/docs/table-design/cold-hot-separation.md
@@ -211,8 +211,6 @@ Furthermore, the garbage data on objects is not immediately
cleaned up. The BE p
## TODOs
-- Currently, there is no way to query tables associated with a specific
storage policy.
-
- Some remote occupancy metrics may not have comprehensive update retrieval.
## FAQs
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manu
(doris) branch branch-2.0 updated: [fix](if) handle result status of if function (#38426)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new b92aae73ed2 [fix](if) handle result status of if function (#38426)
b92aae73ed2 is described below
commit b92aae73ed249e077dcbdc9ce054885f3d217a7e
Author: camby
AuthorDate: Mon Jul 29 11:16:19 2024 +0800
[fix](if) handle result status of if function (#38426)
---
be/src/vec/exec/join/vjoin_node_base.cpp | 7 +++
be/src/vec/functions/if.cpp | 105 +--
2 files changed, 64 insertions(+), 48 deletions(-)
diff --git a/be/src/vec/exec/join/vjoin_node_base.cpp
b/be/src/vec/exec/join/vjoin_node_base.cpp
index 912f7826fe2..bd4143e3a73 100644
--- a/be/src/vec/exec/join/vjoin_node_base.cpp
+++ b/be/src/vec/exec/join/vjoin_node_base.cpp
@@ -193,6 +193,13 @@ Status VJoinNodeBase::_build_output_block(Block*
origin_block, Block* output_blo
auto result_column_id = -1;
RETURN_IF_ERROR(_output_expr_ctxs[i]->execute(origin_block,
&result_column_id));
auto& origin_column =
origin_block->get_by_position(result_column_id).column;
+if (!origin_column) {
+LOG(WARNING)
+<< "BUG!!! VExprContext::execute successfully, but
return null column. "
+<<
origin_block->get_by_position(result_column_id).dump_structure();
+return Status::InternalError(
+"VExprContext::execute inside VJoinNodeBase return
null column");
+}
/// `convert_to_full_column_if_const` will create a pointer to
the origin column if
/// the origin column is not ColumnConst/ColumnArray, this
make the column be not
diff --git a/be/src/vec/functions/if.cpp b/be/src/vec/functions/if.cpp
index 77f1c0ae53b..5c360d670bf 100644
--- a/be/src/vec/functions/if.cpp
+++ b/be/src/vec/functions/if.cpp
@@ -287,23 +287,24 @@ public:
else_col.type->get_type_id(), call);
}
-bool execute_for_null_then_else(FunctionContext* context, Block& block,
-const ColumnWithTypeAndName& arg_cond,
-const ColumnWithTypeAndName& arg_then,
-const ColumnWithTypeAndName& arg_else,
size_t result,
-size_t input_rows_count, Status& status) {
+Status execute_for_null_then_else(FunctionContext* context, Block& block,
+ const ColumnWithTypeAndName& arg_cond,
+ const ColumnWithTypeAndName& arg_then,
+ const ColumnWithTypeAndName& arg_else,
size_t result,
+ size_t input_rows_count, bool& handled) {
bool then_is_null = arg_then.column->only_null();
bool else_is_null = arg_else.column->only_null();
if (!then_is_null && !else_is_null) {
-return false;
+return Status::OK();
}
if (then_is_null && else_is_null) {
block.get_by_position(result).column =
block.get_by_position(result).type->create_column_const_with_default_value(
input_rows_count);
-return true;
+handled = true;
+return Status::OK();
}
const ColumnUInt8* cond_col = typeid_cast(arg_cond.column.get());
@@ -335,16 +336,12 @@ public:
make_nullable_column_if_not(arg_else.column);
}
} else {
-status = Status::InternalError(
+return Status::InternalError(
"Illegal column {} of first argument of function {}.
Must be ColumnUInt8 "
"or ColumnConstUInt8.",
arg_cond.column->get_name(), get_name());
}
-return true;
-}
-
-/// If else is NULL, we create Nullable column with null mask OR-ed
with negated condition.
-if (else_is_null) {
+} else { /// If else is NULL, we create Nullable column with null mask
OR-ed with negated condition.
if (cond_col) {
size_t size = input_rows_count;
auto& null_map_data = cond_col->get_data();
@@ -380,33 +377,32 @@ public:
input_rows_count);
}
} else {
-status = Status::InternalError(
+return Status::InternalError(
(doris) branch branch-2.1 updated: [opt](mow) opt mow lookup with sequence column (#38287) (#38406)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 715bcd13f1f [opt](mow) opt mow lookup with sequence column (#38287)
(#38406)
715bcd13f1f is described below
commit 715bcd13f1fdb59646942a016238cec31b7b1472
Author: camby
AuthorDate: Tue Jul 30 09:46:09 2024 +0800
[opt](mow) opt mow lookup with sequence column (#38287) (#38406)
---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index 49616c63e5c..9040cbf3e27 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -1481,7 +1481,8 @@ Status
SegmentIterator::_lookup_ordinal_from_pk_index(const RowCursor& key, bool
// for mow with cluster key table, we should get key range from short key
index.
DCHECK(_segment->_tablet_schema->cluster_key_idxes().empty());
-if (has_seq_col) {
+// if full key is exact_match, the primary key without sequence column
should also the same
+if (has_seq_col && !exact_match) {
size_t seq_col_length =
_segment->_tablet_schema->column(_segment->_tablet_schema->sequence_col_idx())
.length() +
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [opt](mow) opt mow lookup with sequence column (#38287) (#38407)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 0ad9600a52f [opt](mow) opt mow lookup with sequence column (#38287)
(#38407)
0ad9600a52f is described below
commit 0ad9600a52fecc3786ccfb543541fc0a9f5659f0
Author: camby
AuthorDate: Tue Jul 30 09:46:31 2024 +0800
[opt](mow) opt mow lookup with sequence column (#38287) (#38407)
---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index 919c6678463..4f819318fd2 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -1388,7 +1388,7 @@ Status
SegmentIterator::_lookup_ordinal_from_pk_index(const RowCursor& key, bool
// The sequence column needs to be removed from primary key index when
comparing key
bool has_seq_col = _segment->_tablet_schema->has_sequence_col();
-if (has_seq_col) {
+if (has_seq_col && !exact_match) {
size_t seq_col_length =
_segment->_tablet_schema->column(_segment->_tablet_schema->sequence_col_idx())
.length() +
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Enhancement](bitmapfilter) Support bitmap filter to apply zone_map index to filter pages (#14635)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7873bc95a6 [Enhancement](bitmapfilter) Support bitmap filter to apply
zone_map index to filter pages (#14635)
7873bc95a6 is described below
commit 7873bc95a6aae30596d7697cd3fa199030ea74c4
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Thu Dec 1 10:41:09 2022 +0800
[Enhancement](bitmapfilter) Support bitmap filter to apply zone_map index
to filter pages (#14635)
---
be/src/exprs/bitmapfilter_predicate.h | 35 ++
be/src/exprs/runtime_filter.cpp| 21 --
be/src/exprs/runtime_filter.h | 3 +
be/src/olap/bitmap_filter_predicate.h | 45 +++--
be/src/olap/olap_common.h | 2 +
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 1 +
be/src/util/bitmap_value.h | 76 +-
be/src/vec/exec/scan/new_olap_scan_node.cpp| 1 +
be/src/vec/exec/scan/new_olap_scan_node.h | 1 +
be/src/vec/exec/scan/new_olap_scanner.cpp | 2 +
be/src/vec/runtime/shared_hash_table_controller.h | 2 +-
11 files changed, 151 insertions(+), 38 deletions(-)
diff --git a/be/src/exprs/bitmapfilter_predicate.h
b/be/src/exprs/bitmapfilter_predicate.h
index c9a2efc621..dbc9447034 100644
--- a/be/src/exprs/bitmapfilter_predicate.h
+++ b/be/src/exprs/bitmapfilter_predicate.h
@@ -17,7 +17,8 @@
#pragma once
-#include "common/object_pool.h"
+#include
+
#include "gutil/integral_types.h"
#include "runtime/define_primitive_type.h"
#include "runtime/primitive_type.h"
@@ -30,14 +31,15 @@ class BitmapFilterFuncBase {
public:
virtual void insert(const void* data) = 0;
virtual void insert_many(const std::vector bitmaps) =
0;
-virtual bool find(uint64 data) const = 0;
-virtual bool is_empty() = 0;
+virtual bool empty() = 0;
virtual Status assign(BitmapValue* bitmap_value) = 0;
virtual void light_copy(BitmapFilterFuncBase* other) { _not_in =
other->_not_in; };
virtual uint16_t find_fixed_len_olap_engine(const char* data, const uint8*
nullmap,
uint16_t* offsets, int number)
= 0;
virtual void find_batch(const char* data, const uint8* nullmap, int number,
uint8* results) const = 0;
+virtual size_t size() const = 0;
+bool is_not_in() const { return _not_in; }
void set_not_in(bool not_in) { _not_in = not_in; }
virtual ~BitmapFilterFuncBase() = default;
@@ -51,7 +53,7 @@ class BitmapFilterFunc : public BitmapFilterFuncBase {
public:
using CppType = typename PrimitiveTypeTraits::CppType;
-BitmapFilterFunc() : _bitmap_value(std::make_shared()),
_empty(true) {}
+BitmapFilterFunc() : _bitmap_value(std::make_shared()) {}
~BitmapFilterFunc() override = default;
@@ -59,15 +61,13 @@ public:
void insert_many(const std::vector bitmaps) override;
-bool find(uint64 data) const override { return _not_in ^
_bitmap_value->contains(data); }
-
uint16_t find_fixed_len_olap_engine(const char* data, const uint8*
nullmap, uint16_t* offsets,
int number) override;
void find_batch(const char* data, const uint8* nullmap, int number,
uint8* results) const override;
-bool is_empty() override { return _empty; }
+bool empty() override { return _bitmap_value->empty(); }
Status assign(BitmapValue* bitmap_value) override {
*_bitmap_value = *bitmap_value;
@@ -76,9 +76,25 @@ public:
void light_copy(BitmapFilterFuncBase* bloomfilter_func) override;
+size_t size() const override { return _bitmap_value->cardinality(); }
+
+uint64_t max() { return _bitmap_value->max(nullptr); }
+
+uint64_t min() { return _bitmap_value->min(nullptr); }
+
+bool contains_any(CppType left, CppType right) {
+if (right < 0) {
+return false;
+}
+return _bitmap_value->contains_any(std::max(left, (CppType)0), right);
+}
+
+std::shared_ptr get_inner_bitmap() { return _bitmap_value; }
+
private:
std::shared_ptr _bitmap_value;
-bool _empty;
+
+bool find(CppType data) const { return _not_in ^ (data >= 0 &&
_bitmap_value->contains(data)); }
};
template
@@ -88,7 +104,6 @@ void BitmapFilterFunc::insert(const void* data) {
}
*_bitmap_value |= *reinterpret_cast(data);
-_empty = false;
}
template
@@ -97,7 +112,6 @@ void BitmapFilterFunc::insert_many(const
std::vector b
return;
}
_bitmap_value->fastunion(bitmaps);
-_empty = false;
}
template
@@ -136,7 +150,6 @
(doris) branch branch-2.0 updated: [conf](mysql) opt mysql network timeout to 600s #32545 (#32739)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 2ae85e42846 [conf](mysql) opt mysql network timeout to 600s #32545
(#32739)
2ae85e42846 is described below
commit 2ae85e42846f7cebf66243e7edbeb2c1724d48be
Author: camby
AuthorDate: Fri Mar 29 11:12:59 2024 +0800
[conf](mysql) opt mysql network timeout to 600s #32545 (#32739)
---
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 55066fab565..efcfec7570e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -604,11 +604,11 @@ public class SessionVariable implements Serializable,
Writable {
// The number of seconds to wait for a block to be written to a connection
before aborting the write
@VariableMgr.VarAttr(name = NET_WRITE_TIMEOUT)
-public int netWriteTimeout = 60;
+public int netWriteTimeout = 600;
// The number of seconds to wait for a block to be written to a connection
before aborting the write
@VariableMgr.VarAttr(name = NET_READ_TIMEOUT)
-public int netReadTimeout = 60;
+public int netReadTimeout = 600;
// The current time zone
@VariableMgr.VarAttr(name = TIME_ZONE, needForward = true)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-1.2-lts updated: [pick-1.2][fix](mysql) fix mysql channel infinite blocking (#32741)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 74123e40bba [pick-1.2][fix](mysql) fix mysql channel infinite blocking
(#32741)
74123e40bba is described below
commit 74123e40bba52702bd96abde3ff347c0cf1e9d07
Author: camby
AuthorDate: Fri Mar 29 11:19:55 2024 +0800
[pick-1.2][fix](mysql) fix mysql channel infinite blocking (#32741)
* [fix](mysql) fix mysql channel infinite blocking (#28808)
Call the Channels blocking method with timeout instead.
Using session variables net_write_timeout and net_read_timeout as the
timeout parameter.
* [conf](mysql) opt mysql network timeout to 600s #32545
-
Co-authored-by: fornaix
---
.../src/main/java/org/apache/doris/mysql/MysqlChannel.java | 14 ++
.../src/main/java/org/apache/doris/qe/ConnectContext.java | 9 -
.../src/main/java/org/apache/doris/qe/SessionVariable.java | 4 ++--
3 files changed, 20 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlChannel.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlChannel.java
index 5172b243bc2..25d23af70e2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlChannel.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlChannel.java
@@ -30,6 +30,7 @@ import org.xnio.channels.Channels;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
+import java.util.concurrent.TimeUnit;
import javax.net.ssl.SSLEngine;
import javax.net.ssl.SSLEngineResult;
import javax.net.ssl.SSLException;
@@ -75,11 +76,13 @@ public class MysqlChannel {
protected volatile MysqlSerializer serializer;
+private ConnectContext context;
+
protected MysqlChannel() {
// For DummyMysqlChannel
}
-public MysqlChannel(StreamConnection connection) {
+public MysqlChannel(StreamConnection connection, ConnectContext context) {
Preconditions.checkNotNull(connection);
this.sequenceId = 0;
this.isSend = false;
@@ -100,6 +103,7 @@ public class MysqlChannel {
this.defaultBuffer = ByteBuffer.allocate(16 * 1024);
this.headerByteBuffer = ByteBuffer.allocate(PACKET_HEADER_LEN);
this.sendBuffer = ByteBuffer.allocate(2 * 1024 * 1024);
+this.context = context;
}
public void initSslBuffer() {
@@ -182,7 +186,8 @@ public class MysqlChannel {
}
try {
while (dstBuf.remaining() != 0) {
-int ret = Channels.readBlocking(conn.getSourceChannel(),
dstBuf);
+int ret = Channels.readBlocking(conn.getSourceChannel(),
dstBuf, context.getNetReadTimeout(),
+TimeUnit.SECONDS);
// return -1 when remote peer close the channel
if (ret == -1) {
decryptData(dstBuf, isHeader);
@@ -352,12 +357,13 @@ public class MysqlChannel {
protected void realNetSend(ByteBuffer buffer) throws IOException {
buffer = encryptData(buffer);
long bufLen = buffer.remaining();
-long writeLen = Channels.writeBlocking(conn.getSinkChannel(), buffer);
+long writeLen = Channels.writeBlocking(conn.getSinkChannel(), buffer,
context.getNetWriteTimeout(),
+TimeUnit.SECONDS);
if (bufLen != writeLen) {
throw new IOException("Write mysql packet failed.[write=" +
writeLen
+ ", needToWrite=" + bufLen + "]");
}
-Channels.flushBlocking(conn.getSinkChannel());
+Channels.flushBlocking(conn.getSinkChannel(),
context.getNetWriteTimeout(), TimeUnit.SECONDS);
isSend = true;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java
index d3994c04d9b..4ebf493bdd1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java
@@ -215,7 +215,7 @@ public class ConnectContext {
serverCapability = MysqlCapability.DEFAULT_CAPABILITY;
isKilled = false;
if (connection != null) {
-mysqlChannel = new MysqlChannel(connection);
+mysqlChannel = new MysqlChannel(connection, this);
} else {
mysqlChannel = new DummyMysqlChannel();
}
@@ -672,5 +672,12 @@ public class ConnectContext {
return "stmt[" + stmtId + ", " + DebugUtil.printId(queryId) + "]";
}
+public int getNetReadTimeout() {
+return this.sessionVariable.getNetReadTimeout();
+}
+
+public int getNetWriteTimeout() {
+return this.se
(doris) branch branch-1.2-lts updated: [fix](function) fixed the get_json_string function (#32149)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 64e332b1ea1 [fix](function) fixed the get_json_string function (#32149)
64e332b1ea1 is described below
commit 64e332b1ea1036b694184f5f33d41726a1cc979f
Author: lw112 <131352377+felixw...@users.noreply.github.com>
AuthorDate: Sat Mar 16 20:41:46 2024 +0800
[fix](function) fixed the get_json_string function (#32149)
---
be/src/vec/functions/function_json.cpp | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/be/src/vec/functions/function_json.cpp
b/be/src/vec/functions/function_json.cpp
index d4c46d829a2..11d75110a3d 100644
--- a/be/src/vec/functions/function_json.cpp
+++ b/be/src/vec/functions/function_json.cpp
@@ -23,6 +23,7 @@
#include
#include "exprs/json_functions.h"
+#include "vec/io/io_helper.h"
#ifdef __AVX2__
#include "util/jsonb_parser_simd.h"
#else
@@ -350,7 +351,7 @@ struct GetJsonString {
rapidjson::Value* root = nullptr;
root = get_json_object(json_string, path_string,
&document);
-const int max_string_len = (l_size <= 65535) ? 65535 : (l_size +
1);
+const int max_string_len = DEFAULT_MAX_JSON_SIZE;
if (root == nullptr || root->IsNull()) {
StringOP::push_null_string(i, res_data, res_offsets, null_map);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (aa3a4c373b1 -> 256981a7cf4)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from aa3a4c373b1 [fix](compile) Conditional inclusion of Hadoop HDFS header based on compilation flag (#32663) add 256981a7cf4 [fix](metrics) fix compaction_used_permits are negative numbers (#32440) No new revisions were added by this update. Summary of changes: be/src/olap/olap_server.cpp | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](index) bitmap and bloomfilter index should not do light index change (#35225)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 746c6207fca [fix](index) bitmap and bloomfilter index should not do
light index change (#35225)
746c6207fca is described below
commit 746c6207fca8aef5f66c29b903d407995fdf1923
Author: camby
AuthorDate: Wed May 29 10:09:31 2024 +0800
[fix](index) bitmap and bloomfilter index should not do light index change
(#35225)
---
.../olap/rowset/segment_v2/bitmap_index_reader.cpp | 3 +
.../segment_v2/bloom_filter_index_reader.cpp | 4 +
be/src/olap/schema_change.cpp | 12 +++
be/src/olap/tablet_schema.cpp | 18
be/src/olap/tablet_schema.h| 2 +
.../test_index_ddl_fault_injection.out | 22 +
.../test_index_ddl_fault_injection.groovy | 97 ++
7 files changed, 158 insertions(+)
diff --git a/be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
b/be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
index c76de68b7ba..fe3609f01ad 100644
--- a/be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
@@ -24,6 +24,7 @@
#include
#include "olap/types.h"
+#include "util/debug_points.h"
#include "vec/columns/column.h"
#include "vec/common/string_ref.h"
#include "vec/data_types/data_type.h"
@@ -53,6 +54,8 @@ Status BitmapIndexReader::_load(bool use_page_cache, bool
kept_in_memory,
}
Status BitmapIndexReader::new_iterator(BitmapIndexIterator** iterator) {
+DBUG_EXECUTE_IF("BitmapIndexReader::new_iterator.fail",
+{ return Status::InternalError("new_iterator for bitmap
index failed"); });
*iterator = new BitmapIndexIterator(this);
return Status::OK();
}
diff --git a/be/src/olap/rowset/segment_v2/bloom_filter_index_reader.cpp
b/be/src/olap/rowset/segment_v2/bloom_filter_index_reader.cpp
index c9b74c034ee..0857c1890c4 100644
--- a/be/src/olap/rowset/segment_v2/bloom_filter_index_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/bloom_filter_index_reader.cpp
@@ -22,6 +22,7 @@
#include "olap/rowset/segment_v2/bloom_filter.h"
#include "olap/types.h"
+#include "util/debug_points.h"
#include "vec/columns/column.h"
#include "vec/common/string_ref.h"
#include "vec/data_types/data_type.h"
@@ -46,6 +47,9 @@ Status BloomFilterIndexReader::_load(bool use_page_cache,
bool kept_in_memory) {
}
Status
BloomFilterIndexReader::new_iterator(std::unique_ptr*
iterator) {
+DBUG_EXECUTE_IF("BloomFilterIndexReader::new_iterator.fail", {
+return Status::InternalError("new_iterator for bloom filter index
failed");
+});
iterator->reset(new BloomFilterIndexIterator(this));
return Status::OK();
}
diff --git a/be/src/olap/schema_change.cpp b/be/src/olap/schema_change.cpp
index c5ae7793c47..02f60a993a1 100644
--- a/be/src/olap/schema_change.cpp
+++ b/be/src/olap/schema_change.cpp
@@ -765,6 +765,9 @@ Status
SchemaChangeHandler::_do_process_alter_tablet_v2(const TAlterTabletReqV2&
for (const auto& column : request.columns) {
base_tablet_schema->append_column(TabletColumn(column));
}
+// The request only include column info, do not include bitmap or
bloomfilter index info,
+// So we also need to copy index info from the real base tablet
+
base_tablet_schema->update_index_info_from(*base_tablet->tablet_schema());
}
// Use tablet schema directly from base tablet, they are the newest
schema, not contain
// dropped column during light weight schema change.
@@ -1299,6 +1302,15 @@ Status SchemaChangeHandler::_parse_request(const
SchemaChangeParams& sc_params,
if (column_mapping->expr != nullptr) {
*sc_directly = true;
return Status::OK();
+} else if (column_mapping->ref_column >= 0) {
+const auto& column_new = new_tablet_schema->column(i);
+const auto& column_old =
base_tablet_schema->column(column_mapping->ref_column);
+// index changed
+if (column_new.is_bf_column() != column_old.is_bf_column() ||
+column_new.has_bitmap_index() !=
column_old.has_bitmap_index()) {
+*sc_directly = true;
+return Status::OK();
+}
}
}
diff --git a/be/src/olap/tablet_schema.cpp b/be/src/olap/tablet_schema.cpp
index d423ec4d709..a44abedc354 100644
--- a/be/src/olap/tablet_schema.cpp
+++ b/be/src/olap/tablet_schema.cpp
@@ -974,6 +974,24 @@ void TabletSchema::copy_from(const TabletSchema&
tablet_schema) {
_table_i
(doris) branch master updated: [fix](short circurt) fix return default value issue (#34186)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e000330e616 [fix](short circurt) fix return default value issue
(#34186)
e000330e616 is described below
commit e000330e616c4a4c2313be48b95c43f7e4a113ad
Author: lw112 <131352377+felixw...@users.noreply.github.com>
AuthorDate: Wed May 29 20:10:50 2024 +0800
[fix](short circurt) fix return default value issue (#34186)
---
.../vec/data_types/serde/data_type_nullable_serde.cpp | 10 +-
.../test_compaction_uniq_keys_row_store.out | 8
.../insert_into_table/partial_update_seq_col.out | Bin 1412 -> 1416 bytes
regression-test/data/point_query_p0/test_rowstore.out | 6 ++
.../test_partial_update_insert_seq_col.out| Bin 1412 -> 1416 bytes
.../partial_update/test_partial_update_seq_col.out| Bin 1411 -> 1415 bytes
.../test_partial_update_seq_col_delete.out| Bin 1526 -> 1530 bytes
.../suites/point_query_p0/test_rowstore.groovy| 9 +
8 files changed, 24 insertions(+), 9 deletions(-)
diff --git a/be/src/vec/data_types/serde/data_type_nullable_serde.cpp
b/be/src/vec/data_types/serde/data_type_nullable_serde.cpp
index 43ee1dce5d9..254c3d74ab4 100644
--- a/be/src/vec/data_types/serde/data_type_nullable_serde.cpp
+++ b/be/src/vec/data_types/serde/data_type_nullable_serde.cpp
@@ -238,13 +238,13 @@ void DataTypeNullableSerDe::write_one_cell_to_jsonb(const
IColumn& column, Jsonb
Arena* mem_pool, int32_t
col_id,
int row_num) const {
auto& nullable_col = assert_cast(column);
+result.writeKey(col_id);
if (nullable_col.is_null_at(row_num)) {
-// do not insert to jsonb
-return;
+result.writeNull();
+} else {
+
nested_serde->write_one_cell_to_jsonb(nullable_col.get_nested_column(), result,
mem_pool,
+ col_id, row_num);
}
-result.writeKey(col_id);
-nested_serde->write_one_cell_to_jsonb(nullable_col.get_nested_column(),
result, mem_pool,
- col_id, row_num);
}
void DataTypeNullableSerDe::read_one_cell_from_jsonb(IColumn& column, const
JsonbValue* arg) const {
diff --git
a/regression-test/data/compaction/test_compaction_uniq_keys_row_store.out
b/regression-test/data/compaction/test_compaction_uniq_keys_row_store.out
index cedf0dbe9bd..7c163c62d33 100644
--- a/regression-test/data/compaction/test_compaction_uniq_keys_row_store.out
+++ b/regression-test/data/compaction/test_compaction_uniq_keys_row_store.out
@@ -18,10 +18,10 @@
3 2017-10-01 2017-10-01 2017-10-01T11:11:11.026
2017-10-01T11:11:11.016 Beijing 10 1 2020-01-04T00:00
2020-01-04T00:002017-10-01T11:11:11.110 2017-10-01T11:11:11.150111
2020-01-04T00:001 33 21
-- !point_select --
-3 2017-10-01 2017-10-01 2017-10-01T11:11:11.027
2017-10-01T11:11:11.017 Beijing 10 1 1970-01-01T00:00
1970-01-01T00:001970-01-01T00:00:00.111 1970-01-01T00:00
2020-01-05T00:001 34 20
+3 2017-10-01 2017-10-01 2017-10-01T11:11:11.027
2017-10-01T11:11:11.017 Beijing 10 1 \N \N \N \N
2020-01-05T00:001 34 20
-- !point_select --
-4 2017-10-01 2017-10-01 2017-10-01T11:11:11.028
2017-10-01T11:11:11.018 Beijing 10 1 1970-01-01T00:00
1970-01-01T00:001970-01-01T00:00:00.111 1970-01-01T00:00
2020-01-05T00:001 34 20
+4 2017-10-01 2017-10-01 2017-10-01T11:11:11.028
2017-10-01T11:11:11.018 Beijing 10 1 \N \N \N \N
2020-01-05T00:001 34 20
-- !point_select --
1 2017-10-01 2017-10-01 2017-10-01T11:11:11.021
2017-10-01T11:11:11.011 Beijing 10 1 2020-01-01T00:00
2020-01-01T00:002017-10-01T11:11:11.170 2017-10-01T11:11:11.110111
2020-01-01T00:001 30 20
@@ -42,8 +42,8 @@
3 2017-10-01 2017-10-01 2017-10-01T11:11:11.026
2017-10-01T11:11:11.016 Beijing 10 1 2020-01-04T00:00
2020-01-04T00:002017-10-01T11:11:11.110 2017-10-01T11:11:11.150111
2020-01-04T00:001 33 21
-- !point_select --
-3 2017-10-01 2017-10-01 2017-10-01T11:11:11.027
2017-10-01T11:11:11.017 Beijing 10 1 1970-01-01T00:00
1970-01-01T00:001970-01-01T00:00:00.111 1970-01-01T00:00
2020-01-05T00:001 34 20
+3 2017-10-01 2017-10-01 2017-10-01T11:11:11.027
2017-
(doris) branch master updated (a8df1d7e145 -> cc1fef18005)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a8df1d7e145 [Test](regression-test): Normalize S3 http url (#35666) add cc1fef18005 [regression-test](load) add something like $.tag.[a.b] key's json case (#35134) No new revisions were added by this update. Summary of changes: .../data/load_p0/stream_load/test_json_load.out| 3 +++ .../load_p0/stream_load/test_special_key_json.json | 1 + .../load_p0/stream_load/test_json_load.groovy | 26 ++ 3 files changed, 30 insertions(+) create mode 100644 regression-test/data/load_p0/stream_load/test_special_key_json.json - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](compile) fix two compile errors on MacOS (#33834) (#34005)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 743fb62a2c4 [fix](compile) fix two compile errors on MacOS (#33834)
(#34005)
743fb62a2c4 is described below
commit 743fb62a2c42cc5cc662583c235f7336d5e6ddef
Author: camby
AuthorDate: Thu Apr 25 19:39:35 2024 +0800
[fix](compile) fix two compile errors on MacOS (#33834) (#34005)
---
be/CMakeLists.txt | 5 +
be/src/runtime/CMakeLists.txt | 2 +-
.../aggregate_function_group_array_intersect.h| 8
build.sh | 5 +
regression-test/pipeline/performance/compile.sh | 1 +
5 files changed, 16 insertions(+), 5 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index f5d99c69962..2082aa8def8 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -72,6 +72,7 @@ option(USE_LIBCPP "Use libc++" OFF)
option(USE_MEM_TRACKER, "Use memory tracker" ON)
option(USE_UNWIND "Use libunwind" ON)
option(USE_JEMALLOC "Use jemalloc" ON)
+option(USE_JEMALLOC_HOOK "Use jemalloc hook" ON)
if (OS_MACOSX)
set(GLIBC_COMPATIBILITY OFF)
set(USE_LIBCPP ON)
@@ -87,6 +88,7 @@ message(STATUS "GLIBC_COMPATIBILITY is
${GLIBC_COMPATIBILITY}")
message(STATUS "USE_LIBCPP is ${USE_LIBCPP}")
message(STATUS "USE_MEM_TRACKER is ${USE_MEM_TRACKER}")
message(STATUS "USE_JEMALLOC is ${USE_JEMALLOC}")
+message(STATUS "USE_JEMALLOC_HOOK is ${USE_JEMALLOC_HOOK}")
message(STATUS "USE_UNWIND is ${USE_UNWIND}")
message(STATUS "ENABLE_PCH is ${ENABLE_PCH}")
@@ -341,6 +343,9 @@ endif()
if (USE_JEMALLOC)
add_compile_options(-DUSE_JEMALLOC)
endif()
+if (USE_JEMALLOC_HOOK)
+add_definitions(-DUSE_JEMALLOC_HOOK)
+endif()
# Compile with libunwind
if (USE_UNWIND)
diff --git a/be/src/runtime/CMakeLists.txt b/be/src/runtime/CMakeLists.txt
index a0b3b799a76..3bfec93edfc 100644
--- a/be/src/runtime/CMakeLists.txt
+++ b/be/src/runtime/CMakeLists.txt
@@ -25,7 +25,7 @@ set(EXECUTABLE_OUTPUT_PATH "${BUILD_DIR}/src/runtime")
file(GLOB_RECURSE RUNTIME_FILES CONFIGURE_DEPENDS *.cpp *.cc)
-if (NOT USE_JEMALLOC OR NOT USE_MEM_TRACKER)
+if (NOT USE_JEMALLOC OR NOT USE_MEM_TRACKER OR NOT USE_JEMALLOC_HOOK)
list(REMOVE_ITEM RUNTIME_FILES
${CMAKE_CURRENT_SOURCE_DIR}/memory/jemalloc_hook.cpp)
endif()
diff --git
a/be/src/vec/aggregate_functions/aggregate_function_group_array_intersect.h
b/be/src/vec/aggregate_functions/aggregate_function_group_array_intersect.h
index 03c1639c45a..5d627782f25 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_group_array_intersect.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_group_array_intersect.h
@@ -256,11 +256,11 @@ public:
read_pod_binary(is_set_contains_null, buf);
data.value->change_contains_null_value(is_set_contains_null);
read_pod_binary(data.init, buf);
-size_t size;
+UInt64 size;
read_var_uint(size, buf);
T element;
-for (size_t i = 0; i < size; ++i) {
+for (UInt64 i = 0; i < size; ++i) {
read_int_binary(element, buf);
data.value->insert(static_cast(&element));
}
@@ -484,11 +484,11 @@ public:
read_pod_binary(is_set_contains_null, buf);
data.value->change_contains_null_value(is_set_contains_null);
read_pod_binary(data.init, buf);
-size_t size;
+UInt64 size;
read_var_uint(size, buf);
StringRef element;
-for (size_t i = 0; i < size; ++i) {
+for (UInt64 i = 0; i < size; ++i) {
element = read_string_binary_into(*arena, buf);
data.value->insert((void*)element.data, element.size);
}
diff --git a/build.sh b/build.sh
index efa808652b6..99de71173ac 100755
--- a/build.sh
+++ b/build.sh
@@ -346,6 +346,9 @@ fi
if [[ -z "${USE_JEMALLOC}" ]]; then
USE_JEMALLOC='ON'
fi
+if [[ -z "${USE_JEMALLOC_HOOK}" ]]; then
+USE_JEMALLOC_HOOK='OFF'
+fi
if [[ -z "${USE_BTHREAD_SCANNER}" ]]; then
USE_BTHREAD_SCANNER='OFF'
fi
@@ -446,6 +449,7 @@ echo "Get params:
STRIP_DEBUG_INFO-- ${STRIP_DEBUG_INFO}
USE_MEM_TRACKER -- ${USE_MEM_TRACKER}
USE_JEMALLOC-- ${USE_JEMALLOC}
+USE_JEMALLOC_HOOK -- ${USE_JEMALLOC_HOOK}
USE_BTHREAD_SCANNER -- ${USE_BTHREAD_SCANNER}
ENABLE_STACKTRACE -- ${ENABLE_STACKTRACE}
DENABLE_CLANG_COVERAGE -- ${DENABLE_CLANG_COVERAGE}
@@ -544,6 +548,
(doris) branch branch-2.1 updated: [fix](compile) fix two compile errors on MacOS (#33834) (#34149)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 10e098845df [fix](compile) fix two compile errors on MacOS (#33834)
(#34149)
10e098845df is described below
commit 10e098845df781d4f8f2e2065eca246968ab7c98
Author: camby
AuthorDate: Fri Apr 26 17:02:44 2024 +0800
[fix](compile) fix two compile errors on MacOS (#33834) (#34149)
---
.../aggregate_function_group_array_intersect.h| 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/be/src/vec/aggregate_functions/aggregate_function_group_array_intersect.h
b/be/src/vec/aggregate_functions/aggregate_function_group_array_intersect.h
index 03c1639c45a..5d627782f25 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_group_array_intersect.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_group_array_intersect.h
@@ -256,11 +256,11 @@ public:
read_pod_binary(is_set_contains_null, buf);
data.value->change_contains_null_value(is_set_contains_null);
read_pod_binary(data.init, buf);
-size_t size;
+UInt64 size;
read_var_uint(size, buf);
T element;
-for (size_t i = 0; i < size; ++i) {
+for (UInt64 i = 0; i < size; ++i) {
read_int_binary(element, buf);
data.value->insert(static_cast(&element));
}
@@ -484,11 +484,11 @@ public:
read_pod_binary(is_set_contains_null, buf);
data.value->change_contains_null_value(is_set_contains_null);
read_pod_binary(data.init, buf);
-size_t size;
+UInt64 size;
read_var_uint(size, buf);
StringRef element;
-for (size_t i = 0; i < size; ++i) {
+for (UInt64 i = 0; i < size; ++i) {
element = read_string_binary_into(*arena, buf);
data.value->insert((void*)element.data, element.size);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-1.2-lts updated: fix query external table make BE crash (#34477)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new e3b1916b11f fix query external table make BE crash (#34477)
e3b1916b11f is described below
commit e3b1916b11f2b6a758763ece89020fb50810ca0b
Author: camby
AuthorDate: Thu May 9 10:28:47 2024 +0800
fix query external table make BE crash (#34477)
---
.../java/org/apache/doris/planner/external/ExternalFileScanNode.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/ExternalFileScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/ExternalFileScanNode.java
index 124acdb71ac..b89e5db4a66 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/ExternalFileScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/ExternalFileScanNode.java
@@ -339,7 +339,6 @@ public class ExternalFileScanNode extends ExternalScanNode {
ParamCreateContext context = contexts.get(i);
FileScanProviderIf scanProvider = scanProviders.get(i);
setDefaultValueExprs(scanProvider, context);
-setColumnPositionMappingForTextFile(scanProvider, context);
finalizeParamsForLoad(context, analyzer);
createScanRangeLocations(context, scanProvider);
this.inputSplitsNum += scanProvider.getInputSplitNum();
@@ -358,6 +357,7 @@ public class ExternalFileScanNode extends ExternalScanNode {
if (scanProvider instanceof IcebergScanProvider) {
((IcebergScanProvider)
scanProvider).updateRequiredSlots(context);
}
+setColumnPositionMappingForTextFile(scanProvider, context);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](replication_allocation) fix two problems for force_olap_table_replication_allocation #34554 (#34659)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new a3d185479b9 [fix](replication_allocation) fix two problems for
force_olap_table_replication_allocation #34554 (#34659)
a3d185479b9 is described below
commit a3d185479b9262bf11a907373608518c5aeddb64
Author: camby
AuthorDate: Mon May 13 10:57:57 2024 +0800
[fix](replication_allocation) fix two problems for
force_olap_table_replication_allocation #34554 (#34659)
---
.../main/java/org/apache/doris/common/util/DynamicPartitionUtil.java | 3 ++-
.../main/java/org/apache/doris/mysql/privilege/UserPropertyMgr.java | 5 +++--
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java
index ca3086c6ede..e26d0cbd9f2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java
@@ -630,7 +630,8 @@ public class DynamicPartitionUtil {
// check replication_allocation first, then replciation_num
ReplicaAllocation replicaAlloc = null;
-if
(properties.containsKey(DynamicPartitionProperty.REPLICATION_ALLOCATION)) {
+if (!Config.force_olap_table_replication_allocation.isEmpty()
+||
properties.containsKey(DynamicPartitionProperty.REPLICATION_ALLOCATION)) {
replicaAlloc =
PropertyAnalyzer.analyzeReplicaAllocation(properties, "dynamic_partition");
properties.remove(DynamicPartitionProperty.REPLICATION_ALLOCATION);
analyzedProperties.put(DynamicPartitionProperty.REPLICATION_ALLOCATION,
replicaAlloc.toCreateStmt());
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserPropertyMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserPropertyMgr.java
index 8f634bccbe3..4fb873d404d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserPropertyMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserPropertyMgr.java
@@ -137,9 +137,10 @@ public class UserPropertyMgr implements Writable {
Set tags = existProperty.getCopiedResourceTags();
// only root and admin can return empty tag.
// empty tag means user can access all backends.
-// for normal user, if tag is empty, use default tag.
+// for normal user, if tag is empty and not set
force_olap_table_replication_allocation, use default tag.
if (tags.isEmpty() && !(qualifiedUser.equalsIgnoreCase(Auth.ROOT_USER)
-|| qualifiedUser.equalsIgnoreCase(Auth.ADMIN_USER))) {
+|| qualifiedUser.equalsIgnoreCase(Auth.ADMIN_USER))
+&& Config.force_olap_table_replication_allocation.isEmpty()) {
tags = Sets.newHashSet(Tag.DEFAULT_BACKEND_TAG);
}
return tags;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (75ad0ff6760 -> 3f4e548ba5d)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 75ad0ff6760 [Improve]delete match_element_xx and add fe config for create inverted index on array (#34766) add 3f4e548ba5d fix backup and restore failed between force_replication_allocation setted and not setted clusters (#34608) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/java/org/apache/doris/analysis/RestoreStmt.java | 5 + fe/fe-core/src/main/java/org/apache/doris/backup/BackupJobInfo.java | 4 fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java| 4 3 files changed, 13 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated (96e1280f4b8 -> f1eaf165075)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 96e1280f4b8 [branch-2.0](resource)fix check available fail when s3 aws_token is set and reset as, sk faild on be. #34655 add f1eaf165075 fix backup and restore failed between force_replication_allocation setted and not setted clusters (#34624) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/java/org/apache/doris/analysis/RestoreStmt.java | 5 + fe/fe-core/src/main/java/org/apache/doris/backup/BackupJobInfo.java | 4 fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java| 4 3 files changed, 13 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](profile) task type not the same in observer and master (#39245)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5bb0c688d44 [fix](profile) task type not the same in observer and
master (#39245)
5bb0c688d44 is described below
commit 5bb0c688d440f8b3c1f5031bac939b8baf1e15d6
Author: camby
AuthorDate: Wed Aug 21 14:19:45 2024 +0800
[fix](profile) task type not the same in observer and master (#39245)
---
.../src/main/java/org/apache/doris/qe/StmtExecutor.java | 11 +++
1 file changed, 11 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index 8a1a2f2a606..64b216052ab 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -714,6 +714,10 @@ public class StmtExecutor {
if (logicalPlan instanceof Forward) {
redirectStatus = ((Forward) logicalPlan).toRedirectStatus();
if (isForwardToMaster()) {
+// before forward to master, we also need to set
profileType in this node
+if (logicalPlan instanceof InsertIntoTableCommand) {
+profileType = ProfileType.LOAD;
+}
if (context.getCommand() == MysqlCommand.COM_STMT_PREPARE)
{
throw new UserException("Forward master command is not
supported for prepare statement");
}
@@ -956,6 +960,13 @@ public class StmtExecutor {
analyze(context.getSessionVariable().toThrift());
if (isForwardToMaster()) {
+// before forward to master, we also need to set
profileType in this node
+if (parsedStmt instanceof InsertStmt) {
+InsertStmt insertStmt = (InsertStmt) parsedStmt;
+if (!insertStmt.getQueryStmt().isExplain()) {
+profileType = ProfileType.LOAD;
+}
+}
if (context.getCommand() == MysqlCommand.COM_STMT_PREPARE)
{
throw new UserException("Forward master command is not
supported for prepare statement");
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [opt](log) Remove unnecessary log for analysis (#38944)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 4ecf77e7b7f [opt](log) Remove unnecessary log for analysis (#38944)
4ecf77e7b7f is described below
commit 4ecf77e7b7fccc7338421a57239e63edd943659d
Author: camby
AuthorDate: Thu Aug 8 23:18:01 2024 +0800
[opt](log) Remove unnecessary log for analysis (#38944)
---
.../src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java | 1 -
1 file changed, 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
index 1eec0ee93f9..d93b0b74e54 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
@@ -105,7 +105,6 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
double scaleFactor = (double) totalRowCount / (double) pair.second;
// might happen if row count in fe metadata hasn't been updated yet
if (Double.isInfinite(scaleFactor) || Double.isNaN(scaleFactor)) {
-LOG.warn("Scale factor is infinite or Nan, will set scale factor
to 1.");
scaleFactor = 1;
tabletIds = Collections.emptyList();
pair.second = totalRowCount;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [opt](log) Remove unnecessary warning log (#37093) (#38941)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new d98c0aff8b5 [opt](log) Remove unnecessary warning log (#37093) (#38941)
d98c0aff8b5 is described below
commit d98c0aff8b599677ef1fd669086b67ba22085c0b
Author: camby
AuthorDate: Fri Aug 9 14:35:36 2024 +0800
[opt](log) Remove unnecessary warning log (#37093) (#38941)
---
fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java
index 1cbe41d9107..9b933f73ff1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java
@@ -202,8 +202,7 @@ public final class QeProcessorImpl implements QeProcessor {
} else {
result.setStatus(new TStatus(TStatusCode.RUNTIME_ERROR));
}
-LOG.warn("ReportExecStatus() runtime error, query {} with type {}
does not exist",
-DebugUtil.printId(params.query_id), params.query_type);
+
return result;
}
try {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [mac](compile) fix compile error on mac (#37726)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 9556c07a167 [mac](compile) fix compile error on mac (#37726)
9556c07a167 is described below
commit 9556c07a167dba87394b4acd0b1dbdd3a6f60504
Author: camby
AuthorDate: Mon Jul 15 10:19:42 2024 +0800
[mac](compile) fix compile error on mac (#37726)
---
be/src/runtime/group_commit_mgr.cpp | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/be/src/runtime/group_commit_mgr.cpp
b/be/src/runtime/group_commit_mgr.cpp
index 3dd64f154cf..fd9b96b0c1f 100644
--- a/be/src/runtime/group_commit_mgr.cpp
+++ b/be/src/runtime/group_commit_mgr.cpp
@@ -142,7 +142,8 @@ Status LoadBlockQueue::get_block(RuntimeState*
runtime_state, vectorized::Block*
<< ", runtime_state=" << runtime_state;
}
}
-_get_cond.wait_for(l,
std::chrono::milliseconds(std::min(left_milliseconds, 1L)));
+_get_cond.wait_for(l, std::chrono::milliseconds(
+ std::min(left_milliseconds,
static_cast(1;
}
if (runtime_state->is_cancelled()) {
auto st = Status::Cancelled(runtime_state->cancel_reason());
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](colocate) fix colocate join while multi tables (#37729)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 09ba4ac8ff5 [fix](colocate) fix colocate join while multi tables
(#37729)
09ba4ac8ff5 is described below
commit 09ba4ac8ff54da4150c762e3fcf2292dad991da2
Author: camby
AuthorDate: Mon Jul 15 15:01:04 2024 +0800
[fix](colocate) fix colocate join while multi tables (#37729)
---
.../org/apache/doris/nereids/util/JoinUtils.java | 9 +++
.../test_colocate_join_of_column_order.groovy | 29 ++
2 files changed, 32 insertions(+), 6 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java
index 5062bedc67c..770905bf199 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java
@@ -325,12 +325,9 @@ public class JoinUtils {
SlotReference leftSlot = (SlotReference) leftChild;
SlotReference rightSlot = (SlotReference) rightChild;
-Integer leftIndex = null;
-Integer rightIndex = null;
-if (leftSlot.getTable().isPresent() &&
leftSlot.getTable().get().getId() == leftHashSpec.getTableId()) {
-leftIndex =
leftHashSpec.getExprIdToEquivalenceSet().get(leftSlot.getExprId());
-rightIndex =
rightHashSpec.getExprIdToEquivalenceSet().get(rightSlot.getExprId());
-} else {
+Integer leftIndex =
leftHashSpec.getExprIdToEquivalenceSet().get(leftSlot.getExprId());
+Integer rightIndex =
rightHashSpec.getExprIdToEquivalenceSet().get(rightSlot.getExprId());
+if (leftIndex == null) {
leftIndex =
rightHashSpec.getExprIdToEquivalenceSet().get(leftSlot.getExprId());
rightIndex =
leftHashSpec.getExprIdToEquivalenceSet().get(rightSlot.getExprId());
}
diff --git
a/regression-test/suites/correctness_p0/test_colocate_join_of_column_order.groovy
b/regression-test/suites/correctness_p0/test_colocate_join_of_column_order.groovy
index 663b7da02d6..efef9969506 100644
---
a/regression-test/suites/correctness_p0/test_colocate_join_of_column_order.groovy
+++
b/regression-test/suites/correctness_p0/test_colocate_join_of_column_order.groovy
@@ -79,4 +79,33 @@ suite("test_colocate_join_of_column_order") {
sql """ DROP TABLE IF EXISTS `test_colocate_join_of_column_order_t1`; """
sql """ DROP TABLE IF EXISTS `test_colocate_join_of_column_order_t2`; """
+
+// multi tables
+sql """ DROP TABLE IF EXISTS `test_colocate_join_of_column_order_ta`; """
+sql """
+CREATE TABLE IF NOT EXISTS `test_colocate_join_of_column_order_ta` (
`c1` bigint NULL, `c2` bigint NULL)
+DISTRIBUTED BY HASH(c1) PROPERTIES ( "replication_num" = "1",
"colocate_with" = "group_column_order3");
+"""
+sql """ DROP TABLE IF EXISTS `test_colocate_join_of_column_order_tb`; """
+sql """
+CREATE TABLE IF NOT EXISTS `test_colocate_join_of_column_order_tb` (
`c1` bigint NULL, `c2` bigint NULL)
+DISTRIBUTED BY HASH(c1) PROPERTIES ( "replication_num" = "1",
"colocate_with" = "group_column_order3");
+"""
+sql """ DROP TABLE IF EXISTS `test_colocate_join_of_column_order_tc`; """
+sql """
+CREATE TABLE IF NOT EXISTS `test_colocate_join_of_column_order_tc` (
`c1` bigint NULL, `c2` bigint NULL)
+DISTRIBUTED BY HASH(c1) PROPERTIES ( "replication_num" = "1",
"colocate_with" = "group_column_order3");
+"""
+sql """insert into test_colocate_join_of_column_order_ta values(1,1);"""
+sql """insert into test_colocate_join_of_column_order_tb values(1,1);"""
+sql """insert into test_colocate_join_of_column_order_tc values(1,1);"""
+
+explain {
+sql("""select /*+ set_var(disable_join_reorder=true) */ * from
test_colocate_join_of_column_order_ta join [shuffle] (select cast((c2 + 1) as
bigint) c2 from test_colocate_join_of_column_order_tb)
test_colocate_join_of_column_order_tb on
test_colocate_join_of_column_order_ta.c1 =
test_colocate_join_of_column_order_tb.c2 join [shuffle]
test_colocate_join_of_column_order_tc on
test_colocate_join_of_column_order_tb.c2 =
test_colocate_join_of_column_order_tc.c1;""");
+contains "COLOCATE"
+}
+
+sql """ DROP TABLE IF EXISTS `test_colocate_join_of_column_order_ta`; """
+sql """ DROP TABLE IF EXISTS `test_colocate_join_of_column_order_tb`; """
+sql """ DROP TABLE IF EXISTS `test_colocate_join_of_column_order_tc`; """
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-1.2-lts updated: [fix](timeout) unify query_timeout, use setting from user property first (#37626)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new f57627285f7 [fix](timeout) unify query_timeout, use setting from user
property first (#37626)
f57627285f7 is described below
commit f57627285f796249c680acfb441cad31f9705f2a
Author: Yulei-Yang
AuthorDate: Tue Jul 16 19:24:07 2024 +0800
[fix](timeout) unify query_timeout, use setting from user property first
(#37626)
---
.../org/apache/doris/mysql/AcceptListener.java | 2 +-
.../java/org/apache/doris/qe/ConnectContext.java | 28 +++---
2 files changed, 9 insertions(+), 21 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/AcceptListener.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/AcceptListener.java
index 4f059c7c751..6fec03bc565 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/AcceptListener.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/AcceptListener.java
@@ -78,7 +78,7 @@ public class AcceptListener implements
ChannelListener 0) {
+sessionVariable.setQueryTimeoutS(queryTimeout);
+}
}
private StatementContext statementContext;
@@ -571,23 +571,11 @@ public class ConnectContext {
killConnection = true;
}
} else {
-if (userQueryTimeout > 0) {
-// user set query_timeout property
-if (delta > userQueryTimeout * 1000) {
-LOG.warn("kill query timeout, remote: {}, query timeout:
{}",
-getMysqlChannel().getRemoteHostPortString(),
userQueryTimeout);
+if (delta > sessionVariable.getQueryTimeoutS() * 1000) {
+LOG.warn("kill query timeout, remote: {}, query timeout: {}",
+getMysqlChannel().getRemoteHostPortString(),
sessionVariable.getQueryTimeoutS());
-killFlag = true;
-}
-} else {
-// default use session query_timeout
-if (delta > sessionVariable.getQueryTimeoutS() * 1000) {
-LOG.warn("kill query timeout, remote: {}, query timeout:
{}",
-getMysqlChannel().getRemoteHostPortString(),
sessionVariable.getQueryTimeoutS());
-
-// Only kill
-killFlag = true;
-}
+killFlag = true;
}
}
if (killFlag) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](audit) duplicate audit log for multi-statements at the same query (#37933)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new cc1770ca359 [fix](audit) duplicate audit log for multi-statements at
the same query (#37933)
cc1770ca359 is described below
commit cc1770ca359b2d6680c92a51946f69ccd256bf5d
Author: camby
AuthorDate: Wed Jul 17 19:32:41 2024 +0800
[fix](audit) duplicate audit log for multi-statements at the same query
(#37933)
---
.../java/org/apache/doris/qe/AuditLogHelper.java | 33
.../java/org/apache/doris/qe/ConnectProcessor.java | 90 +-
2 files changed, 21 insertions(+), 102 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java
index dcfedc26792..b0a038dbf5e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java
@@ -25,6 +25,7 @@ import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.Config;
import org.apache.doris.common.util.DebugUtil;
import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.plugin.AuditEvent.AuditEventBuilder;
import org.apache.doris.plugin.AuditEvent.EventType;
import org.apache.doris.qe.QueryState.MysqlStateType;
import org.apache.doris.service.FrontendOptions;
@@ -44,7 +45,13 @@ public class AuditLogHelper {
long elapseMs = endTime - ctx.getStartTime();
SpanContext spanContext =
Span.fromContext(Context.current()).getSpanContext();
-ctx.getAuditEventBuilder().setEventType(EventType.AFTER_QUERY)
+AuditEventBuilder auditEventBuilder = ctx.getAuditEventBuilder();
+auditEventBuilder.reset();
+auditEventBuilder.setTimestamp(ctx.getStartTime())
+.setClientIp(ctx.getMysqlChannel().getRemoteHostPortString())
+
.setUser(ClusterNamespace.getNameFromFullName(ctx.getQualifiedUser()))
+.setSqlHash(ctx.getSqlHash())
+.setEventType(EventType.AFTER_QUERY)
.setDb(ClusterNamespace.getNameFromFullName(ctx.getDatabase()))
.setState(ctx.getState().toString())
.setErrorCode(ctx.getState().getErrorCode() == null ? 0 :
ctx.getState().getErrorCode().getCode())
@@ -78,10 +85,10 @@ public class AuditLogHelper {
if (elapseMs > Config.qe_slow_log_ms) {
String sqlDigest = DigestUtils.md5Hex(((Queriable)
parsedStmt).toDigest());
-ctx.getAuditEventBuilder().setSqlDigest(sqlDigest);
+auditEventBuilder.setSqlDigest(sqlDigest);
}
}
-ctx.getAuditEventBuilder().setIsQuery(true);
+auditEventBuilder.setIsQuery(true);
if (ctx.getQueryDetail() != null) {
ctx.getQueryDetail().setEventTime(endTime);
ctx.getQueryDetail().setEndTime(endTime);
@@ -91,35 +98,35 @@ public class AuditLogHelper {
ctx.setQueryDetail(null);
}
} else {
-ctx.getAuditEventBuilder().setIsQuery(false);
+auditEventBuilder.setIsQuery(false);
}
-ctx.getAuditEventBuilder().setIsNereids(ctx.getState().isNereids);
+auditEventBuilder.setIsNereids(ctx.getState().isNereids);
-
ctx.getAuditEventBuilder().setFeIp(FrontendOptions.getLocalHostAddress());
+auditEventBuilder.setFeIp(FrontendOptions.getLocalHostAddress());
// We put origin query stmt at the end of audit log, for parsing the
log more convenient.
if (!ctx.getState().isQuery() && (parsedStmt != null &&
parsedStmt.needAuditEncryption())) {
-ctx.getAuditEventBuilder().setStmt(parsedStmt.toSql());
+auditEventBuilder.setStmt(parsedStmt.toSql());
} else {
if (parsedStmt instanceof InsertStmt && !((InsertStmt)
parsedStmt).needLoadManager()
&& ((InsertStmt) parsedStmt).isValuesOrConstantSelect()) {
// INSERT INTO VALUES may be very long, so we only log at most
1K bytes.
int length = Math.min(1024, origStmt.length());
-ctx.getAuditEventBuilder().setStmt(origStmt.substring(0,
length));
+auditEventBuilder.setStmt(origStmt.substring(0, length));
} else {
-ctx.getAuditEventBuilder().setStmt(origStmt);
+auditEventBuilder.setStmt(origStmt);
}
}
if (!Env.getCurrentEnv().isMaster()) {
if (ctx.executor.isForwardToMaster()) {
-
ctx.getAuditEventBuilder().setState(ctx.executor.getProxyStatus());
+auditEventBuilder.setState(ctx.executor.getP
(doris) branch branch-2.1 updated: [fix](deps) fix NoSuchMethodError: newInstanceFromKeytab when use kerberos (#37322)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 969f7532d64 [fix](deps) fix NoSuchMethodError: newInstanceFromKeytab
when use kerberos (#37322)
969f7532d64 is described below
commit 969f7532d64b8bb2f679a8f66facd98cffb400d7
Author: Yulei-Yang
AuthorDate: Mon Jul 8 10:19:02 2024 +0800
[fix](deps) fix NoSuchMethodError: newInstanceFromKeytab when use kerberos
(#37322)
---
build.sh | 2 ++
thirdparty/build-thirdparty.sh | 2 ++
2 files changed, 4 insertions(+)
diff --git a/build.sh b/build.sh
index efa808652b6..6cb08cdefe2 100755
--- a/build.sh
+++ b/build.sh
@@ -662,6 +662,8 @@ if [[ "${BUILD_SPARK_DPP}" -eq 1 ]]; then
fi
if [[ "${OUTPUT_BE_BINARY}" -eq 1 ]]; then
+# need remove old version hadoop jars if $DORIS_OUTPUT be used multiple
times, otherwise will cause jar conflict
+rm -rf "${DORIS_OUTPUT}/be/lib/hadoop_hdfs"
install -d "${DORIS_OUTPUT}/be/bin" \
"${DORIS_OUTPUT}/be/conf" \
"${DORIS_OUTPUT}/be/lib" \
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index f17af7571bb..2df09895d08 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -1673,6 +1673,8 @@ build_hadoop_libs() {
echo "THIRDPARTY_INSTALLED=${TP_INSTALL_DIR}" >env.sh
./build.sh
+rm -rf "${TP_INSTALL_DIR}/include/hadoop_hdfs/"
+rm -rf "${TP_INSTALL_DIR}/lib/hadoop_hdfs/"
mkdir -p "${TP_INSTALL_DIR}/include/hadoop_hdfs/"
mkdir -p "${TP_INSTALL_DIR}/lib/hadoop_hdfs/"
cp -r ./hadoop-dist/target/hadoop-libhdfs-3.3.6/*
"${TP_INSTALL_DIR}/lib/hadoop_hdfs/"
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](deps) fix NoSuchMethodError: newInstanceFromKeytab issue when use kerberos (#37323)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 039c245a401 [fix](deps) fix NoSuchMethodError: newInstanceFromKeytab
issue when use kerberos (#37323)
039c245a401 is described below
commit 039c245a4013563bc11dd1d8ae2966c44d8b44e4
Author: Yulei-Yang
AuthorDate: Mon Jul 8 10:19:18 2024 +0800
[fix](deps) fix NoSuchMethodError: newInstanceFromKeytab issue when use
kerberos (#37323)
---
build.sh | 2 ++
thirdparty/build-thirdparty.sh | 2 ++
2 files changed, 4 insertions(+)
diff --git a/build.sh b/build.sh
index 43ae8d8d2e8..f216dfe44b3 100755
--- a/build.sh
+++ b/build.sh
@@ -631,6 +631,8 @@ if [[ "${BUILD_SPARK_DPP}" -eq 1 ]]; then
fi
if [[ "${OUTPUT_BE_BINARY}" -eq 1 ]]; then
+# need remove old version hadoop jars if $DORIS_OUTPUT be used multiple
times, otherwise will cause jar conflict
+rm -rf "${DORIS_OUTPUT}/be/lib/hadoop_hdfs"
install -d "${DORIS_OUTPUT}/be/bin" \
"${DORIS_OUTPUT}/be/conf" \
"${DORIS_OUTPUT}/be/lib" \
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index aeb4e580e00..cd9c655fedc 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -1624,6 +1624,8 @@ build_hadoop_libs() {
echo "THIRDPARTY_INSTALLED=${TP_INSTALL_DIR}" >env.sh
./build.sh
+rm -rf "${TP_INSTALL_DIR}/include/hadoop_hdfs/"
+rm -rf "${TP_INSTALL_DIR}/lib/hadoop_hdfs/"
mkdir -p "${TP_INSTALL_DIR}/include/hadoop_hdfs/"
mkdir -p "${TP_INSTALL_DIR}/lib/hadoop_hdfs/"
cp -r ./hadoop-dist/target/hadoop-libhdfs-3.3.6/*
"${TP_INSTALL_DIR}/lib/hadoop_hdfs/"
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (5dd27802096 -> 7216734f582)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 5dd27802096 [change](index) change enable_create_bitmap_index_as_inverted_index default to true (#36692) add 7216734f582 [fix)(colocate join) fix wrong use of colocate join (#37361) No new revisions were added by this update. Summary of changes: .../properties/ChildOutputPropertyDeriver.java | 2 +- .../properties/ChildrenPropertiesRegulator.java| 2 +- .../LogicalOlapScanToPhysicalOlapScan.java | 12 ++-- .../org/apache/doris/nereids/util/JoinUtils.java | 53 -- .../test_colocate_join_of_column_order.groovy | 82 ++ 5 files changed, 137 insertions(+), 14 deletions(-) create mode 100644 regression-test/suites/correctness_p0/test_colocate_join_of_column_order.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch bulkload created (now 9eccbdbef3)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch bulkload in repository https://gitbox.apache.org/repos/asf/doris.git at 9eccbdbef3 [typo](docs) fix fqdn doc error (#20171) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](Nereids) fixed the limit offset error pick 39316 (#41878)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new e62e47700df [fix](Nereids) fixed the limit offset error pick 39316
(#41878)
e62e47700df is described below
commit e62e47700dfd929051b27d06b98dde9a6b7753d7
Author: lw112 <131352377+felixw...@users.noreply.github.com>
AuthorDate: Wed Oct 16 10:38:05 2024 +0800
[fix](Nereids) fixed the limit offset error pick 39316 (#41878)
---
.../glue/translator/PhysicalPlanTranslator.java| 33 ++---
.../processor/post/AddOffsetIntoDistribute.java| 42 --
.../nereids/processor/post/PlanPostProcessors.java | 1 -
.../data/nereids_syntax_p0/test_limit.out | 6
.../suites/nereids_syntax_p0/test_limit.groovy | 36 +++
5 files changed, 71 insertions(+), 47 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index c0d1aeb917f..b9a6c169c20 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -1812,10 +1812,35 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor
physicalLimit, PlanTranslatorContext context) {
PlanFragment inputFragment = physicalLimit.child(0).accept(this,
context);
PlanNode child = inputFragment.getPlanRoot();
-child.setLimit(MergeLimits.mergeLimit(physicalLimit.getLimit(),
physicalLimit.getOffset(), child.getLimit()));
-// TODO: plan node don't support limit
-// child.setOffset(MergeLimits.mergeOffset(physicalLimit.getOffset(),
child.getOffset()));
-updateLegacyPlanIdToPhysicalPlan(child, physicalLimit);
+
+if (physicalLimit.getPhase().isLocal()) {
+child.setLimit(MergeLimits.mergeLimit(physicalLimit.getLimit(),
physicalLimit.getOffset(),
+child.getLimit()));
+} else if (physicalLimit.getPhase().isGlobal()) {
+if (!(child instanceof ExchangeNode)) {
+ExchangeNode exchangeNode = new
ExchangeNode(context.nextPlanNodeId(), child);
+exchangeNode.setLimit(physicalLimit.getLimit());
+exchangeNode.setOffset(physicalLimit.getOffset());
+exchangeNode.setPartitionType(TPartitionType.UNPARTITIONED);
+exchangeNode.setNumInstances(1);
+PlanFragment fragment = new
PlanFragment(context.nextFragmentId(), exchangeNode,
+DataPartition.UNPARTITIONED);
+inputFragment.setDestination(exchangeNode);
+inputFragment.setOutputPartition(DataPartition.UNPARTITIONED);
+DataStreamSink sink = new DataStreamSink(exchangeNode.getId());
+sink.setOutputPartition(DataPartition.UNPARTITIONED);
+inputFragment.setSink(sink);
+context.addPlanFragment(fragment);
+inputFragment = fragment;
+} else {
+ExchangeNode exchangeNode = (ExchangeNode) child;
+
exchangeNode.setLimit(MergeLimits.mergeLimit(physicalLimit.getLimit(),
physicalLimit.getOffset(),
+exchangeNode.getLimit()));
+
exchangeNode.setOffset(MergeLimits.mergeOffset(physicalLimit.getOffset(),
exchangeNode.getOffset()));
+}
+}
+
+updateLegacyPlanIdToPhysicalPlan(inputFragment.getPlanRoot(),
physicalLimit);
return inputFragment;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
deleted file mode 100644
index dc817321298..000
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
+++ /dev/null
@@ -1,42 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See t
(doris) branch master updated (23bd54da4e3 -> f65223fdb3e)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 23bd54da4e3 [Enhancement]use awaitility.await() (#41514) add f65223fdb3e [fix](index compaction) fix fd leak and mem leak while index compaction (#41915) No new revisions were added by this update. Summary of changes: be/src/olap/compaction.cpp| 7 +++ .../olap/rowset/segment_v2/inverted_index_compaction.cpp | 15 +++ be/src/olap/rowset/segment_v2/inverted_index_compaction.h | 4 +++- 3 files changed, 13 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](metrics) fix compaction_used_permits are negative numbers #32440 (#42526)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new bb50ce193f2 [fix](metrics) fix compaction_used_permits are negative
numbers #32440 (#42526)
bb50ce193f2 is described below
commit bb50ce193f2c0974f36067de79cbffc2c1e19369
Author: camby
AuthorDate: Tue Oct 29 10:59:16 2024 +0800
[fix](metrics) fix compaction_used_permits are negative numbers #32440
(#42526)
---
be/src/olap/olap_server.cpp | 18 +++---
be/src/olap/task/engine_storage_migration_task.h | 3 +--
be/src/runtime/fragment_mgr.cpp | 2 +-
be/src/runtime/runtime_state.h | 1 -
4 files changed, 13 insertions(+), 11 deletions(-)
diff --git a/be/src/olap/olap_server.cpp b/be/src/olap/olap_server.cpp
index 3455fb267fd..5b89bdd3b1e 100644
--- a/be/src/olap/olap_server.cpp
+++ b/be/src/olap/olap_server.cpp
@@ -994,14 +994,18 @@ Status
StorageEngine::_submit_compaction_task(TabletSharedPtr tablet,
(compaction_type == CompactionType::CUMULATIVE_COMPACTION)
? _cumu_compaction_thread_pool
: _base_compaction_thread_pool;
-auto st = thread_pool->submit_func(
-[tablet, compaction = std::move(compaction), compaction_type,
permits, this]() {
-tablet->execute_compaction(*compaction);
-_permit_limiter.release(permits);
-_pop_tablet_from_submitted_compaction(tablet,
compaction_type);
-});
+auto st = thread_pool->submit_func([tablet, compaction =
std::move(compaction),
+compaction_type, permits, force,
this]() {
+tablet->execute_compaction(*compaction);
+if (!force) {
+_permit_limiter.release(permits);
+}
+_pop_tablet_from_submitted_compaction(tablet, compaction_type);
+});
if (!st.ok()) {
-_permit_limiter.release(permits);
+if (!force) {
+_permit_limiter.release(permits);
+}
_pop_tablet_from_submitted_compaction(tablet, compaction_type);
return Status::InternalError(
"failed to submit compaction task to thread pool, "
diff --git a/be/src/olap/task/engine_storage_migration_task.h
b/be/src/olap/task/engine_storage_migration_task.h
index f25fecbd178..7f0ab051ca6 100644
--- a/be/src/olap/task/engine_storage_migration_task.h
+++ b/be/src/olap/task/engine_storage_migration_task.h
@@ -18,9 +18,8 @@
#ifndef DORIS_BE_SRC_OLAP_TASK_ENGINE_STORAGE_MIGRATION_TASK_H
#define DORIS_BE_SRC_OLAP_TASK_ENGINE_STORAGE_MIGRATION_TASK_H
-#include
-
#include
+#include
#include
#include
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index 303ba449325..f9051610dbf 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -681,7 +681,7 @@ Status FragmentMgr::_get_query_ctx(const Params& params,
TUniqueId query_id, boo
// This may be a first fragment request of the query.
// Create the query fragments context.
query_ctx = QueryContext::create_shared(params.fragment_num_on_host,
_exec_env,
-
params.query_options,params.is_nereids);
+params.query_options,
params.is_nereids);
query_ctx->query_id = query_id;
RETURN_IF_ERROR(DescriptorTbl::create(&(query_ctx->obj_pool),
params.desc_tbl,
&(query_ctx->desc_tbl)));
diff --git a/be/src/runtime/runtime_state.h b/be/src/runtime/runtime_state.h
index d63dfc4d732..970a28d22ea 100644
--- a/be/src/runtime/runtime_state.h
+++ b/be/src/runtime/runtime_state.h
@@ -450,7 +450,6 @@ public:
: 0;
}
-
bool is_nereids() const;
private:
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](crash) be crash because of duplicate __PARTIAL_UPDATE_AUTO_INC_COLUMN__ (#41172)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c2517147886 [fix](crash) be crash because of duplicate
__PARTIAL_UPDATE_AUTO_INC_COLUMN__ (#41172)
c2517147886 is described below
commit c2517147886f9d57fe0694c65a3cae42b6174cb8
Author: camby
AuthorDate: Wed Sep 25 22:29:51 2024 +0800
[fix](crash) be crash because of duplicate
__PARTIAL_UPDATE_AUTO_INC_COLUMN__ (#41172)
---
be/src/common/consts.h | 1 +
be/src/olap/partial_update_info.cpp | 3 ++-
be/src/vec/sink/vtablet_block_convertor.cpp | 9 +++--
3 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/be/src/common/consts.h b/be/src/common/consts.h
index dfee235e37b..2ec9ae12679 100644
--- a/be/src/common/consts.h
+++ b/be/src/common/consts.h
@@ -29,6 +29,7 @@ const std::string BLOCK_TEMP_COLUMN_SCANNER_FILTERED =
"__TEMP__scanner_filtered
const std::string ROWID_COL = "__DORIS_ROWID_COL__";
const std::string ROW_STORE_COL = "__DORIS_ROW_STORE_COL__";
const std::string DYNAMIC_COLUMN_NAME = "__DORIS_DYNAMIC_COL__";
+const std::string PARTIAL_UPDATE_AUTO_INC_COL =
"__PARTIAL_UPDATE_AUTO_INC_COLUMN__";
/// The maximum precision representable by a 4-byte decimal (Decimal4Value)
constexpr int MAX_DECIMAL32_PRECISION = 9;
diff --git a/be/src/olap/partial_update_info.cpp
b/be/src/olap/partial_update_info.cpp
index b8e528e99e1..247353103df 100644
--- a/be/src/olap/partial_update_info.cpp
+++ b/be/src/olap/partial_update_info.cpp
@@ -19,6 +19,7 @@
#include
+#include "common/consts.h"
#include "olap/base_tablet.h"
#include "olap/olap_common.h"
#include "olap/rowset/rowset.h"
@@ -306,7 +307,7 @@ Status PartialUpdateReadPlan::fill_missing_columns(
assert_cast(missing_col.get());
auto_inc_column->insert(
(assert_cast(
-
block->get_by_name("__PARTIAL_UPDATE_AUTO_INC_COLUMN__").column.get()))->get_element(idx));
+
block->get_by_name(BeConsts::PARTIAL_UPDATE_AUTO_INC_COL).column.get()))->get_element(idx));
} else {
// If the control flow reaches this branch, the column
neither has default value
// nor is nullable. It means that the row's delete sign is
marked, and the value
diff --git a/be/src/vec/sink/vtablet_block_convertor.cpp
b/be/src/vec/sink/vtablet_block_convertor.cpp
index 36d9a034cf7..820759af2e4 100644
--- a/be/src/vec/sink/vtablet_block_convertor.cpp
+++ b/be/src/vec/sink/vtablet_block_convertor.cpp
@@ -29,6 +29,7 @@
#include
#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/consts.h"
#include "common/status.h"
#include "runtime/descriptors.h"
#include "runtime/runtime_state.h"
@@ -556,6 +557,10 @@ Status
OlapTableBlockConvertor::_fill_auto_inc_cols(vectorized::Block* block, si
Status
OlapTableBlockConvertor::_partial_update_fill_auto_inc_cols(vectorized::Block*
block,
size_t
rows) {
+// avoid duplicate PARTIAL_UPDATE_AUTO_INC_COL
+if (block->has(BeConsts::PARTIAL_UPDATE_AUTO_INC_COL)) {
+return Status::OK();
+}
auto dst_column = vectorized::ColumnInt64::create();
vectorized::ColumnInt64::Container& dst_values = dst_column->get_data();
size_t null_value_count = rows;
@@ -570,8 +575,8 @@ Status
OlapTableBlockConvertor::_partial_update_fill_auto_inc_cols(vectorized::B
}
block->insert(vectorized::ColumnWithTypeAndName(std::move(dst_column),
std::make_shared>(),
-
"__PARTIAL_UPDATE_AUTO_INC_COLUMN__"));
+
BeConsts::PARTIAL_UPDATE_AUTO_INC_COL));
return Status::OK();
}
-} // namespace doris::vectorized
\ No newline at end of file
+} // namespace doris::vectorized
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: branch-2.0: [fix](mem leak) fe non_heap mem leak while use jdbc catalog #45806 (#47626)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new bef4651fb46 branch-2.0: [fix](mem leak) fe non_heap mem leak while use
jdbc catalog #45806 (#47626)
bef4651fb46 is described below
commit bef4651fb465d945ba93babdef7d1db244c58cb0
Author: lw112 <131352377+felixw...@users.noreply.github.com>
AuthorDate: Mon Feb 10 19:11:47 2025 +0800
branch-2.0: [fix](mem leak) fe non_heap mem leak while use jdbc catalog
#45806 (#47626)
---
.../org/apache/doris/datasource/jdbc/client/JdbcClient.java | 13 ++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
index 455c9bbc4eb..aaae8993fcd 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
@@ -53,6 +53,8 @@ public abstract class JdbcClient {
private static final int HTTP_TIMEOUT_MS = 1;
protected static final int JDBC_DATETIME_SCALE = 6;
+private static final Map classLoaderMap = new
ConcurrentHashMap<>();
+
private String catalog;
protected String dbType;
protected String jdbcUser;
@@ -145,11 +147,16 @@ public abstract class JdbcClient {
}
}
-private void initializeClassLoader(JdbcClientConfig config) {
+private synchronized void initializeClassLoader(JdbcClientConfig config) {
try {
URL[] urls = {new
URL(JdbcResource.getFullDriverUrl(config.getDriverUrl()))};
-ClassLoader parent = getClass().getClassLoader();
-this.classLoader = URLClassLoader.newInstance(urls, parent);
+if (classLoaderMap.containsKey(urls[0])) {
+this.classLoader = classLoaderMap.get(urls[0]);
+} else {
+ClassLoader parent = getClass().getClassLoader();
+this.classLoader = URLClassLoader.newInstance(urls, parent);
+classLoaderMap.put(urls[0], this.classLoader);
+}
} catch (MalformedURLException e) {
throw new RuntimeException("Error loading JDBC driver.", e);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](coldheat) fix missing partition's storage policy in create_table_like stmt (#47145)
This is an automated email from the ASF dual-hosted git repository.
cambyzju pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new fbf3b52385b [fix](coldheat) fix missing partition's storage policy in
create_table_like stmt (#47145)
fbf3b52385b is described below
commit fbf3b52385bab65b42f46c5f956de0ce5265
Author: Yulei-Yang
AuthorDate: Mon Jan 20 10:55:50 2025 +0800
[fix](coldheat) fix missing partition's storage policy in create_table_like
stmt (#47145)
---
.../apache/doris/catalog/ListPartitionInfo.java| 4 ++
.../apache/doris/catalog/RangePartitionInfo.java | 3 ++
.../create_table_use_partition_policy.groovy | 55 +-
3 files changed, 61 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionInfo.java
index 8683005f37e..3c32e336eb2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionInfo.java
@@ -222,6 +222,10 @@ public class ListPartitionInfo extends PartitionInfo {
}
sb.append(")");
+if (!"".equals(getStoragePolicy(entry.getKey( {
+sb.append("(\"storage_policy\" =
\"").append(getStoragePolicy(entry.getKey())).append("\")");
+}
+
if (partitionId != null) {
partitionId.add(entry.getKey());
break;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java
index bea50593c3f..87ac9e20d32 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java
@@ -277,6 +277,9 @@ public class RangePartitionInfo extends PartitionInfo {
sb.append("PARTITION ").append(partitionName).append(" VALUES [");
sb.append(range.lowerEndpoint().toSql());
sb.append(", ").append(range.upperEndpoint().toSql()).append(")");
+if (!"".equals(getStoragePolicy(entry.getKey( {
+sb.append("(\"storage_policy\" =
\"").append(getStoragePolicy(entry.getKey())).append("\")");
+}
if (partitionId != null) {
partitionId.add(entry.getKey());
diff --git
a/regression-test/suites/cold_heat_separation/empty_table_use_policy/create_table_use_partition_policy.groovy
b/regression-test/suites/cold_heat_separation/empty_table_use_policy/create_table_use_partition_policy.groovy
index 7307392147a..3fff811f33b 100644
---
a/regression-test/suites/cold_heat_separation/empty_table_use_policy/create_table_use_partition_policy.groovy
+++
b/regression-test/suites/cold_heat_separation/empty_table_use_policy/create_table_use_partition_policy.groovy
@@ -140,11 +140,64 @@ suite("create_table_use_partition_policy") {
) PARTITION BY RANGE (k1) (
PARTITION p1 VALUES LESS THAN ("2022-01-01 00:00:00.111")
("storage_policy" = "test_create_table_partition_use_policy_1"
,"replication_num"="1"),
PARTITION p2 VALUES LESS THAN ("2022-02-01 00:00:00.111")
("storage_policy" = "test_create_table_partition_use_policy_2"
,"replication_num"="1")
-) DISTRIBUTED BY HASH(k2) BUCKETS 1;
+) DISTRIBUTED BY HASH(k2) BUCKETS 1
+PROPERTIES (
+"replication_num" = "1"
+);
"""
assertEquals(create_table_partition_use_created_policy_2.size(), 1);
+// test create table like
+sql """
+CREATE TABLE create_table_partition_use_created_policy_3 LIKE
create_table_partition_use_created_policy_2;
+"""
+def policy_using_item_count_1 = try_sql """
+SHOW STORAGE POLICY USING for test_create_table_partition_use_policy_1
+"""
+assertEquals(policy_using_item_count_1.size(), 2);
+
+def policy_using_item_count_2 = try_sql """
+SHOW STORAGE POLICY USING for test_create_table_partition_use_policy_2
+"""
+assertEquals(policy_using_item_count_2.size(), 2);
+
+def create_table_partition_use_created_policy_4 = try_sql """
+CREATE TABLE IF NOT EXISTS create_table_partition_use_created_policy_4
+(
+k1 DATE NOT NULL,
+k2 INT,
+V1 VARCHAR(2048) REPLACE
+
(doris) branch master updated (25cd9559652 -> 44cc25494b2)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 25cd9559652 [feat](nereids) add session variable disable_nereids_expression_rules (#44831) add 44cc25494b2 [Improve](nereids) use hash set replace three set in DiscreteValue to improve in predicate performance (#45181) No new revisions were added by this update. Summary of changes: .../rules/expression/rules/RangeInference.java| 2 +- .../doris/nereids/trees/expressions/InPredicate.java | 11 +++ .../nereids/rules/expression/SimplifyRangeTest.java | 19 ++- 3 files changed, 30 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (f97b23cb778 -> 2f1691c7049)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f97b23cb778 [fix](p2) fix the failed of cold_heat_separation_p2 (#48199) add 2f1691c7049 [fix](broker) fix fd always timeout (#48494) No new revisions were added by this update. Summary of changes: .../org/apache/doris/broker/hdfs/ClientContextManager.java | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43004-branch-3.0 updated (eee58c7e6ad -> eb32c29ff50)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch auto-pick-43004-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from eee58c7e6ad [fix](build) Fix Mac compilation error caused by namespace conflict in find_symbols.h (#43004) add c9e3e18afe7 branch-3.0: [Fix](sc) Don't save base tablet's compaction jobs to new tablet's `TabletJobInfoPB` #48960 (#49035) add 5f03c9025f7 branch-3.0: [fix](statistics)Control memory use for sample partition column and key column. #46534 (#48922) add bc1f6b32b94 branch-3.0: [improvement](statistics)Add session variable for partition sample count. #48218 (#49091) add 3d1eb35bbff branch-3.0: [feat](binlog) filter the async mv binlogs #49028 (#49098) add 40805f2e4fc branch-3.0: [fix](nereids)fix the cascadesContext.getMemo()==null #48771 (#49094) add e73c5bde214 branch-3.0: [fix](auth)fix check internal priv when drop db of external catalog (#47876) add 484625df8f9 branch-3.0: [opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782 (#48855) add baff13ef466 branch-3.0: [fix](Nereids) fix substring with only one parameter #48957 (#49029) add cdbb07a57a4 branch-3.0: [fix](Nereids) fix split part with regex not exist in source string #48895 (#48909) add 380d5355a21 branch-3.0: [fix](sql cache) fix prepare statement with sql cache throw NullPointerException #48902 (#48976) add 1e9bea28fd1 branch-3.0: [improve](restore) Link existing rowset files with source rowset id #48435 (#48999) add 58760b1de5c branch-3.0: [feat](storage vault) Add object storage op check when creating resource #48073 (#48880) add cb9c5076ef9 branch-3.0: [improvement](statistics)Improve analyze partition column and key column corner case. #48757 (#49100) add a88ef51f38d [fix](pipeline) Do not schedule finalized task again (#49007) add eb32c29ff50 Merge branch 'branch-3.0' into auto-pick-43004-branch-3.0 No new revisions were added by this update. Summary of changes: be/src/cloud/pb_convert.cpp| 8 + be/src/olap/snapshot_manager.cpp | 8 +- be/src/olap/tablet_meta.cpp| 12 +- be/src/olap/tablet_meta.h | 1 + be/src/pipeline/task_scheduler.cpp | 7 +- be/src/runtime/snapshot_loader.cpp | 890 ++--- be/src/runtime/snapshot_loader.h | 3 + be/src/service/backend_service.cpp | 8 +- be/src/util/stopwatch.hpp | 5 +- cloud/src/meta-service/meta_service_job.cpp| 12 +- .../java/org/apache/doris/analysis/DropDbStmt.java | 10 +- .../org/apache/doris/binlog/BinlogConfigCache.java | 70 +- .../org/apache/doris/binlog/BinlogManager.java | 22 + .../org/apache/doris/catalog/AzureResource.java| 58 +- .../java/org/apache/doris/catalog/S3Resource.java | 84 +- .../doris/common/NereidsSqlCacheManager.java | 19 + .../org/apache/doris/fs/obj/AzureObjStorage.java | 47 +- .../java/org/apache/doris/fs/obj/ObjStorage.java | 4 + .../java/org/apache/doris/fs/obj/S3ObjStorage.java | 88 ++ .../org/apache/doris/nereids/NereidsPlanner.java | 15 +- .../org/apache/doris/nereids/StatementContext.java | 2 +- .../apache/doris/nereids/memo/StructInfoMap.java | 9 +- .../exploration/mv/MaterializedViewUtils.java | 2 +- .../functions/executable/StringArithmetic.java | 10 +- .../trees/plans/commands/ExecuteCommand.java | 12 +- .../java/org/apache/doris/qe/ConnectProcessor.java | 11 + .../java/org/apache/doris/qe/SessionVariable.java | 14 + .../apache/doris/statistics/BaseAnalysisTask.java | 2 +- .../apache/doris/statistics/OlapAnalysisTask.java | 386 ++--- .../doris/statistics/StatisticConstants.java | 4 + .../doris/statistics/util/StatisticsUtil.java | 18 + .../apache/doris/catalog/AzureResourceTest.java| 57 ++ .../org/apache/doris/catalog/S3ResourceTest.java | 35 + .../doris/nereids/memo/StructInfoMapTest.java | 15 +- .../doris/statistics/OlapAnalysisTaskTest.java | 628 ++- gensrc/proto/olap_file.proto | 7 + .../test_insert_table_with_dump_nereids_memo.out} | Bin 125 -> 126 bytes .../doris/regression/action/TestAction.groovy | 2 +- .../vault_p0/conf/regression-conf-custom.groovy| 1 - .../test_ddl_database_external_auth.groovy}| 46 +- .../ccr_syncer_p1/test_backup_restore.groovy | 9 +- ...test_insert_table_with_dump_nereids_memo.groovy | 58 ++ .../cache/parse_sql_from_sql_cache.groovy | 7 +- .../cache/prepare_stmt_with_sql_cache.groovy | 33 +- .../fold_constant_string_arithmatic.groovy | 9 + 45 files changed, 2018 insertions(+), 730 deletions(-) create mode 100644
(doris) branch auto-pick-48494-branch-3.0 updated (9ed6901499c -> 139a78cd1dc)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch auto-pick-48494-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 9ed6901499c [fix](broker) fix fd always timeout (#48494) add b5937f512d0 branch-3.0: [fix](group commit) group commit support generated column #48538 (#48582) add 9b2e203ad11 branch-3.0: [opt](fe) Avoid load `retrying_service_config.json` multi times for class MetaServiceClient #48487 (#48566) add 3b55eb56cef branch-3.0: [fix](jdbc test) fix some jdbc catalog test when cloud mode #48559 (#48584) add b51d3018258 branch-3.0-pick: [Opt](partial update) Add some cases for partial update #48161 (#48544) add 00ea4ad0df2 branch-3.0: [fix](p2) fix the failed of cold_heat_separation_p2 #48199 (#48516) add b1a11192072 branch-3.0: [fix](jdbc catalog) Use factory methods to return mapped types instead 2 #48541 (#48585) add 643698bd01b [fix](beut) fix the unit test for calculating the compaction score (#48660) add a94cb02849d [cherry-pick](branch-30) fix unstable test case (#48141) (#48610) add 987600d5a3b [test](beut) pick ColumnHelper to branch-3.0 (#48692) add a9843ff8f47 [fix](array_avg)pick array_avg core (#48691) add a25e4133540 [fix](regression) fix case bug limit_push_down in branch-3.0 (#48608) add 1a2aff22209 branch-3.0: [Bug](compile) fix compile error in master branch about GCC #42936 (#48665) add efb090be69e [cherry-pick](serde) Fix the behavior of serializing ip/date types nested in complex types (#48685) add efd5439f3df [Fix](test) Fix test_cold_data_compaction_fault_injection (#48708) add 3eaa03d2052 branch-3.0: Revert "[fix](nereids) Add unique id to non foldable expr… (#48703) add 738fb88ee79 branch-3.0: [fix](regression)Fix insert p2 cases. #48636 (#48683) add 49ed7c8f165 [fix](cloud) fix evict_in_advance ut for branch-3.0 (#48730) add 5989c2b0352 [branch-30](ut) fix window_funnel UT failure (#48724) add 2fdc378e1aa [cherry-pick](jsonb) fix invalid jsonb value write into segment file which make be crash (#48731) add 3936408ec88 branch-3.0: [case](auth)fix no such property jdbcUser #48622 (#48633) add cd7e03f59b2 branch-3.0: [fix](index build) Correct inverted index behavior after dynamically adding a column #48389 (#48547) add 68ffec777ad branch-3.0: [fix](build index) Forbid building indexes for ngram BF indexes #47041 (#47146) add d036421157a branch-3.0: [test](stable) fix some unstable hive and mc cases #48697 (#48769) add ac7bc80eeeb branch-3.0 [fix] (inverted index) Fix UTF-8 4-byte truncation issue and add configuration to control correct term writing (#48657) (#48792) add 75103ae89d8 [test](fe) Add session variable enable_reserve_memory for testing (#48806) add 6ddff3dec12 [improve](ipv6) Enhance ipv6 type to accept uint128 strings in netwo… (#48802) add 92ab9be5a54 branch-3.0: [case](auth)grant usage_priv to user when in vault mode #47316 (#48413) add e531acbd585 branch 3.0: [fix](regression) topn-filter unstable case #47797 (#48367) add 48e65ebba6e branch-3.0: [case](mtmv)The impact of validating the base table and m… (#48549) add 77c6b81ef28 branch-3.0: [fix](file-cache) Clear file cache when tablet meta not found in ms #48318 (#48437) add 230774bdf6f branch-3.0: [test](kerberos)Add hdfs(tvf),outfile and export test (#48433) (#48814) add f4aefd058cd branch-3.0: [fix](Nereids) nested window function with order by raise exception #48492 (#48676) add 9fb783187b9 branch-3.0: [fix](regression) fix cache p0 regression tests #48515 (#48561) add 10595e5b46b branch-3.0: [fix](nereids)fix unstable test_correlated_filter_removed case #48520 (#48659) add 9ec20604558 branch-3.0: [fix](nereids) fix create/alter view column name force cast to lower case #48569 (#48600) add 17c7309d548 branch-3.0: [fix](Nereids) divide const folding for decimalv3 return type is not correct #48641 (#48679) add 634f2d5e03d branch-3.0: [fix](Nereids) string constant folding process regex delim by mistake #48783 (#48822) add 9fadab4544f branch-3.0: [enhancement](threadpool) reduce thread pool for arrow flight and spill io threads #48530 (#48555) add 8e77cc94be9 branch-3.0: [fix](cloud) Modify the log level to prevent too many logs #48524 (#48796) add d5c6a0c8cfe branch-3.0: [Chore](client) Do not log in thrift exception when ADDRESS_SANITIZER is defined #48430 (#48453) add 1ac212cdd16 branch-3.0: [fix](memtable) revert #47860 and fix #47610 #48489 (#48637) add 5dfcabf1de4 branch-3.0: [fix](cloud) fix filecache warmup crash due to spurious wakeup #48623 (#48654) add 31342672dc9 branch-3.0: [fix](cloud) speed up file cache initializtion #48687 (#48798) add 79df82ba620 branch-3.0: [fix](func) Fix be core dump caused by mem out of bound #48693 (#48845) add f7ed94
(doris) branch master updated (ca24024ff8d -> 1d2ae9fb884)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ca24024ff8d [Enhancement] (nereids)implement CreateUserCommand in nereids (#47554) add 1d2ae9fb884 [feat](bi) support default_init_catalog user property for external catalog (#49658) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/mysql/MysqlProto.java| 16 .../org/apache/doris/mysql/privilege/Auth.java | 9 + .../mysql/privilege/CommonUserProperties.java | 12 ++ .../apache/doris/mysql/privilege/UserProperty.java | 21 ++ .../doris/mysql/privilege/UserPropertyMgr.java | 9 + .../org/apache/doris/catalog/UserPropertyTest.java | 44 + .../org/apache/doris/mysql/MysqlProtoTest.java | 45 +- .../apache/doris/planner/ResourceTagQueryTest.java | 2 +- 8 files changed, 156 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
