(doris) branch branch-2.0 updated: [Fix](Nereids) fix leading with different be instance number (#36613) (#36661)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new b3f33de9023 [Fix](Nereids) fix leading with different be instance
number (#36613) (#36661)
b3f33de9023 is described below
commit b3f33de9023dedba40383ec626971b86f8a0043f
Author: LiBinfeng <46676950+libinfeng...@users.noreply.github.com>
AuthorDate: Mon Jul 1 15:01:52 2024 +0800
[Fix](Nereids) fix leading with different be instance number (#36613)
(#36661)
cherry-pick: #36613
Problem:
When use different be number to test leading explain shape plan,
physical distribute plan would differ due to different be numbers
Solved:
Disable physical distribute node showing in fix leading cases
---
.../data/nereids_p0/hint/fix_leading.out | 77 --
.../suites/nereids_p0/hint/fix_leading.groovy | 7 +-
2 files changed, 31 insertions(+), 53 deletions(-)
diff --git a/regression-test/data/nereids_p0/hint/fix_leading.out
b/regression-test/data/nereids_p0/hint/fix_leading.out
index 9b75ff8d907..32cd1820944 100644
--- a/regression-test/data/nereids_p0/hint/fix_leading.out
+++ b/regression-test/data/nereids_p0/hint/fix_leading.out
@@ -1,23 +1,18 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !select1 --
PhysicalResultSink
---PhysicalDistribute
-PhysicalProject
---hashJoin[INNER_JOIN](t1.c1 = t3.c3)(t1.c1 = t4.c4)
-NestedLoopJoin[CROSS_JOIN]
---PhysicalOlapScan[t1]
---PhysicalDistribute
-filter((t2.c2 = t2.c2))
---PhysicalOlapScan[t2]
-PhysicalDistribute
---NestedLoopJoin[CROSS_JOIN](t4.c4 = t3.c3)(t3.c3 = t4.c4)
-PhysicalOlapScan[t3]
-PhysicalDistribute
---PhysicalOlapScan[t4]
+--hashJoin[INNER_JOIN](t1.c1 = t3.c3)(t1.c1 = t4.c4)
+NestedLoopJoin[CROSS_JOIN]
+--PhysicalOlapScan[t1]
+--filter((t2.c2 = t2.c2))
+PhysicalOlapScan[t2]
+NestedLoopJoin[CROSS_JOIN](t4.c4 = t3.c3)(t3.c3 = t4.c4)
+--PhysicalOlapScan[t3]
+--PhysicalOlapScan[t4]
Hint log:
Used: leading({ t1 t2 } { t3 t4 } )
-UnUsed:
+UnUsed:
SyntaxError:
-- !select2_1_1 --
@@ -231,50 +226,36 @@ SyntaxError:
PhysicalResultSink
--hashAgg[GLOBAL]
hashAgg[LOCAL]
---PhysicalProject
-NestedLoopJoin[RIGHT_OUTER_JOIN](t3.c3 > 500)
---PhysicalDistribute[DistributionSpecGather]
-PhysicalProject
---NestedLoopJoin[LEFT_OUTER_JOIN](t1.c1 < 200)(t1.c1 > 500)
-PhysicalProject
---PhysicalOlapScan[t1]
-PhysicalDistribute[DistributionSpecReplicated]
---PhysicalProject
-filter((t2.c2 > 500))
---PhysicalOlapScan[t2]
---PhysicalDistribute[DistributionSpecGather]
-PhysicalProject
---PhysicalOlapScan[t3]
+--NestedLoopJoin[RIGHT_OUTER_JOIN](t3.c3 > 500)
+NestedLoopJoin[LEFT_OUTER_JOIN](t1.c1 < 200)(t1.c1 > 500)
+--PhysicalOlapScan[t1]
+--filter((t2.c2 > 500))
+PhysicalOlapScan[t2]
+PhysicalOlapScan[t3]
Hint log:
Used: leading(t1 t2 t3 )
-UnUsed:
+UnUsed:
SyntaxError:
-- !select6_1 --
PhysicalResultSink
--hashAgg[GLOBAL]
-PhysicalDistribute[DistributionSpecGather]
---hashAgg[LOCAL]
-hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t6.c6)) otherCondition=()
---hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t2.c2) and (t1.c1 =
t3.c3) and (t1.c1 = t4.c4) and (t1.c1 = t5.c5)) otherCondition=()
-PhysicalOlapScan[t1]
-PhysicalDistribute[DistributionSpecHash]
---hashJoin[INNER_JOIN] hashCondition=((t2.c2 = t4.c4) and (t2.c2 =
t5.c5) and (t3.c3 = t4.c4) and (t3.c3 = t5.c5)) otherCondition=()
-hashJoin[INNER_JOIN] hashCondition=((t2.c2 = t3.c3))
otherCondition=()
---PhysicalOlapScan[t2]
---PhysicalDistribute[DistributionSpecHash]
-PhysicalOlapScan[t3]
-PhysicalDistribute[DistributionSpecHash]
---hashJoin[INNER_JOIN] hashCondition=((t4.c4 = t5.c5))
otherCondition=()
-PhysicalOlapScan[t4]
-PhysicalDistribute[DistributionSpecHash]
---PhysicalOlapScan[t5]
---PhysicalDistribute[DistributionSpecHash]
-PhysicalOlapScan[t6]
+hashAgg[LOCAL]
+--hashJoin[INNER_JOIN](t1.c1 = t6.c6)
+hashJoin[INNER_JOIN](t1.c1 = t2.c2)(t1.c1 = t3.c3)(t1.c1 =
t4.c4)(t1.c1 = t5.c5)
+--PhysicalOlapScan[t1]
+--NestedLoopJoin[CROSS_JOIN](t3.c3 = t5.c5)(t2.c2 = t4.c4)(t2.c2 =
t5.c5)(t5.c5 = t2.c2)(t4.c4 = t2.c2)(t5.c5 = t3.c3)(t4.c4 = t3.c3)(t3.c3 =
t4.c4)
+NestedLoopJoin[CROSS_JOIN](t3.c3
(doris) branch master updated (127987f307f -> d6892a10a7f)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 127987f307f [opt](arena) lazy memory allocation in arena (#36498) add d6892a10a7f [improvement](jdbc catalog)Rename config from `disable_jdbc_sqlserver_encrypt` to `force_sqlserver_jdbc_encrypt_false` (#37015) No new revisions were added by this update. Summary of changes: fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 6 +++--- fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [branch-2.0][improvement](sqlserver catalog) Can configure whether to use encryption when connecting to SQL Server using the catalog (#36940)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 4c4f0be576c [branch-2.0][improvement](sqlserver catalog) Can configure
whether to use encryption when connecting to SQL Server using the catalog
(#36940)
4c4f0be576c is described below
commit 4c4f0be576c68f0b981866404bd2f7e3ca4a3b0e
Author: zy-kkk
AuthorDate: Mon Jul 1 15:04:32 2024 +0800
[branch-2.0][improvement](sqlserver catalog) Can configure whether to use
encryption when connecting to SQL Server using the catalog (#36940)
pick #36659
pick #37015
---
fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 4
fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java | 3 +++
2 files changed, 7 insertions(+)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 9e401b16588..dcdcc7dd035 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -148,6 +148,10 @@ public class Config extends ConfigBase {
+ "The default is * to allow all, if set to empty, all are
not allowed"})
public static String jdbc_driver_secure_path = "*";
+@ConfField(description = {"强制 SQLServer Jdbc Catalog 加密为 false",
+"Force SQLServer Jdbc Catalog encrypt to false"})
+public static boolean force_sqlserver_jdbc_encrypt_false = false;
+
@ConfField(mutable = true, masterOnly = true, description = {"broker load
时，单个节点上 load 执行计划的默认并行度",
"The default parallelism of the load execution plan on a single
node when the broker load is submitted"})
public static int default_load_parallelism = 1;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
index a5ee2c2b2a8..51d2d9b2a9c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
@@ -360,6 +360,9 @@ public class JdbcResource extends Resource {
newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl,
"reWriteBatchedInserts", "false", "true");
}
if (dbType.equals(SQLSERVER)) {
+if (Config.force_sqlserver_jdbc_encrypt_false) {
+newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl,
"encrypt", "true", "false");
+}
newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl,
"useBulkCopyForBatchInsert", "false", "true");
}
return newJdbcUrl;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Fix](regression) Fix p0 case `test_unique_table_auto_inc_concurrent` (#37048)
This is an automated email from the ASF dual-hosted git repository.
zhangchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new daa579b6e58 [Fix](regression) Fix p0 case
`test_unique_table_auto_inc_concurrent` (#37048)
daa579b6e58 is described below
commit daa579b6e5845054a9290cfc4c6a1eb7610da3cd
Author: bobhan1
AuthorDate: Mon Jul 1 15:11:53 2024 +0800
[Fix](regression) Fix p0 case `test_unique_table_auto_inc_concurrent`
(#37048)
since we can't wait all transaction publish finish for insert statement,
we just check the uniqueness of the generated auto-increment values
---
.../data/data_model_p0/unique/test_unique_auto_inc_concurrent.out | 3 ---
.../suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy | 3 ++-
2 files changed, 2 insertions(+), 4 deletions(-)
diff --git
a/regression-test/data/data_model_p0/unique/test_unique_auto_inc_concurrent.out
b/regression-test/data/data_model_p0/unique/test_unique_auto_inc_concurrent.out
index 03819c9a717..c803ca86189 100644
---
a/regression-test/data/data_model_p0/unique/test_unique_auto_inc_concurrent.out
+++
b/regression-test/data/data_model_p0/unique/test_unique_auto_inc_concurrent.out
@@ -1,10 +1,7 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !sql --
-150150
-- !sql --
-300300
-- !sql --
-450450
diff --git
a/regression-test/suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy
b/regression-test/suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy
index bf6d584b2af..4793fb52343 100644
---
a/regression-test/suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy
+++
b/regression-test/suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy
@@ -46,8 +46,9 @@ suite("test_unique_table_auto_inc_concurrent") {
}
threads.each { thread -> thread.join() }
+sql "sync"
-qt_sql "select count(id), count(distinct id) from ${table1};"
+qt_sql "select id, count(*) from ${table1} group by id having count(*)
> 1;"
}
run_test(15, 1, 10)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (daa579b6e58 -> 5e8aebb8c39)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from daa579b6e58 [Fix](regression) Fix p0 case
`test_unique_table_auto_inc_concurrent` (#37048)
add 5e8aebb8c39 [test](migrate) move 2 cases from p2 to p0 (#37004)
No new revisions were added by this update.
Summary of changes:
.../hive_textfile_array_all_types/create_table.hql | 27 ++
.../hive_textfile_array_all_types/data.tar.gz | Bin 0 -> 625 bytes
.../run.sh | 0
.../hive_textfile_array_delimiter/create_table.hql | 32 ++
.../hive_textfile_array_delimiter/data.tar.gz | Bin 0 -> 690 bytes
.../run.sh | 0
.../hive_textfile_nestedarray/create_table.hql | 16 +
.../hive_textfile_nestedarray/data.tar.gz | Bin 0 -> 280 bytes
.../run.sh | 0
.../multi_catalog/logs1_parquet/create_table.hql | 39 +++
.../data/multi_catalog/logs1_parquet/run.sh| 22 ++
.../multi_catalog/one_partition/create_table.hql | 22 ++
.../data/multi_catalog/one_partition/data.tar.gz | Bin 0 -> 296 bytes
.../{datev2_csv => one_partition}/run.sh | 0
.../test_csv_format_error/create_table.hql | 68
.../test_csv_format_error/data.tar.gz | Bin 0 -> 151583 bytes
.../{datev2_csv => test_csv_format_error}/run.sh | 0
.../test_date_string_partition/create_table.hql| 25 ++
.../test_date_string_partition/data.tar.gz | Bin 0 -> 353 bytes
.../run.sh | 0
.../multi_catalog/two_partition/create_table.hql | 25 ++
.../data/multi_catalog/two_partition/data.tar.gz | Bin 0 -> 375 bytes
.../{datev2_csv => two_partition}/run.sh | 0
.../data/statistics/statistics/create_table.hql| 33 ++
.../scripts/data/statistics/statistics/data.tar.gz | Bin 0 -> 3956 bytes
.../hive/scripts/data/statistics/statistics/run.sh | 13 +
.../data/tpch_1000_parquet/part/create_table.hql | 24 ++
.../scripts/data/tpch_1000_parquet/part/run.sh | 22 ++
.../hive/test_hive_default_partition.out | 174 +++
.../hive/test_hive_to_array.out| 21 ++
.../hive/test_text_garbled_file.out| Bin 296830 -> 593565 bytes
.../hive/test_hive_default_partition.groovy| 17 +-
.../hive/test_hive_statistic.groovy| 344 +
.../hive/test_hive_statistic_timeout.groovy| 23 +-
.../hive/test_hive_to_array.groovy | 17 +-
.../hive/test_text_garbled_file.groovy | 47 +++
.../hive/test_hive_statistic.groovy| 338
.../hive/test_text_garbled_file.groovy | 46 ---
38 files changed, 993 insertions(+), 402 deletions(-)
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_all_types/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_all_types/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{datev2_orc
=> hive_textfile_array_all_types}/run.sh (100%)
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_delimiter/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_delimiter/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{datev2_orc
=> hive_textfile_array_delimiter}/run.sh (100%)
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_nestedarray/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_nestedarray/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{datev2_orc
=> hive_textfile_nestedarray}/run.sh (100%)
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/logs1_parquet/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/logs1_parquet/run.sh
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/one_partition/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/one_partition/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{datev2_csv
=> one_partition}/run.sh (100%)
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_csv_format_error/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_csv_format_error/data.tar.gz
copy
docker/thirdparties/docker-compose/hive
(doris) branch branch-2.0 updated: [pick20][opt](mow) reduce memory usage for mow table compaction (#36865) (#36998)
This is an automated email from the ASF dual-hosted git repository.
zhangchen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new d5a6106fd40 [pick20][opt](mow) reduce memory usage for mow table
compaction (#36865) (#36998)
d5a6106fd40 is described below
commit d5a6106fd402116797eccde1c92fcc9432a0bd8c
Author: camby
AuthorDate: Mon Jul 1 15:32:21 2024 +0800
[pick20][opt](mow) reduce memory usage for mow table compaction (#36865)
(#36998)
cherry-pick https://github.com/apache/doris/pull/36865 to branch-2.0
---
be/src/common/config.cpp | 2 +
be/src/common/config.h | 2 +
be/src/olap/compaction.cpp | 51 +++---
be/src/olap/tablet.cpp | 8 +++-
be/src/olap/utils.h| 2 +
regression-test/pipeline/external/conf/be.conf | 1 +
regression-test/pipeline/p0/conf/be.conf | 1 +
regression-test/pipeline/p1/conf/be.conf | 1 +
8 files changed, 44 insertions(+), 24 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 604535825fb..9e8f226dc0d 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1098,6 +1098,8 @@ DEFINE_mInt64(LZ4_HC_compression_level, "9");
DEFINE_mBool(enable_merge_on_write_correctness_check, "true");
// rowid conversion correctness check when compaction for mow table
DEFINE_mBool(enable_rowid_conversion_correctness_check, "false");
+// missing rows correctness check when compaction for mow table
+DEFINE_mBool(enable_missing_rows_correctness_check, "false");
// When the number of missing versions is more than this value, do not directly
// retry the publish and handle it through async publish.
DEFINE_mInt32(mow_publish_max_discontinuous_version_num, "20");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 6e7f2ff490a..35e3a620288 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1144,6 +1144,8 @@ DECLARE_mInt64(LZ4_HC_compression_level);
DECLARE_mBool(enable_merge_on_write_correctness_check);
// rowid conversion correctness check when compaction for mow table
DECLARE_mBool(enable_rowid_conversion_correctness_check);
+// missing rows correctness check when compaction for mow table
+DECLARE_mBool(enable_missing_rows_correctness_check);
// When the number of missing versions is more than this value, do not directly
// retry the publish and handle it through async publish.
DECLARE_mInt32(mow_publish_max_discontinuous_version_num);
diff --git a/be/src/olap/compaction.cpp b/be/src/olap/compaction.cpp
index 5ca06457366..1e1ad3e2775 100644
--- a/be/src/olap/compaction.cpp
+++ b/be/src/olap/compaction.cpp
@@ -802,8 +802,17 @@ Status Compaction::modify_rowsets(const
Merger::Statistics* stats) {
_tablet->enable_unique_key_merge_on_write()) {
Version version = _tablet->max_version();
DeleteBitmap output_rowset_delete_bitmap(_tablet->tablet_id());
-std::set missed_rows;
-std::map>> location_map;
+std::unique_ptr missed_rows;
+if (config::enable_missing_rows_correctness_check &&
!allow_delete_in_cumu_compaction() &&
+compaction_type() == ReaderType::READER_CUMULATIVE_COMPACTION) {
+missed_rows = std::make_unique();
+LOG(INFO) << "RowLocation Set inited succ for tablet:" <<
_tablet->tablet_id();
+}
+std::unique_ptr>
location_map;
+if (config::enable_rowid_conversion_correctness_check) {
+location_map = std::make_unique>();
+LOG(INFO) << "Location Map inited succ for tablet:" <<
_tablet->tablet_id();
+}
// Convert the delete bitmap of the input rowsets to output rowset.
// New loads are not blocked, so some keys of input rowsets might
// be deleted during the time. We need to deal with delete bitmap
@@ -811,13 +820,12 @@ Status Compaction::modify_rowsets(const
Merger::Statistics* stats) {
// TODO(LiaoXin): check if there are duplicate keys
std::size_t missed_rows_size = 0;
_tablet->calc_compaction_output_rowset_delete_bitmap(
-_input_rowsets, _rowid_conversion, 0, version.second + 1,
&missed_rows,
-&location_map, _tablet->tablet_meta()->delete_bitmap(),
+_input_rowsets, _rowid_conversion, 0, version.second + 1,
missed_rows.get(),
+location_map.get(), _tablet->tablet_meta()->delete_bitmap(),
&output_rowset_delete_bitmap);
-if (!allow_delete_in_cumu_compaction()) {
-missed_rows_size = missed_rows.size();
-if (compaction_type() == ReaderType::READER_CUMULATIVE_COMPACTION
&& stats != nullptr &&
-stats->merged_rows != missed_rows_size &&
+if (missed_rows) {
+missed_rows
(doris) branch master updated: [opt](catalog) add some profile for parquet reader and change meta cache config (#37040)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 346ccae56ec [opt](catalog) add some profile for parquet reader and
change meta cache config (#37040)
346ccae56ec is described below
commit 346ccae56ec009355cd5298e482cd05f7f4214c4
Author: Mingyu Chen
AuthorDate: Mon Jul 1 15:32:31 2024 +0800
[opt](catalog) add some profile for parquet reader and change meta cache
config (#37040)
## Proposed changes
This PR mainly changes:
1. add new BE config `enable_parquet_page_index`
Default is true, if set to false, the parquet reader will not use page
index to filter data.
This is only for debug purpose, in case sometimes the page index
filter wrong data.
2. Add new FE config `max_hive_partition_table_cache_num`
Separator from `max_hive_table_cache_num`. This config is used to set
the max cache number
of `partitionValuesCache`, which is for partition values list of
partitioned hive table.
3. Reduce the default expire time of meta cache from 86400 to 24400.
The hive table partitioned by day, if the expire time of cache is 86400,
it may always fetching the stale cache value after a day pass by.
---
be/src/common/config.cpp | 5 +
be/src/common/config.h | 2 ++
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 23 +-
be/src/vec/exec/format/parquet/vparquet_reader.h | 4
.../main/java/org/apache/doris/common/Config.java | 13
.../doris/datasource/hive/HiveMetaStoreCache.java | 6 +++---
6 files changed, 41 insertions(+), 12 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index c2274fd169b..7166b39dda8 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1313,6 +1313,11 @@ DEFINE_Bool(enable_file_logger, "true");
// The minimum row group size when exporting Parquet files. default 128MB
DEFINE_Int64(min_row_group_size, "134217728");
+// If set to false, the parquet reader will not use page index to filter data.
+// This is only for debug purpose, in case sometimes the page index
+// filter wrong data.
+DEFINE_mBool(enable_parquet_page_index, "true");
+
// clang-format off
#ifdef BE_TEST
// test s3
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 9920b65fe52..dbb5b716b78 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1399,6 +1399,8 @@ DECLARE_Bool(enable_file_logger);
// The minimum row group size when exporting Parquet files.
DECLARE_Int64(min_row_group_size);
+DECLARE_mBool(enable_parquet_page_index);
+
#ifdef BE_TEST
// test s3
DECLARE_String(test_s3_resource);
diff --git a/be/src/vec/exec/format/parquet/vparquet_reader.cpp
b/be/src/vec/exec/format/parquet/vparquet_reader.cpp
index 629f272ef72..25421d80b0e 100644
--- a/be/src/vec/exec/format/parquet/vparquet_reader.cpp
+++ b/be/src/vec/exec/format/parquet/vparquet_reader.cpp
@@ -148,6 +148,10 @@ void ParquetReader::_init_profile() {
ADD_CHILD_COUNTER_WITH_LEVEL(_profile, "FileNum", TUnit::UNIT,
parquet_profile, 1);
_parquet_profile.page_index_filter_time =
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "PageIndexFilterTime",
parquet_profile, 1);
+_parquet_profile.read_page_index_time =
+ADD_CHILD_TIMER_WITH_LEVEL(_profile, "PageIndexReadTime",
parquet_profile, 1);
+_parquet_profile.parse_page_index_time =
+ADD_CHILD_TIMER_WITH_LEVEL(_profile, "PageIndexParseTime",
parquet_profile, 1);
_parquet_profile.row_group_filter_time =
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "RowGroupFilterTime",
parquet_profile, 1);
@@ -747,25 +751,32 @@ Status ParquetReader::_process_page_index(const
tparquet::RowGroup& row_group,
return Status::OK();
}
PageIndex page_index;
-if (!_has_page_index(row_group.columns, page_index)) {
+if (!config::enable_parquet_page_index ||
!_has_page_index(row_group.columns, page_index)) {
read_whole_row_group();
return Status::OK();
}
std::vector col_index_buff(page_index._column_index_size);
size_t bytes_read = 0;
Slice result(col_index_buff.data(), page_index._column_index_size);
-RETURN_IF_ERROR(
-_file_reader->read_at(page_index._column_index_start, result,
&bytes_read, _io_ctx));
+{
+SCOPED_RAW_TIMER(&_statistics.read_page_index_time);
+RETURN_IF_ERROR(_file_reader->read_at(page_index._column_index_start,
result, &bytes_read,
+ _io_ctx));
+}
_column_statistics.read_bytes += bytes_read;
auto& schema_desc = _file_metadata->schema();
std::vector skipped_row_ranges;
std::vect
(doris) branch branch-2.1 updated: [pick21][opt](mow) reduce memory usage for mow table compaction (#36865) (#36968)
This is an automated email from the ASF dual-hosted git repository.
zhangchen pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 798d9d6fc60 [pick21][opt](mow) reduce memory usage for mow table
compaction (#36865) (#36968)
798d9d6fc60 is described below
commit 798d9d6fc60a7cca8bb3f975041fe5064d62b739
Author: camby
AuthorDate: Mon Jul 1 15:33:18 2024 +0800
[pick21][opt](mow) reduce memory usage for mow table compaction (#36865)
(#36968)
cherry-pick https://github.com/apache/doris/pull/36865 to branch-2.1
---
be/src/common/config.cpp | 2 ++
be/src/common/config.h | 2 ++
be/src/olap/compaction.cpp | 49 +++---
be/src/olap/tablet.cpp | 8 +++--
be/src/olap/utils.h| 2 ++
regression-test/pipeline/external/conf/be.conf | 1 +
regression-test/pipeline/p0/conf/be.conf | 1 +
regression-test/pipeline/p1/conf/be.conf | 1 +
8 files changed, 43 insertions(+), 23 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 7a8c63db748..910bf69609e 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1098,6 +1098,8 @@ DEFINE_mInt64(LZ4_HC_compression_level, "9");
DEFINE_mBool(enable_merge_on_write_correctness_check, "true");
// rowid conversion correctness check when compaction for mow table
DEFINE_mBool(enable_rowid_conversion_correctness_check, "false");
+// missing rows correctness check when compaction for mow table
+DEFINE_mBool(enable_missing_rows_correctness_check, "false");
// When the number of missing versions is more than this value, do not directly
// retry the publish and handle it through async publish.
DEFINE_mInt32(mow_publish_max_discontinuous_version_num, "20");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 177bb03e02b..2d0dc128a2a 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1159,6 +1159,8 @@
DECLARE_mInt64(variant_threshold_rows_to_estimate_sparse_column);
DECLARE_mBool(enable_merge_on_write_correctness_check);
// rowid conversion correctness check when compaction for mow table
DECLARE_mBool(enable_rowid_conversion_correctness_check);
+// missing rows correctness check when compaction for mow table
+DECLARE_mBool(enable_missing_rows_correctness_check);
// When the number of missing versions is more than this value, do not directly
// retry the publish and handle it through async publish.
DECLARE_mInt32(mow_publish_max_discontinuous_version_num);
diff --git a/be/src/olap/compaction.cpp b/be/src/olap/compaction.cpp
index 05c4cbcb56a..677681f712f 100644
--- a/be/src/olap/compaction.cpp
+++ b/be/src/olap/compaction.cpp
@@ -895,8 +895,17 @@ Status Compaction::modify_rowsets(const
Merger::Statistics* stats) {
_tablet->tablet_schema()->cluster_key_idxes().empty()) {
Version version = _tablet->max_version();
DeleteBitmap output_rowset_delete_bitmap(_tablet->tablet_id());
-std::set missed_rows;
-std::map>> location_map;
+std::unique_ptr missed_rows;
+if (config::enable_missing_rows_correctness_check &&
!allow_delete_in_cumu_compaction() &&
+compaction_type() == ReaderType::READER_CUMULATIVE_COMPACTION) {
+missed_rows = std::make_unique();
+LOG(INFO) << "RowLocation Set inited succ for tablet:" <<
_tablet->tablet_id();
+}
+std::unique_ptr>
location_map;
+if (config::enable_rowid_conversion_correctness_check) {
+location_map = std::make_unique>();
+LOG(INFO) << "Location Map inited succ for tablet:" <<
_tablet->tablet_id();
+}
// Convert the delete bitmap of the input rowsets to output rowset.
// New loads are not blocked, so some keys of input rowsets might
// be deleted during the time. We need to deal with delete bitmap
@@ -904,13 +913,12 @@ Status Compaction::modify_rowsets(const
Merger::Statistics* stats) {
// TODO(LiaoXin): check if there are duplicate keys
std::size_t missed_rows_size = 0;
_tablet->calc_compaction_output_rowset_delete_bitmap(
-_input_rowsets, _rowid_conversion, 0, version.second + 1,
&missed_rows,
-&location_map, _tablet->tablet_meta()->delete_bitmap(),
+_input_rowsets, _rowid_conversion, 0, version.second + 1,
missed_rows.get(),
+location_map.get(), _tablet->tablet_meta()->delete_bitmap(),
&output_rowset_delete_bitmap);
-if (!allow_delete_in_cumu_compaction()) {
-missed_rows_size = missed_rows.size();
-if (compaction_type() == ReaderType::READER_CUMULATIVE_COMPACTION
&&
-_tablet->tablet_state() == TABLET_RUNNING && stats != nullptr
&&
+
(doris) branch master updated (346ccae56ec -> 55f5fce1f8d)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 346ccae56ec [opt](catalog) add some profile for parquet reader and change meta cache config (#37040) add 55f5fce1f8d [fix](Nereids) simplify window expression should inherit data type (#37061) No new revisions were added by this update. Summary of changes: .../doris/nereids/rules/rewrite/SimplifyWindowExpression.java | 4 +++- .../simplify_window_expression/simplify_window_expression.out | 8 .../simplify_window_expression/simplify_window_expression.groovy | 7 +-- 3 files changed, 16 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-kafka-connector) branch master updated: [feature]Add kafka to doris container test case (#33)
This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-kafka-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 528c901 [feature]Add kafka to doris container test case (#33)
528c901 is described below
commit 528c901202067e0b851a626917906337d61d2270
Author: wudongliang <46414265+donglian...@users.noreply.github.com>
AuthorDate: Mon Jul 1 16:22:47 2024 +0800
[feature]Add kafka to doris container test case (#33)
---
.../workflows/kafka2doris-e2ecase.yaml | 33 ++-
.licenserc.yaml| 1 +
pom.xml| 24 +++
.../connector/e2e/doris/DorisContainerService.java | 29 +++
.../e2e/doris/DorisContainerServiceImpl.java | 188
.../connector/e2e/kafka/KafkaContainerService.java | 37
.../e2e/kafka/KafkaContainerServiceImpl.java | 239 +
.../e2e/sink/AbstractKafka2DorisSink.java | 125 +++
.../stringconverter/AbstractStringE2ESinkTest.java | 62 ++
.../e2e/sink/stringconverter/StringMsgE2ETest.java | 95
.../e2e/string_converter/string_msg_connector.json | 21 ++
.../e2e/string_converter/string_msg_tab.sql| 12 ++
12 files changed, 857 insertions(+), 9 deletions(-)
diff --git a/.licenserc.yaml b/.github/workflows/kafka2doris-e2ecase.yaml
similarity index 61%
copy from .licenserc.yaml
copy to .github/workflows/kafka2doris-e2ecase.yaml
index ca50638..f30f5c1 100644
--- a/.licenserc.yaml
+++ b/.github/workflows/kafka2doris-e2ecase.yaml
@@ -15,15 +15,30 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+#
+---
+name: E2E Test CI
+on:
+ pull_request:
+ push:
-header:
- license:
-spdx-id: Apache-2.0
-copyright-owner: Apache Software Foundation
+jobs:
+ build-extension:
+name: "e2e test case"
+runs-on: ubuntu-latest
+defaults:
+ run:
+shell: bash
+steps:
+ - name: Checkout
+uses: actions/checkout@master
- paths-ignore:
-- 'LICENSE'
-- '.gitignore'
-- 'src/test/resources/decode/avro/**'
+ - name: Setup java
+uses: actions/setup-java@v2
+with:
+ distribution: adopt
+ java-version: '8'
- comment: on-failure
\ No newline at end of file
+ - name: Doris E2E Test
+run: |
+ mvn test -Dtest='org.apache.doris.kafka.connector.e2e.**'
\ No newline at end of file
diff --git a/.licenserc.yaml b/.licenserc.yaml
index ca50638..07a811b 100644
--- a/.licenserc.yaml
+++ b/.licenserc.yaml
@@ -25,5 +25,6 @@ header:
- 'LICENSE'
- '.gitignore'
- 'src/test/resources/decode/avro/**'
+- 'src/test/resources/e2e/**'
comment: on-failure
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
index 29005cb..fd99b7d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -91,6 +91,7 @@
4.5.13
2.3
2.2.0
+1.17.6
@@ -294,6 +295,18 @@
+
+org.testcontainers
+testcontainers
+${testcontainers.version}
+test
+
+
+org.testcontainers
+kafka
+${testcontainers.version}
+test
+
@@ -422,6 +435,17 @@
+
+
+org.apache.maven.plugins
+maven-surefire-plugin
+2.22.2
+
+
+
**/org/apache/doris/kafka/connector/e2e/**
+
+
+
diff --git
a/src/test/java/org/apache/doris/kafka/connector/e2e/doris/DorisContainerService.java
b/src/test/java/org/apache/doris/kafka/connector/e2e/doris/DorisContainerService.java
new file mode 100644
index 000..79ec94d
--- /dev/null
+++
b/src/test/java/org/apache/doris/kafka/connector/e2e/doris/DorisContainerService.java
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing
(doris) branch master updated: [profile](fe)update format of min/hour/sec unit in profile (#37010)
This is an automated email from the ASF dual-hosted git repository.
englefly pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9ca7ffc89e9 [profile](fe)update format of min/hour/sec unit in profile
(#37010)
9ca7ffc89e9 is described below
commit 9ca7ffc89e9c7d6fd86b1d4f314f02661b4022a2
Author: minghong
AuthorDate: Mon Jul 1 16:27:25 2024 +0800
[profile](fe)update format of min/hour/sec unit in profile (#37010)
## Proposed changes
it helps us to find the point in profile which operator consumes long
time
Issue Number: close #xxx
---
.../src/main/java/org/apache/doris/common/util/DebugUtil.java | 6 +++---
.../src/test/java/org/apache/doris/common/util/DebugUtilTest.java | 8
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/DebugUtil.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/DebugUtil.java
index 2a52420a96d..937c74cac66 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/DebugUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/DebugUtil.java
@@ -71,17 +71,17 @@ public class DebugUtil {
boolean hour = false;
boolean minute = false;
if (newValue >= HOUR) {
-builder.append(newValue / HOUR).append("h");
+builder.append(newValue / HOUR).append("hour");
newValue %= HOUR;
hour = true;
}
if (newValue >= MINUTE) {
-builder.append(newValue / MINUTE).append("m");
+builder.append(newValue / MINUTE).append("min");
newValue %= MINUTE;
minute = true;
}
if (!hour && newValue >= SECOND) {
-builder.append(newValue / SECOND).append("s");
+builder.append(newValue / SECOND).append("sec");
newValue %= SECOND;
}
if (!hour && !minute) {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/util/DebugUtilTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/util/DebugUtilTest.java
index 54a3f4c388b..aa599783f18 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/common/util/DebugUtilTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/common/util/DebugUtilTest.java
@@ -50,16 +50,16 @@ public class DebugUtilTest {
@Test
public void testGetPrettyStringMs() {
// 6hour1min
-Assert.assertEquals(DebugUtil.getPrettyStringMs(21660222), "6h1m");
+Assert.assertEquals("6hour1min",
DebugUtil.getPrettyStringMs(21660222));
// 1min222ms
-Assert.assertEquals(DebugUtil.getPrettyStringMs(60222), "1m");
+Assert.assertEquals("1min", DebugUtil.getPrettyStringMs(60222));
// 2s222ms
-Assert.assertEquals(DebugUtil.getPrettyStringMs(), "2s222ms");
+Assert.assertEquals("2sec222ms", DebugUtil.getPrettyStringMs());
// 22ms
-Assert.assertEquals(DebugUtil.getPrettyStringMs(22), "22ms");
+Assert.assertEquals("22ms", DebugUtil.getPrettyStringMs(22));
}
@Test
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (9ca7ffc89e9 -> 9ec572f5381)
This is an automated email from the ASF dual-hosted git repository. englefly pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 9ca7ffc89e9 [profile](fe)update format of min/hour/sec unit in profile (#37010) add 9ec572f5381 [fix](nereids) ColStatsMeta.partitionUpdateRows npe (#37044) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (9ec572f5381 -> dc369cd1309)
This is an automated email from the ASF dual-hosted git repository. englefly pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 9ec572f5381 [fix](nereids) ColStatsMeta.partitionUpdateRows npe (#37044) add dc369cd1309 [feat](nereids) support explain delete from clause (#36782) No new revisions were added by this update. Summary of changes: .../doris/nereids/parser/LogicalPlanBuilder.java | 16 ++- .../trees/plans/commands/DeleteFromCommand.java| 92 +++- .../plans/commands/DeleteFromUsingCommand.java | 88 +++ .../suites/nereids_p0/explain/explain_dml.groovy | 122 + 4 files changed, 236 insertions(+), 82 deletions(-) create mode 100644 regression-test/suites/nereids_p0/explain/explain_dml.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](map)fix upgrade behavior from 1.2 version (#36937)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6f432784833 [fix](map)fix upgrade behavior from 1.2 version (#36937)
6f432784833 is described below
commit 6f4327848335fb75ea0bc0ff2f86f10f9ec215c9
Author: amory
AuthorDate: Mon Jul 1 16:58:41 2024 +0800
[fix](map)fix upgrade behavior from 1.2 version (#36937)
fix core dump when upgrading from 1.2.x to 2.0.x with map datatype column
---
be/src/olap/rowset/segment_v2/column_reader.cpp | 1 +
be/src/olap/rowset/segment_v2/column_writer.cpp | 7 ++-
be/src/olap/tablet_schema.cpp | 6 --
3 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/column_reader.cpp
b/be/src/olap/rowset/segment_v2/column_reader.cpp
index 392917e0d83..d0f2830712d 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/column_reader.cpp
@@ -128,6 +128,7 @@ Status ColumnReader::create_map(const ColumnReaderOptions&
opts, const ColumnMet
const io::FileReaderSPtr& file_reader,
std::unique_ptr* reader) {
// map reader now has 3 sub readers for key, value, offsets(scalar),
null(scala)
+DCHECK(meta.children_columns_size() == 3 || meta.children_columns_size()
== 4);
std::unique_ptr key_reader;
RETURN_IF_ERROR(ColumnReader::create(opts, meta.children_columns(0),
meta.children_columns(0).num_rows(),
file_reader,
diff --git a/be/src/olap/rowset/segment_v2/column_writer.cpp
b/be/src/olap/rowset/segment_v2/column_writer.cpp
index 016a932b276..798cbe08261 100644
--- a/be/src/olap/rowset/segment_v2/column_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/column_writer.cpp
@@ -206,6 +206,11 @@ Status ColumnWriter::create_map_writer(const
ColumnWriterOptions& opts, const Ta
io::FileWriter* file_writer,
std::unique_ptr* writer) {
DCHECK(column->get_subtype_count() == 2);
+if (column->get_subtype_count() < 2) {
+return Status::InternalError(
+"If you upgraded from version 1.2.*, please DROP the MAP
columns and then "
+"ADD the MAP columns back.");
+}
// create key & value writer
std::vector> inner_writer_list;
for (int i = 0; i < 2; ++i) {
@@ -1141,4 +1146,4 @@ Status MapColumnWriter::write_inverted_index() {
return Status::OK();
}
-} // namespace doris::segment_v2
\ No newline at end of file
+} // namespace doris::segment_v2
diff --git a/be/src/olap/tablet_schema.cpp b/be/src/olap/tablet_schema.cpp
index cee6f3e2d2e..ec887f14a91 100644
--- a/be/src/olap/tablet_schema.cpp
+++ b/be/src/olap/tablet_schema.cpp
@@ -553,7 +553,8 @@ void TabletColumn::init_from_pb(const ColumnPB& column) {
CHECK(column.children_columns_size() == 1) << "ARRAY type has more
than 1 children types.";
}
if (_type == FieldType::OLAP_FIELD_TYPE_MAP) {
-CHECK(column.children_columns_size() == 2) << "MAP type has more than
2 children types.";
+DCHECK(column.children_columns_size() == 2) << "MAP type has more than
2 children types.";
+LOG(WARNING) << "MAP type has more than 2 children types.";
}
for (size_t i = 0; i < column.children_columns_size(); i++) {
TabletColumn child_column;
@@ -623,7 +624,8 @@ void TabletColumn::to_schema_pb(ColumnPB* column) const {
CHECK(_sub_columns.size() == 1) << "ARRAY type has more than 1
children types.";
}
if (_type == FieldType::OLAP_FIELD_TYPE_MAP) {
-CHECK(_sub_columns.size() == 2) << "MAP type has more than 2 children
types.";
+DCHECK(_sub_columns.size() == 2) << "MAP type has more than 2 children
types.";
+LOG(WARNING) << "MAP type has more than 2 children types.";
}
for (size_t i = 0; i < _sub_columns.size(); i++) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (6f432784833 -> 707cb40d008)
This is an automated email from the ASF dual-hosted git repository. starocean999 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6f432784833 [fix](map)fix upgrade behavior from 1.2 version (#36937) add 707cb40d008 [feature](nereids)use mtmv to match legacy mv (#33699) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/analysis/Expr.java |2 +- .../apache/doris/catalog/AggregateFunction.java|3 +- .../java/org/apache/doris/catalog/FunctionSet.java |4 + .../doris/nereids/analyzer/UnboundFunction.java| 48 +- .../parser/LogicalPlanBuilderForSyncMv.java| 194 +++ .../apache/doris/nereids/parser/NereidsParser.java |7 + .../org/apache/doris/nereids/rules/RuleSet.java|2 + .../org/apache/doris/nereids/rules/RuleType.java |1 + .../mv/AbstractMaterializedViewAggregateRule.java |3 +- .../mv/AbstractMaterializedViewRule.java | 22 + .../mv/InitMaterializationContextHook.java | 160 +++ .../mv/MaterializedViewOnlyScanRule.java | 34 +- .../exploration/mv/MaterializedViewUtils.java | 67 + .../exploration/mv/SyncMaterializationContext.java | 117 ++ .../exploration/mv/rollup/DirectRollupHandler.java |8 +- .../mv/SelectMaterializedIndexWithAggregate.java | 30 + .../SelectMaterializedIndexWithoutAggregate.java | 15 + .../functions/agg/MultiDistinctCount.java | 10 +- .../functions/agg/NullableAggregateFunction.java | 14 + .../trees/plans/commands/info/AlterViewInfo.java |5 +- .../trees/plans/commands/info/BaseViewInfo.java| 40 - .../trees/plans/commands/info/CreateViewInfo.java |5 +- .../apache/doris/nereids/types/AggStateType.java | 23 +- .../org/apache/doris/nereids/util/PlanUtils.java | 40 + .../java/org/apache/doris/qe/SessionVariable.java | 16 + .../rules/rewrite/mv/SelectMvIndexTest.java|1 + .../rules/rewrite/mv/SelectRollupIndexTest.java|1 + .../planner/MaterializedViewFunctionTest.java |1 + .../mv_p0/agg_have_dup_base/agg_have_dup_base.out |7 +- regression-test/data/mv_p0/k1s2m3/k1s2m3.out | 12 +- .../multi_slot_k1p2ap3ps/multi_slot_k1p2ap3ps.out |3 +- .../multi_slot_multi_mv/multi_slot_multi_mv.out|4 +- .../mv_ignore_predicate/mv_ignore_predicate.out|5 +- .../mv_p0/test_upper_alias/test_upper_alias.out|1 + .../single_table_without_aggregate.out | 116 ++ .../data/nereids_syntax_p0/mv/agg_mv_test.dat | 26 + .../nereids_syntax_p0/mv/aggregate/agg_sync_mv.out | 1501 .../nereids_syntax_p0/mv/newMv/aggHaveDupBase.out |7 +- .../nereids_syntax_p0/mv/newMv/multi_slot4.out |1 + .../data/nereids_syntax_p0/mv/ut/aggOnAggMV3.out |3 +- .../data/nereids_syntax_p0/rollup/agg.out |2 +- regression-test/data/rollup_p0/test_rollup_agg.out |2 +- .../agg_have_dup_base/agg_have_dup_base.groovy | 22 + .../mv_p0/agg_state/test_agg_state_max_by.groovy | 26 + .../suites/mv_p0/case_ignore/case_ignore.groovy| 13 + .../suites/mv_p0/count_star/count_star.groovy | 27 +- .../suites/mv_p0/k1ap2spa/k1ap2spa.groovy | 10 + regression-test/suites/mv_p0/k1s2m3/k1s2m3.groovy | 37 + .../mv_p0/k1s2m3_auto_inc/k1s2m3_auto_inc.groovy |9 + .../multi_agg_with_same_slot.groovy| 25 + .../mv_p0/multi_slot_k123p/multi_slot_k123p.groovy | 13 +- .../multi_slot_k1a2p2ap3p.groovy |9 + .../multi_slot_k1a2p2ap3ps.groovy | 14 + .../multi_slot_k1p2ap3p/multi_slot_k1p2ap3p.groovy |9 + .../multi_slot_k1p2ap3ps.groovy| 10 + .../multi_slot_multi_mv/multi_slot_multi_mv.groovy | 39 + .../mv_ignore_predicate/mv_ignore_predicate.groovy | 37 +- .../mv_p0/mv_percentile/mv_percentile.groovy | 50 +- .../suites/mv_p0/mv_with_view/mv_with_view.groovy | 45 +- .../suites/mv_p0/null_insert/null_insert.groovy| 11 + .../mv_p0/routine_load_hll/routine_load_hll.groovy |9 + .../ssb/multiple_no_where/multiple_no_where.groovy | 62 + .../mv_p0/ssb/multiple_ssb/multiple_ssb.groovy | 66 + .../multiple_ssb_between.groovy| 56 + .../suites/mv_p0/ssb/q_1_1/q_1_1.groovy| 16 + .../suites/mv_p0/ssb/q_2_1/q_2_1.groovy| 15 + .../suites/mv_p0/ssb/q_3_1/q_3_1.groovy| 19 + .../suites/mv_p0/ssb/q_4_1/q_4_1.groovy| 18 +- .../suites/mv_p0/ssb/q_4_1_r1/q_4_1_r1.groovy | 17 + .../suites/mv_p0/sum_count/sum_count.groovy| 34 +- .../mv_p0/sum_divede_count/sum_devide_count.groovy | 24 + .../suites/mv_p0/test_28741/test_28741.groovy |9 + .../test_approx_count_distinct.groovy | 19 + .../suites/mv_p0/test_base/test_base.groovy| 18 +- .../mv_p0/test_casewhen/test_casewh
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [doc](delete) delete support predicates on unique key's value column since 2.0 (#807)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new c514350f85 [doc](delete) delete support predicates on unique key's value column since 2.0 (#807) c514350f85 is described below commit c514350f8574ab6bc0db984982328d6ead8b918b Author: zhannngchen <48427519+zhannngc...@users.noreply.github.com> AuthorDate: Mon Jul 1 17:49:06 2024 +0800 [doc](delete) delete support predicates on unique key's value column since 2.0 (#807) The doc such part is outdated - Co-authored-by: Luzhijing <82810928+luzhij...@users.noreply.github.com> --- docs/data-operate/delete/delete-manual.md | 4 ++-- .../current/data-operate/delete/delete-manual.md | 4 ++-- .../version-2.0/data-operate/delete/delete-manual.md | 4 ++-- .../version-2.1/data-operate/delete/delete-manual.md | 4 ++-- versioned_docs/version-2.0/data-operate/delete/delete-manual.md | 4 ++-- versioned_docs/version-2.1/data-operate/delete/delete-manual.md | 4 ++-- 6 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/data-operate/delete/delete-manual.md b/docs/data-operate/delete/delete-manual.md index c14da40e68..77db76b32b 100644 --- a/docs/data-operate/delete/delete-manual.md +++ b/docs/data-operate/delete/delete-manual.md @@ -59,7 +59,7 @@ DELETE FROM table_name [table_alias] ### Note -- When using the table model AGGREGATE, UNIQUE, you can only specify conditions on the key column. +- When using the table model Aggregate, you can only specify conditions on the key column. - If the selected key column does not exist in a rollup, it cannot be deleted. @@ -299,4 +299,4 @@ mysql show delete from test_db; | test_tbl | p4| 2020-04-15 23:09:53 | k1 GT "80" | FINISHED | +---+---+-+-+--+ 2 rows in set (0.00 sec) -``` \ No newline at end of file +``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/delete/delete-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/delete/delete-manual.md index 6eaf288ab4..145058267c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/delete/delete-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/delete/delete-manual.md @@ -54,7 +54,7 @@ DELETE FROM table_name [table_alias] ### 注意事项 -- 使用表模型 AGGREGATE、UNIQUE 时，只能指定 Key 列上的条件。 +- 使用表模型 Aggregate 时，只能指定 Key 列上的条件。 - 当选定的 Key 列不存在于某个 Rollup 中时，无法进行 Delete。 @@ -286,4 +286,4 @@ mysql show delete from test_db; | test_tbl | p4| 2020-04-15 23:09:53 | k1 GT "80" | FINISHED | +---+---+-+-+--+ 2 rows in set (0.00 sec) -``` \ No newline at end of file +``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/data-operate/delete/delete-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/data-operate/delete/delete-manual.md index 4f296ebd6e..e593a5bc06 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/data-operate/delete/delete-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/data-operate/delete/delete-manual.md @@ -54,7 +54,7 @@ DELETE FROM table_name [table_alias] ### 注意事项 -- 使用表模型 AGGREGATE、UNIQUE 时，只能指定 Key 列上的条件。 +- 使用表模型 Aggregate 时，只能指定 Key 列上的条件。 - 当选定的 Key 列不存在于某个 Rollup 中时，无法进行 Delete。 @@ -286,4 +286,4 @@ mysql show delete from test_db; | test_tbl | p4| 2020-04-15 23:09:53 | k1 GT "80" | FINISHED | +---+---+-+-+--+ 2 rows in set (0.00 sec) -``` \ No newline at end of file +``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/delete/delete-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/delete/delete-manual.md index 6eaf288ab4..145058267c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/delete/delete-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/delete/delete-manual.md @@ -54,7 +54,7 @@ DELETE FROM table_name [table_alias] ### 注意事项 -- 使用表模型 AGGREGATE、UNIQUE 时，只能指定 Key 列上的条件。 +- 使用表模型 Aggregate 时，只能指定 Key 列上的条件。 - 当选定的 Key 列不存在于某个 Rollup 中时，无法进行 Delete。 @@ -286,4 +286,4 @@ mysql show delete from test_db; | test_tbl | p4| 2020-04-15 23:09:53 | k1 GT "80" | FINISHED | +---+---+-+-+--+ 2 rows in set (0.00 sec) -``` \ No newline at end of file +``` diff --git a/versioned_docs/version-2.0/data-operate/delete/delete-manual.md b/versioned_docs/version-2.
(doris) branch luzhijing-patch-2 created (now 93925c68173)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a change to branch luzhijing-patch-2 in repository https://gitbox.apache.org/repos/asf/doris.git at 93925c68173 Update README.md This branch includes the following new commits: new 93925c68173 Update README.md The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/01: Update README.md
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch luzhijing-patch-2 in repository https://gitbox.apache.org/repos/asf/doris.git commit 93925c6817305e68520a5331b8879d5949e3b700 Author: Luzhijing <82810928+luzhij...@users.noreply.github.com> AuthorDate: Mon Jul 1 18:11:56 2024 +0800 Update README.md --- README.md | 19 +-- 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index bf0dc168857..c95ad01fad8 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,7 @@ under the License. # Apache Doris [](https://www.apache.org/licenses/LICENSE-2.0.html) [](https://github.com/apache/doris/releases) +[](https://ossrank.com/p/516) [](https://ci-builds.apache.org/job/Doris/job/doris_daily_enable_vectorized) [](https://github.com/apache/doris) [ @@ -32,7 +33,7 @@ under the License. Apache Doris is an MPP-based real-time data warehouse known for its high query speed. For queries on large datasets, it returns results in sub-seconds. It supports both high-concurrency point queries and high-throughput complex analysis. It can be used for report analysis, ad-hoc queries, unified data warehouse building, and data lake query acceleration. Based on Apache Doris, users can build applications for user behavior analysis, A/B testing platform, log analysis, and e-commerce orde [...] -Please visit our [official download page](https://doris.apache.org/download/) to get the latest release version. +Please visit our 🔗[official download page](https://doris.apache.org/download/) to get the latest release version. The current stable version is the 2.0.x series, and the latest version is the 2.1.x series. For production, it is recommended to use the latest version of the 2.0.x series. And if used for POC or testing, it is recommended to use the latest version of the 2.1.x series. @@ -110,10 +111,14 @@ Apache Doris uses Adaptive Query Execution technology to dynamically adjust the ### 🚅 Query Optimizer -In terms of optimizers, Doris uses a combination of CBO and RBO. RBO supports constant folding, subquery rewriting, predicate pushdown and CBO supports Join Reorder. The Doris CBO is under continuous optimization for more accurate statistical information collection and derivation, and more accurate cost model prediction. +In terms of optimizers, Doris uses a combination of CBO and RBO. RBO supports constant folding, subquery rewriting, predicate pushdown. The Doris CBO is under continuous optimization for more accurate statistical information collection and derivation, and more accurate cost model prediction. +The query optimizer in V2.0 has a richer statistical base and adopts the Cascades framework. It is capable of self-tuning in most query scenarios and supports all 99 SQLs in TPC-DS, so users can expect high performance without any fine-tuning or SQL rewriting. -**Technical Overview**: 🔗[Introduction to Apache Doris](https://doris.apache.org/docs/dev/summary/basic-summary) +The query optimizer in V2.1 comes with enhanced statistics-based inference and enumeration framework. We have upgraded the cost model and expanded the optimization rules to serve the needs of more use cases. + + +**Technical Overview**: 🔗[Introduction to Apache Doris](https://doris.apache.org/docs/get-starting/what-is-apache-doris) ## 🎆 Why choose Apache Doris? @@ -133,7 +138,7 @@ In terms of optimizers, Doris uses a combination of CBO and RBO. RBO supports co **Apache Doris has graduated from Apache incubator successfully and become a Top-Level Project in June 2022**. -Currently, the Apache Doris community has gathered more than 600 contributors from over 200 companies in different industries, and the number of monthly active contributors exceeds 100. +Currently, the Apache Doris community has gathered more than 600 contributors from over 200 companies in different industries, and the number of monthly active contributors exceeds 120. [](https://www.apiseven.com/en/contributor-graph?chart=contributorMonthlyActivity&repo=apache/doris) @@ -156,17 +161,19 @@ Add your company logo at Apache Doris Website: 🔗[Add Your Company](https://gi All Documentation 🔗[
(doris) branch master updated (6a9f003944c -> 98ac57c3af4)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6a9f003944c [improve](json)improve json support empty keys (#36762) add 98ac57c3af4 [fix](stmt) fix show create table consistency (#37074) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](auth)fix mtmv name to resolve conflicts (#36958)
This is an automated email from the ASF dual-hosted git repository.
hellostephen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 22ecb9540b4 [fix](auth)fix mtmv name to resolve conflicts (#36958)
22ecb9540b4 is described below
commit 22ecb9540b4770f54cdd22f84a4ae6b05b58fdd8
Author: zfr95 <87513668+zfr9...@users.noreply.github.com>
AuthorDate: Mon Jul 1 20:26:52 2024 +0800
[fix](auth)fix mtmv name to resolve conflicts (#36958)
## Proposed changes
[fix](auth)fix mtmv name to resolve conflicts
---
.../test_master_slave_consistency_auth.groovy | 49 --
.../suites/auth_p0/test_select_column_auth.groovy | 33 ---
2 files changed, 46 insertions(+), 36 deletions(-)
diff --git
a/regression-test/suites/auth_p0/test_master_slave_consistency_auth.groovy
b/regression-test/suites/auth_p0/test_master_slave_consistency_auth.groovy
index 379ea68f3ce..9e5239f6434 100644
--- a/regression-test/suites/auth_p0/test_master_slave_consistency_auth.groovy
+++ b/regression-test/suites/auth_p0/test_master_slave_consistency_auth.groovy
@@ -34,11 +34,16 @@ suite ("test_follower_consistent_auth","p0,auth") {
String user = 'test_follower_consistent_user'
String pwd = 'C123_567p'
-String dbName = 'test_select_column_auth_db'
-String tableName = 'test_select_column_auth_table'
-String role = 'test_select_column_auth_role'
-String wg = 'test_select_column_auth_wg'
-String rg = 'test_select_column_auth_rg'
+String dbName = 'test_follower_consistent_db'
+String tableName = 'test_follower_consistent_table'
+String role = 'test_follower_consistent_role'
+String wg = 'test_follower_consistent_wg'
+String rg = 'test_follower_consistent_rg'
+String mv_name = 'test_follower_consistent_mv'
+String mtmv_name = 'test_follower_consistent_mtmv'
+String view_name = 'test_follower_consistent_view'
+String rollup_name = 'test_follower_consistent_rollup'
+String catalog_name = 'test_follower_consistent_catalog'
try_sql("DROP role ${role}")
sql """CREATE ROLE ${role}"""
sql """drop WORKLOAD GROUP if exists '${wg}'"""
@@ -76,12 +81,12 @@ suite ("test_follower_consistent_auth","p0,auth") {
);
"""
-sql """create view ${dbName}.v1 as select * from
${dbName}.${tableName};"""
-sql """alter table ${dbName}.${tableName} add rollup
rollup1(username)"""
+sql """create view ${dbName}.${view_name} as select * from
${dbName}.${tableName};"""
+sql """alter table ${dbName}.${tableName} add rollup
${rollup_name}(username)"""
sleep(5 * 1000)
-sql """create materialized view mv1 as select username from
${dbName}.${tableName}"""
+sql """create materialized view ${mv_name} as select username from
${dbName}.${tableName}"""
sleep(5 * 1000)
-sql """CREATE MATERIALIZED VIEW ${dbName}.mtmv1
+sql """CREATE MATERIALIZED VIEW ${dbName}.${mtmv_name}
BUILD IMMEDIATE REFRESH AUTO ON MANUAL
DISTRIBUTED BY RANDOM BUCKETS 1
PROPERTIES ('replication_num' = '1')
@@ -93,13 +98,13 @@ suite ("test_follower_consistent_auth","p0,auth") {
(3, "333");
"""
sql """CREATE USER '${user}' IDENTIFIED BY '${pwd}'"""
-sql """refresh MATERIALIZED VIEW ${dbName}.mtmv1 auto"""
+sql """refresh MATERIALIZED VIEW ${dbName}.${mtmv_name} auto"""
sql """grant select_priv on regression_test to ${user}"""
connect(user=user, password="${pwd}", url=context.config.jdbcUrl) {
try {
-sql "SHOW CATALOG RECYCLE BIN WHERE NAME = 'test'"
+sql "SHOW CATALOG RECYCLE BIN WHERE NAME = '${catalog_name}'"
} catch (Exception e) {
log.info(e.getMessage())
assertTrue(e.getMessage().contains("Admin_priv"))
@@ -107,7 +112,7 @@ suite ("test_follower_consistent_auth","p0,auth") {
}
connect(user=user, password="${pwd}", url=new_jdbc_url) {
try {
-sql "SHOW CATALOG RECYCLE BIN WHERE NAME = 'test'"
+sql "SHOW CATALOG RECYCLE BIN WHERE NAME = '${catalog_name}'"
} catch (Exception e) {
log.info(e.getMessage())
assertTrue(e.getMessage().contains("Admin_priv"))
@@ -157,7 +162,7 @@ suite ("test_follower_consistent_auth","p0,auth") {
connect(user=user, password="${pwd}", url=context.config.jdbcUrl) {
try {
-sql "select username from ${dbName}.v1"
+sql "select username from ${dbName}.${view_name}"
} catch (Exception e) {
log.info(e.getMessage())
assertTrue(e.getMessage().contains("Admin_priv,Selec
(doris) branch master updated (22ecb9540b4 -> 6c03335406f)
This is an automated email from the ASF dual-hosted git repository. hellostephen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 22ecb9540b4 [fix](auth)fix mtmv name to resolve conflicts (#36958) add 6c03335406f [fix](mtmv)Fix mtmv name to resolve conflicts in regression test (#36902) No new revisions were added by this update. Summary of changes: .../mv/dimension_2_join_agg/dimension_join_agg_negative.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (798d9d6fc60 -> 89b4918e885)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from 798d9d6fc60 [pick21][opt](mow) reduce memory usage for mow table compaction (#36865) (#36968) add 89b4918e885 [branch-2.1](stmt) fix show create table consistency #37074 (#37079) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [branch-2.1](function) fix date_format and from_unixtime core when meet long format string (#35883) (#36158)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 72c20d3ccc2 [branch-2.1](function) fix date_format and from_unixtime
core when meet long format string (#35883) (#36158)
72c20d3ccc2 is described below
commit 72c20d3ccc270e301108dadfeef8e36c93f6ca03
Author: zclllyybb
AuthorDate: Mon Jul 1 20:35:31 2024 +0800
[branch-2.1](function) fix date_format and from_unixtime core when meet
long format string (#35883) (#36158)
pick #35883
---
be/src/olap/types.h| 16 +-
.../serde/data_type_datetimev2_serde.cpp | 16 +-
be/src/vec/functions/date_time_transforms.h| 10 ---
be/src/vec/runtime/vdatetime_value.cpp | 35 ++
be/src/vec/runtime/vdatetime_value.h | 17 +--
.../data/datatype_p0/date/test_from_unixtime.out | 6 +++-
.../datetime_functions/test_date_function.out | 3 ++
.../datatype_p0/date/test_from_unixtime.groovy | 1 +
.../datetime_functions/test_date_function.groovy | 1 +
9 files changed, 68 insertions(+), 37 deletions(-)
diff --git a/be/src/olap/types.h b/be/src/olap/types.h
index edc1fdabd7a..d2de658da59 100644
--- a/be/src/olap/types.h
+++ b/be/src/olap/types.h
@@ -1236,11 +1236,11 @@ struct
FieldTypeTraits
CppType tmp = *reinterpret_cast(src);
DateV2Value value =
binary_cast>(tmp);
-string format = "%Y-%m-%d";
-string res;
-res.resize(12);
-res.reserve(12);
-value.to_format_string(format.c_str(), format.size(), res.data());
+std::string format = "%Y-%m-%d";
+std::string res;
+res.resize(12 + SAFE_FORMAT_STRING_MARGIN);
+value.to_format_string_conservative(format.c_str(), format.size(),
res.data(),
+12 + SAFE_FORMAT_STRING_MARGIN);
return res;
}
@@ -1277,9 +1277,9 @@ struct
FieldTypeTraits
binary_cast>(tmp);
string format = "%Y-%m-%d %H:%i:%s.%f";
string res;
-res.resize(30);
-res.reserve(30);
-value.to_format_string(format.c_str(), format.size(), res.data());
+res.resize(30 + SAFE_FORMAT_STRING_MARGIN);
+value.to_format_string_conservative(format.c_str(), format.size(),
res.data(),
+30 + SAFE_FORMAT_STRING_MARGIN);
return res;
}
diff --git a/be/src/vec/data_types/serde/data_type_datetimev2_serde.cpp
b/be/src/vec/data_types/serde/data_type_datetimev2_serde.cpp
index 73e859f985a..ec70b127af9 100644
--- a/be/src/vec/data_types/serde/data_type_datetimev2_serde.cpp
+++ b/be/src/vec/data_types/serde/data_type_datetimev2_serde.cpp
@@ -20,7 +20,6 @@
#include
#include // IWYU pragma: keep
-#include
#include "vec/columns/column_const.h"
#include "vec/io/io_helper.h"
@@ -32,8 +31,7 @@ enum {
DIVISOR_FOR_NANO = 10
};
-namespace doris {
-namespace vectorized {
+namespace doris::vectorized {
static const int64_t timestamp_threshold = -2177481943;
static const int64_t timestamp_diff = 343;
static const int64_t micr_to_nano_second = 1000;
@@ -57,8 +55,9 @@ Status
DataTypeDateTimeV2SerDe::serialize_one_cell_to_json(const IColumn& column
if (options.date_olap_format) {
std::string format = "%Y-%m-%d %H:%i:%s.%f";
-char buf[30];
-val.to_format_string(format.c_str(), format.size(), buf);
+char buf[30 + SAFE_FORMAT_STRING_MARGIN];
+val.to_format_string_conservative(format.c_str(), format.size(), buf,
+ 30 + SAFE_FORMAT_STRING_MARGIN);
std::string s = std::string(buf);
bw.write(s.c_str(), s.length());
} else {
@@ -132,7 +131,7 @@ void
DataTypeDateTimeV2SerDe::read_column_from_arrow(IColumn& column,
auto& col_data = static_cast&>(column).get_data();
int64_t divisor = 1;
if (arrow_array->type()->id() == arrow::Type::TIMESTAMP) {
-auto concrete_array = dynamic_cast(arrow_array);
+const auto* concrete_array = dynamic_cast(arrow_array);
const auto type =
std::static_pointer_cast(arrow_array->type());
switch (type->unit()) {
case arrow::TimeUnit::type::SECOND: {
@@ -176,7 +175,7 @@ template
Status DataTypeDateTimeV2SerDe::_write_column_to_mysql(const IColumn& column,
MysqlRowBuffer& result,
int row_idx, bool
col_const) const {
-auto& data = assert_cast&>(column).get_data();
+const auto& data = assert_cast&>(column).get_data();
const auto col_index = index_check_const(row_idx, col_const);
DateV2Value date_val =
binary_cas
(doris) branch master updated: [fix](function) fix nereids fold constant wrong result of abs (#37065)
This is an automated email from the ASF dual-hosted git repository.
zhangstar333 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f3b93f40e9f [fix](function) fix nereids fold constant wrong result of
abs (#37065)
f3b93f40e9f is described below
commit f3b93f40e9f3ef4d2dca969c27ad7339cd9bc7e5
Author: zclllyybb
AuthorDate: Mon Jul 1 21:19:01 2024 +0800
[fix](function) fix nereids fold constant wrong result of abs (#37065)
## Proposed changes
Issue Number: close #xxx
before:
```sql
mysql [optest]>select abs(cast(-9223372036854775808 as BIGINT));
+---+
| abs(cast(-9223372036854775808 as BIGINT)) |
+---+
| -9223372036854775808 |
+---+
1 row in set (0.00 sec)
```
now:
```sql
mysql [optest]>select abs(cast(-9223372036854775808 as BIGINT));
+---+
| abs(cast(-9223372036854775808 as BIGINT)) |
+---+
| 9223372036854775808 |
+---+
1 row in set (0.01 sec)
```
---
.../functions/executable/ExecutableFunctions.java | 8 +--
.../functions/ExecutableFunctionsTest.java | 64 ++
2 files changed, 68 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/ExecutableFunctions.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/ExecutableFunctions.java
index aad56942f37..2e84542fd04 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/ExecutableFunctions.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/ExecutableFunctions.java
@@ -47,22 +47,22 @@ public class ExecutableFunctions {
*/
@ExecFunction(name = "abs", argTypes = {"TINYINT"}, returnType =
"SMALLINT")
public static Expression abs(TinyIntLiteral literal) {
-return new SmallIntLiteral((byte) Math.abs(literal.getValue()));
+return new SmallIntLiteral((short) Math.abs(literal.getValue()));
}
@ExecFunction(name = "abs", argTypes = {"SMALLINT"}, returnType = "INT")
public static Expression abs(SmallIntLiteral literal) {
-return new IntegerLiteral((short) Math.abs(literal.getValue()));
+return new IntegerLiteral(Math.abs(literal.getValue()));
}
@ExecFunction(name = "abs", argTypes = {"INT"}, returnType = "BIGINT")
public static Expression abs(IntegerLiteral literal) {
-return new BigIntLiteral(Math.abs(literal.getValue()));
+return new BigIntLiteral(Math.abs((long) literal.getValue()));
}
@ExecFunction(name = "abs", argTypes = {"BIGINT"}, returnType = "LARGEINT")
public static Expression abs(BigIntLiteral literal) {
-return new LargeIntLiteral(new
BigInteger(Long.toString(Math.abs(literal.getValue();
+return new
LargeIntLiteral(BigInteger.valueOf(literal.getValue()).abs());
}
@ExecFunction(name = "abs", argTypes = {"LARGEINT"}, returnType =
"LARGEINT")
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/ExecutableFunctionsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/ExecutableFunctionsTest.java
new file mode 100644
index 000..6c2e9f144be
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/ExecutableFunctionsTest.java
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions;
+
+import
org.apache.doris.nereids.trees.expressions.functions.executable.ExecutableFunctions;
+import org.apache.doris.nereids.trees.expressions.literal.BigIntLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.
(doris) branch master updated (f3b93f40e9f -> fc65bc43730)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f3b93f40e9f [fix](function) fix nereids fold constant wrong result of abs (#37065) add fc65bc43730 [opt](split) add max wait time of getting splits (#36843) No new revisions were added by this update. Summary of changes: be/src/pipeline/exec/file_scan_operator.cpp| 3 ++- be/src/vec/exec/scan/split_source_connector.cpp| 2 +- be/src/vec/exec/scan/split_source_connector.h | 15 +++ be/src/vec/exec/scan/vfile_scanner.cpp | 2 -- be/src/vec/exec/scan/vfile_scanner.h | 1 - .../org/apache/doris/datasource/FileQueryScanNode.java | 3 ++- .../java/org/apache/doris/datasource/SplitSource.java | 18 ++ .../main/java/org/apache/doris/qe/SessionVariable.java | 17 + 8 files changed, 39 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (fc65bc43730 -> 27152b28e09)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from fc65bc43730 [opt](split) add max wait time of getting splits (#36843)
add 27152b28e09 [test](tvf) move p2 tvf tests from p2 to p0 (#37081)
No new revisions were added by this update.
Summary of changes:
.../data/external_table_p0/tvf/test_tvf_view.out | 28 ++
.../external_table_p2/tvf/test_tvf_view_p2.out | 28 --
.../tvf/test_tvf_view.groovy} | 12 +-
.../tvf/test_tvf_view_count.groovy}| 10
4 files changed, 39 insertions(+), 39 deletions(-)
create mode 100644 regression-test/data/external_table_p0/tvf/test_tvf_view.out
delete mode 100644
regression-test/data/external_table_p2/tvf/test_tvf_view_p2.out
rename regression-test/suites/{external_table_p2/tvf/test_tvf_view_p2.groovy
=> external_table_p0/tvf/test_tvf_view.groovy} (85%)
rename
regression-test/suites/{external_table_p2/tvf/test_tvf_view_count_p2.groovy =>
external_table_p0/tvf/test_tvf_view_count.groovy} (79%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (72c20d3ccc2 -> e686e85f273)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from 72c20d3ccc2 [branch-2.1](function) fix date_format and from_unixtime core when meet long format string (#35883) (#36158) add e686e85f273 [opt](split) add max wait time of getting splits (#36842) No new revisions were added by this update. Summary of changes: be/src/pipeline/exec/file_scan_operator.cpp| 3 ++- be/src/vec/exec/scan/new_file_scan_node.cpp| 3 ++- be/src/vec/exec/scan/split_source_connector.cpp| 2 +- be/src/vec/exec/scan/split_source_connector.h | 15 +++ be/src/vec/exec/scan/vfile_scanner.cpp | 3 --- be/src/vec/exec/scan/vfile_scanner.h | 1 - .../org/apache/doris/datasource/FileQueryScanNode.java | 3 ++- .../java/org/apache/doris/datasource/SplitSource.java | 18 ++ .../main/java/org/apache/doris/qe/SessionVariable.java | 17 + 9 files changed, 41 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (1a87cfc4cd9 -> a2534a6d42b)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1a87cfc4cd9 [test](migrate) move test_hive_text_complex_type from p2 to p0 (#37007) add a2534a6d42b [fix](parquet) prevent parquet page reader print much warning logs (#37011) No new revisions were added by this update. Summary of changes: be/src/util/thrift_util.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (27152b28e09 -> 1a87cfc4cd9)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 27152b28e09 [test](tvf) move p2 tvf tests from p2 to p0 (#37081)
add 1a87cfc4cd9 [test](migrate) move test_hive_text_complex_type from p2
to p0 (#37007)
No new revisions were added by this update.
Summary of changes:
.../hive_text_complex_type/create_table.hql| 27 +
.../hive_text_complex_type/data.tar.gz | Bin 0 -> 560 bytes
.../run.sh | 0
.../hive_text_complex_type2/create_table.hql | 21 +++
.../hive_text_complex_type2/data.tar.gz| Bin 0 -> 366 bytes
.../run.sh | 0
.../hive_text_complex_type3/create_table.hql | 24
.../hive_text_complex_type3/data.tar.gz| Bin 0 -> 977 bytes
.../run.sh | 0
.../create_table.hql | 33 +++
.../hive_text_complex_type_delimiter/data.tar.gz | Bin 0 -> 568 bytes
.../run.sh | 0
.../create_table.hql | 27 +
.../hive_text_complex_type_delimiter2/data.tar.gz | Bin 0 -> 376 bytes
.../run.sh | 0
.../create_table.hql | 26 +
.../hive_text_complex_type_delimiter3/data.tar.gz | Bin 0 -> 978 bytes
.../run.sh | 0
.../parquet_predicate_table/create_table.hql | 18 ++
.../parquet_predicate_table/data.tar.gz| Bin 0 -> 828 bytes
.../run.sh | 0
.../hive/test_hive_text_complex_type.out | 65 +
.../hive/test_hive_text_complex_type.groovy| 15 +++--
23 files changed, 250 insertions(+), 6 deletions(-)
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{par_fields_in_file_orc
=> hive_text_complex_type}/run.sh (100%)
mode change 100755 => 100644
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type2/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type2/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{par_fields_in_file_orc
=> hive_text_complex_type2}/run.sh (100%)
mode change 100755 => 100644
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type3/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type3/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{par_fields_in_file_orc
=> hive_text_complex_type3}/run.sh (100%)
mode change 100755 => 100644
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type_delimiter/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type_delimiter/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{par_fields_in_file_orc
=> hive_text_complex_type_delimiter}/run.sh (100%)
mode change 100755 => 100644
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type_delimiter2/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type_delimiter2/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{par_fields_in_file_orc
=> hive_text_complex_type_delimiter2}/run.sh (100%)
mode change 100755 => 100644
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type_delimiter3/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_text_complex_type_delimiter3/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{par_fields_in_file_orc
=> hive_text_complex_type_delimiter3}/run.sh (100%)
mode change 100755 => 100644
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_predicate_table/create_table.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_predicate_table/data.tar.gz
copy
docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/{par_fields_in_file_orc
=> parquet_predicate_tab
(doris) branch branch-2.1 updated (e686e85f273 -> 6425ce8a891)
This is an automated email from the ASF dual-hosted git repository. eldenmoon pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from e686e85f273 [opt](split) add max wait time of getting splits (#36842) add 6425ce8a891 [Fix](Prepared Statment) use fixed charset to init StringLiteral (#37084) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/analysis/StringLiteral.java| 5 - .../doris/nereids/trees/expressions/literal/Literal.java | 9 +++-- .../java/org/apache/doris/qe/MysqlConnectProcessor.java | 4 +++- regression-test/data/prepared_stmt_p0/prepared_stmt.out | 9 ++--- .../suites/prepared_stmt_p0/prepared_stmt.groovy | 16 +++- 5 files changed, 35 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](regression) fix txn_insert case (#36892)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 66b54723734 [fix](regression) fix txn_insert case (#36892)
66b54723734 is described below
commit 66b547237342eb3beb656a916632324280bd9d81
Author: meiyi
AuthorDate: Tue Jul 2 09:22:37 2024 +0800
[fix](regression) fix txn_insert case (#36892)
## Proposed changes
The delete predicate on mow table is implementated in insert, so in txn
insert, it is an insert rather than delete operation, the commit order
is changed, so the result is changed.
---
regression-test/data/insert_p0/txn_insert.out | 2 --
regression-test/suites/insert_p0/txn_insert.groovy| 2 +-
.../txn_insert_concurrent_insert_duplicate.groovy | 13 +++--
.../insert_p2/txn_insert_concurrent_insert_ud.groovy | 15 +--
.../insert_p2/txn_insert_concurrent_insert_unique.groovy | 15 +--
.../insert_p2/txn_insert_concurrent_insert_update.groovy | 15 +--
.../suites/insert_p2/txn_insert_with_schema_change.groovy | 1 +
7 files changed, 36 insertions(+), 27 deletions(-)
diff --git a/regression-test/data/insert_p0/txn_insert.out
b/regression-test/data/insert_p0/txn_insert.out
index 257bdcc0311..c1278f7fe2e 100644
--- a/regression-test/data/insert_p0/txn_insert.out
+++ b/regression-test/data/insert_p0/txn_insert.out
@@ -1211,14 +1211,12 @@
5 1 15
-- !select_cu2 --
-1 2 121
2 2 122
3 \N 30
4 4 14
5 \N 15
-- !select_cu3 --
-1 2 111
2 2 112
3 \N 130
4 4 14
diff --git a/regression-test/suites/insert_p0/txn_insert.groovy
b/regression-test/suites/insert_p0/txn_insert.groovy
index 66b7be4e659..c403f4c132c 100644
--- a/regression-test/suites/insert_p0/txn_insert.groovy
+++ b/regression-test/suites/insert_p0/txn_insert.groovy
@@ -739,10 +739,10 @@ suite("txn_insert") {
sql """ delete from ${unique_table}_2 where id <= 1; """
sql """ commit """
-sql """ delete from ${unique_table}_3 where id <= 1; """
sql """ insert into ${unique_table}_3(id, score) select id,
score from ${unique_table}_0; """
sql """ insert into ${unique_table}_3(id, score) select id,
score from ${unique_table}_1; """
sql """ update ${unique_table}_3 set score = score + 100 where
id in (select id from ${unique_table}_0); """
+sql """ delete from ${unique_table}_3 where id <= 1; """
} catch (Throwable e) {
logger.warn("column update failed", e)
assertTrue(false)
diff --git
a/regression-test/suites/insert_p2/txn_insert_concurrent_insert_duplicate.groovy
b/regression-test/suites/insert_p2/txn_insert_concurrent_insert_duplicate.groovy
index 493b85064a5..048a07fb817 100644
---
a/regression-test/suites/insert_p2/txn_insert_concurrent_insert_duplicate.groovy
+++
b/regression-test/suites/insert_p2/txn_insert_concurrent_insert_duplicate.groovy
@@ -148,14 +148,13 @@ suite("txn_insert_concurrent_insert_duplicate") {
CompletableFuture.allOf(futuresArray).get(30, TimeUnit.MINUTES)
sql """ sync """
-logger.info("errors: " + errors)
+logger.info("error num: " + errors.size() + ", errors: " + errors)
def result = sql """ select count() from ${tableName}_0 """
-logger.info("result: ${result}, expected: ${6001215 * threadNum}")
-assertTrue(result[0][0] >= 6001215 * threadNum)
-result = sql """ select count() from ${tableName}_1 """
-logger.info("result: ${result}")
-assertEquals(2999666 * threadNum, result[0][0])
+logger.info("${tableName}_0 row count: ${result}, expected: ${6001215 *
threadNum}")
+
+def result2 = sql """ select count() from ${tableName}_1 """
+logger.info("${tableName}_1 row count: ${result2}, expected: ${2999666 *
threadNum}")
def tables = sql """ show tables from $dbName """
logger.info("tables: $tables")
@@ -166,5 +165,7 @@ suite("txn_insert_concurrent_insert_duplicate") {
}
}
+assertTrue(result[0][0] >= 6001215 * threadNum)
+assertEquals(2999666 * threadNum, result2[0][0])
assertEquals(0, errors.size())
}
diff --git
a/regression-test/suites/insert_p2/txn_insert_concurrent_insert_ud.groovy
b/regression-test/suites/insert_p2/txn_insert_concurrent_insert_ud.groovy
index 555f4b3511a..787c556dc3d 100644
--- a/regression-test/suites/insert_p2/txn_insert_concurrent_insert_ud.groovy
+++ b/regression-test/suites/insert_p2/txn_insert_concurrent_insert_ud.groovy
@@ -161,14 +161,15 @@ suite("txn_insert_concurrent_insert_ud") {
CompletableFuture.allOf(futuresArray).get(30, TimeUnit.MINUTES)
sql """ sync """
-logger.info("errors: " + er
(doris) branch master updated: [fix](routine-load) fix auto resume invalid when FE leader change (#37071)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7e3cfb65aa8 [fix](routine-load) fix auto resume invalid when FE leader
change (#37071)
7e3cfb65aa8 is described below
commit 7e3cfb65aa8e13d9bd1840c518b5e2038dd74af2
Author: hui lai <1353307...@qq.com>
AuthorDate: Tue Jul 2 09:33:49 2024 +0800
[fix](routine-load) fix auto resume invalid when FE leader change (#37071)
We meet routine load pause and never be auto resume even if it meet the
conditions.
```
Id: 134305
Name:
lineitem_balance_dup_persistent_weekly_persistent_flow_weekly
CreateTime: 2024-06-27 19:54:13
PauseTime: 2024-06-28 23:02:46
EndTime: NULL
DbName: regression_test_stress_load_long_duration_load
TableName: lineitem_balance_dup_persistent_weekly
IsMultiTable: false
State: PAUSED
DataSourceType: KAFKA
CurrentTaskNum: 0
JobProperties:
{"max_batch_rows":"55","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","column_separator":"','","line_delimiter":"\n","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"10","max_batch_size":"409715200","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"","whereExpr":"*","desired_co
[...]
DataSourceProperties:
{"topic":"test-topic-persistent-weekly-new","currentKafkaPartitions":"","brokerList":"xxx"}
CustomProperties:
{"kafka_default_offsets":"OFFSET_BEGINNING","group.id":"test-consumer-group","client.id":"test-client-id"}
Statistic:
{"receivedBytes":2234836231654,"runningTxns":[],"errorRows":0,"committedTaskNum":1019074,"loadedRows":11693905636,"loadRowsRate":119675,"abortedTaskNum":13556,"errorRowsAfterResumed":0,"totalRows":11693905636,"unselectedRows":0,"receivedBytesRate":22871277,"taskExecuteTimeMs":97713660}
Progress:
{"0":"81666390","1":"81605244","2":"80934894","3":"81531594","4":"81866067","5":"80841194","6":"81229045","7":"80854534","8":"81305844","9":"81384530","10":"81016926","11":"81018762","12":"81586996","13":"81028852","14":"80836728","15":"81536307","16":"81191324","17":"80790892","18":"81518108","19":"80853947","20":"80944134","21":"81567859","22":"80967795","23":"80962887","24":"81444757","25":"81182803","26":"81081053","27":"81374984","28":"81089548","29":"811612
[...]
Lag:
{"0":-1,"1":-1,"2":-1,"3":-1,"4":-1,"5":-1,"6":-1,"7":-1,"8":-1,"9":-1,"10":-1,"11":-1,"12":-1,"13":-1,"14":-1,"15":-1,"16":-1,"17":-1,"18":-1,"19":-1,"20":-1,"21":-1,"22":-1,"23":-1,"24":-1,"25":-1,"26":-1,"27":-1,"28":-1,"29":-1,"30":-1,"31":-1,"32":-1,"33":-1,"34":-1,"35":-1,"36":-1,"37":-1,"38":-1,"39":-1,"40":-1,"41":-1,"42":-1,"43":-1,"44":-1,"45":-1,"46":-1,"47":-1,"48":-1,"49":-1,"50":-1,"51":-1,"52":-1,"53":-1,"54":-1,"55":-1,"56":-1,"57":-1,"58":-1,"59"
[...]
ReasonOfStateChanged:
ErrorLogUrls:
OtherMsg:
User: root
Comment:
```
If routine load pause and FE leader changes at the same time,
pauseReason will be null if FE leader changes, so auto resume logic will
never be triggered:
```
if (jobRoutine.pauseReason != null
&& jobRoutine.pauseReason.getCode() !=
InternalErrorCode.MANUAL_PAUSE_ERR
&& jobRoutine.pauseReason.getCode() !=
InternalErrorCode.TOO_MANY_FAILURE_ROWS_ERR
&& jobRoutine.pauseReason.getCode() !=
InternalErrorCode.CANNOT_RESUME_ERR) {
```
---
.../main/java/org/apache/doris/load/routineload/RoutineLoadJob.java | 5 +
1 file changed, 5 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
index d870922e2d7..51f31837780 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
@@ -242,11 +242,16 @@ public abstract class RoutineLoadJob
@SerializedName("pg")
protected RoutineLoadProgress progress;
+@SerializedName("lrt")
protected long latestResumeTimestamp; // the latest resume time
+@SerializedName("art")
protected long autoResumeCount;
// some other msg which need to show to user;
+@SerializedName("om")
protected String otherMsg = "";
+@SerializedName("pr")
protected ErrorReason pauseReason;
+@SerializedName("cr")
protected ErrorReason cancelReason;
@Se
(doris) branch master updated: [fix](merge-on-write) when full clone failed, duplicate key might occur (#37001)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c972ea55bf8 [fix](merge-on-write) when full clone failed, duplicate
key might occur (#37001)
c972ea55bf8 is described below
commit c972ea55bf8c405df2c2dbf0ec02d8a1717a76ed
Author: zhannngchen <48427519+zhannngc...@users.noreply.github.com>
AuthorDate: Tue Jul 2 09:53:49 2024 +0800
[fix](merge-on-write) when full clone failed, duplicate key might occur
(#37001)
## Proposed changes
Issue Number: close #xxx
introduced by #31268
full clone failure may produce duplicate keys in mow table
the bug would be triggered in the following condition:
1. replica 0 miss version
2. replica 0 try to do full clone from other replicas
3. the full clone failed and the delete bitmap is overrided incorrectly
4. replica 0 try to do incremental clone again and this time the clone
succeed
5. incremental clone can't fix the delete bitmap overrided by previous
failed full clone
6. duplicate key occurred
solution:
for full clone, don't override the delete bitmap, use `merge()` method
instead.
---
be/src/olap/snapshot_manager.cpp | 9 ++
be/src/olap/tablet.cpp | 8 ++
be/src/olap/task/engine_clone_task.cpp | 2 +-
.../test_mow_full_clone_exception.out | 37 ++
.../test_mow_full_clone_exception.groovy | 137 +
5 files changed, 192 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/snapshot_manager.cpp b/be/src/olap/snapshot_manager.cpp
index eab49997599..e2f117b54e2 100644
--- a/be/src/olap/snapshot_manager.cpp
+++ b/be/src/olap/snapshot_manager.cpp
@@ -451,6 +451,15 @@ Status SnapshotManager::_create_snapshot_files(const
TabletSharedPtr& ref_tablet
}
}
+
DBUG_EXECUTE_IF("SnapshotManager.create_snapshot_files.allow_inc_clone", {
+auto tablet_id = dp->param("tablet_id", 0);
+auto is_full_clone = dp->param("is_full_clone", false);
+if (ref_tablet->tablet_id() == tablet_id && is_full_clone) {
+LOG(INFO) << "injected full clone for tabelt: " <<
tablet_id;
+res = Status::InternalError("fault injection error");
+}
+});
+
// be would definitely set it as true no matter has missed version
or not, we could
// just check whether the missed version is empty or not
int64_t version = -1;
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index d49d56ef2d3..8b6ebc2c395 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -404,6 +404,14 @@ Status Tablet::revise_tablet_meta(const
std::vector& to_add,
break; // while (keys_type() == UNIQUE_KEYS &&
enable_unique_key_merge_on_write())
}
+DBUG_EXECUTE_IF("Tablet.revise_tablet_meta_fail", {
+auto ptablet_id = dp->param("tablet_id", 0);
+if (tablet_id() == ptablet_id) {
+LOG(INFO) << "injected revies_tablet_meta failure for tabelt: " <<
ptablet_id;
+calc_bm_status = Status::InternalError("fault injection error");
+}
+});
+
// error handling
if (!calc_bm_status.ok()) {
if (is_incremental_clone) {
diff --git a/be/src/olap/task/engine_clone_task.cpp
b/be/src/olap/task/engine_clone_task.cpp
index 590fdd1b4a7..a05a640dcfe 100644
--- a/be/src/olap/task/engine_clone_task.cpp
+++ b/be/src/olap/task/engine_clone_task.cpp
@@ -875,7 +875,7 @@ Status EngineCloneTask::_finish_full_clone(Tablet* tablet,
}
}
if (tablet->enable_unique_key_merge_on_write()) {
-tablet->tablet_meta()->delete_bitmap() =
cloned_tablet_meta->delete_bitmap();
+
tablet->tablet_meta()->delete_bitmap().merge(cloned_tablet_meta->delete_bitmap());
}
return tablet->revise_tablet_meta(to_add, to_delete, false);
// TODO(plat1ko): write cooldown meta to remote if this replica is
cooldown replica
diff --git
a/regression-test/data/unique_with_mow_p0/test_mow_full_clone_exception.out
b/regression-test/data/unique_with_mow_p0/test_mow_full_clone_exception.out
new file mode 100644
index 000..f11c60b41e1
--- /dev/null
+++ b/regression-test/data/unique_with_mow_p0/test_mow_full_clone_exception.out
@@ -0,0 +1,37 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !sql --
+1 10
+2 200
+3 30
+4 400
+5 500
+6 600
+7 7
+8 8
+9 9
+10 10
+
+-- !sql --
+1 10
+2 200
+3 30
+4 400
+5 500
+6 600
+7 7
+8 8
+9 9
+10 10
+
+-- !sql --
+1 10
+2 200
+3 30
+4 400
+5 500
+6
(doris) branch master updated: [Fix]add set thread num config for wg flush pool (#37028)
This is an automated email from the ASF dual-hosted git repository.
wangbo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9ec4c54c7a3 [Fix]add set thread num config for wg flush pool (#37028)
9ec4c54c7a3 is described below
commit 9ec4c54c7a33a63c2646fb20f2cddea5952ef0e5
Author: wangbo
AuthorDate: Tue Jul 2 09:58:57 2024 +0800
[Fix]add set thread num config for wg flush pool (#37028)
## Proposed changes
calculate workload group's mem table flush pool 's thread num by cpu and
disk num, other wise it may cause mem table flush cost more memory than
not enable workload group.
---
be/src/common/config.cpp | 3 ++
be/src/common/config.h | 4 ++
be/src/olap/delta_writer_v2.cpp | 2 +-
be/src/olap/storage_engine.cpp | 3 +-
be/src/olap/storage_engine.h | 4 ++
be/src/runtime/query_context.cpp | 6 +--
be/src/runtime/query_context.h | 4 +-
be/src/runtime/workload_group/workload_group.cpp | 53
be/src/runtime/workload_group/workload_group.h | 4 +-
be/src/vec/sink/writer/async_result_writer.cpp | 27
10 files changed, 64 insertions(+), 46 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 7166b39dda8..78afc756af8 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -691,6 +691,9 @@ DEFINE_Int32(high_priority_flush_thread_num_per_store, "6");
// max_flush_thread_num_per_cpu * num_cpu)
DEFINE_Int32(max_flush_thread_num_per_cpu, "4");
+DEFINE_mInt32(wg_flush_thread_num_per_store, "6");
+DEFINE_mInt32(wg_flush_thread_num_per_cpu, "4");
+
// config for tablet meta checkpoint
DEFINE_mInt32(tablet_meta_checkpoint_min_new_rowsets_num, "10");
DEFINE_mInt32(tablet_meta_checkpoint_min_interval_secs, "600");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index dbb5b716b78..1a9e3291db5 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -743,6 +743,10 @@ DECLARE_Int32(high_priority_flush_thread_num_per_store);
// max_flush_thread_num_per_cpu * num_cpu)
DECLARE_Int32(max_flush_thread_num_per_cpu);
+// workload group flush pool params
+DECLARE_mInt32(wg_flush_thread_num_per_store);
+DECLARE_mInt32(wg_flush_thread_num_per_cpu);
+
// config for tablet meta checkpoint
DECLARE_mInt32(tablet_meta_checkpoint_min_new_rowsets_num);
DECLARE_mInt32(tablet_meta_checkpoint_min_interval_secs);
diff --git a/be/src/olap/delta_writer_v2.cpp b/be/src/olap/delta_writer_v2.cpp
index 80978280b92..3f2f7bf99fa 100644
--- a/be/src/olap/delta_writer_v2.cpp
+++ b/be/src/olap/delta_writer_v2.cpp
@@ -128,7 +128,7 @@ Status DeltaWriterV2::init() {
RETURN_IF_ERROR(_rowset_writer->init(context));
ThreadPool* wg_thread_pool_ptr = nullptr;
if (_state->get_query_ctx()) {
-wg_thread_pool_ptr =
_state->get_query_ctx()->get_non_pipe_exec_thread_pool();
+wg_thread_pool_ptr =
_state->get_query_ctx()->get_memtable_flush_pool();
}
RETURN_IF_ERROR(_memtable_writer->init(_rowset_writer, _tablet_schema,
_partial_update_info,
wg_thread_pool_ptr,
diff --git a/be/src/olap/storage_engine.cpp b/be/src/olap/storage_engine.cpp
index 5d50bb5f4df..90093241ad2 100644
--- a/be/src/olap/storage_engine.cpp
+++ b/be/src/olap/storage_engine.cpp
@@ -233,8 +233,9 @@ Status StorageEngine::_open() {
auto dirs = get_stores();
RETURN_IF_ERROR(load_data_dirs(dirs));
+_disk_num = dirs.size();
_memtable_flush_executor = std::make_unique();
-_memtable_flush_executor->init(dirs.size());
+_memtable_flush_executor->init(_disk_num);
_calc_delete_bitmap_executor =
std::make_unique();
_calc_delete_bitmap_executor->init();
diff --git a/be/src/olap/storage_engine.h b/be/src/olap/storage_engine.h
index 94cf142a8c1..5ddd888db6d 100644
--- a/be/src/olap/storage_engine.h
+++ b/be/src/olap/storage_engine.h
@@ -131,6 +131,8 @@ public:
int64_t memory_limitation_bytes_per_thread_for_schema_change() const;
+int get_disk_num() { return _disk_num; }
+
protected:
void _evict_querying_rowset();
void _evict_quring_rowset_thread_callback();
@@ -153,6 +155,8 @@ protected:
scoped_refptr _evict_quering_rowset_thread;
int64_t _memory_limitation_bytes_for_schema_change;
+
+int _disk_num {-1};
};
class StorageEngine final : public BaseStorageEngine {
diff --git a/be/src/runtime/query_context.cpp b/be/src/runtime/query_context.cpp
index 18d565dcfef..dd7cf4f55b8 100644
--- a/be/src/runtime/query_context.cpp
+++ b/be/src/runtime/query_context.cpp
@@ -326,9 +326,9 @@ doris::pipeline::TaskScheduler*
QueryContext::get_pipe_exec_scheduler() {
return _exec_env->pipeline_task_
(doris) branch master updated: [fix](nereids) fix This stopwatch is already running (#37095)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d8da2987a0f [fix](nereids) fix This stopwatch is already running
(#37095)
d8da2987a0f is described below
commit d8da2987a0fb3306d67d5a5dea41fa96d424360e
Author: meiyi
AuthorDate: Tue Jul 2 10:01:27 2024 +0800
[fix](nereids) fix This stopwatch is already running (#37095)
If plan throws exception, the stop watch is not stopped, and when
execute next sql will get exception:
2024-07-01 19:05:01,679 WARN (mysql-nio-pool-21|660)
[StmtExecutor.execute():601] Analyze failed. stmt[10651860,
6cea9e53220940ee-b324d18af20d9ab7]
org.apache.doris.common.NereidsException: errCode = 2, detailMessage = This
stopwatch is already running.
at
org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:755)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:584)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:537)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.nereids.trees.plans.commands.ExecuteCommand.run(ExecuteCommand.java:82)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:727)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:584)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:537)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.MysqlConnectProcessor.handleExecute(MysqlConnectProcessor.java:201)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.MysqlConnectProcessor.handleExecute(MysqlConnectProcessor.java:246)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:291)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:342)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52)
~[doris-fe.jar:1.2-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.doris.common.AnalysisException: errCode = 2,
detailMessage = This stopwatch is already running.
... 15 more
Caused by: java.lang.IllegalStateException: This stopwatch is already
running.
at
com.google.common.base.Preconditions.checkState(Preconditions.java:512)
~[guava-32.1.2-jre.jar:?]
at com.google.common.base.Stopwatch.start(Stopwatch.java:166)
~[guava-32.1.2-jre.jar:?]
at
org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:140)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.initPlan(InsertIntoTableCommand
.java:155) ~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.runInternal(InsertIntoTableComm
and.java:226) ~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.run(InsertIntoTableCommand.java
:105) ~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:727)
~[doris-fe.jar:1.2-SNAPSHOT]
... 14 more
---
.../src/main/java/org/apache/doris/nereids/NereidsPlanner.java | 10 +++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
index e0d6f2f7589..97377c7ba26 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
@@ -138,9 +138,13 @@ public class NereidsPlanner extends Planner {
PhysicalProperties requireProperties = buildInitRequireProperties();
statementContext.getStopwatch().start();
-boolean showPlanProcess =
showPlanProcess(queryStmt.getExplainOptions());
-Plan resultPlan = plan(parsedPlan, requireProperties, explainLevel,
showPlanProcess);
-statementContext.getStopwatch().stop();
+Plan resultPlan = null;
+try {
+boolean showPlanProcess =
showPlanProcess(queryStmt.getExplainOptions());
+resultPlan = plan(parsedPlan, requireProperties, explainLevel,
showPlanProcess);
+
svn commit: r70083 - /dev/doris/doris-shade/2.1.1/ /release/doris/doris-shade/2.1.1/
Author: diwu Date: Tue Jul 2 02:08:48 2024 New Revision: 70083 Log: move doris shade 2.1.1 to release Added: release/doris/doris-shade/2.1.1/ - copied from r70082, dev/doris/doris-shade/2.1.1/ Removed: dev/doris/doris-shade/2.1.1/ - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](recycler) Fix DCHECK in ObjectListIterator (#37112)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6807f757bed [fix](recycler) Fix DCHECK in ObjectListIterator (#37112)
6807f757bed is described below
commit 6807f757bed4c36b67512dd36226e3a1bbda327e
Author: plat1ko
AuthorDate: Tue Jul 2 10:13:22 2024 +0800
[fix](recycler) Fix DCHECK in ObjectListIterator (#37112)
Fix DCHECK in ObjectListIterator
---
cloud/src/recycler/azure_obj_client.cpp | 2 +-
cloud/src/recycler/s3_obj_client.cpp| 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/cloud/src/recycler/azure_obj_client.cpp
b/cloud/src/recycler/azure_obj_client.cpp
index 60cd79abb95..62386808d08 100644
--- a/cloud/src/recycler/azure_obj_client.cpp
+++ b/cloud/src/recycler/azure_obj_client.cpp
@@ -83,7 +83,7 @@ public:
try {
auto resp = client_->ListBlobs(req_);
has_more_ = resp.NextPageToken.HasValue();
-DCHECK(!has_more_ || resp.Blobs.empty()) << has_more_ << ' ' <<
resp.Blobs.empty();
+DCHECK(!(has_more_ && resp.Blobs.empty())) << has_more_ << ' ' <<
resp.Blobs.empty();
req_.ContinuationToken = std::move(resp.NextPageToken);
results_.reserve(resp.Blobs.size());
for (auto&& item : std::ranges::reverse_view(resp.Blobs)) {
diff --git a/cloud/src/recycler/s3_obj_client.cpp
b/cloud/src/recycler/s3_obj_client.cpp
index e95a7409475..23e55711096 100644
--- a/cloud/src/recycler/s3_obj_client.cpp
+++ b/cloud/src/recycler/s3_obj_client.cpp
@@ -77,7 +77,7 @@ public:
const_cast(outcome.GetResult().GetNextContinuationToken(;
auto&& content = outcome.GetResult().GetContents();
-DCHECK(!has_more_ || content.empty()) << has_more_ << ' ' <<
content.empty();
+DCHECK(!(has_more_ && content.empty())) << has_more_ << ' ' <<
content.empty();
results_.reserve(content.size());
for (auto&& obj : std::ranges::reverse_view(content)) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [branch-2.1][improvement](sqlserver catalog) Configurable whether to use encrypt when connecting to SQL Server using the catalog (#36971)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 3f382b797ad [branch-2.1][improvement](sqlserver catalog) Configurable
whether to use encrypt when connecting to SQL Server using the catalog (#36971)
3f382b797ad is described below
commit 3f382b797ad0bcf05fdaf0e38d130f803eb04cc3
Author: zy-kkk
AuthorDate: Tue Jul 2 10:14:43 2024 +0800
[branch-2.1][improvement](sqlserver catalog) Configurable whether to use
encrypt when connecting to SQL Server using the catalog (#36971)
pick (#36659)
pick #37015
In previous versions, we used druid as the default JDBC connection pool,
which can use custom decryption to parse the certificate when SQL Server
encryption is turned on. However, in the new version, after changing
HikariCP as the default connection pool, the SQLServer certificate
cannot be parsed, so encryption needs to be turned off for normal use.
Therefore, a parameter is added to decide whether to disable SQLServer
encryption. It is not disabled by default.
---
fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 4
fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java | 3 +++
2 files changed, 7 insertions(+)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index e1b29b7f849..b51ab170f02 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -162,6 +162,10 @@ public class Config extends ConfigBase {
"MySQL Jdbc Catalog mysql does not support pushdown functions"})
public static String[] jdbc_mysql_unsupported_pushdown_functions =
{"date_trunc", "money_format", "negative"};
+@ConfField(description = {"强制 SQLServer Jdbc Catalog 加密为 false",
+"Force SQLServer Jdbc Catalog encrypt to false"})
+public static boolean force_sqlserver_jdbc_encrypt_false = false;
+
@ConfField(mutable = true, masterOnly = true, description = {"broker load
时，单个节点上 load 执行计划的默认并行度",
"The default parallelism of the load execution plan on a single
node when the broker load is submitted"})
public static int default_load_parallelism = 8;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
index e8498d0a2d7..bd14740a6f8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
@@ -353,6 +353,9 @@ public class JdbcResource extends Resource {
newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl,
"reWriteBatchedInserts", "false", "true");
}
if (dbType.equals(SQLSERVER)) {
+if (Config.force_sqlserver_jdbc_encrypt_false) {
+newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl,
"encrypt", "true", "false");
+}
newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl,
"useBulkCopyForBatchInsert", "false", "true");
}
return newJdbcUrl;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](group commit) remove non pipeline code (#37018)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ae7caba54d2 [fix](group commit) remove non pipeline code (#37018)
ae7caba54d2 is described below
commit ae7caba54d2a9e30a0c6e7e157c04e1d7f0d81f6
Author: meiyi
AuthorDate: Tue Jul 2 10:29:02 2024 +0800
[fix](group commit) remove non pipeline code (#37018)
remove non pipilene code in group_commit_mgr
---
be/src/runtime/group_commit_mgr.cpp | 67 +++--
be/src/runtime/group_commit_mgr.h | 4 +--
2 files changed, 21 insertions(+), 50 deletions(-)
diff --git a/be/src/runtime/group_commit_mgr.cpp
b/be/src/runtime/group_commit_mgr.cpp
index 111780c9a42..8a81c942fd3 100644
--- a/be/src/runtime/group_commit_mgr.cpp
+++ b/be/src/runtime/group_commit_mgr.cpp
@@ -266,14 +266,12 @@ Status GroupCommitTable::get_first_block_load_queue(
if (base_schema_version == inner_block_queue->schema_version) {
if (inner_block_queue->add_load_id(load_id,
put_block_dep).ok()) {
load_block_queue = inner_block_queue;
-
return Status::OK();
}
} else {
return Status::DataQualityError(
"schema version not match, maybe a schema change
is in process. "
-"Please "
-"retry this load manually.");
+"Please retry this load manually.");
}
}
}
@@ -284,13 +282,13 @@ Status GroupCommitTable::get_first_block_load_queue(
if (try_to_get_matched_queue().ok()) {
return Status::OK();
}
+create_plan_dep->block();
+_create_plan_deps.push_back(create_plan_dep);
if (!_is_creating_plan_fragment) {
_is_creating_plan_fragment = true;
-create_plan_dep->block();
RETURN_IF_ERROR(_thread_pool->submit_func([&, be_exe_version,
mem_tracker,
dep = create_plan_dep] {
Defer defer {[&, dep = dep]() {
-dep->set_ready();
std::unique_lock l(_lock);
for (auto it : _create_plan_deps) {
it->set_ready();
@@ -303,9 +301,6 @@ Status GroupCommitTable::get_first_block_load_queue(
LOG(WARNING) << "create group commit load error, st=" <<
st.to_string();
}
}));
-} else {
-create_plan_dep->block();
-_create_plan_deps.push_back(create_plan_dep);
}
return try_to_get_matched_queue();
}
@@ -313,16 +308,14 @@ Status GroupCommitTable::get_first_block_load_queue(
Status GroupCommitTable::_create_group_commit_load(int be_exe_version,
std::shared_ptr mem_tracker) {
Status st = Status::OK();
-TStreamLoadPutRequest request;
-UniqueId load_id = UniqueId::gen_uid();
-TUniqueId tload_id;
-bool is_pipeline = true;
TStreamLoadPutResult result;
std::string label;
int64_t txn_id;
TUniqueId instance_id;
{
SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER(mem_tracker);
+UniqueId load_id = UniqueId::gen_uid();
+TUniqueId tload_id;
tload_id.__set_hi(load_id.hi);
tload_id.__set_lo(load_id.lo);
std::regex reg("-");
@@ -330,6 +323,7 @@ Status GroupCommitTable::_create_group_commit_load(int
be_exe_version,
std::stringstream ss;
ss << "insert into doris_internal_table_id(" << _table_id << ") WITH
LABEL " << label
<< " select * from group_commit(\"table_id\"=\"" << _table_id <<
"\")";
+TStreamLoadPutRequest request;
request.__set_load_sql(ss.str());
request.__set_loadId(tload_id);
request.__set_label(label);
@@ -355,51 +349,36 @@ Status GroupCommitTable::_create_group_commit_load(int
be_exe_version,
return st;
}
st = Status::create(result.status);
+if (st.ok() && !result.__isset.pipeline_params) {
+st = Status::InternalError("Non-pipeline is disabled!");
+}
if (!st.ok()) {
LOG(WARNING) << "create group commit load error, st=" <<
st.to_string();
return st;
}
auto schema_version = result.base_schema_version;
-is_pipeline = result.__isset.pipeline_params;
-auto& params = result.params;
auto& pipeline_params = result.pipeline_params;
-if (!is_pipeline) {
-DCHECK(params.fragment.output_sink.olap_table_sink.db_id ==
_db_id);
-txn_id = params.txn_conf.txn_id;
-instance_id = params.params.fragment_instance_id;
-} else {
-DCHECK(pipeline_params.fragment.output_
(doris) branch branch-2.1 updated: [opt](log) Remove unnecessary warning log (#37093)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 011f203d717 [opt](log) Remove unnecessary warning log (#37093)
011f203d717 is described below
commit 011f203d717aab2fb37218b6bb9dedbf97892023
Author: zhiqiang
AuthorDate: Tue Jul 2 10:53:36 2024 +0800
[opt](log) Remove unnecessary warning log (#37093)
When enable_profile = true or report_succeed=true, it is very likely
fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java::reportExecStatus
will print much warning log. It is not necesary.
---
fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java
index a4cd867a31e..8bc189321bd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java
@@ -245,8 +245,7 @@ public final class QeProcessorImpl implements QeProcessor {
} else {
result.setStatus(new TStatus(TStatusCode.RUNTIME_ERROR));
}
-LOG.warn("ReportExecStatus() runtime error, query {} with type {}
does not exist",
-DebugUtil.printId(params.query_id), params.query_type);
+
return result;
}
try {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (606ce442438 -> 06f875726b4)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 606ce442438 [opt](nereids) add join shuffle type in explain shape plan (#36712) add 06f875726b4 [opt](function) opt pad function process the input const column (#36863) No new revisions were added by this update. Summary of changes: be/src/vec/functions/function_string.h | 151 + 1 file changed, 78 insertions(+), 73 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (06f875726b4 -> 6c03ca1641e)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 06f875726b4 [opt](function) opt pad function process the input const column (#36863) add 6c03ca1641e [opt](function) opt ParseUrl function by process the input const column (#36882) No new revisions were added by this update. Summary of changes: be/src/vec/columns/column_const.h | 9 +++ be/src/vec/functions/function_string.h | 134 + 2 files changed, 93 insertions(+), 50 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (6c03ca1641e -> ab8920d6cda)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6c03ca1641e [opt](function) opt ParseUrl function by process the input const column (#36882) add ab8920d6cda [Readme](Community)Update README.md (#37088) No new revisions were added by this update. Summary of changes: README.md | 19 +-- 1 file changed, 13 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (ab8920d6cda -> 0d758137ad9)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ab8920d6cda [Readme](Community)Update README.md (#37088) add 0d758137ad9 [regression](kerberos)fix regression pipeline env when write hosts (#37057) No new revisions were added by this update. Summary of changes: docker/thirdparties/run-thirdparties-docker.sh | 2 ++ 1 file changed, 2 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](null safe equal join) fix coredump if both sides of the conjunct is not nullable #36263 (#37073)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a commit to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.1 by this push: new 6789f5bc80d [fix](null safe equal join) fix coredump if both sides of the conjunct is not nullable #36263 (#37073) 6789f5bc80d is described below commit 6789f5bc80db8742efbf54ac2a5562325914ff91 Author: TengJianPing <18241664+jackte...@users.noreply.github.com> AuthorDate: Tue Jul 2 11:01:55 2024 +0800 [fix](null safe equal join) fix coredump if both sides of the conjunct is not nullable #36263 (#37073) --- be/src/pipeline/exec/hashjoin_build_sink.cpp | 10 +- be/src/pipeline/exec/hashjoin_probe_operator.cpp | 8 +- .../data/query_p0/join/rqg/rqg12257/rqg12257.out | 5 + .../query_p0/join/rqg/rqg12257/rqg12257.groovy | 341 + 4 files changed, 360 insertions(+), 4 deletions(-) diff --git a/be/src/pipeline/exec/hashjoin_build_sink.cpp b/be/src/pipeline/exec/hashjoin_build_sink.cpp index 56480096e3d..ccfd934af99 100644 --- a/be/src/pipeline/exec/hashjoin_build_sink.cpp +++ b/be/src/pipeline/exec/hashjoin_build_sink.cpp @@ -486,8 +486,14 @@ Status HashJoinBuildSinkOperatorX::init(const TPlanNode& tnode, RuntimeState* st const auto vexpr = _build_expr_ctxs.back()->root(); /// null safe equal means null = null is true, the operator in SQL should be: <=>. -const bool is_null_safe_equal = eq_join_conjunct.__isset.opcode && -eq_join_conjunct.opcode == TExprOpcode::EQ_FOR_NULL; +const bool is_null_safe_equal = +eq_join_conjunct.__isset.opcode && +(eq_join_conjunct.opcode == TExprOpcode::EQ_FOR_NULL) && +// For a null safe equal join, FE may generate a plan that +// both sides of the conjuct are not nullable, we just treat it +// as a normal equal join conjunct. +(eq_join_conjunct.right.nodes[0].is_nullable || + eq_join_conjunct.left.nodes[0].is_nullable); const bool should_convert_to_nullable = is_null_safe_equal && !eq_join_conjunct.right.nodes[0].is_nullable && diff --git a/be/src/pipeline/exec/hashjoin_probe_operator.cpp b/be/src/pipeline/exec/hashjoin_probe_operator.cpp index 00cf6a65eb0..002a79f2db2 100644 --- a/be/src/pipeline/exec/hashjoin_probe_operator.cpp +++ b/be/src/pipeline/exec/hashjoin_probe_operator.cpp @@ -540,13 +540,17 @@ Status HashJoinProbeOperatorX::init(const TPlanNode& tnode, RuntimeState* state) RETURN_IF_ERROR(vectorized::VExpr::create_expr_tree(eq_join_conjunct.left, ctx)); _probe_expr_ctxs.push_back(ctx); bool null_aware = eq_join_conjunct.__isset.opcode && - eq_join_conjunct.opcode == TExprOpcode::EQ_FOR_NULL; + eq_join_conjunct.opcode == TExprOpcode::EQ_FOR_NULL && + (eq_join_conjunct.right.nodes[0].is_nullable || + eq_join_conjunct.left.nodes[0].is_nullable); probe_not_ignore_null[conjuncts_index] = null_aware || (_probe_expr_ctxs.back()->root()->is_nullable() && probe_dispose_null); conjuncts_index++; const bool is_null_safe_equal = eq_join_conjunct.__isset.opcode && -eq_join_conjunct.opcode == TExprOpcode::EQ_FOR_NULL; +(eq_join_conjunct.opcode == TExprOpcode::EQ_FOR_NULL) && + (eq_join_conjunct.right.nodes[0].is_nullable || + eq_join_conjunct.left.nodes[0].is_nullable); /// If it's right anti join, /// we should convert the probe to nullable if the build side is nullable. diff --git a/regression-test/data/query_p0/join/rqg/rqg12257/rqg12257.out b/regression-test/data/query_p0/join/rqg/rqg12257/rqg12257.out new file mode 100644 index 000..e82718978b6 --- /dev/null +++ b/regression-test/data/query_p0/join/rqg/rqg12257/rqg12257.out @@ -0,0 +1,5 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !rqg12257 -- + +-- !rqg12257_2 -- + diff --git a/regression-test/suites/query_p0/join/rqg/rqg12257/rqg12257.groovy b/regression-test/suites/query_p0/join/rqg/rqg12257/rqg12257.groovy new file mode 100644 index 000..c04a1f460d4 --- /dev/null +++ b/regression-test/suites/query_p0/join/rqg/rqg12257/rqg12257.groovy @@ -0,0 +1,341 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache Li
(doris-flink-connector) branch master updated: (Fix)[case] Fix unstable cases (#415)
This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new d4af6308 (Fix)[case] Fix unstable cases (#415)
d4af6308 is described below
commit d4af630843b4605f93686517d1f3939f300c40d2
Author: wudi <676366...@qq.com>
AuthorDate: Tue Jul 2 11:06:25 2024 +0800
(Fix)[case] Fix unstable cases (#415)
---
.../doris/flink/sink/copy/BatchStageLoad.java | 5 +++
.../doris/flink/sink/copy/DorisCopyWriter.java | 4 +-
.../java/org/apache/doris/flink/sink/TestUtil.java | 42 +++
.../flink/sink/batch/TestDorisBatchStreamLoad.java | 23 ++-
.../doris/flink/sink/copy/TestDorisCopyWriter.java | 48 +++---
5 files changed, 75 insertions(+), 47 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/copy/BatchStageLoad.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/copy/BatchStageLoad.java
index 0080dee1..be8adcb0 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/copy/BatchStageLoad.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/copy/BatchStageLoad.java
@@ -427,4 +427,9 @@ public class BatchStageLoad implements Serializable {
public void setHttpClientBuilder(HttpClientBuilder httpClientBuilder) {
this.httpClientBuilder = httpClientBuilder;
}
+
+@VisibleForTesting
+public boolean isLoadThreadAlive() {
+return loadThreadAlive;
+}
}
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/copy/DorisCopyWriter.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/copy/DorisCopyWriter.java
index 469b3f2d..b47f2ebe 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/copy/DorisCopyWriter.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/copy/DorisCopyWriter.java
@@ -197,7 +197,7 @@ public class DorisCopyWriter
}
@VisibleForTesting
-public void setBatchStageLoad(BatchStageLoad batchStageLoad) {
-this.batchStageLoad = batchStageLoad;
+public BatchStageLoad getBatchStageLoad() {
+return batchStageLoad;
}
}
diff --git
a/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/TestUtil.java
b/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/TestUtil.java
new file mode 100644
index ..9858e05e
--- /dev/null
+++
b/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/TestUtil.java
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.sink;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.util.function.SupplierWithException;
+
+import java.util.concurrent.TimeoutException;
+
+public class TestUtil {
+
+public static void waitUntilCondition(
+SupplierWithException condition,
+Deadline timeout,
+long retryIntervalMillis,
+String errorMsg)
+throws Exception {
+while (timeout.hasTimeLeft() && !(Boolean) condition.get()) {
+long timeLeft = Math.max(0L, timeout.timeLeft().toMillis());
+Thread.sleep(Math.min(retryIntervalMillis, timeLeft));
+}
+
+if (!timeout.hasTimeLeft()) {
+throw new TimeoutException(errorMsg);
+}
+}
+}
diff --git
a/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/batch/TestDorisBatchStreamLoad.java
b/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/batch/TestDorisBatchStreamLoad.java
index d73ff440..6080db2d 100644
---
a/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/batch/TestDorisBatchStreamLoad.java
+++
b/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/batch/TestDorisBatchStreamLoad.java
@@ -18,13 +18,13 @@
package org.apache.doris.flink.sink.batch;
import org.apache.flink.api.common.time.Deadline;
-import org.apache.flink.util.function.SupplierWithException;
import org.apache.doris.flink.cfg.DorisExecuti
(doris) branch branch-2.1 updated: [chore](multi catalog) Print serde properties when show create hive-external-table (#34966) (#37096)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new a9764b9d39f [chore](multi catalog) Print serde properties when show
create hive-external-table (#34966) (#37096)
a9764b9d39f is described below
commit a9764b9d39ff61d547b7b750d96b0a53be01a726
Author: Mingyu Chen
AuthorDate: Tue Jul 2 11:07:10 2024 +0800
[chore](multi catalog) Print serde properties when show create
hive-external-table (#34966) (#37096)
bp #34966
Co-authored-by: xy720 <22125576+xy...@users.noreply.github.com>
---
.../datasource/hive/HiveMetaStoreClientHelper.java | 7 +++
.../hive/ddl/test_hive_show_create_table.groovy| 54 ++
2 files changed, 61 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClientHelper.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClientHelper.java
index aa3ff1ccee8..2e7693619b8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClientHelper.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClientHelper.java
@@ -756,6 +756,13 @@ public class HiveMetaStoreClientHelper {
output.append("ROW FORMAT SERDE\n")
.append(String.format(" '%s'\n",
descriptor.getSerdeInfo().getSerializationLib()));
}
+if (descriptor.getSerdeInfo().isSetParameters()) {
+output.append("WITH SERDEPROPERTIES (\n")
+
.append(descriptor.getSerdeInfo().getParameters().entrySet().stream()
+.map(entry -> String.format(" '%s' = '%s'",
entry.getKey(), entry.getValue()))
+.collect(Collectors.joining(",\n")))
+.append(")\n");
+}
if (descriptor.isSetInputFormat()) {
output.append("STORED AS INPUTFORMAT\n")
.append(String.format(" '%s'\n",
descriptor.getInputFormat()));
diff --git
a/regression-test/suites/external_table_p0/hive/ddl/test_hive_show_create_table.groovy
b/regression-test/suites/external_table_p0/hive/ddl/test_hive_show_create_table.groovy
new file mode 100644
index 000..0b2af3e95f8
--- /dev/null
+++
b/regression-test/suites/external_table_p0/hive/ddl/test_hive_show_create_table.groovy
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_hive_show_create_table",
"p0,external,hive,external_docker,external_docker_hive") {
+String enabled = context.config.otherConfigs.get("enableHiveTest")
+if (enabled != null && enabled.equalsIgnoreCase("true")) {
+String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+String hms_port = context.config.otherConfigs.get("hive2HmsPort")
+String hdfs_port = context.config.otherConfigs.get("hive2HdfsPort")
+String catalog_name = "test_hive_show_create_table"
+String table_name = "table_with_pars";
+
+sql """drop catalog if exists ${catalog_name};"""
+
+sql """
+create catalog if not exists ${catalog_name} properties (
+'type'='hms',
+'hive.metastore.uris' =
'thrift://${externalEnvIp}:${hms_port}',
+'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}'
+);
+"""
+logger.info("catalog " + catalog_name + " created")
+sql """switch ${catalog_name};"""
+logger.info("switched to catalog " + catalog_name)
+sql """use `default`;"""
+
+def serde = "'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'"
+def serdeFormat = "'serialization.format' = '|'"
+def filedDelim = "'field.delim' = '|'"
+def inputFormat = "'org.apache.hadoop.mapred.TextInputFormat'"
+def outputFormat =
"'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'"
+def create_tbl_res = sql """ show create table table_with_pars """
+logger.info("${create_tbl_res}")
+
(doris-flink-connector) branch master updated: [fix](cdc)fix mongodb sync cause ClassCastException when user specifies the `_id` field manually (#410)
This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new a24898c5 [fix](cdc)fix mongodb sync cause ClassCastException when user
specifies the `_id` field manually (#410)
a24898c5 is described below
commit a24898c5d1952dea50a9e6c1b2d2c41b8bc00b39
Author: North Lin <37775475+qg-...@users.noreply.github.com>
AuthorDate: Tue Jul 2 11:07:40 2024 +0800
[fix](cdc)fix mongodb sync cause ClassCastException when user specifies the
`_id` field manually (#410)
---
.../flink/tools/cdc/mongodb/MongoDBDatabaseSync.java | 5 -
.../mongodb/serializer/MongoJsonDebeziumDataChange.java | 16 ++--
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBDatabaseSync.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBDatabaseSync.java
index 7c0b6706..fe7f33d0 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBDatabaseSync.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBDatabaseSync.java
@@ -133,7 +133,8 @@ public class MongoDBDatabaseSync extends DatabaseSync {
private ArrayList sampleData(MongoCollection
collection, Long sampleNum) {
ArrayList query = new ArrayList<>();
query.add(new Document("$sample", new Document("size", sampleNum)));
-return collection.aggregate(query).into(new ArrayList<>());
+// allowDiskUse to avoid mongo 'Sort exceeded memory limit' error
+return collection.aggregate(query).allowDiskUse(true).into(new
ArrayList<>());
}
private static String buildConnectionString(
@@ -159,6 +160,8 @@ public class MongoDBDatabaseSync extends DatabaseSync {
String username = config.get(MongoDBSourceOptions.USERNAME);
String password = config.get(MongoDBSourceOptions.PASSWORD);
String database = config.get(MongoDBSourceOptions.DATABASE);
+// note: just to unify job name, no other use.
+config.setString("database-name", database);
String collection = config.get(MongoDBSourceOptions.COLLECTION);
if (StringUtils.isBlank(collection)) {
collection = config.get(TABLE_NAME);
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumDataChange.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumDataChange.java
index 4b20ebd6..8048e38a 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumDataChange.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumDataChange.java
@@ -126,7 +126,13 @@ public class MongoJsonDebeziumDataChange extends
CdcDataChange implements Change
public Map extractAfterRow(JsonNode recordRoot) {
JsonNode dataNode = recordRoot.get(FIELD_DATA);
Map rowMap = extractRow(dataNode);
-String objectId = ((Map)
rowMap.get(ID_FIELD)).get(OID_FIELD).toString();
+String objectId;
+// if user specifies the `_id` field manually, the $oid field may not
exist
+if (rowMap.get(ID_FIELD) instanceof Map) {
+objectId = ((Map)
rowMap.get(ID_FIELD)).get(OID_FIELD).toString();
+} else {
+objectId = rowMap.get(ID_FIELD).toString();
+}
rowMap.put(ID_FIELD, objectId);
return rowMap;
}
@@ -135,7 +141,13 @@ public class MongoJsonDebeziumDataChange extends
CdcDataChange implements Change
throws JsonProcessingException {
String documentKey = extractJsonNode(recordRoot, FIELD_DOCUMENT_KEY);
JsonNode jsonNode = objectMapper.readTree(documentKey);
-String objectId = extractJsonNode(jsonNode.get(ID_FIELD), OID_FIELD);
+String objectId;
+// if user specifies the `_id` field manually, the $oid field may not
exist
+if (jsonNode.get(ID_FIELD).has(OID_FIELD)) {
+objectId = extractJsonNode(jsonNode.get(ID_FIELD), OID_FIELD);
+} else {
+objectId = jsonNode.get(ID_FIELD).asText();
+}
Map row = new HashMap<>();
row.put(ID_FIELD, objectId);
return row;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (0d758137ad9 -> b7385ce3fc2)
This is an automated email from the ASF dual-hosted git repository. huajianlan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 0d758137ad9 [regression](kerberos)fix regression pipeline env when write hosts (#37057) add b7385ce3fc2 [enhancement](nereids) speedup sql cache with variable (#37090) No new revisions were added by this update. Summary of changes: .../doris/common/NereidsSqlCacheManager.java | 49 +++- .../org/apache/doris/nereids/NereidsPlanner.java | 73 +++--- .../org/apache/doris/nereids/SqlCacheContext.java | 88 +++--- .../nereids/trees/plans/ComputeResultSet.java | 55 ++ .../plans/physical/PhysicalEmptyRelation.java | 39 +- .../plans/physical/PhysicalOneRowRelation.java | 49 +++- .../trees/plans/physical/PhysicalResultSink.java | 18 - .../trees/plans/physical/PhysicalSqlCache.java | 11 ++- .../java/org/apache/doris/qe/StmtExecutor.java | 8 +- .../cache/parse_sql_from_sql_cache.groovy | 30 +--- 10 files changed, 277 insertions(+), 143 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/ComputeResultSet.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (b7385ce3fc2 -> 3eda6b0381a)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from b7385ce3fc2 [enhancement](nereids) speedup sql cache with variable (#37090) add 3eda6b0381a [Fix](Scanner) Fix reinitialization of TabletReader (#37059) No new revisions were added by this update. Summary of changes: be/src/vec/exec/scan/scanner_scheduler.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [branch-2.1] Pick "[Fix](regression) Fix p0 case test_unique_table_auto_inc_concurrent #37048" (#37050)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new c66975b2442 [branch-2.1] Pick "[Fix](regression) Fix p0 case
test_unique_table_auto_inc_concurrent #37048" (#37050)
c66975b2442 is described below
commit c66975b2442f0455978df9da6505dbc14a49befb
Author: bobhan1
AuthorDate: Tue Jul 2 11:15:50 2024 +0800
[branch-2.1] Pick "[Fix](regression) Fix p0 case
test_unique_table_auto_inc_concurrent #37048" (#37050)
## Proposed changes
pick https://github.com/apache/doris/pull/37048
---
.../data/data_model_p0/unique/test_unique_auto_inc_concurrent.out | 3 ---
.../suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy | 3 ++-
2 files changed, 2 insertions(+), 4 deletions(-)
diff --git
a/regression-test/data/data_model_p0/unique/test_unique_auto_inc_concurrent.out
b/regression-test/data/data_model_p0/unique/test_unique_auto_inc_concurrent.out
index 03819c9a717..c803ca86189 100644
---
a/regression-test/data/data_model_p0/unique/test_unique_auto_inc_concurrent.out
+++
b/regression-test/data/data_model_p0/unique/test_unique_auto_inc_concurrent.out
@@ -1,10 +1,7 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !sql --
-150150
-- !sql --
-300300
-- !sql --
-450450
diff --git
a/regression-test/suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy
b/regression-test/suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy
index bf6d584b2af..4793fb52343 100644
---
a/regression-test/suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy
+++
b/regression-test/suites/data_model_p0/unique/test_unique_auto_inc_concurrent.groovy
@@ -46,8 +46,9 @@ suite("test_unique_table_auto_inc_concurrent") {
}
threads.each { thread -> thread.join() }
+sql "sync"
-qt_sql "select count(id), count(distinct id) from ${table1};"
+qt_sql "select id, count(*) from ${table1} group by id having count(*)
> 1;"
}
run_test(15, 1, 10)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris-flink-connector) branch master updated: [improve]improve group commit logic (#413)
This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new ba21aabd [improve]improve group commit logic (#413)
ba21aabd is described below
commit ba21aabd0be664141e44581c5d16d97e9ff5467f
Author: Petrichor <31833513+vinle...@users.noreply.github.com>
AuthorDate: Tue Jul 2 11:26:40 2024 +0800
[improve]improve group commit logic (#413)
---
.../flink/sink/batch/DorisBatchStreamLoad.java | 14 +-
.../doris/flink/sink/writer/DorisStreamLoad.java | 44 +++
.../doris/flink/sink/writer/LoadConstants.java | 1 +
.../apache/doris/flink/sink/DorisSinkITCase.java | 51 ++
4 files changed, 99 insertions(+), 11 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchStreamLoad.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchStreamLoad.java
index fbc6daa0..375e4335 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchStreamLoad.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchStreamLoad.java
@@ -68,6 +68,7 @@ import static
org.apache.doris.flink.sink.writer.LoadConstants.ARROW;
import static org.apache.doris.flink.sink.writer.LoadConstants.CSV;
import static org.apache.doris.flink.sink.writer.LoadConstants.FORMAT_KEY;
import static org.apache.doris.flink.sink.writer.LoadConstants.GROUP_COMMIT;
+import static
org.apache.doris.flink.sink.writer.LoadConstants.GROUP_COMMIT_OFF_MODE;
import static
org.apache.doris.flink.sink.writer.LoadConstants.LINE_DELIMITER_DEFAULT;
import static
org.apache.doris.flink.sink.writer.LoadConstants.LINE_DELIMITER_KEY;
@@ -122,7 +123,11 @@ public class DorisBatchStreamLoad implements Serializable {
LINE_DELIMITER_KEY,
LINE_DELIMITER_DEFAULT))
.getBytes();
}
-this.enableGroupCommit = loadProps.containsKey(GROUP_COMMIT);
+this.enableGroupCommit =
+loadProps.containsKey(GROUP_COMMIT)
+&& !loadProps
+.getProperty(GROUP_COMMIT)
+.equalsIgnoreCase(GROUP_COMMIT_OFF_MODE);
this.executionOptions = executionOptions;
this.flushQueue = new
LinkedBlockingDeque<>(executionOptions.getFlushQueueSize());
if (StringUtils.isNotBlank(dorisOptions.getTableIdentifier())) {
@@ -283,7 +288,12 @@ public class DorisBatchStreamLoad implements Serializable {
Throwable resEx = new Throwable();
int retry = 0;
while (retry <= executionOptions.getMaxRetries()) {
-LOG.info("stream load started for {} on host {}", label,
hostPort);
+if (enableGroupCommit) {
+LOG.info("stream load started with group commit on host
{}", hostPort);
+} else {
+LOG.info("stream load started for {} on host {}", label,
hostPort);
+}
+
try (CloseableHttpClient httpClient =
httpClientBuilder.build()) {
try (CloseableHttpResponse response =
httpClient.execute(putBuilder.build())) {
int statusCode =
response.getStatusLine().getStatusCode();
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/DorisStreamLoad.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/DorisStreamLoad.java
index 676de3df..4cbcb431 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/DorisStreamLoad.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/DorisStreamLoad.java
@@ -59,6 +59,7 @@ import static
org.apache.doris.flink.sink.writer.LoadConstants.ARROW;
import static org.apache.doris.flink.sink.writer.LoadConstants.CSV;
import static org.apache.doris.flink.sink.writer.LoadConstants.FORMAT_KEY;
import static org.apache.doris.flink.sink.writer.LoadConstants.GROUP_COMMIT;
+import static
org.apache.doris.flink.sink.writer.LoadConstants.GROUP_COMMIT_OFF_MODE;
import static
org.apache.doris.flink.sink.writer.LoadConstants.LINE_DELIMITER_DEFAULT;
import static
org.apache.doris.flink.sink.writer.LoadConstants.LINE_DELIMITER_KEY;
@@ -131,7 +132,11 @@ public class DorisStreamLoad implements Serializable {
LINE_DELIMITER_KEY,
LINE_DELIMITER_DEFAULT))
.getBytes();
}
-enableGroupCommit = streamLoadProp.containsKey(GROUP_COMMIT);
+this.enableGroupCommit =
+streamLoadProp.containsKey(GROUP_COMMIT)
+&& !streamLoadProp
+
(doris) branch master updated (3eda6b0381a -> da3aff7878a)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3eda6b0381a [Fix](Scanner) Fix reinitialization of TabletReader (#37059) add da3aff7878a [Bug](QueryStatistics) use std::atomic for returned_rows to save data (#37072) No new revisions were added by this update. Summary of changes: be/src/runtime/query_statistics.cpp | 2 +- be/src/runtime/query_statistics.h | 8 +--- 2 files changed, 6 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (da3aff7878a -> 71c3fbae300)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from da3aff7878a [Bug](QueryStatistics) use std::atomic for returned_rows to save data (#37072) add 71c3fbae300 [fix](mtmv)fix MTMVTask log has null pointer (#36840) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/job/task/AbstractTask.java | 13 + .../src/test/java/org/apache/doris/mtmv/MTMVTest.java | 4 ++-- 2 files changed, 15 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [test](external)move hive cases from p2 to p0 (#36855)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new cd8d707e120 [test](external)move hive cases from p2 to p0 (#36855) cd8d707e120 is described below commit cd8d707e120060071047957d81c5f4d997b5f789 Author: zy-kkk AuthorDate: Tue Jul 2 12:56:19 2024 +0800 [test](external)move hive cases from p2 to p0 (#36855) test_hive_same_db_table_name test_hive_special_char_partition test_complex_types test_wide_table --- .../test_complex_types/create_table.hql| 138 + .../data/multi_catalog/test_complex_types/run.sh | 22 + .../test_hive_same_db_table_name/create_table.hql | 22 + .../test_hive_same_db_table_name/data.tar.gz | Bin 0 -> 183 bytes .../test_hive_same_db_table_name/run.sh| 12 + .../create_table.hql | 22 + .../test_hive_special_char_partition/data.tar.gz | Bin 0 -> 894 bytes .../test_hive_special_char_partition/run.sh| 12 + .../multi_catalog/test_wide_table/create_table.hql | 654 + .../data/multi_catalog/test_wide_table/run.sh | 22 + .../hive/test_complex_types.out| 54 +- .../hive/test_hive_same_db_table_name.out | 15 + .../hive/test_hive_special_char_partition.out | 50 ++ .../external_table_p0/hive/test_wide_table.out | 45 ++ .../external_table_p2/hive/test_wide_table.out | 23 - .../hive/test_complex_types.groovy | 39 +- .../hive/test_hive_same_db_table_name.groovy | 31 +- .../hive/test_hive_special_char_partition.groovy | 31 +- .../hive/test_wide_table.groovy| 33 +- 19 files changed, 1132 insertions(+), 93 deletions(-) diff --git a/docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_complex_types/create_table.hql b/docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_complex_types/create_table.hql new file mode 100644 index 000..f22d7ff246b --- /dev/null +++ b/docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_complex_types/create_table.hql @@ -0,0 +1,138 @@ +create database if not exists multi_catalog; + +use multi_catalog; + +CREATE TABLE `byd`( + `id` int, + `capacity` array, + `singles` map, + `favor` struct) +ROW FORMAT SERDE + 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' +STORED AS INPUTFORMAT + 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' +OUTPUTFORMAT + 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' +LOCATION + '/user/doris/suites/multi_catalog/byd' +TBLPROPERTIES ( + 'transient_lastDdlTime'='1690356922'); + +msck repair table byd; + +CREATE TABLE `complex_offsets_check`( + `id` int, + `array1` array, + `array2` array>, + `map1` map, + `struct1` struct) +ROW FORMAT SERDE + 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' +STORED AS INPUTFORMAT + 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' +OUTPUTFORMAT + 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' +LOCATION + '/user/doris/suites/multi_catalog/complex_offsets_check' +TBLPROPERTIES ( + 'transient_lastDdlTime'='1690974653'); + +msck repair table complex_offsets_check; + +CREATE TABLE `parquet_all_types`( + `t_null_string` string, + `t_null_varchar` varchar(65535), + `t_null_char` char(10), + `t_null_decimal_precision_2` decimal(2,1), + `t_null_decimal_precision_4` decimal(4,2), + `t_null_decimal_precision_8` decimal(8,4), + `t_null_decimal_precision_17` decimal(17,8), + `t_null_decimal_precision_18` decimal(18,8), + `t_null_decimal_precision_38` decimal(38,16), + `t_empty_string` string, + `t_string` string, + `t_empty_varchar` varchar(65535), + `t_varchar` varchar(65535), + `t_varchar_max_length` varchar(65535), + `t_char` char(10), + `t_int` int, + `t_bigint` bigint, + `t_float` float, + `t_double` double, + `t_boolean_true` boolean, + `t_boolean_false` boolean, + `t_decimal_precision_2` decimal(2,1), + `t_decimal_precision_4` decimal(4,2), + `t_decimal_precision_8` decimal(8,4), + `t_decimal_precision_17` decimal(17,8), + `t_decimal_precision_18` decimal(18,8), + `t_decimal_precision_38` decimal(38,16), + `t_binary` binary, + `t_map_string` map, + `t_map_varchar` map, + `t_map_char` map, + `t_map_int` map, + `t_map_bigint` map, + `t_map_float` map, + `t_map_double` map, + `t_map_boolean` map, + `t_map_decimal_precision_2` map, + `t_map_decimal_precision_4` map, + `t_map_decimal_precision_8` map, + `t_map_decimal_precision_17` map, + `t_map_decimal_precision_18` map, + `t_map_decimal_precision_38` map, + `t_array_string` array, + `t_array_int` array, + `t_array_bigint` array, + `t_array_float` array, + `
(doris) branch master updated: [fix](statistics)Fix partition record update rows bug. Remove useless code. (#37087)
This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1f45e99b8b2 [fix](statistics)Fix partition record update rows bug.
Remove useless code. (#37087)
1f45e99b8b2 is described below
commit 1f45e99b8b2b056e10ce31956fde3038047fa4bd
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Tue Jul 2 12:59:18 2024 +0800
[fix](statistics)Fix partition record update rows bug. Remove useless code.
(#37087)
Fix partition record update rows bug.
When enable_partition_analyze set to false, we shouldn't update column's
update rows map, only update it when enable_partition_analyze is true.
Because when it's false, the analyze task didn't collect partition level
stats. If we update column's update rows in this case, it will cause
auto analyze failed to trigger partition collection because the health
checker thought the update rows are update-to-date.
---
.../java/org/apache/doris/analysis/AnalyzeStmt.java | 5 -
.../org/apache/doris/analysis/ShowAnalyzeStmt.java | 1 +
.../main/java/org/apache/doris/qe/ShowExecutor.java | 3 ++-
.../org/apache/doris/statistics/AnalysisInfo.java| 20 +++-
.../apache/doris/statistics/AnalysisInfoBuilder.java | 19 +--
.../org/apache/doris/statistics/AnalysisManager.java | 16
.../doris/statistics/StatisticsAutoCollector.java| 2 +-
.../org/apache/doris/statistics/TableStatsMeta.java | 10 ++
.../apache/doris/statistics/AnalysisManagerTest.java | 2 +-
.../doris/statistics/AnalysisTaskExecutorTest.java | 3 ---
.../org/apache/doris/statistics/AnalyzeTest.java | 2 --
.../apache/doris/statistics/HistogramTaskTest.java | 2 --
.../suites/statistics/test_partition_stats.groovy| 6 +-
13 files changed, 36 insertions(+), 55 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java
index 08bc1957800..e97de804376 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java
@@ -20,7 +20,6 @@ package org.apache.doris.analysis;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
-import org.apache.doris.statistics.AnalysisInfo.AnalysisMode;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
import org.apache.doris.statistics.AnalysisInfo.ScheduleType;
import org.apache.doris.statistics.util.StatisticsUtil;
@@ -57,10 +56,6 @@ public class AnalyzeStmt extends StatementBase {
return analyzeProperties.getProperties();
}
-public AnalysisMode getAnalysisMode() {
-return analyzeProperties.isIncremental() ? AnalysisMode.INCREMENTAL :
AnalysisMode.FULL;
-}
-
public AnalysisType getAnalysisType() {
return analyzeProperties.getAnalysisType();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowAnalyzeStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowAnalyzeStmt.java
index f660d6eeb3c..f015cdb0555 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowAnalyzeStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowAnalyzeStmt.java
@@ -63,6 +63,7 @@ public class ShowAnalyzeStmt extends ShowStmt {
.add("start_time")
.add("end_time")
.add("priority")
+.add("enable_partition")
.build();
private long jobId;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
index 62e8e6dad5e..ef75ed75193 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
@@ -2914,7 +2914,7 @@ public class ShowExecutor {
private void handleShowAnalyze() {
ShowAnalyzeStmt showStmt = (ShowAnalyzeStmt) stmt;
-List results =
Env.getCurrentEnv().getAnalysisManager().showAnalysisJob(showStmt);
+List results =
Env.getCurrentEnv().getAnalysisManager().findAnalysisJobs(showStmt);
List> resultRows = Lists.newArrayList();
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("-MM-dd
HH:mm:ss");
for (AnalysisInfo analysisInfo : results) {
@@ -2951,6 +2951,7 @@ public class ShowExecutor {
row.add(startTime.format(formatter));
row.add(endTime.format(formatter));
row.add(analysisInfo.priority.name());
+row.add(String.valueOf(analysisInfo.enablePartition));
resultRows.add(row);
} catch (Exception e) {
(doris) branch branch-2.1 updated: [opt](nereids)using mv's derived stats #35721 (#37098)
This is an automated email from the ASF dual-hosted git repository.
englefly pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 0c8934642e9 [opt](nereids)using mv's derived stats #35721 (#37098)
0c8934642e9 is described below
commit 0c8934642e9e1840f455021f458bbbaf3d2bfbb8
Author: minghong
AuthorDate: Tue Jul 2 13:54:05 2024 +0800
[opt](nereids)using mv's derived stats #35721 (#37098)
pick from #35721
(cherry picked from commit a481d79b1e258f7b6711157bb0d5e4c224e680e5)
## Proposed changes
Issue Number: close #xxx
---
.../org/apache/doris/nereids/stats/StatsCalculator.java | 15 +++
1 file changed, 15 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java
index 6a8d1234050..4d4f68bb29a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java
@@ -760,6 +760,21 @@ public class StatsCalculator extends
DefaultPlanVisitor {
// 2. Consider the influence of runtime filter
// 3. Get NDV and column data size from StatisticManger,
StatisticManager doesn't support it now.
private Statistics computeCatalogRelation(CatalogRelation catalogRelation)
{
+if (catalogRelation instanceof LogicalOlapScan) {
+LogicalOlapScan olap = (LogicalOlapScan) catalogRelation;
+if (olap.getSelectedIndexId() != olap.getTable().getBaseIndexId())
{
+// mv is selected, return its estimated stats
+Optional optStats =
cascadesContext.getStatementContext()
+.getStatistics(olap.getRelationId());
+if (optStats.isPresent()) {
+double actualRowCount =
catalogRelation.getTable().getRowCountForNereids();
+if (actualRowCount > optStats.get().getRowCount()) {
+return optStats.get();
+}
+}
+}
+}
+
List output = catalogRelation.getOutput();
ImmutableSet.Builder slotSetBuilder =
ImmutableSet.builderWithExpectedSize(output.size());
for (Slot slot : output) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Nereids) null type in result set will be cast to tinyint (#37019)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b262f985342 [fix](Nereids) null type in result set will be cast to
tinyint (#37019)
b262f985342 is described below
commit b262f9853421ccdaa0f6e98243dd114eb23e06fd
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Tue Jul 2 14:01:51 2024 +0800
[fix](Nereids) null type in result set will be cast to tinyint (#37019)
---
.../nereids/rules/analysis/BindExpression.java | 34 +---
.../doris/nereids/util/TypeCoercionUtils.java | 3 +-
.../data/mtmv_p0/test_create_with_null_type.out| 7 +++
.../mtmv_p0/test_create_with_null_type.groovy | 64 ++
4 files changed, 100 insertions(+), 8 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
index c1080adf3b7..ad8dec4ac7c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
@@ -42,6 +42,7 @@ import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.BoundStar;
+import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.DefaultValueSlot;
import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.ExprId;
@@ -89,8 +90,11 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalTVFRelation;
import org.apache.doris.nereids.trees.plans.logical.UsingJoin;
import org.apache.doris.nereids.trees.plans.visitor.InferPlanOutputAlias;
import org.apache.doris.nereids.types.BooleanType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.NullType;
import org.apache.doris.nereids.types.StructField;
import org.apache.doris.nereids.types.StructType;
+import org.apache.doris.nereids.types.TinyIntType;
import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.nereids.util.PlanUtils;
import org.apache.doris.nereids.util.TypeCoercionUtils;
@@ -208,22 +212,38 @@ public class BindExpression implements
AnalysisRuleFactory {
private LogicalResultSink
bindResultSink(MatchingContext> ctx) {
LogicalSink sink = ctx.root;
+Plan child = sink.child();
+List output = child.getOutput();
+List castNullToTinyInt =
Lists.newArrayListWithCapacity(output.size());
+boolean needProject = false;
+for (Slot slot : output) {
+DataType newType = TypeCoercionUtils.replaceSpecifiedType(
+slot.getDataType(), NullType.class, TinyIntType.INSTANCE);
+if (!newType.equals(slot.getDataType())) {
+needProject = true;
+castNullToTinyInt.add(new Alias(new Cast(slot, newType)));
+} else {
+castNullToTinyInt.add(slot);
+}
+}
+if (needProject) {
+child = new LogicalProject<>(castNullToTinyInt, child);
+}
if (ctx.connectContext.getState().isQuery()) {
-List outputExprs =
sink.child().getOutput().stream()
+List outputExprs = child.getOutput().stream()
.map(NamedExpression.class::cast)
.collect(ImmutableList.toImmutableList());
-return new LogicalResultSink<>(outputExprs, sink.child());
+return new LogicalResultSink<>(outputExprs, child);
}
// Should infer column name for expression when query command
-final ImmutableListMultimap.Builder
exprIdToIndexMapBuilder =
-ImmutableListMultimap.builder();
-List childOutput = sink.child().getOutput();
+final ImmutableListMultimap.Builder
exprIdToIndexMapBuilder = ImmutableListMultimap.builder();
+List childOutput = child.getOutput();
for (int index = 0; index < childOutput.size(); index++) {
exprIdToIndexMapBuilder.put(childOutput.get(index).getExprId(),
index);
}
InferPlanOutputAlias aliasInfer = new
InferPlanOutputAlias(childOutput);
-List output = aliasInfer.infer(sink.child(),
exprIdToIndexMapBuilder.build());
-return new LogicalResultSink<>(output, sink.child());
+List sinkExpr = aliasInfer.infer(child,
exprIdToIndexMapBuilder.build());
+return new LogicalResultSink<>(sinkExpr, child);
}
private LogicalSubQueryAlias
bindSubqueryAlias(MatchingContext> ctx) {
diff --git
a/fe/fe-core/sr
(doris) branch master updated: [Chore](runtime-filter) enlarge sync filter size rpc timeout limit (#37103)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fa5e7008d5d [Chore](runtime-filter) enlarge sync filter size rpc
timeout limit (#37103)
fa5e7008d5d is described below
commit fa5e7008d5d918f30e0580ce7bda03512b3d6aec
Author: Pxl
AuthorDate: Tue Jul 2 14:16:39 2024 +0800
[Chore](runtime-filter) enlarge sync filter size rpc timeout limit (#37103)
## Proposed changes
enlarge sync filter size rpc timeout limit
rf will failed when rpc timeout, so we need enlarge limit
```
sync filter size meet error, filter: RuntimeFilter: (id = 3, type =
in_or_bloomfilter, need_local_merge: false, is_broadcast: false,
build_bf_cardinality: true
```
---
be/src/common/config.cpp | 1 +
be/src/exprs/runtime_filter.cpp| 4 ++--
be/src/exprs/runtime_filter.h | 2 +-
be/src/exprs/runtime_filter_slots.h| 2 +-
regression-test/suites/query_p0/join/test_join5.groovy | 1 +
5 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 78afc756af8..92303473ad6 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -255,6 +255,7 @@ DEFINE_Int32(doris_scanner_thread_pool_queue_size,
"102400");
// default thrift client connect timeout(in seconds)
DEFINE_mInt32(thrift_connect_timeout_seconds, "3");
DEFINE_mInt32(fetch_rpc_timeout_seconds, "30");
+
// default thrift client retry interval (in milliseconds)
DEFINE_mInt64(thrift_client_retry_interval_ms, "1000");
// max message size of thrift request
diff --git a/be/src/exprs/runtime_filter.cpp b/be/src/exprs/runtime_filter.cpp
index e38a80a143e..2dcfc97b096 100644
--- a/be/src/exprs/runtime_filter.cpp
+++ b/be/src/exprs/runtime_filter.cpp
@@ -1054,7 +1054,7 @@ public:
: Base(req, callback), _dependency(std::move(dependency)) {}
};
-Status IRuntimeFilter::send_filter_size(uint64_t local_filter_size) {
+Status IRuntimeFilter::send_filter_size(RuntimeState* state, uint64_t
local_filter_size) {
DCHECK(is_producer());
if (_need_local_merge) {
@@ -1105,7 +1105,7 @@ Status IRuntimeFilter::send_filter_size(uint64_t
local_filter_size) {
request->set_filter_size(local_filter_size);
request->set_filter_id(_filter_id);
-callback->cntl_->set_timeout_ms(wait_time_ms());
+callback->cntl_->set_timeout_ms(std::min(3600, state->execution_timeout())
* 1000);
stub->send_filter_size(closure->cntl_.get(), closure->request_.get(),
closure->response_.get(),
closure.get());
diff --git a/be/src/exprs/runtime_filter.h b/be/src/exprs/runtime_filter.h
index a78d732b687..390a61bfe1a 100644
--- a/be/src/exprs/runtime_filter.h
+++ b/be/src/exprs/runtime_filter.h
@@ -229,7 +229,7 @@ public:
// push filter to remote node or push down it to scan_node
Status publish(bool publish_local = false);
-Status send_filter_size(uint64_t local_filter_size);
+Status send_filter_size(RuntimeState* state, uint64_t local_filter_size);
RuntimeFilterType type() const { return _runtime_filter_type; }
diff --git a/be/src/exprs/runtime_filter_slots.h
b/be/src/exprs/runtime_filter_slots.h
index b5b04a1ebac..0bf8a33f9f2 100644
--- a/be/src/exprs/runtime_filter_slots.h
+++ b/be/src/exprs/runtime_filter_slots.h
@@ -55,7 +55,7 @@ public:
// send_filter_size may call dependency->sub(), so we call
set_dependency firstly for all rf to avoid dependency set_ready repeatedly
for (auto* runtime_filter : _runtime_filters) {
if (runtime_filter->need_sync_filter_size()) {
-
RETURN_IF_ERROR(runtime_filter->send_filter_size(hash_table_size));
+RETURN_IF_ERROR(runtime_filter->send_filter_size(state,
hash_table_size));
}
}
return Status::OK();
diff --git a/regression-test/suites/query_p0/join/test_join5.groovy
b/regression-test/suites/query_p0/join/test_join5.groovy
index 62be496372d..4323575870f 100644
--- a/regression-test/suites/query_p0/join/test_join5.groovy
+++ b/regression-test/suites/query_p0/join/test_join5.groovy
@@ -16,6 +16,7 @@
// under the License.
suite("test_join5", "query,p0") {
+sql "set runtime_filter_wait_time_ms = 5"
def DBname = "regression_test_join5"
sql "DROP DATABASE IF EXISTS ${DBname}"
sql "CREATE DATABASE IF NOT EXISTS ${DBname}"
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [enhance](mtmv)when calculating the availability of MTMV, no longer c… (#36617)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new ebc1afed4af [enhance](mtmv)when calculating the availability of MTMV,
no longer c… (#36617)
ebc1afed4af is described below
commit ebc1afed4afcf42153ce8f6593b1ff6466c8d95a
Author: zhangdong <493738...@qq.com>
AuthorDate: Tue Jul 2 14:24:33 2024 +0800
[enhance](mtmv)when calculating the availability of MTMV, no longer c…
(#36617)
pick: #36507
---
.../main/java/org/apache/doris/mtmv/MTMVRewriteUtil.java | 4 ++--
.../java/org/apache/doris/mtmv/MTMVRewriteUtilTest.java | 15 ++-
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRewriteUtil.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRewriteUtil.java
index 03a1aefeba1..209fd5da0f6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRewriteUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRewriteUtil.java
@@ -62,8 +62,8 @@ public class MTMVRewriteUtil {
return res;
}
// check mv is normal
-if (!(mtmv.getStatus().getState() == MTMVState.NORMAL
-&& mtmv.getStatus().getRefreshState() ==
MTMVRefreshState.SUCCESS)) {
+if (mtmv.getStatus().getState() != MTMVState.NORMAL
+|| mtmv.getStatus().getRefreshState() ==
MTMVRefreshState.INIT) {
return res;
}
Map> partitionMappings = null;
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRewriteUtilTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRewriteUtilTest.java
index 864478933c0..40797760b70 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRewriteUtilTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRewriteUtilTest.java
@@ -256,7 +256,20 @@ public class MTMVRewriteUtilTest {
};
Collection mtmvCanRewritePartitions = MTMVRewriteUtil
.getMTMVCanRewritePartitions(mtmv, ctx, currentTimeMills);
-Assert.assertEquals(0, mtmvCanRewritePartitions.size());
+Assert.assertEquals(1, mtmvCanRewritePartitions.size());
}
+@Test
+public void testGetMTMVCanRewritePartitionsRefreshStateInit() {
+new Expectations() {
+{
+status.getRefreshState();
+minTimes = 0;
+result = MTMVRefreshState.INIT;
+}
+};
+Collection mtmvCanRewritePartitions = MTMVRewriteUtil
+.getMTMVCanRewritePartitions(mtmv, ctx, currentTimeMills);
+Assert.assertEquals(0, mtmvCanRewritePartitions.size());
+}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](mtmv) mtmv add case and fix bug (#36019) (#36600)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 496024ffb55 [fix](mtmv) mtmv add case and fix bug (#36019) (#36600)
496024ffb55 is described below
commit 496024ffb553045d0802cd536552856b6aa4c690
Author: zhangdong <493738...@qq.com>
AuthorDate: Tue Jul 2 14:24:53 2024 +0800
[fix](mtmv) mtmv add case and fix bug (#36019) (#36600)
pick from: #36019
---
.../org/apache/doris/analysis/AddRollupClause.java | 2 +-
.../apache/doris/analysis/DropRollupClause.java| 2 +-
.../apache/doris/analysis/RollupRenameClause.java | 2 +-
.../trees/plans/commands/info/CreateMTMVInfo.java | 2 +
.../trees/plans/commands/info/CreateTableInfo.java | 12 ++-
.../data/mtmv_p0/test_auto_bucket_mtmv.out | 6 ++
regression-test/data/mtmv_p0/test_bitmap_mtmv.out | 6 ++
.../data/mtmv_p0/test_bloom_filter_mtmv.out| 6 ++
.../data/mtmv_p0/test_colocate_with_mtmv.out | 6 ++
.../data/mtmv_p0/test_compression_mtmv.out | 6 ++
.../data/mtmv_p0/test_create_mv_mtmv.out | 16
.../data/mtmv_p0/test_create_rollup_mtmv.out | 16
.../data/mtmv_p0/test_create_view_mtmv.out | 15
.../data/mtmv_p0/test_inverted_mtmv.out| 6 ++
.../data/mtmv_p0/test_light_schema_change_mtmv.out | 6 ++
regression-test/data/mtmv_p0/test_n_gram_mtmv.out | 6 ++
.../mtmv_p0/test_refresh_partition_num_mtmv.out| 6 ++
.../data/mtmv_p0/test_schema_version_mtmv.out | 6 ++
.../mtmv_p0/test_set_partition_version_mtmv.out| 6 ++
.../data/mtmv_p0/test_tablet_type_mtmv.out | 6 ++
.../data/mtmv_p0/test_version_info_mtmv.out| 6 ++
.../test_alter_distribution_type_mtmv.groovy | 60 ++
.../suites/mtmv_p0/test_auto_bucket_mtmv.groovy| 64 +++
.../suites/mtmv_p0/test_bitmap_mtmv.groovy | 79 +++
.../suites/mtmv_p0/test_block_rule_mtmv.groovy | 70 +
.../suites/mtmv_p0/test_bloom_filter_mtmv.groovy | 86
.../suites/mtmv_p0/test_colocate_with_mtmv.groovy | 77 ++
.../suites/mtmv_p0/test_compression_mtmv.groovy| 68
.../suites/mtmv_p0/test_create_mv_mtmv.groovy | 90 +
.../suites/mtmv_p0/test_create_rollup_mtmv.groovy | 91 ++
.../suites/mtmv_p0/test_create_view_mtmv.groovy| 72 +
.../suites/mtmv_p0/test_inverted_mtmv.groovy | 80 +++
.../mtmv_p0/test_light_schema_change_mtmv.groovy | 64 +++
.../suites/mtmv_p0/test_limit_op_mtmv.groovy | 66 +++-
.../suites/mtmv_p0/test_n_gram_mtmv.groovy | 79 +++
.../mtmv_p0/test_refresh_partition_num_mtmv.groovy | 70 +
.../suites/mtmv_p0/test_schema_version_mtmv.groovy | 64 +++
.../mtmv_p0/test_sequence_column_mtmv.groovy | 56 +
.../mtmv_p0/test_set_partition_version_mtmv.groovy | 73 +
.../suites/mtmv_p0/test_tablet_type_mtmv.groovy| 64 +++
.../suites/mtmv_p0/test_version_info_mtmv.groovy | 64 +++
41 files changed, 1519 insertions(+), 63 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/AddRollupClause.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/AddRollupClause.java
index 03e4c868fb6..a9df4fedc81 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/AddRollupClause.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/AddRollupClause.java
@@ -91,7 +91,7 @@ public class AddRollupClause extends AlterTableClause {
@Override
public boolean allowOpMTMV() {
-return false;
+return true;
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRollupClause.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRollupClause.java
index c3d41a4bf75..292dc6c9ada 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRollupClause.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRollupClause.java
@@ -50,7 +50,7 @@ public class DropRollupClause extends AlterTableClause {
@Override
public boolean allowOpMTMV() {
-return false;
+return true;
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/RollupRenameClause.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/RollupRenameClause.java
index b3af2ec09d8..37544ba6edd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/RollupRenameClause.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/RollupRenameClause.java
@@ -65,7 +65,7 @@ public class RollupRenameClause extends AlterTableClause {
@Override
public boolean allowOpM
(doris) branch master updated: [fix](auth)support check priv when tvf use resource (#36928)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new af3122244ef [fix](auth)support check priv when tvf use resource
(#36928)
af3122244ef is described below
commit af3122244ef3ad99f24069a00805e90a7f488b13
Author: zhangdong <493738...@qq.com>
AuthorDate: Tue Jul 2 14:25:21 2024 +0800
[fix](auth)support check priv when tvf use resource (#36928)
current, follow sql not check priv, we need check `usage priv` of
`test_resource`
```
SELECT * FROM S3 (
"uri" = "xxx",
"format" = "orc",
"resource" = "test_resource",
"use_path_style" = "true"
);
---
.../org/apache/doris/analysis/AlterPolicyStmt.java | 1 +
.../apache/doris/analysis/CreatePolicyStmt.java| 1 +
.../java/org/apache/doris/common/ErrorCode.java| 4 ++
.../nereids/rules/rewrite/CheckPrivileges.java | 9
.../functions/table/TableValuedFunction.java | 5 +++
.../ExternalFileTableValuedFunction.java | 17
.../doris/tablefunction/TableValuedFunctionIf.java | 5 +++
.../tvf/test_s3_tvf_with_resource.groovy | 48 ++
8 files changed, 90 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterPolicyStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterPolicyStmt.java
index c8128e2bcbd..db9c10f34f2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterPolicyStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/AlterPolicyStmt.java
@@ -53,6 +53,7 @@ public class AlterPolicyStmt extends DdlStmt {
super.analyze(analyzer);
// check auth
+// check if can alter policy and use storage_resource
if (!Env.getCurrentEnv().getAccessManager()
.checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreatePolicyStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreatePolicyStmt.java
index 8aedccb6e75..0c3e760e576 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreatePolicyStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreatePolicyStmt.java
@@ -102,6 +102,7 @@ public class CreatePolicyStmt extends DdlStmt {
+ "Enable it by setting
'enable_storage_policy=true' in fe.conf");
}
// check auth
+// check if can create policy and use storage_resource
if (!Env.getCurrentEnv().getAccessManager()
.checkGlobalPriv(ConnectContext.get(),
PrivPredicate.ADMIN)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
index e66bdef133c..c3864b45aa2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
@@ -83,6 +83,10 @@ public enum ErrorCode {
ERR_SPECIFIC_ALL_ACCESS_DENIED_ERROR(1223, new byte[] {'4', '2', '0', '0',
'0'}, "Access denied; you need all "
+ " %s privilege(s) for this operation"),
+
+ERR_RESOURCE_ACCESS_DENIED_ERROR(1222, new byte[]{'4', '2', '0', '0',
'0'}, "Access denied; you need (at least "
++ "one of) the (%s) privilege(s) on resource %s for this
operation"),
+
ERR_LOCAL_VARIABLE(1228, new byte[]{'H', 'Y', '0', '0', '0'}, "Variable
'%s' is a SESSION variable and can't be "
+ "used with SET GLOBAL"),
ERR_GLOBAL_VARIABLE(1229, new byte[]{'H', 'Y', '0', '0', '0'}, "Variable
'%s' is a GLOBAL variable and should be "
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CheckPrivileges.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CheckPrivileges.java
index 5c82dfac651..74609694431 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CheckPrivileges.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CheckPrivileges.java
@@ -27,9 +27,11 @@ import org.apache.doris.nereids.jobs.JobContext;
import org.apache.doris.nereids.rules.analysis.UserAuthentication;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
+import
org.apache.doris.nereids.trees.expressions.functions.table.TableValuedFunction;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
import org.apache.doris.nereids.trees.plans.logical.L
(doris) branch branch-2.1 updated: [fix](parquet) prevent parquet page reader print much warning logs (#37012)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 859a7c80b8b [fix](parquet) prevent parquet page reader print much
warning logs (#37012)
859a7c80b8b is described below
commit 859a7c80b8bdef46cf30ad65057605ea269a4429
Author: Ashin Gau
AuthorDate: Tue Jul 2 14:33:01 2024 +0800
[fix](parquet) prevent parquet page reader print much warning logs (#37012)
bp #37011
---
be/src/util/thrift_util.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/util/thrift_util.h b/be/src/util/thrift_util.h
index aff3a9ab101..9f4792ff64b 100644
--- a/be/src/util/thrift_util.h
+++ b/be/src/util/thrift_util.h
@@ -147,7 +147,7 @@ Status deserialize_thrift_msg(const uint8_t* buf, uint32_t*
len, bool compact,
try {
deserialized_msg->read(tproto.get());
} catch (std::exception& e) {
-return Status::InternalError("Couldn't deserialize thrift msg:\n{}",
e.what());
+return Status::InternalError("Couldn't deserialize thrift
msg:\n{}", e.what());
} catch (...) {
// TODO: Find the right exception for 0 bytes
return Status::InternalError("Unknown exception");
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Chore](exec) remove some unused code (#37094)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f517022d32d [Chore](exec) remove some unused code (#37094)
f517022d32d is described below
commit f517022d32d0d7997a9c5db6fc5cfd2b23642d85
Author: Pxl
AuthorDate: Tue Jul 2 14:39:18 2024 +0800
[Chore](exec) remove some unused code (#37094)
## Proposed changes
remove some unused code
---
be/src/pipeline/exec/exchange_sink_operator.cpp | 1 -
be/src/vec/data_types/data_type_decimal.h | 20 +---
2 files changed, 5 insertions(+), 16 deletions(-)
diff --git a/be/src/pipeline/exec/exchange_sink_operator.cpp
b/be/src/pipeline/exec/exchange_sink_operator.cpp
index 198bc555024..2a8aa56dc62 100644
--- a/be/src/pipeline/exec/exchange_sink_operator.cpp
+++ b/be/src/pipeline/exec/exchange_sink_operator.cpp
@@ -108,7 +108,6 @@ Status ExchangeSinkLocalState::open(RuntimeState* state) {
SCOPED_TIMER(_open_timer);
RETURN_IF_ERROR(Base::open(state));
auto& p = _parent->cast();
-_part_type = p._part_type;
SCOPED_CONSUME_MEM_TRACKER(_mem_tracker.get());
int local_size = 0;
diff --git a/be/src/vec/data_types/data_type_decimal.h
b/be/src/vec/data_types/data_type_decimal.h
index a8e9184739a..7b6ea532b74 100644
--- a/be/src/vec/data_types/data_type_decimal.h
+++ b/be/src/vec/data_types/data_type_decimal.h
@@ -288,16 +288,6 @@ public:
return x % get_scale_multiplier();
}
-T max_whole_value() const { return get_scale_multiplier(max_precision() -
scale) - T(1); }
-
-bool can_store_whole(T x) const {
-T max = max_whole_value();
-if (x > max || x < T(-max)) {
-return false;
-}
-return true;
-}
-
/// @returns multiplier for U to become T with correct scale
template
T scale_factor_for(const DataTypeDecimal& x, bool) const {
@@ -388,19 +378,19 @@ const DataTypeDecimal* check_decimal(const IDataType&
data_type) {
}
inline UInt32 get_decimal_scale(const IDataType& data_type, UInt32
default_value = 0) {
-if (auto* decimal_type = check_decimal(data_type)) {
+if (const auto* decimal_type = check_decimal(data_type)) {
return decimal_type->get_scale();
}
-if (auto* decimal_type = check_decimal(data_type)) {
+if (const auto* decimal_type = check_decimal(data_type)) {
return decimal_type->get_scale();
}
-if (auto* decimal_type = check_decimal(data_type)) {
+if (const auto* decimal_type = check_decimal(data_type)) {
return decimal_type->get_scale();
}
-if (auto* decimal_type = check_decimal(data_type)) {
+if (const auto* decimal_type = check_decimal(data_type)) {
return decimal_type->get_scale();
}
-if (auto* decimal_type = check_decimal(data_type)) {
+if (const auto* decimal_type = check_decimal(data_type)) {
return decimal_type->get_scale();
}
return default_value;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Compatibility](agg-state) do not throw eception when DataTypeAggState meet old be exec version (#37104)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9f80a4b68f5 [Compatibility](agg-state) do not throw eception when
DataTypeAggState meet old be exec version (#37104)
9f80a4b68f5 is described below
commit 9f80a4b68f540fb65286c683b21ae5f6a826cdd8
Author: Pxl
AuthorDate: Tue Jul 2 14:58:25 2024 +0800
[Compatibility](agg-state) do not throw eception when DataTypeAggState meet
old be exec version (#37104)
## Proposed changes
do not throw eception when DataTypeAggState meet old be exec version
this change is used to avoid memtable meet eception and core dump
---
be/src/vec/data_types/data_type_agg_state.h | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/data_types/data_type_agg_state.h
b/be/src/vec/data_types/data_type_agg_state.h
index 6b4fdc64562..96b147c8e96 100644
--- a/be/src/vec/data_types/data_type_agg_state.h
+++ b/be/src/vec/data_types/data_type_agg_state.h
@@ -43,11 +43,13 @@ public:
_be_exec_version(be_exec_version) {
_agg_function =
AggregateFunctionSimpleFactory::instance().get(_function_name, _sub_types,
_result_is_nullable);
-if (_agg_function == nullptr ||
-!BeExecVersionManager::check_be_exec_version(be_exec_version)) {
+if (_agg_function == nullptr) {
throw Exception(ErrorCode::INVALID_ARGUMENT,
"DataTypeAggState function get failed, type={}",
do_get_name());
}
+if (!BeExecVersionManager::check_be_exec_version(be_exec_version)) {
+LOG(WARNING) << "meet old agg-state, be_exec_version=" <<
be_exec_version;
+}
_agg_function->set_version(be_exec_version);
_agg_serialized_type = _agg_function->get_serialized_type();
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
