This is an automated email from the ASF dual-hosted git repository.
starocean999 pushed a change to branch dev_rec
in repository https://gitbox.apache.org/repos/asf/doris.git
discard cfbca67a303 update fe part
discard 35c6846f920 update fe
discard 7e19594e325 update fe code
discard b66b3a8f2e8 fix fe plan
discard 559ba4fa8fe fix some bug in fe
discard ec953dcb469 recursive cte fe part
discard 4baea2668d2 add TRecCTEScanNode
discard a8ab9095c7b add result_expr_lists to TRecCTENode
discard ae1bd108fef update thrift
discard ce7d354a0c4 update thrift
discard dda76b4bc81 add rec cte thrift struct
add cb1e3bd9f8c [feat](nereids) opt condition expression in filter / join
/ if /case when (#57025)
add 69f82ad67db [Feature](variable) Support session variable lc_time_names
(#56915)
add bd8a44f2db0 [fix](case) fix case bug (#56871)
add d9daa81c18e [chore](regression) add conf for run ccr cases in
regression (#57222)
add 8f22466b6f8 [opt](inverted index) default index format uses v3 (#57140)
add 3eca43c56a3 [Chore](compile) fix compile fail coz
base64_encode_part_num (#57225)
add b9bac89b66f [improve](sort) opt heap sort without runtime predicate
(#57149)
add 6719ea0f144 [chore](checker) Add skeleton for mvcc meta checker
(#57174)
add 5a9cb5aa255 [optimize](function) modify to_base64, from_base64, unhex
to avoid a extra copy (#57182)
add 1553bc61dc8 [fix](outfile) fix analysis error when specifying parquet
schema (#57232)
add dce846d5576 [Fix](function) Fix wrong nullable of floor/ceil functions
(#57238)
add 4a4c2b223de [chore](cloud) Log the large value info (#57237)
add c4193bdb676 [feat](regression) Add dockers to manage multi docker
clusters (#57236)
add 72385a5f364 [fix](cloud) Change CloudGlobalTransactionMgr some log to
debug level (#57168)
add 6ee5dae4950 [fix](cloud) Fix cloud balance warm up lost idx file
(#57114)
add 641b3861ee4 [Fix](job) add thread num config for streaming task exec
(#57230)
add 8e919ba0aa1 [fix](olap) Crash caused by out-of-bounds PODArray access
(#57239)
add 324fd59bd09 [opt](nereids) print slotReference in explain verbose
(#57167)
add 0b20f0313ee [improvement](tvf load)add the data_{1..200}.csv wildcard
in tvf load (#56705)
add d739136485c [fix](paimon)Support user-defined S3 config prefixes and
unify to HDFS S3A protocol (#57116)
add 95c55bbc217 [fix](case) make case stable (#57040)
add e1884e4d323 [fix](cloud) fix correct pipeline task number calculation
in cloud mode (#57129)
add 08a5143b8c1 [opt](arm) Improve BlockBloomFilter performance with NEON
intrinsics (#57183)
add 75d45605d59 [opt](rowset) Remote fetch rowsets to avoid -230 error
when capturing rowsets (#52995)
add 3c8cc786ba1 [External](parquet) pass non predicates column's offset
index to RowGroupReader (#55795)
add 008b51bb157 [Bug](memtracker) fix wrong memtracker scope on
PipelineTask destructor (#57276)
add 821aae2e769 [feat](deps) add hadoop 3.4.2 (#57181)
add fad1907e74b [chore](cloud) Add find_derived_instance_ids function
(#57274)
add 890c778bacb [feature](maxcompute)support maxcompte catalog read
project-schema-table. (#57012)
add 4f00ca1cbad [feature](analyzer) support * as json_object's parameter
(#57256)
new 27e65644219 add rec cte thrift struct
new 2752b757ffc update thrift
new 5382755d725 update thrift
new a83d5dfe266 add result_expr_lists to TRecCTENode
new f850db41e9e add TRecCTEScanNode
new dbd56506e3c recursive cte fe part
new 839cec06dc6 fix some bug in fe
new d23c1c2ae27 fix fe plan
new 9177b565fca update fe code
new ac0ee244300 update fe
new 15e2609bc98 update fe part
new 4c06602b742 update fe
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (cfbca67a303)
\
N -- N -- N refs/heads/dev_rec (4c06602b742)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/build-thirdparty.yml | 6 +
be/benchmark/benchmark_bit_pack.hpp | 8 +-
be/benchmark/benchmark_block_bloom_filter.hpp | 97 ++++
be/benchmark/benchmark_main.cpp | 2 +
be/benchmark/benchmark_string.hpp | 394 +++++++++++++
be/src/cloud/cloud_full_compaction.cpp | 5 +-
be/src/cloud/cloud_internal_service.cpp | 72 ++-
be/src/cloud/cloud_schema_change_job.cpp | 18 +-
be/src/cloud/cloud_tablet.cpp | 107 ++--
be/src/cloud/cloud_tablet.h | 28 +-
be/src/cloud/cloud_tablet_mgr.cpp | 17 +-
be/src/cloud/cloud_tablet_mgr.h | 2 +-
be/src/common/config.cpp | 1 +
be/src/common/config.h | 1 +
be/src/exprs/block_bloom_filter.hpp | 109 ++--
be/src/exprs/block_bloom_filter_avx_impl.cc | 8 +-
be/src/exprs/block_bloom_filter_impl.cc | 58 +-
be/src/io/cache/block_file_cache_downloader.cpp | 16 +-
be/src/io/fs/azure_obj_storage_client.cpp | 2 +-
be/src/olap/base_tablet.cpp | 41 +-
be/src/olap/base_tablet.h | 85 ++-
be/src/olap/full_compaction.cpp | 5 +-
be/src/olap/merger.cpp | 8 +-
be/src/olap/parallel_scanner_builder.cpp | 43 +-
be/src/olap/parallel_scanner_builder.h | 9 +-
be/src/olap/rowset/rowset_reader_context.h | 2 +-
.../inverted_index/tokenizer/tokenizer.h | 1 -
be/src/olap/rowset_version_mgr.cpp | 449 ++++++++++++++
be/src/olap/schema_change.cpp | 17 +-
be/src/olap/snapshot_manager.cpp | 19 +-
be/src/olap/tablet.cpp | 99 +---
be/src/olap/tablet.h | 13 +-
be/src/olap/tablet_meta.cpp | 47 +-
be/src/olap/tablet_meta.h | 12 +
be/src/olap/tablet_reader.cpp | 15 +-
be/src/olap/tablet_reader.h | 16 +-
be/src/olap/task/engine_checksum_task.cpp | 12 +-
be/src/olap/task/engine_storage_migration_task.cpp | 7 +-
be/src/pipeline/exec/olap_scan_operator.cpp | 17 +-
be/src/pipeline/exec/olap_scan_operator.h | 2 +-
be/src/pipeline/exec/sort_sink_operator.cpp | 2 +-
be/src/pipeline/pipeline_task.cpp | 19 +-
be/src/runtime/runtime_state.cpp | 1 +
be/src/runtime/runtime_state.h | 2 +
be/src/service/internal_service.cpp | 68 +++
be/src/service/internal_service.h | 5 +
be/src/vec/common/sort/heap_sorter.cpp | 64 +-
be/src/vec/common/sort/heap_sorter.h | 13 +-
be/src/vec/common/sort/sorter.cpp | 21 +-
be/src/vec/common/sort/sorter.h | 2 +-
be/src/vec/core/sort_cursor.h | 5 +
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 20 +-
be/src/vec/exec/scan/olap_scanner.cpp | 41 +-
be/src/vec/exec/scan/olap_scanner.h | 2 +-
be/src/vec/functions/date_time_transforms.h | 15 +-
.../function_date_or_datetime_to_string.cpp | 78 ++-
.../vec/functions/function_datetime_floor_ceil.cpp | 2 +-
.../vec/functions/function_other_types_to_date.cpp | 2 +-
be/src/vec/functions/function_string.cpp | 142 ++---
be/src/vec/olap/vcollect_iterator.cpp | 2 +-
be/src/vec/runtime/vdatetime_value.cpp | 34 ++
be/src/vec/runtime/vdatetime_value.h | 4 +
.../cloud/cloud_tablet_query_prefer_cache_test.cpp | 14 +-
.../cloud_tablet_query_with_tolerance_test.cpp | 12 +-
be/test/olap/segcompaction_mow_test.cpp | 3 +-
be/test/vec/function/function_string_test.cpp | 19 +-
cloud/src/common/config.h | 1 +
cloud/src/meta-store/keys.cpp | 51 ++
cloud/src/meta-store/keys.h | 5 +
cloud/src/meta-store/meta_reader.cpp | 37 ++
cloud/src/meta-store/meta_reader.h | 5 +
cloud/src/meta-store/txn_kv.cpp | 32 +
cloud/src/recycler/meta_checker.cpp | 526 +++++++++--------
cloud/src/recycler/meta_checker.h | 81 ++-
cloud/src/recycler/recycler_service.cpp | 5 +-
cloud/src/snapshot/snapshot_manager.cpp | 4 +
cloud/src/snapshot/snapshot_manager.h | 3 +
cloud/test/keys_test.cpp | 92 +++
cloud/test/meta_reader_test.cpp | 97 ++++
docker/runtime/doris-compose/Readme.md | 56 ++
docker/runtime/doris-compose/cluster.py | 86 ++-
docker/runtime/doris-compose/command.py | 116 +++-
docker/runtime/doris-compose/resource/common.sh | 37 ++
docker/runtime/doris-compose/resource/init_be.sh | 6 +-
.../runtime/doris-compose/resource/init_cloud.sh | 35 +-
docker/runtime/doris-compose/resource/init_fe.sh | 17 +-
docker/runtime/doris-compose/utils.py | 6 +-
.../main/java/org/apache/doris/common/Config.java | 9 +-
.../org/apache/doris/analysis/OutFileClause.java | 17 +-
.../org/apache/doris/analysis/SlotDescriptor.java | 41 +-
.../apache/doris/cloud/catalog/CloudReplica.java | 16 +
.../cloud/datasource/CloudInternalCatalog.java | 6 +-
.../doris/cloud/system/CloudSystemInfoService.java | 11 +-
.../transaction/CloudGlobalTransactionMgr.java | 18 +-
.../apache/doris/common/util/PropertyAnalyzer.java | 12 +-
.../java/org/apache/doris/common/util/S3Util.java | 88 +++
.../maxcompute/MaxComputeExternalCatalog.java | 104 ++--
.../maxcompute/MaxComputeExternalTable.java | 19 +-
.../maxcompute/MaxComputeSchemaCacheValue.java | 8 +-
.../datasource/maxcompute/McStructureHelper.java | 221 +++++++
.../maxcompute/source/MaxComputeScanNode.java | 4 +-
.../property/constants/MCProperties.java | 20 +
.../metastore/AbstractPaimonProperties.java | 49 +-
.../IcebergFileSystemMetaStoreProperties.java | 4 +-
.../metastore/IcebergHMSMetaStoreProperties.java | 3 -
.../property/metastore/IcebergRestProperties.java | 7 +-
.../PaimonAliyunDLFMetaStoreProperties.java | 6 +-
.../PaimonFileSystemMetaStoreProperties.java | 7 +-
.../metastore/PaimonHMSMetaStoreProperties.java | 6 +-
.../metastore/PaimonRestMetaStoreProperties.java | 2 +-
.../storage/AbstractS3CompatibleProperties.java | 41 --
.../property/storage/AzureProperties.java | 7 +
.../property/storage/BrokerProperties.java | 8 +
.../property/storage/HdfsCompatibleProperties.java | 7 +
.../property/storage/LocalProperties.java | 6 +
.../property/storage/StorageProperties.java | 62 +-
.../org/apache/doris/fs/obj/AzureObjStorage.java | 2 +-
.../java/org/apache/doris/fs/obj/S3ObjStorage.java | 2 +-
.../apache/doris/fs/remote/dfs/DFSFileSystem.java | 3 +-
.../job/scheduler/StreamingTaskScheduler.java | 8 +-
.../doris/nereids/glue/LogicalPlanAdapter.java | 2 +-
.../glue/translator/PhysicalPlanTranslator.java | 4 +-
.../glue/translator/PlanTranslatorContext.java | 6 +-
.../jobs/rewrite/TopDownVisitorRewriteJob.java | 2 -
.../doris/nereids/parser/LogicalPlanBuilder.java | 34 +-
.../doris/nereids/rules/analysis/AnalyzeCTE.java | 4 +-
.../nereids/rules/analysis/BindExpression.java | 10 +-
.../nereids/rules/analysis/ExpressionAnalyzer.java | 41 +-
.../nereids/rules/analysis/SubqueryToApply.java | 3 +-
.../rules/expression/ExpressionOptimization.java | 4 +-
.../rules/expression/ExpressionRewrite.java | 58 +-
.../rules/expression/ExpressionRewriteContext.java | 28 +
.../rules/expression/ExpressionRuleType.java | 2 +-
.../rules/CaseWhenToCompoundPredicate.java | 64 +-
...WhenToIf.java => CondReplaceNullWithFalse.java} | 48 +-
.../rules/expression/rules/ConditionRewrite.java | 183 ++++++
.../expression/rules/NullSafeEqualToEqual.java | 49 +-
.../rules/ReplaceNullWithFalseForCond.java | 131 -----
.../rules/SimplifyArithmeticComparisonRule.java | 2 +-
...eChildToPhysicalRecursiveCteRecursiveChild.java | 1 +
.../doris/nereids/rules/rewrite/CTEInline.java | 22 +-
.../nereids/rules/rewrite/ConstantPropagation.java | 78 +--
.../nereids/rules/rewrite/EliminateFilter.java | 14 +-
.../rules/rewrite/PushProjectIntoUnion.java | 2 +-
.../trees/copier/LogicalPlanDeepCopier.java | 2 +-
.../executable/DateTimeExtractAndTransform.java | 28 +-
.../logical/LogicalRecursiveCteRecursiveChild.java | 27 +-
.../plans/logical/LogicalRecursiveCteScan.java | 2 +-
.../nereids/trees/plans/logical/LogicalUnion.java | 2 +-
.../trees/plans/physical/PhysicalRecursiveCte.java | 4 +
.../PhysicalRecursiveCteRecursiveChild.java | 28 +-
.../org/apache/doris/planner/RecursiveCteNode.java | 8 +-
.../main/java/org/apache/doris/qe/Coordinator.java | 1 +
.../org/apache/doris/qe/CoordinatorContext.java | 1 +
.../java/org/apache/doris/qe/SessionVariable.java | 54 ++
.../doris/qe/runtime/ThriftPlansBuilder.java | 22 +-
.../apache/doris/service/FrontendServiceImpl.java | 47 +-
.../org/apache/doris/system/SystemInfoService.java | 14 +-
.../cloud/system/CloudSystemInfoServiceTest.java | 645 ++++++++++++++++++++-
.../apache/doris/common/PropertyAnalyzerTest.java | 90 +--
.../org/apache/doris/common/util/S3UtilTest.java | 252 ++++++++
.../metastore/AbstractPaimonPropertiesTest.java | 89 +++
.../PaimonRestMetaStorePropertiesTest.java | 14 +-
.../parser/UnboundFunctionWithUnboundStarTest.java | 106 ++++
.../nereids/rules/analysis/AnalyzeCTETest.java | 148 +++++
.../rules/analysis/ExpressionAnalyzerTest.java | 36 ++
.../expression/ExpressionRewriteTestHelper.java | 3 +-
.../rules/CaseWhenToCompoundPredicateTest.java | 28 +
.../rules/CondReplaceNullWithFalseTest.java | 145 +++++
.../expression/rules/ExpressionRewriteSqlTest.java | 2 +-
.../expression/rules/NullSafeEqualToEqualTest.java | 54 +-
.../rules/ReplaceNullWithFalseForCondTest.java | 110 ----
.../doris/nereids/rules/rewrite/CTEInlineTest.java | 84 +++
.../nereids/rules/rewrite/ColumnPruningTest.java | 30 +
.../rules/rewrite/ConstantPropagationTest.java | 25 +-
.../apache/doris/system/SystemInfoServiceTest.java | 61 ++
gensrc/proto/internal_service.proto | 17 +
gensrc/thrift/PaloInternalService.thrift | 3 +
regression-test/conf/regression-conf.groovy | 6 +
.../sql-functions/doc_date_functions_test.out | 351 +++++++++++
...ty_data.out => test_outfile_parquet_schema.out} | 10 +-
.../data/external_table_p0/tvf/hdfs_data_1.txt | 1 +
.../data/external_table_p0/tvf/hdfs_data_2.txt | 1 +
.../data/external_table_p0/tvf/hdfs_data_3.txt | 1 +
.../tvf/test_s3_tvf_number_range.out | 57 ++
.../maxcompute/test_max_compute_schema.out | 258 +++++++++
...n.out => test_cloud_version_already_merged.out} | 8 +-
.../test_storage_format_default.out} | 2 +-
.../json_function/test_query_json_object.out | 16 +-
.../extend_infer_equal_predicate.out | 8 +-
.../data/query_p0/system/test_table_properties.out | 12 +-
.../test_show_create_table_and_views_nereids.out | 8 +-
.../org/apache/doris/regression/Config.groovy | 1 +
.../org/apache/doris/regression/suite/Suite.groovy | 138 ++++-
.../doris/regression/suite/SuiteCluster.groovy | 43 ++
regression-test/pipeline/p0/conf/be.conf | 1 +
.../cloud_p0/balance/test_balance_warm_up.groovy | 86 ++-
.../suites/correctness_p0/test_time_round.groovy | 6 +
.../suites/demo_p0/test_external_ms_cluster.groovy | 83 +++
.../sql-functions/doc_date_functions_test.groovy | 454 +++++++++++++++
.../outfile/test_outfile_parquet_schema.groovy | 93 +++
.../tvf/test_s3_tvf_number_range.groovy | 398 +++++++++++++
.../maxcompute/test_max_compute_schema.groovy | 239 ++++++++
.../cloud/test_cloud_version_already_merged.groovy | 142 +++++
.../test_index_change_on_renamed_column.groovy | 2 +-
...est_cumulative_compaction_with_format_v2.groovy | 2 +-
.../test_mor_table_with_format_v2.groovy | 4 +-
.../test_mow_table_with_format_v2.groovy | 2 +-
.../test_storage_format_default.groovy | 91 +++
.../test_ingestion_load_alter_column.groovy | 4 +-
.../datatype/test_date_implicit_cast.groovy | 8 +-
.../json_function/test_query_json_object.groovy | 5 +-
.../suites/nereids_syntax_p0/explain.groovy | 2 +-
.../suites/query_p0/explain/explain_alias.groovy | 56 ++
.../predefine/test_multi_index_file.groovy | 2 +-
.../variant_p0/predefine/variant_with_mow.groovy | 4 +-
thirdparty/CHANGELOG.md | 7 +
thirdparty/build-thirdparty.sh | 120 +++-
thirdparty/download-thirdparty.sh | 24 -
thirdparty/patches/googletest-release-1.11.0.patch | 71 ---
thirdparty/patches/libevent-1532.patch | 188 ------
.../libevent-keepalive-accepted-socket.patch | 17 -
thirdparty/patches/libevent.patch | 193 ------
thirdparty/vars.sh | 39 +-
224 files changed, 8569 insertions(+), 2253 deletions(-)
create mode 100644 be/benchmark/benchmark_block_bloom_filter.hpp
create mode 100644 be/benchmark/benchmark_string.hpp
create mode 100644 be/src/olap/rowset_version_mgr.cpp
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/McStructureHelper.java
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/{CaseWhenToIf.java
=> CondReplaceNullWithFalse.java} (50%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/ConditionRewrite.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/ReplaceNullWithFalseForCond.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/common/util/S3UtilTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/AbstractPaimonPropertiesTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/UnboundFunctionWithUnboundStarTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/CondReplaceNullWithFalseTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/ReplaceNullWithFalseForCondTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/CTEInlineTest.java
copy regression-test/data/export_p0/outfile/{csv/test_outfile_empty_data.out
=> test_outfile_parquet_schema.out} (54%)
create mode 100644 regression-test/data/external_table_p0/tvf/hdfs_data_1.txt
create mode 100644 regression-test/data/external_table_p0/tvf/hdfs_data_2.txt
create mode 100644 regression-test/data/external_table_p0/tvf/hdfs_data_3.txt
create mode 100644
regression-test/data/external_table_p0/tvf/test_s3_tvf_number_range.out
create mode 100644
regression-test/data/external_table_p2/maxcompute/test_max_compute_schema.out
copy
regression-test/data/fault_injection_p0/cloud/{test_cloud_sc_self_retry_with_stop_token.out
=> test_cloud_version_already_merged.out} (75%)
copy regression-test/data/{correctness/test_crossjoin_inlineview_slot.out =>
inverted_index_p0/storage_format/test_storage_format_default.out} (96%)
create mode 100644
regression-test/suites/demo_p0/test_external_ms_cluster.groovy
create mode 100644
regression-test/suites/export_p0/outfile/test_outfile_parquet_schema.groovy
create mode 100644
regression-test/suites/external_table_p0/tvf/test_s3_tvf_number_range.groovy
create mode 100644
regression-test/suites/external_table_p2/maxcompute/test_max_compute_schema.groovy
create mode 100644
regression-test/suites/fault_injection_p0/cloud/test_cloud_version_already_merged.groovy
create mode 100644
regression-test/suites/inverted_index_p0/storage_format/test_storage_format_default.groovy
create mode 100644 regression-test/suites/query_p0/explain/explain_alias.groovy
delete mode 100644 thirdparty/patches/googletest-release-1.11.0.patch
delete mode 100644 thirdparty/patches/libevent-1532.patch
delete mode 100644 thirdparty/patches/libevent-keepalive-accepted-socket.patch
delete mode 100644 thirdparty/patches/libevent.patch
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
