This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch auto-pick-55544-branch-4.0
in repository https://gitbox.apache.org/repos/asf/doris.git
discard ebead07123e [feat](txn) Add txn write amplification brpc metrics
(#55544)
omit e031d4e6ae5 4.0.0-rc02
add f4d164d547d [fix](nereids)prevent null exception in ShowTableCommand
(#56239)
add 0fbab818735 [Fix](s3) Fix hdfs will throw an exception if the URL
verification fails (#56268)
add a793b19ef35 [fix](es) Handle object type in Elasticsearch mapping when
table is empty (#56152)
add fca7aff86ef [fix](olap) A crash caused by an incorrect storage type in
the IN predicate (#56309)
add ebd27d8e883 [fix](github) force use text diff for .out files (#56330)
add 2148a6d60b3 [fix] (agg_strategy) fix result wrong when the
multi_distinct_func and count distinct multi expr exists same time (#56271)
add 8470c2f4151 [feature](info) Append RunningTasks into BE info (#56277)
add cd475b224b1 [opt](paimon) Optimize Paimon Time Travel Implementation
and Fix Schema Consistency Issues (#56023)
add af71338d043 [fix](cloud) recycle orphan partitions (#56306)
add db09cef4009 [Chore](errmsg) Avoid extra error stacktrace for error in
scanner (#56335)
add 616eba6d050 [opt](inverted index) add debug logs (#56280)
add 4fc6bdb569e [fix](function) fix convert_tz function fe time_zone range
problem (#56228)
add 8fad28d108c [fix](function) A crash caused by nullptr in
json_exists_path (#56331)
add 441d45f7490 [fix](inverted index) fix boolean query for NOT operator
(#56329)
add 8b1618fea18 [Chore](errmsg) refine error message of str_to_date
(#56349)
add f66069adf9e [fix](ci) performance enable_sql_cache=false (#56285)
add cfd990d0775 [ci](branch-4.0) adjust pipeline for 4.0 (#56296)
add b022236337e [log](minor) Add debug logs (#56339)
add 840a32aad99 [chore](test)Icebebrg test uses doris's fileio (#56193)
add 54f7b735a7f [opt](query-progress) add more log to debug (#56333)
add ec2a6f63f1f [opt](iceberg) opt the error msg of iceberg scan node
(#56327)
add a39b6433da3 [chore](cloud) Add check snapshot for checker (#56290)
add c73d225f396 [feat](streaming job) Introduce streaming job for
incremental load (#56175)
add a4a4c4a34c0 [feat](txn) Add txn write amplification brpc metrics
(#55544)
add 6aa8e6ef268 [feat](snapshot) clone instance return image url (#56310)
add 2ac4fd5c1b1 [feat](snapshot) modify cloud snapshot commands (#56293)
add 73c006aab59 [metrics](warmup) add bvar for skipped warmup rowsets
(#56301)
add be3ba367a22 [opt](warmup) make warm_up_rowset async (#55236)
add 011b55b3495 [fix](filecache) microbench crash when FileBufferBuilder
(#56204)
add 01451ada116 [fix](cloud) Rowset meta save reference instance id
(#56291)
add 57bd84a4f21 [fix](cloud) create_instance notify refresh instance
(#56341)
add 8274895972f [fix](load) fix s3 load check failed when no source file
found (#56319)
add 1c7c7b2400e [improve](bin) do not chmod if current user has x and r
privilige (#56351)
add 35a06509319 [fix](test) Fix recycler test for hdfs and kerberos config
(#56357)
add 7946376c74f [fix](case) bug (#56364)
add 9840b20a0a7 [fix](case) bug with same table in same db (#56363)
add 110543e8c01 [fix](case) bug (#56362)
add 228e5fc62fe [fix](case) fix ingestion load cases (#55361) (#56359)
add 319bc7d5f6d [chore](jsonb) Remove unused code from ObjectVal (#56334)
add 54a81965e03 [chore](type) remove to_string from DataType part I
(#56197)
add 8b18f2d5b05 [ci](fix) fix 4.0.x auto pick (#56385)
add 8ee6089278b [Refactor](util) Remove date_cast utils (#56343)
add a076fea62af [feature](function) support MySQL dialect of function
substr and position (#55914)
add fbe93b92186 [Chore](log) Fix wrong warning logs (#56394)
add edfe2171885 [fix](nereids) tpcds query64 shape unstable (#55920)
add c89de1cfefa [fix](test) add order by items in test_json_function
(#56347)
add 5edc39ab0a2 [fix](routine load) delete redundant thrift field (#56302)
add 73c1bf14b76 4.0.0-rc02
add 51435d1363f branch-4.0: [fix](test) fix paimon dlf rest catalog case
#56381 (#56422)
add 43b9fd6876b branch-4.0: [fix](mysql) optimize some error handling
logic #56416 (#56455)
add 358d5523ff1 branch-4.0: [feat](oss)Support DeltaLake VPC Endpoint
Format (dlf / deltalake-vpc) #56473 (#56477)
add f9a46e9afa2 branch-4.0: [feat](catalog)AWS Glue supports S3 access via
IAM AssumeRole. #56311 (#56418)
add 3d03ecb8750 branch-4.0: [Bug](datetype) add PGenericType for varbinary
type #56375 (#56404)
add 016f6756b5d branch-4.0: [fix](checker) Txn is old due to scan too much
kv #56419 (#56474)
add 3e8d8313ecd branch-4.0: [refactor](compress) Unify the decompressin
error msg #56299 (#56480)
add 44ee3b7d104 branch-4.0: [Fix](warmup) Fix coredump in
`CloudTablet::complete_rowset_segment_warmup` due to capture by reference
#56395 (#56458)
add c5b8626b968 branch-4.0: [Opt](cloud) use unordered_map to reduce meta
lock held time in `delete_expired_stale_rowsets` #53467 (#56475)
add f2d5f2aa161 branch-4.0: [fix](outfile) fix small file output with bz2
compression #56368 (#56495)
add ccbc1e26f56 branch-4.0: [fix](vcg) Fix vcg regression case due to pick
code #56411 (#56503)
add a63f3729c82 [fix](top rf) fix external table top rf lack of
consideration of nulls first conditions. (#56392) (#56489)
add 9bf402d86af branch-4.0: [fix](inverted index) support dict_compression
configuration for variant type #56399 (#56435)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (ebead07123e)
\
N -- N -- N refs/heads/auto-pick-55544-branch-4.0 (9bf402d86af)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.gitattributes | 2 +-
.github/workflows/auto-cherry-pick.yml | 2 +-
.github/workflows/comment-to-trigger-teamcity.yml | 4 +
be/src/agent/task_worker_pool.cpp | 9 +-
be/src/cloud/cloud_meta_mgr.cpp | 36 +-
be/src/cloud/cloud_meta_mgr.h | 5 +
be/src/cloud/cloud_schema_change_job.cpp | 14 +-
be/src/cloud/cloud_snapshot_mgr.cpp | 2 +-
be/src/cloud/cloud_tablet.cpp | 11 +-
be/src/cloud/cloud_warm_up_manager.cpp | 47 +-
be/src/cloud/cloud_warm_up_manager.h | 6 +
be/src/cloud/config.cpp | 2 +
be/src/cloud/config.h | 2 +
be/src/cloud/pb_convert.cpp | 12 +
be/src/common/config.cpp | 2 +-
be/src/common/config.h | 2 +-
be/src/exec/decompressor.cpp | 36 +-
be/src/exec/schema_scanner.cpp | 6 +
.../schema_cluster_snapshot_properties_scanner.cpp | 86 ++++
.../schema_cluster_snapshot_properties_scanner.h} | 39 +-
.../schema_cluster_snapshots_scanner.cpp | 265 +++++++++++
.../schema_cluster_snapshots_scanner.h} | 36 +-
.../exec/schema_scanner/schema_tablets_scanner.cpp | 11 +-
be/src/io/tools/file_cache_microbench.cpp | 88 +++-
be/src/olap/base_tablet.cpp | 27 +-
be/src/olap/cumulative_compaction_policy.cpp | 8 +-
be/src/olap/cumulative_compaction_policy.h | 5 +-
.../cumulative_compaction_time_series_policy.cpp | 8 +-
.../cumulative_compaction_time_series_policy.h | 3 +-
be/src/olap/data_dir.cpp | 4 +-
be/src/olap/in_list_predicate.h | 6 +
be/src/olap/olap_server.cpp | 2 +-
be/src/olap/rowset/rowset_meta.h | 3 +-
.../query_v2/boolean_query/boolean_weight.h | 83 +++-
.../inverted_index/query_v2/composite_reader.h | 25 +-
.../query_v2/intersection_scorer.cpp | 75 ++-
.../inverted_index/query_v2/intersection_scorer.h | 21 +-
.../query_v2/match_all_docs_scorer.h | 98 ++++
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 11 +-
be/src/olap/tablet.cpp | 25 +-
be/src/olap/tablet.h | 2 +-
be/src/olap/tablet_meta.cpp | 107 ++---
be/src/olap/tablet_meta.h | 36 +-
be/src/olap/task/engine_clone_task.cpp | 2 +-
be/src/olap/version_graph.cpp | 199 +-------
be/src/olap/version_graph.h | 225 ++++++++-
be/src/pipeline/exec/exchange_source_operator.cpp | 11 +-
be/src/runtime/exec_env.h | 2 +-
be/src/runtime/exec_env_init.cpp | 4 +-
be/src/runtime/fragment_mgr.cpp | 8 +-
be/src/runtime/fragment_mgr.h | 1 +
be/src/runtime/query_context.cpp | 2 +
.../routine_load/routine_load_task_executor.cpp | 20 +-
be/src/util/block_compression.cpp | 5 +-
be/src/util/datetype_cast.hpp | 114 -----
be/src/util/jsonb_document.cpp | 6 +-
be/src/util/jsonb_document.h | 92 +---
be/src/util/quantile_state.cpp | 2 +-
be/src/util/quantile_state.h | 2 +-
be/src/vec/data_types/data_type.cpp | 33 +-
be/src/vec/data_types/data_type.h | 16 +-
be/src/vec/data_types/data_type_agg_state.h | 13 -
be/src/vec/data_types/data_type_array.cpp | 57 ---
be/src/vec/data_types/data_type_array.h | 3 -
be/src/vec/data_types/data_type_bitmap.cpp | 12 -
be/src/vec/data_types/data_type_bitmap.h | 10 -
be/src/vec/data_types/data_type_date.cpp | 35 --
be/src/vec/data_types/data_type_date.h | 11 +-
.../data_types/data_type_date_or_datetime_v2.cpp | 78 ---
.../vec/data_types/data_type_date_or_datetime_v2.h | 21 +-
be/src/vec/data_types/data_type_date_time.cpp | 39 --
be/src/vec/data_types/data_type_date_time.h | 13 +-
be/src/vec/data_types/data_type_decimal.cpp | 67 ---
be/src/vec/data_types/data_type_decimal.h | 7 +-
be/src/vec/data_types/data_type_factory.cpp | 3 +
be/src/vec/data_types/data_type_hll.cpp | 14 -
be/src/vec/data_types/data_type_hll.h | 3 -
be/src/vec/data_types/data_type_ipv4.cpp | 28 --
be/src/vec/data_types/data_type_ipv4.h | 10 -
be/src/vec/data_types/data_type_ipv6.cpp | 25 -
be/src/vec/data_types/data_type_ipv6.h | 11 -
be/src/vec/data_types/data_type_jsonb.cpp | 22 -
be/src/vec/data_types/data_type_jsonb.h | 3 -
be/src/vec/data_types/data_type_map.cpp | 45 --
be/src/vec/data_types/data_type_map.h | 2 -
be/src/vec/data_types/data_type_nullable.cpp | 28 --
be/src/vec/data_types/data_type_nullable.h | 2 -
be/src/vec/data_types/data_type_number.cpp | 57 ---
be/src/vec/data_types/data_type_number.h | 8 -
be/src/vec/data_types/data_type_number_base.cpp | 52 --
be/src/vec/data_types/data_type_number_base.h | 34 +-
be/src/vec/data_types/data_type_quantilestate.cpp | 10 -
be/src/vec/data_types/data_type_quantilestate.h | 5 -
be/src/vec/data_types/data_type_string.cpp | 18 -
be/src/vec/data_types/data_type_string.h | 3 -
be/src/vec/data_types/data_type_struct.cpp | 35 --
be/src/vec/data_types/data_type_struct.h | 2 -
be/src/vec/data_types/data_type_time.cpp | 24 -
be/src/vec/data_types/data_type_time.h | 12 +-
be/src/vec/data_types/data_type_varbinary.cpp | 18 -
be/src/vec/data_types/data_type_varbinary.h | 3 -
be/src/vec/data_types/data_type_variant.cpp | 12 -
be/src/vec/data_types/data_type_variant.h | 2 -
.../data_types/serde/data_type_bitmap_serde.cpp | 11 +
.../vec/data_types/serde/data_type_bitmap_serde.h | 2 +
.../vec/data_types/serde/data_type_hll_serde.cpp | 8 +
be/src/vec/data_types/serde/data_type_hll_serde.h | 2 +
.../data_types/serde/data_type_nullable_serde.cpp | 6 +-
.../serde/data_type_quantilestate_serde.h | 7 +
.../data_types/serde/data_type_varbinary_serde.cpp | 6 +
.../data_types/serde/data_type_varbinary_serde.h | 2 +
be/src/vec/exec/scan/file_scanner.cpp | 11 +-
be/src/vec/exec/scan/olap_scanner.cpp | 9 +-
be/src/vec/exprs/vexpr.h | 3 +-
be/src/vec/exprs/vtopn_pred.h | 45 ++
be/src/vec/functions/function.h | 4 +-
be/src/vec/functions/function_convert_tz.cpp | 123 ++---
.../function_date_or_datetime_to_something.h | 1 -
.../vec/functions/function_datetime_floor_ceil.cpp | 66 ++-
be/src/vec/functions/function_jsonb.cpp | 2 +-
.../vec/functions/function_other_types_to_date.cpp | 401 ++++++----------
be/src/vec/functions/function_string.cpp | 1 +
be/src/vec/functions/simple_function_factory.h | 5 +-
be/src/vec/olap/vcollect_iterator.cpp | 4 -
be/src/vec/runtime/vdata_stream_recvr.cpp | 27 ++
be/src/vec/runtime/vdata_stream_recvr.h | 2 +
be/src/vec/sink/vdata_stream_sender.cpp | 13 +
be/src/vec/sink/vdata_stream_sender.h | 1 +
be/test/cloud/cloud_meta_mgr_test.cpp | 22 +-
...ma_cluster_snapshot_properties_scanner_test.cpp | 50 ++
.../schema_cluster_snapshots_scanner_test.cpp | 67 +++
.../compaction_delete_bitmap_calculator_test.cpp | 4 +-
be/test/olap/delete_handler_test.cpp | 2 +-
.../inverted_index/query_v2/boolean_query_test.cpp | 78 +++
.../operator/streaming_agg_operator_test.cpp | 6 +-
be/test/vec/data_types/data_type_bitmap_test.cpp | 49 --
be/test/vec/data_types/data_type_hll_test.cpp | 49 --
.../data_types/data_type_quantile_state_test.cpp | 32 --
be/test/vec/data_types/data_type_struct_test.cpp | 2 +-
.../serde/data_type_from_string_test.cpp | 6 +-
.../data_types/serde/data_type_to_string_test.cpp | 8 +-
.../serde/data_type_write_to_jsonb_test.cpp | 12 +-
be/test/vec/function/cast/cast_to_string.cpp | 2 +-
be/test/vec/function/function_test_util.cpp | 30 +-
be/test/vec/function/function_time_test.cpp | 45 +-
be/test/vec/jsonb/jsonb_document_test.cpp | 66 +++
bin/start_be.sh | 6 +-
cloud/script/start.sh | 4 +-
cloud/src/common/bvars.cpp | 5 +
cloud/src/common/bvars.h | 3 +
cloud/src/common/config.h | 1 +
cloud/src/meta-service/meta_service.h | 13 +
cloud/src/meta-service/meta_service_resource.cpp | 59 ++-
cloud/src/meta-service/meta_service_txn.cpp | 142 ++++++
cloud/src/meta-store/keys.cpp | 13 +-
cloud/src/meta-store/keys.h | 7 +
cloud/src/meta-store/mem_txn_kv.cpp | 4 +-
cloud/src/recycler/checker.cpp | 40 +-
cloud/src/recycler/checker.h | 9 +
cloud/src/recycler/checker_snapshot.cpp | 59 +++
cloud/src/recycler/recycler.cpp | 36 +-
cloud/src/snapshot/snapshot_manager.cpp | 9 +
cloud/src/snapshot/snapshot_manager.h | 6 +
cloud/test/keys_test.cpp | 52 ++
cloud/test/meta_service_job_test.cpp | 224 +++++++++
cloud/test/recycle_versioned_keys_test.cpp | 148 +++---
cloud/test/recycler_test.cpp | 57 +++
.../elasticsearch/scripts/es_init.sh | 2 +
.../scripts/index/es7_test_object.json | 16 +
.../scripts/create_preinstalled_scripts/run81.hql | 48 ++
.../create_preinstalled_scripts/paimon/run09.sql | 24 +-
.../main/java/org/apache/doris/common/Config.java | 5 +
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 4 +
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 27 +-
.../ConfigurationAWSCredentialsProvider.java | 23 +-
.../amazonaws/glue/catalog/util/AWSGlueConfig.java | 2 +
.../apache/doris/analysis/InvertedIndexUtil.java | 2 +-
.../org/apache/doris/analysis/SchemaTableType.java | 5 +-
.../doris/catalog/BuiltinScalarFunctions.java | 2 +-
.../main/java/org/apache/doris/catalog/Env.java | 7 +-
.../java/org/apache/doris/catalog/SchemaTable.java | 22 +
.../cloud/catalog/CloudInstanceStatusChecker.java | 35 +-
.../apache/doris/cloud/rpc/MetaServiceClient.java | 6 +
.../apache/doris/cloud/rpc/MetaServiceProxy.java | 5 +
.../doris/cloud/snapshot/CloudSnapshotHandler.java | 12 +
.../transaction/CloudGlobalTransactionMgr.java | 14 +
.../apache/doris/cloud/transaction/TxnUtil.java | 37 ++
.../apache/doris/common/proc/BackendsProcDir.java | 5 +-
.../doris/common/profile/ProfileManager.java | 25 +-
.../org/apache/doris/datasource/es/EsUtil.java | 2 +
.../datasource/iceberg/source/IcebergScanNode.java | 13 +-
.../datasource/paimon/PaimonExternalTable.java | 27 +-
.../apache/doris/datasource/paimon/PaimonUtil.java | 221 +--------
.../datasource/paimon/source/PaimonScanNode.java | 32 +-
.../doris/datasource/property/ParamRules.java | 51 ++
.../metastore/AWSGlueMetaStoreBaseProperties.java | 18 +-
.../metastore/HMSGlueMetaStoreProperties.java | 16 +-
.../metastore/IcebergGlueMetaStoreProperties.java | 32 +-
.../storage/AbstractS3CompatibleProperties.java | 11 +-
.../property/storage/HdfsPropertiesUtils.java | 12 +-
.../datasource/property/storage/OSSProperties.java | 5 +-
.../datasource/property/storage/S3Properties.java | 26 +-
.../main/java/org/apache/doris/fs/FileSystem.java | 14 +
.../java/org/apache/doris/fs/GlobListResult.java | 57 +++
.../java/org/apache/doris/fs/obj/S3ObjStorage.java | 98 +++-
.../org/apache/doris/fs/remote/S3FileSystem.java | 8 +
.../org/apache/doris/job/base/AbstractJob.java | 20 +-
.../main/java/org/apache/doris/job/base/Job.java | 7 +
.../doris/job/base/JobExecutionConfiguration.java | 19 +-
.../TaskType.java => base/JobProperties.java} | 6 +-
.../org/apache/doris/job/common/FailureReason.java | 71 +++
.../org/apache/doris/job/common/JobStatus.java | 12 +-
.../java/org/apache/doris/job/common/TaskType.java | 3 +-
.../doris/job/executor/DispatchTaskHandler.java | 3 +-
.../doris/job/executor/TimerJobSchedulerTask.java | 3 +-
.../doris/job/extensions/insert/InsertJob.java | 35 +-
.../doris/job/extensions/insert/InsertTask.java | 1 +
.../insert/streaming/StreamingInsertJob.java | 530 +++++++++++++++++++++
.../insert/streaming/StreamingInsertTask.java | 247 ++++++++++
.../insert/streaming/StreamingJobProperties.java | 94 ++++
.../streaming/StreamingJobSchedulerTask.java | 159 +++++++
.../insert/streaming/StreamingJobStatistic.java} | 30 +-
.../StreamingTaskTxnCommitAttachment.java | 80 ++++
.../org/apache/doris/job/manager/JobManager.java | 73 ++-
.../doris/job/manager/StreamingTaskManager.java | 55 +++
.../{common/TaskType.java => offset/Offset.java} | 8 +-
.../doris/job/offset/SourceOffsetProvider.java | 85 ++++
.../job/offset/SourceOffsetProviderFactory.java | 48 ++
.../JobStatus.java => offset/s3/S3Offset.java} | 51 +-
.../job/offset/s3/S3SourceOffsetProvider.java | 188 ++++++++
.../apache/doris/job/scheduler/JobScheduler.java | 6 +-
.../job/scheduler/StreamingTaskScheduler.java | 117 +++++
.../doris/load/loadv2/BrokerLoadPendingTask.java | 2 +
.../apache/doris/load/loadv2/LoadStatistic.java | 8 +
.../doris/load/routineload/KafkaTaskInfo.java | 1 -
.../doris/load/routineload/RoutineLoadJob.java | 4 -
.../org/apache/doris/master/ReportHandler.java | 13 +-
.../java/org/apache/doris/mysql/MysqlProto.java | 19 +-
.../org/apache/doris/nereids/CascadesContext.java | 14 +-
.../org/apache/doris/nereids/NereidsPlanner.java | 7 +-
.../java/org/apache/doris/nereids/memo/Group.java | 10 +-
.../apache/doris/nereids/memo/GroupExpression.java | 2 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 60 ++-
.../nereids/rules/analysis/NormalizeAggregate.java | 19 +-
.../rules/rewrite/DistinctAggregateRewriter.java | 2 +-
.../functions/agg/MultiDistinctCount.java | 30 +-
.../functions/agg/MultiDistinctGroupConcat.java | 26 +-
.../functions/agg/MultiDistinctSum.java | 26 +-
.../functions/agg/MultiDistinctSum0.java | 23 +-
.../functions/agg/MultiDistinction.java | 1 -
.../executable/DateTimeExtractAndTransform.java | 94 ++--
.../expressions/functions/scalar/ConvertTz.java | 6 +-
.../expressions/functions/scalar/DateTrunc.java | 11 +-
.../expressions/functions/scalar/LastDay.java | 7 +-
.../expressions/functions/scalar/MakeDate.java | 4 +-
.../expressions/functions/scalar/StrToDate.java | 16 +-
.../expressions/functions/scalar/ToMonday.java | 6 +-
.../apache/doris/nereids/trees/plans/PlanType.java | 8 +-
...java => AdminCreateClusterSnapshotCommand.java} | 14 +-
...ava => AdminSetAutoClusterSnapshotCommand.java} | 46 +-
...minSetClusterSnapshotFeatureSwitchCommand.java} | 62 +--
.../commands/AdminShowClusterSnapshotCommand.java | 126 -----
.../trees/plans/commands/AlterJobCommand.java | 136 ++++++
.../trees/plans/commands/CreateJobCommand.java | 12 +
.../trees/plans/commands/ResumeJobCommand.java | 9 +-
.../trees/plans/commands/ShowClustersCommand.java | 47 +-
.../trees/plans/commands/ShowTableCommand.java | 3 +-
.../trees/plans/commands/WarmUpClusterCommand.java | 34 +-
.../trees/plans/commands/info/CreateJobInfo.java | 68 ++-
.../commands/insert/InsertIntoTableCommand.java | 11 +
.../plans/commands/insert/OlapInsertExecutor.java | 14 +
.../trees/plans/visitor/CommandVisitor.java | 35 +-
.../org/apache/doris/nereids/types/DataType.java | 14 +
.../org/apache/doris/persist/gson/GsonUtils.java | 9 +-
.../java/org/apache/doris/qe/ConnectProcessor.java | 15 -
.../java/org/apache/doris/qe/SessionVariable.java | 3 +-
.../resource/computegroup/ComputeGroupMgr.java | 10 +-
.../org/apache/doris/statistics/Statistics.java | 10 +-
.../main/java/org/apache/doris/system/Backend.java | 10 +
.../apache/doris/transaction/TransactionState.java | 9 +-
.../doris/datasource/property/ParamRulesTest.java | 34 +-
.../AWSGlueMetaStoreBasePropertiesTest.java | 14 +-
.../datasource/property/metastore/HMSGlueIT.java | 49 ++
.../metastore/HMSGlueMetaStorePropertiesTest.java | 2 +-
.../property/metastore/IcebergGlueIT.java | 51 ++
.../property/storage/OSSPropertiesTest.java | 6 +
.../doris/nereids/parser/NereidsParserTest.java | 126 +++++
.../nereids/rules/expression/FoldConstantTest.java | 41 +-
... => AdminCreateClusterSnapshotCommandTest.java} | 12 +-
...=> AdminSetAutoClusterSnapshotCommandTest.java} | 29 +-
...etClusterSnapshotFeatureSwitchCommandTest.java} | 11 +-
...inShowClusterSnapshotPropertiesCommandTest.java | 101 ----
.../doris/utframe/DemoMultiBackendsTest.java | 8 +-
gensrc/proto/cloud.proto | 29 ++
gensrc/proto/olap_file.proto | 2 +
gensrc/thrift/BackendService.thrift | 1 -
gensrc/thrift/Descriptors.thrift | 2 +
gensrc/thrift/MasterService.thrift | 2 +
.../data/export_p0/test_outfile_csv_compress.out | 60 +++
.../data/external_table_p0/es/test_es_query.out | 12 +
.../hive/test_hive_get_schema_from_table.out | 2 +-
.../hive/test_hive_topn_rf_null.out | 489 +++++++++++++++++++
.../paimon/paimon_time_travel.out | 98 ++++
.../external_table_p0/tvf/test_s3_tvf_globlist.out | 35 ++
.../iceberg/test_glue_rest_s3tables.out | 33 --
...te_insert.out => test_s3tables_glue_insert.out} | 0
...out => test_s3tables_glue_insert_overwrite.out} | 0
...ut => test_s3tables_glue_insert_partitions.out} | 0
.../paimon/test_paimon_dlf_rest_catalog.out | 8 +-
.../streaming_job/test_streaming_insert_job.out | 22 +
.../data/nereids_function_p0/scalar_function/P.out | 174 +++++++
.../data/nereids_function_p0/scalar_function/S.out | 116 +++++
.../fold_constant_string_arithmatic.out | 202 ++++++++
.../join/initial_join_order/initial_join_order.out | 4 +
.../nereids_rules_p0/agg_strategy/agg_strategy.out | 22 +
.../test_in_no_concurrent.out} | 20 +-
.../datetime_functions/test_date_function.out | 4 +-
.../json_functions/test_json_exists_path.out | 5 +
.../shape_check/tpcds_sf100/rf_prune/query72.out | 4 +
.../data/shape_check/tpcds_sf100/shape/query72.out | 4 +
.../shape_check/tpcds_sf1000/shape/query64.out | 135 +++---
.../shape_check/tpcds_sf1000/shape/query72.out | 4 +
regression-test/pipeline/common/teamcity-utils.sh | 5 +-
regression-test/pipeline/performance/compile.sh | 2 +-
.../pipeline/performance/conf/session_variables | 0
.../performance/conf/session_variables.sql | 1 +
regression-test/pipeline/performance/deploy.sh | 3 +
regression-test/pipeline/performance/prepare.sh | 4 +-
.../pipeline/performance/run-clickbench.sh | 6 +-
regression-test/pipeline/performance/run-load.sh | 10 +-
regression-test/pipeline/performance/run-tpcds.sh | 6 +-
regression-test/pipeline/performance/run-tpch.sh | 6 +-
regression-test/pipeline/vault_p0/prepare.sh | 4 +-
.../plugins/cloud_recycler_plugin.groovy | 319 ++++++++++---
.../virtual_compute_group/test_vcg.groovy | 16 +-
.../virtual_compute_group/test_vcg_metrics.groovy | 2 +-
.../suites/cloud_p0/recycler/test_checker.groovy | 108 ++---
.../recycler/test_recycler_inverted_index.groovy | 2 +-
.../cloud_p0/snapshot/test_snapshot_command.groovy | 25 +-
.../suites/correctness/test_str_to_date.groovy | 4 +
.../suites/correctness/test_timev2_fold.groovy | 8 +
.../suites/export/test_array_export.groovy | 28 +-
.../export_p0/test_outfile_csv_compress.groovy | 47 ++
.../external_table_p0/es/test_es_query.groovy | 2 +
.../hive/test_hive_get_schema_from_table.groovy | 6 +-
.../hive/test_hive_topn_rf_null.groovy | 79 +++
.../test_iceberg_partition_evolution.groovy | 2 +-
.../paimon/paimon_time_travel.groovy | 119 ++++-
.../tvf/test_s3_tvf_globlist.groovy | 54 +++
...les.groovy => test_s3tables_glue_insert.groovy} | 21 +-
... => test_s3tables_glue_insert_overwrite.groovy} | 16 +-
...=> test_s3tables_glue_insert_partitions.groovy} | 22 +-
.../iceberg/test_s3tables_insert_overwrite.groovy | 7 +-
.../iceberg/test_s3tables_write_insert.groovy | 7 +-
.../iceberg/test_s3tables_write_partitions.groovy | 5 +
.../paimon/test_paimon_dlf_rest_catalog.groovy | 8 +-
.../iceberg_on_hms_and_filesystem_and_dlf.groovy | 5 +-
.../test_inverted_index_v3.groovy | 28 ++
.../streaming_job/test_streaming_insert_job.groovy | 159 +++++++
.../broker_load/test_s3_load_with_no_file.groovy | 74 +++
.../ingestion_load/test_ingestion_load.groovy | 2 +-
.../test_ingestion_load_alter_column.groovy | 11 +-
.../test_ingestion_load_alter_partition.groovy | 11 +-
.../test_ingestion_load_drop_table.groovy | 4 +-
.../test_ingestion_load_multi_table.groovy | 4 +-
.../test_ingestion_load_with_partition.groovy | 2 +-
.../load_p0/spark_load/test_spark_load.groovy | 1 +
.../nereids_function_p0/scalar_function/P.groovy | 12 +
.../nereids_function_p0/scalar_function/S.groovy | 8 +
.../fold_constant_string_arithmatic.groovy | 110 +++++
.../nereids_p0/show/test_nereids_show_views.groovy | 57 +++
.../agg_strategy/agg_strategy.groovy | 23 +
.../nereids_rules_p0/agg_strategy/load.groovy | 2 +-
.../test_in_no_concurrent.groovy | 67 +++
.../datetime_functions/test_date_function.groovy | 2 +-
.../json_functions/test_json_exists_path.groovy | 32 ++
.../json_functions/test_json_function.groovy | 2 +-
.../query_p0/system/test_query_sys_tables.groovy | 2 +-
.../tpcds_sf100/noStatsRfPrune/query64.groovy | 5 +-
.../tpcds_sf100/no_stats_shape/query64.groovy | 3 +-
.../tpcds_sf100/rf_prune/query64.groovy | 1 +
.../shape_check/tpcds_sf100/shape/query64.groovy | 3 +-
.../shape_check/tpcds_sf1000/hint/query64.groovy | 3 +
.../shape_check/tpcds_sf1000/shape/query64.groovy | 1 +
.../tpcds_sf10t_orc/shape/query64.groovy | 1 +
385 files changed, 8965 insertions(+), 3699 deletions(-)
create mode 100644
be/src/exec/schema_scanner/schema_cluster_snapshot_properties_scanner.cpp
copy
be/src/{olap/rowset/segment_v2/inverted_index/query_v2/intersection_scorer.h =>
exec/schema_scanner/schema_cluster_snapshot_properties_scanner.h} (50%)
create mode 100644
be/src/exec/schema_scanner/schema_cluster_snapshots_scanner.cpp
copy
be/src/{olap/rowset/segment_v2/inverted_index/query_v2/intersection_scorer.h =>
exec/schema_scanner/schema_cluster_snapshots_scanner.h} (53%)
create mode 100644
be/src/olap/rowset/segment_v2/inverted_index/query_v2/match_all_docs_scorer.h
delete mode 100644 be/src/util/datetype_cast.hpp
delete mode 100644 be/src/vec/data_types/data_type_number.cpp
create mode 100644
be/test/exec/schema_scanner/schema_cluster_snapshot_properties_scanner_test.cpp
create mode 100644
be/test/exec/schema_scanner/schema_cluster_snapshots_scanner_test.cpp
create mode 100644 cloud/src/recycler/checker_snapshot.cpp
create mode 100644
docker/thirdparties/docker-compose/elasticsearch/scripts/index/es7_test_object.json
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run81.hql
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/GlobListResult.java
copy fe/fe-core/src/main/java/org/apache/doris/job/{common/TaskType.java =>
base/JobProperties.java} (90%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/common/FailureReason.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertTask.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingJobProperties.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingJobSchedulerTask.java
copy
fe/fe-core/src/main/java/org/apache/doris/{nereids/trees/expressions/functions/agg/MultiDistinction.java
=> job/extensions/insert/streaming/StreamingJobStatistic.java} (63%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingTaskTxnCommitAttachment.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/manager/StreamingTaskManager.java
copy fe/fe-core/src/main/java/org/apache/doris/job/{common/TaskType.java =>
offset/Offset.java} (87%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/offset/SourceOffsetProvider.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/offset/SourceOffsetProviderFactory.java
copy fe/fe-core/src/main/java/org/apache/doris/job/{common/JobStatus.java =>
offset/s3/S3Offset.java} (51%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3SourceOffsetProvider.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/scheduler/StreamingTaskScheduler.java
rename
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/{AdminBackupClusterSnapshotCommand.java
=> AdminCreateClusterSnapshotCommand.java} (90%)
rename
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/{AdminSetClusterSnapshotCommand.java
=> AdminSetAutoClusterSnapshotCommand.java} (72%)
rename
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/{AdminShowClusterSnapshotPropertiesCommand.java
=> AdminSetClusterSnapshotFeatureSwitchCommand.java} (50%)
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AdminShowClusterSnapshotCommand.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterJobCommand.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/HMSGlueIT.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/IcebergGlueIT.java
rename
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/{AdminBackupClusterSnapshotCommandTest.java
=> AdminCreateClusterSnapshotCommandTest.java} (87%)
rename
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/{AdminSetClusterSnapshotCommandTest.java
=> AdminSetAutoClusterSnapshotCommandTest.java} (78%)
rename
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/{AdminShowClusterSnapshotCommandTest.java
=> AdminSetClusterSnapshotFeatureSwitchCommandTest.java} (85%)
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AdminShowClusterSnapshotPropertiesCommandTest.java
create mode 100644
regression-test/data/external_table_p0/hive/test_hive_topn_rf_null.out
create mode 100644
regression-test/data/external_table_p0/tvf/test_s3_tvf_globlist.out
delete mode 100644
regression-test/data/external_table_p2/iceberg/test_glue_rest_s3tables.out
copy
regression-test/data/external_table_p2/iceberg/{test_s3tables_write_insert.out
=> test_s3tables_glue_insert.out} (100%)
copy
regression-test/data/external_table_p2/iceberg/{test_s3tables_insert_overwrite.out
=> test_s3tables_glue_insert_overwrite.out} (100%)
copy
regression-test/data/external_table_p2/iceberg/{test_s3tables_write_partitions.out
=> test_s3tables_glue_insert_partitions.out} (100%)
create mode 100644
regression-test/data/job_p0/streaming_job/test_streaming_insert_job.out
create mode 100644
regression-test/data/nereids_p0/expression/fold_constant/fold_constant_string_arithmatic.out
copy
regression-test/data/{external_table_p2/paimon/test_paimon_dlf_rest_catalog.out
=> query_p0/sql_functions/conditional_functions/test_in_no_concurrent.out} (54%)
delete mode 100644 regression-test/pipeline/performance/conf/session_variables
create mode 100644
regression-test/pipeline/performance/conf/session_variables.sql
create mode 100644
regression-test/suites/external_table_p0/hive/test_hive_topn_rf_null.groovy
create mode 100644
regression-test/suites/external_table_p0/tvf/test_s3_tvf_globlist.groovy
rename
regression-test/suites/external_table_p2/iceberg/{test_glue_rest_s3tables.groovy
=> test_s3tables_glue_insert.groovy} (98%)
copy
regression-test/suites/external_table_p2/iceberg/{test_s3tables_insert_overwrite.groovy
=> test_s3tables_glue_insert_overwrite.groovy} (98%)
copy
regression-test/suites/external_table_p2/iceberg/{test_s3tables_write_partitions.groovy
=> test_s3tables_glue_insert_partitions.groovy} (87%)
create mode 100644
regression-test/suites/job_p0/streaming_job/test_streaming_insert_job.groovy
create mode 100644
regression-test/suites/load_p0/broker_load/test_s3_load_with_no_file.groovy
create mode 100644
regression-test/suites/nereids_p0/show/test_nereids_show_views.groovy
create mode 100644
regression-test/suites/query_p0/sql_functions/conditional_functions/test_in_no_concurrent.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
