This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch auto-pick-41001-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git
omit d09fc60ff36 [cases](ip) improve cases for ip functions (#41001) add 5b44e2840ff [fix] (build index) fix build index coredump (#43246) (#43298) add 2e3ad171ead branch-3.0: [fix](persist) Fix the generated dropping tmp partition sql (#43124) add 1cc16bffe37 [improvement](external)add some improvements for external scan #38946 (#43310) add e49d73614fb [improve](backup) Reduce backup/restore job log size by compress (#42954) add 0d796fd49cf [fix](test) fix some test cases #43217 (#43341) add 4527cceb238 [branch-3.0](pick) Pick #43063 #43199 (#43322) add db2210077ab [Opt](TabletSchema) reuse TabletColumn info to reduce mem (#42448) (#43326) add e0dfae14505 [opt](ShortCircuit) add more stats info to trace (#42697) (#43327) add 0a21b0e13d0 branch-3.0: [fix](restore) Add synchronized to avoid concurrent modification (#43258) add 1f1bee3063b [chore](agent) log the binary message size of agent tasks #43239 (#43366) add 3b14452b720 [improve](report) split agent batch tasks automaticlly #43257 (#43363) add 1688e657254 branch-3.0: [Bug](runtime-filter) avoid wrong partitial ignore minmax filter (#43195) add 79b5b9dbb3a [fix](bloom filter)Fix drop column with bloom filter (#41369) (#43378) add 77459ab8cc3 [improve](backup) Only compress the running backup/restore job #43177 (#43362) add da61d29f99d [improve](cloud) support arm compilation in cloud mode (#42467) (#43377) add 37efa9641f2 branch-3.0: [fix](auth)Fix after fe restarting, external permissions are lost (#43306) add 1821f972a8e branch-3.0: [fix](hive)fix hive catalog miss partition that have special characters. (#43070) add 5911567826e branch-3.0: [fix](S3FileWriter) Fix boundary issue when multipart upload (#43254) add 6121a1cc8bc branch-3.0: [Improve](auditlog) audit log print real sql in prepared statement (#43321) add c80d0170475 branch-3.0: [Chore](GA)Remove deadlink-check (#42502) add cb0eb0f0a61 [branch-3.0](timezone) Preload time offset in datetime (#42395) (#42608) add 1de897dfcd2 [branch-3.0](insert-overwrite) Support create partition for auto partition table when insert overwrite (#38628) (#42645) add 88d268eacda branch-3.0: [chore](profile) add task state in profile (#42981) add b851fcada20 branch-3.0: [fix](cloud) Fix constant folding for requests sent to BE without dis… (#43110) add 519831b18f4 branch-3.0: [test](stream load) add cases to ensure the correctness of the document (#43111) add 88bc2c2fdad branch-3.0: [fix](compatibility) Fix correctness problem during upgrading (#43182) add 15a8ab516c2 branch-3.0: [fix](new_json_reader)fix new_json_reader core (#43187) add 23f7bb983f5 branch-3.0: [fix](auth)Fix concurrency issue during role manager upgrade (#43193) add 940bbee2fa8 branch-3.0: [fix](regression case) Fix `test_dynamic_partition_with_update` regre… (#43232) add 1433ab8398d branch-3.0: [fix](cloud) Replace `cloud cluster` to `compute group` in error hint (#43379) add 6d8390998a8 branch-3.0: [improve](udf) support load data with udf functions (#43361) add 3a47e3d24a7 branch-3.0: [feat](cloud) S3 storage vault support property `use_path_style` (#43330) add b45bbd729ed branch-3.0: [fix](cloud-mow) Should clear GetDeleteBitmapResponse when geting delete bitmap fail and retry (#43297) add 5e4cf9472bc branch-3.0: [Feature](cloud) Add cloud report for clean up expired tablets (#43233) add 0e2f4756e51 [opt](index compaction)Use RAM dir to create tmp index_writer (#41371) (#41703) add e682fa2fed6 branch-3.0: [Improvement](LDAP Auth)Enhance LDAP authentication with a configurable group filter (#43292) add 46a2f2b56e9 branch-3.0: [fix](ip) fix datatype serde for ipv6 with rowstore (#43251) add e49196b4522 [Test](inverted index) add test data for inverted index p2 case #43287 (#43387) add 70e555e62c6 branch-3.0: [Opt](Serde) optimize serialization to string on variant type (#43320) add b4df6805025 [cherry-pick](branch-3.0)add SessionVariable for enableCooldownReplicaAffinity (#42677) add 5e75a1fdbd5 branch-3.0: [regression](concat) add concat and group_concat test case for special chars (#42503) add b6ecb19a9ee branch-3.0: [fix](log) fix group commit warn log (#42529) add 1c3855f7e69 branch-3.0: [fix](schema-change) Make sc job state updated after the table state is changed (#42879) add f7494dd7408 branch-3.0: [fix](array) fix add_array_null_value for inverted index_builder (#42883) add 4b85b60df91 branch-3.0: [enhancement](cloud) expose file cache capacity for each disk (#42896) add a135b049906 branch-3.0: [fix](cloud) Add more log for backend and disable cloud cluster status (#42901) add b4b6e9ef98a branch-3.0: [fix](tablet io error) fix tablet not increase _io_error_times in some cases. (#42955) add 0e6e34d4bf4 branch-3.0: [fix](cloud)Compatibility with the initial default compute group usage (#42965) add 7abf3c5d2a8 branch-3.0: [fix](heartbeat) fix heartbeat editlog no persist hbTime (#42986) add aa5c1336488 branch-3.0: [test](auth) Add call auth module case (#42861) add 5a8e2182aac branch-3.0: [fix](be) Fix creating hdfs connection coredump within bthread (#43020) add 40c97296c33 branch-3.0: [feature](doris compose) run docker as host user (#43091) add 1a0de848318 branch-3.0: [feature](doris compose) enable fe jvm debug port (#43108) add 1c4b55e218c branch-3.0: [fix](editlog) add column rename binlog during the replay (#43028) add 34ae12ca89e branch-3.0: [test](stats)Add upgrade and downgrade test for analyze (#43047) add 5fba86d9faa branch-3.0: [Enhancement](log) Make log of vlog adjustment clear (#43069) add 1fdf8520827 branch-3.0: [fix](mtmv)Fix the problem where the job does not exist, which prevents the deletion of MTMV (#43324) add 3c8c8e412ac branch-3.0: [fix](array-funcs)fix array funcs which ret can be nullable (#43353) add c7193578807 [fix](be) Fix creating hdfs connection coredump within bthread (#42629) (#43368) add 85f839f9852 branch-3.0: [enhancement](log) logout tablename with temp partitions in backup (#42549) add 6b86e56d5e0 [fix](Export) fix dead lock of parallel Exporting (#43083) (#43402) add ab748efaafa [cherry-pick][test](doc) add table design example in doris's doc to regression test (#42444) (#43023) add ae9c9579062 [enhancement](cloud) optimize block cache lock (#41818) (#43401) add 047efd7f4ce [opt](variable) force update some variable by variable version (#41607) (#43405) add d9495eb1008 [fix](planner) NullLiteral should always having a correct Type and set to be analyzed (#43370) (#43374) add 12ced58719a branch-3.0: [fix](build index)Fix build index empty index file on renamed column (#43392) add e3b03ef9ed9 [fix](crash) fix be crash because of int overflow (#41554) (#43398) add a5722471b48 [cherry-pick](branch-3.0) fix wrong property of S3 resource (#43411) add beb0192228c branch-3.0: [feat](doris compose) diff user use diff start subnet #41996 (#43454) add c71cb2d3dc8 [fix] Implementing match_phrase_edge without index query method (#41658) (#43397) add b979744325d branch-3.0: [fix](cloud-mow) FE should release mow lock when calculate delete bitmap catch exception (#43088) add 643ac395412 [fix](inverted index) Fix incorrect exception handling (#41874) (#43395) add 35071eb9f70 [fix](inverted index) Fix incorrect usage of regexp compile_err (#41944) (#43394) add 3fb99945fbc branch-3.0: [fix](delete) Fix static type dispatch by mistake due to typo (#43022) add 4f81a8efe36 branch-3.0: [fix](cloud) fix core dump in CloudWarmUpManager::handle_jobs (#43404) add e30560bfaf9 [fix](cloud-mow) Fix clear GetDeleteBitmapResponse problem when geting delete bitmap fail and retry (#43457) add 16a83fbba6e [feat](cloud) S3 storage vault support property `use_path_style` (#43060) (#43343) add 9d96a4bb710 [fix](inverted index) multi_match supports any, all, phrase. (#41663) (#43396) add be7d43e9de1 branch-3.0: [fix](ES Catalog)Only like on keyword can be applied to wildcard query (#43399) add 85ef12283b2 [cherry-pick][test](doc) add some `data-admin` example in doris's doc to regression test (#43240) (#43417) add efc2ec58458 [cherry-pick][test](doc) add some import-example in doris's doc to regression test(#43123) (#43416) add 3bd553d1ad7 [fix](cloud-mow) MS should delete the existing keys before rewriting it when processing old version delete bitmap on cu compaction (#43479) add 81ef26d6416 [fix](cloud-mow) print the correct lock owner when geting lock timeout (#43407) add 4a28d3e2592 [fix](cloud) fix read cache block file return stat NOT_FOUND (pick #43220) (#43512) add 50d21c012e1 branch-3.0: [opt](agg function) Signature verification of aggregate function. (#43191) add 55b38edd43d [refactor](profile) Remove profile tree printer code #41766 (#43526) add d32d2603bbc [improvement](paimon)Using table serialization on the jni side #43167 (#43474) add 9f28b4fdcf2 [fix](scan) Avoid memory allocated by buffered_reader from being traced (#41921) (#43478) add c8b2a2ecf3f [feature](array-func)support array_match_all/any (#40605) (#43514) add 803cf4dd0bb branch-3.0: [chore](log) Use correct error type of uneven user behaviour (#43493) add 2995e8f3b9b [Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. (#42004) (#43467) add 282d31a0446 branch-3.0: [Optimize](Expr) Opt getting value of VLitreal (#43248) add 1a6a0a67008 branch-3.0: [enhance](mtmv)MTMV interface optimization (#43328) add 1f4072a34b8 [opt](cloud) Simplify default config file of meta-service (#43381) (#43522) add 3026793ca9f branch-3.0: [test](routine load) add routine load doc case (#43384) add a35758bb90e branch-3.0: [improve](load) pass cancel reason to tablet writer when cancelled (#43534) add 041f01c0b1f branch-3.0: [fix](load) fix broker load progress due to retry (#43535) add 591a60a277b branch-3.0: [test](doc) add test case for error data handling doc (#43529) add 378a46e8c6a branch-3.0: [test](doc) add some remote-storage-example in doris's doc to regression test (#43506) add f8c9b2c338d [branch-3.0](broker-load) fix error in broker-load with read_json_by_line (#41788) (#43471) add e07759240df branch-3.0: [fix](cloud) Add check for alter storage vault type (#43489) add 56c9b99e44b branch-3.0: [test](doc) add some `data-admin` example in doris's doc to regression test (#43415) add 4df1499eac7 branch-3.0: [opt](cloud) Simplify default config file of meta-service (#43434) add b1ffbd076b0 branch-3.0: [fix](cloud) fix read cache block file return stat NOT_FOUND (#43504) add d289395453a [fix](mtmv) Fix the old version of the materialized view error 'Current database is not set'` (#42152) (#42309) add cab76c2337c [fix](mtmv) fix mtmv deadlock issue (#43376) (#43427) add f6444ea4cb7 [Impl](Nereids) add propagateNullLiteral trait for special functions (#42256) (#43490) add 91d95f63be7 [fix](backup) Load backup meta and job info bytes from disk #43276 (#43520) add 9e177c0de80 branch-3.0: [opt](log) change lzo decompress log to debug level (#43582) add 856596f9e6f [Refactor] (inverted index) Refactor Inverted index file writer (#41625) (#43528) add 982a2b67266 branch-3.0: [fix](test) fix clickhouse jdbc catalog func with cast push down case (#43429) add d6407e56656 branch-3.0: [Dependency](Fe) Upgrade HikariCP version to 6.0.0 (#43430) add 2159ffcd51a [Exec)(cache) add element count in LRU cache (#43458) add 848eb73c88d [enhance](regression-test) test alter table partition (#41437) (#43548) add 829b99423d4 [chore](restore) reduce logged unfinished snapshoting task #43076 (#43608) add 441c754183e [cherry-pick](branch-30)fix the exchange not skip offset rows (#42190) (#42490) add cc2d146517f [Fix](segment compaction) fix error using of inverted index file writer in segment compaction #43114 (#43616) add 5658c34e5f5 [feat](restore) Support compressed snapshot meta and job info #43516 (#43570) add a78b722c6b9 [Fix](bug) Is null predicate get error query result (#43619) add ff6ec6c9814 [opt](Variant) avoid unnecessary mem for variant extracted columns (#43567) (#43620) add c7b707f2575 [fix](cloud-mow)Full compaction no need to sync delete bitmap (#43613) add ff8334b9238 [fix](cloud) Add check for alter storage vault type (#43352) (#43495) add 78bfa52953e [cherry-pick](branch-3.0) Pick "[Enhancement](MS) Add fix tablet data size api for meta service (#41782)" (#43460) add 3ef12c7a240 [enhancement](err-msg) Output column info when size invalid in block data convertor (#41535) (#43189) add cd2d79c6e2d branch-3.0: [opt](remote scan) Fix remote scan parallelism (#43625) add 9cc51da52e3 [cherry-pick](branch-30)fix date_floor function return wrong result (#41948) (#43588) add 89ff353ebe2 [opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927 (#42079) add 623c68480ee branch-3.0: [fix](Export) fix issue that `Export` can not specify the columns which have capital letters (#43571) add 014b70909da [fix](segcompaction) fix coredump for inverted index file writer close in segment compaction #43477 (#43587) add 60f17d870ad branch-3.0: [fix] (bloom filter) Fix the bloom filter calculation for date and datetime (#43612) add b7358134e70 [fix] (inverted index) fix index compaction #43046 (#43614) add 19ab0c229ea branch-3.0: [Optimize](Variant) move merge_rowsets_schema to sync_tablet_rowsets to make schema more accurate (#43634) add 47c4604a6a0 branch-3.0: [fix](statistics)Change column statistics table varchar length to 1024. (#43418) add b5b8e49b9bd [cherry-pick](branch-3.0) Pick "[fix](rowset meta) Fix rowset meta size relation (#41022)" (#43414) add 13b4c2054dc [improvement](build index)Optimize failed task check on same tablet (#42295) (#43581) add 93281e285be branch-3.0: [fix](group commit) fix NPE in group commit select backend (#43635) add 1d85a711d15 [feature](Nereids): Optimize Query Plan by Pulling Up Join with Common Child from Union (#42033) (#43591) add c30b7526b30 branch-3.0: [improvement](statistics)Remove useless stats validation check. (#43498) add f5f1f8c5aab branch-3.0: [fix](ngram bloomfilter) fix narrow conversion for ngram bf_size (#43645) add f72ff092231 [fix](compaction) reduce memory cost for cloud compaction of mow table (#43502) (#43633) add f97c054bd75 [fix](cloud) Fix aws sdk default retry policy not taking effect (#43575) (#43648) add 94d0d454953 branch-3.0: [fix](case) Fix test_backup_restore_atomic_with_alter with SYNC (#43639) add 64f71a5e995 [fix](pipeline) only sub_running_sink_operators in close (#43500) (#43649) add 478b36edb75 branch-3.0: [opt](http) enable auth token with BE http request (#43659) add 7d74f0c3117 branch-3.0: [fix](scanner) Fix multiple move the same tablet when NewOlapScanner::create_shared (#43691) add f650a16fd9d [fix](nereids) Solve the problem of pruning wrong partitions in multi-column partition pruning (#43332) (#43664) add 9c672fb1856 [Refactor](inverted index) refactor inverted index interface in tablet schema (#43003) (#43280) (#43669) add aaf422d74db branch-3.0: [fix](Nereids) alias function should use unbound slot as place holder (#43764) add fbf81f4672e branch-3.0: [fix](hive) support query hive view created by spark (#43552) add 3ba7cc52e22 [opt](binlog) Support modify comment binlog #39783 (#43762) add daf4f0a7bf5 [fix](cloud-mow) Add enable config for procesing old version delete bitmap when doing cu compaction (#43615) add 27e005149cd [fix](variant) fix index in variant (#43375) (#43773) add 93dec4d80c5 branch-3.0: [fix](filesystem)Use simple authentication directly in S3FileSystem (#43807) add 6e8a80d94d4 branch-3.0: [Fix](Nereids) fix floor/round/ceil/truncate functions type compute precision problem (#43781) add ae0915721b9 [fix](cloud) Check instance_id valid when use cloud_unique_id degrade format #43253 (#43832) add 1b41de79a75 branch-3.0: [fix](cloud version) fix version of information_schema table backward when query in follower node (#43496) add 8358799978a branch-3.0: [feat](backup) GetSnapshot returns snapshot expiration time (#43829) add d846e4c6e98 branch-3.0: [fix](column_complex) wrong type of Field returned by ColumnComplex (#43718) add 85720ac67eb branch-3.0: [Fix](sk) All sk in log should be encrypted (#43544) (#43802) add 481ce1e129b branch-3.0: [fix](sync point) Fix sync point not reg but construct default value (#43461) (#43652) add 4a530dcfde8 [Fix](test) add sync to bloom filter test case #43840 (#43857) add 9a40cba5f0b [feature](profile)Enable merging of incomplete profiles. (#39560) (#42953) add 8febc99ff8e branch-3.0: [fix](string64) fix coredump caused by ColumnArray<ColumnStr<uint64_t>>::insert_indices_from (#43861) add 830f08a2f86 [fix] (inverted index) Disallow variant columns from using inverted index format v1 (#43599) (#43812) add dfc344cdc53 branch-3.0: [improve](binlog) Reserve ids for binlog type allocation (#43787) add 69b8e2b999c branch-3.0: [fix](cloud-mow) Fix sending commiting rpc to FE twice problem (#43854) add 7ac7dddee68 [fix](syncpoint) Fix incorrect syncpoint 'has_point()' intorduced by (#43461) (#43784) (#43883) add 11a9a48cd4d [branch-3.0](log) Pick all BE execution log reduction (#43579) add ca6b3e7d15c branch-3.0: [minor](rpc) Check client before RPC (#43816) add a0285c158ef branch-3.0: [fix](planner)show tablet command return wrong result when having limit and offset (#43844) add 1f0081dbaa9 [fix](test) fix regression test case in test_broker_load_p2 (#43811) add 5cefb505367 [feature](security) Support block specific query with AST names (#43533) (#43887) add 2658ac80f44 [cherry-pick](branch-3.0) Pick "[Enhancement](checker) Enhance inverted checker (#43294)" (#43869) add 5757cbedde1 branch-3.0: [fix](nereids) fix bug: convert stringLikeLiteral to double #43776 (#43880) add a21b017d035 branch-3.0: [fix](cloud) Fix replay default compute group failed throw exception #43766 (#43889) add a1c8765ff34 [fix](nereids) convert stringLikeLiteral to double should use byte length (#43776) branch-3.0 (#43912) add bf8ce94757d [fix](cloud-mow) fix compaction missed rows dcheck fail (#43909) add 39008de5b6b branch-3.0: [fix](case) fix replay_wal_restart_fe #43705 (#43895) add d21b21cd83d [cases](ip) improve cases for ip functions (#41001) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (d09fc60ff36) \ N -- N -- N refs/heads/auto-pick-41001-branch-3.0 (d21b21cd83d) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .dlc.json | 19 - .github/workflows/deadlink-check.yml | 37 - be/CMakeLists.txt | 4 + be/src/agent/agent_server.cpp | 41 +- be/src/agent/agent_server.h | 8 +- be/src/agent/heartbeat_server.cpp | 69 +- be/src/agent/heartbeat_server.h | 13 +- be/src/agent/task_worker_pool.cpp | 97 +- be/src/agent/task_worker_pool.h | 18 +- be/src/agent/utils.cpp | 49 +- be/src/agent/utils.h | 10 +- be/src/apache-orc | 2 +- be/src/cloud/cloud_base_compaction.cpp | 23 +- be/src/cloud/cloud_compaction_action.cpp | 5 +- be/src/cloud/cloud_cumulative_compaction.cpp | 73 +- be/src/cloud/cloud_cumulative_compaction.h | 2 +- .../cloud/cloud_cumulative_compaction_policy.cpp | 2 +- be/src/cloud/cloud_delete_bitmap_action.cpp | 2 + be/src/cloud/cloud_full_compaction.cpp | 30 +- be/src/cloud/cloud_meta_mgr.cpp | 168 +- be/src/cloud/cloud_meta_mgr.h | 13 +- be/src/cloud/cloud_rowset_builder.cpp | 2 +- be/src/cloud/cloud_rowset_writer.cpp | 15 +- be/src/cloud/cloud_schema_change_job.cpp | 2 +- be/src/cloud/cloud_tablet.cpp | 103 +- be/src/cloud/cloud_tablet.h | 9 +- be/src/cloud/cloud_tablet_mgr.cpp | 83 +- be/src/cloud/cloud_tablet_mgr.h | 19 +- be/src/cloud/cloud_warm_up_manager.cpp | 27 +- be/src/cloud/cloud_warm_up_manager.h | 4 +- be/src/cloud/config.cpp | 1 + be/src/cloud/config.h | 2 + be/src/common/config.cpp | 10 +- be/src/common/config.h | 10 +- be/src/common/status.h | 4 +- be/src/common/utils.h | 3 +- be/src/exec/es/es_scroll_parser.cpp | 2 +- be/src/exec/lzo_decompressor.cpp | 19 +- be/src/exec/olap_common.h | 28 +- .../schema_active_queries_scanner.cpp | 2 +- .../schema_scanner/schema_partitions_scanner.cpp | 2 +- .../exec/schema_scanner/schema_routine_scanner.cpp | 4 +- .../schema_table_options_scanner.cpp | 2 +- .../schema_table_properties_scanner.cpp | 2 +- .../schema_workload_group_privileges.cpp | 4 +- .../schema_workload_groups_scanner.cpp | 4 +- .../schema_workload_sched_policy_scanner.cpp | 4 +- be/src/exec/tablet_info.cpp | 1 + be/src/exprs/runtime_filter_slots.h | 6 +- be/src/http/action/adjust_log_level.cpp | 14 +- be/src/http/action/compaction_score_action.cpp | 2 +- be/src/http/action/http_stream.cpp | 8 +- be/src/http/action/stream_load.cpp | 2 +- be/src/http/action/tablets_info_action.cpp | 20 +- be/src/http/http_client.cpp | 4 + be/src/http/http_client.h | 8 + be/src/http/http_common.h | 2 +- be/src/http/http_handler_with_auth.cpp | 19 +- be/src/http/http_headers.cpp | 3 +- be/src/http/http_headers.h | 3 +- be/src/http/utils.cpp | 11 +- be/src/io/cache/block_file_cache.cpp | 109 +- be/src/io/cache/block_file_cache.h | 46 +- be/src/io/cache/block_file_cache_profile.cpp | 6 +- be/src/io/cache/cached_remote_file_reader.cpp | 2 + be/src/io/cache/file_block.cpp | 32 +- be/src/io/cache/fs_file_cache_storage.cpp | 58 +- be/src/io/cache/fs_file_cache_storage.h | 3 + be/src/io/fs/buffered_reader.cpp | 121 +- be/src/io/fs/buffered_reader.h | 152 +- be/src/io/fs/local_file_reader.cpp | 11 + be/src/io/fs/local_file_system.cpp | 17 +- be/src/io/fs/local_file_writer.cpp | 9 + be/src/io/fs/multi_table_pipe.cpp | 2 +- be/src/io/fs/s3_file_system.cpp | 4 +- be/src/io/fs/s3_file_writer.cpp | 54 +- be/src/io/hdfs_util.cpp | 37 +- be/src/olap/base_compaction.cpp | 2 +- be/src/olap/base_tablet.cpp | 8 +- be/src/olap/base_tablet.h | 6 +- be/src/olap/compaction.cpp | 211 +- be/src/olap/compaction.h | 6 +- be/src/olap/comparison_predicate.h | 13 +- be/src/olap/cumulative_compaction.cpp | 3 +- be/src/olap/delete_handler.cpp | 13 +- be/src/olap/delete_handler.h | 3 + be/src/olap/delta_writer.cpp | 2 +- be/src/olap/lru_cache.cpp | 8 + be/src/olap/lru_cache.h | 4 + be/src/olap/olap_server.cpp | 6 +- be/src/olap/primary_key_index.cpp | 7 +- be/src/olap/primary_key_index.h | 10 +- be/src/olap/rowset/beta_rowset.cpp | 52 +- be/src/olap/rowset/beta_rowset_writer.cpp | 131 +- be/src/olap/rowset/beta_rowset_writer.h | 90 +- be/src/olap/rowset/beta_rowset_writer_v2.cpp | 2 +- be/src/olap/rowset/beta_rowset_writer_v2.h | 3 +- be/src/olap/rowset/rowset.h | 3 +- be/src/olap/rowset/rowset_meta.cpp | 11 +- be/src/olap/rowset/rowset_meta.h | 5 +- be/src/olap/rowset/rowset_writer.h | 19 + be/src/olap/rowset/segcompaction.cpp | 9 +- be/src/olap/rowset/segcompaction.h | 5 + be/src/olap/rowset/segment_creator.cpp | 82 +- be/src/olap/rowset/segment_creator.h | 17 +- .../segment_v2/bloom_filter_index_reader.cpp | 6 +- .../rowset/segment_v2/bloom_filter_index_reader.h | 4 +- be/src/olap/rowset/segment_v2/column_writer.h | 2 +- .../rowset/segment_v2/indexed_column_reader.cpp | 19 +- .../olap/rowset/segment_v2/indexed_column_reader.h | 15 +- .../segment_v2/inverted_index_compaction.cpp | 11 +- .../segment_v2/inverted_index_file_writer.cpp | 14 +- .../rowset/segment_v2/inverted_index_file_writer.h | 39 +- .../rowset/segment_v2/inverted_index_writer.cpp | 17 + .../olap/rowset/segment_v2/inverted_index_writer.h | 19 +- be/src/olap/rowset/segment_v2/segment.cpp | 34 +- be/src/olap/rowset/segment_v2/segment.h | 14 +- be/src/olap/rowset/segment_v2/segment_iterator.cpp | 24 +- be/src/olap/rowset/segment_v2/segment_writer.cpp | 103 +- be/src/olap/rowset/segment_v2/segment_writer.h | 24 +- .../rowset/segment_v2/vertical_segment_writer.cpp | 99 +- .../rowset/segment_v2/vertical_segment_writer.h | 25 +- be/src/olap/rowset/vertical_beta_rowset_writer.cpp | 35 +- be/src/olap/schema_change.cpp | 9 +- be/src/olap/segment_loader.cpp | 7 +- be/src/olap/segment_loader.h | 3 +- be/src/olap/single_replica_compaction.cpp | 13 +- be/src/olap/snapshot_manager.cpp | 7 +- be/src/olap/tablet.cpp | 148 +- be/src/olap/tablet.h | 15 +- be/src/olap/tablet_column_object_pool.cpp | 57 + be/src/olap/tablet_column_object_pool.h | 58 + be/src/olap/tablet_manager.cpp | 2 + be/src/olap/tablet_meta.cpp | 3 + be/src/olap/tablet_meta.h | 11 +- be/src/olap/tablet_schema.cpp | 143 +- be/src/olap/tablet_schema.h | 62 +- be/src/olap/tablet_schema_cache.cpp | 23 +- be/src/olap/task/engine_checksum_task.cpp | 2 +- be/src/olap/task/engine_clone_task.cpp | 6 +- be/src/olap/task/engine_clone_task.h | 6 +- be/src/olap/task/engine_storage_migration_task.cpp | 7 +- be/src/olap/task/index_builder.cpp | 117 +- be/src/olap/wal/wal_manager.cpp | 6 +- be/src/olap/wal/wal_table.cpp | 14 +- be/src/pipeline/exec/aggregation_sink_operator.cpp | 2 +- be/src/pipeline/exec/analytic_source_operator.cpp | 6 +- .../distinct_streaming_aggregation_operator.cpp | 2 +- be/src/pipeline/exec/exchange_sink_operator.cpp | 2 +- be/src/pipeline/exec/exchange_source_operator.cpp | 6 +- be/src/pipeline/exec/olap_scan_operator.cpp | 2 +- be/src/pipeline/exec/operator.cpp | 2 +- be/src/pipeline/exec/operator.h | 3 + be/src/pipeline/exec/result_sink_operator.cpp | 11 +- be/src/pipeline/exec/scan_operator.cpp | 33 +- .../exec/streaming_aggregation_operator.cpp | 2 +- .../local_exchange_sink_operator.cpp | 16 +- .../local_exchange/local_exchange_sink_operator.h | 1 + be/src/pipeline/pipeline_fragment_context.cpp | 17 +- be/src/pipeline/pipeline_task.cpp | 9 + be/src/pipeline/query_cache/query_cache.cpp | 2 +- be/src/pipeline/query_cache/query_cache.h | 1 + be/src/pipeline/task_queue.h | 2 +- be/src/runtime/client_cache.h | 8 +- .../timezone_utils.h => runtime/cluster_info.h} | 43 +- be/src/runtime/exec_env.cpp | 24 +- be/src/runtime/exec_env.h | 20 +- be/src/runtime/exec_env_init.cpp | 10 +- be/src/runtime/fragment_mgr.cpp | 28 +- be/src/runtime/group_commit_mgr.cpp | 26 +- be/src/runtime/load_stream.cpp | 11 + be/src/runtime/load_stream_writer.cpp | 2 +- be/src/runtime/memory/cache_policy.h | 10 +- be/src/runtime/memory/lru_cache_policy.h | 2 + be/src/runtime/query_context.cpp | 5 +- .../routine_load/routine_load_task_executor.cpp | 2 + be/src/runtime/runtime_query_statistics_mgr.cpp | 6 +- be/src/runtime/small_file_mgr.cpp | 6 +- be/src/runtime/snapshot_loader.cpp | 2 +- .../runtime/stream_load/stream_load_executor.cpp | 12 +- .../workload_group/workload_group_manager.cpp | 2 +- be/src/service/backend_service.cpp | 11 +- be/src/service/doris_main.cpp | 8 +- be/src/service/http_service.cpp | 2 +- be/src/service/internal_service.cpp | 13 +- be/src/service/point_query_executor.cpp | 64 +- be/src/service/point_query_executor.h | 10 +- be/src/util/block_compression.cpp | 1 + be/src/util/s3_util.cpp | 23 +- be/src/util/slice.h | 6 + be/src/util/timezone_utils.cpp | 50 +- be/src/util/timezone_utils.h | 3 + .../vec/aggregate_functions/aggregate_function.h | 44 + be/src/vec/columns/column_array.cpp | 8 +- be/src/vec/columns/column_complex.h | 5 +- be/src/vec/columns/column_fixed_length_object.h | 6 +- be/src/vec/columns/column_map.cpp | 2 +- be/src/vec/columns/column_object.cpp | 34 +- be/src/vec/columns/column_string.cpp | 65 +- be/src/vec/columns/column_string.h | 4 +- be/src/vec/common/schema_util.cpp | 37 +- be/src/vec/common/schema_util.h | 5 + be/src/vec/core/field.cpp | 2 +- be/src/vec/core/field.h | 56 +- .../vec/data_types/data_type_fixed_length_object.h | 2 +- be/src/vec/data_types/data_type_jsonb.h | 2 +- be/src/vec/data_types/data_type_object.h | 4 +- be/src/vec/data_types/data_type_string.cpp | 2 +- be/src/vec/data_types/data_type_string.h | 2 +- .../vec/data_types/serde/data_type_ipv6_serde.cpp | 22 +- .../data_types/serde/data_type_object_serde.cpp | 5 +- be/src/vec/exec/format/json/new_json_reader.cpp | 5 + be/src/vec/exec/format/orc/vorc_reader.cpp | 80 +- be/src/vec/exec/format/orc/vorc_reader.h | 15 +- be/src/vec/exec/format/table/paimon_jni_reader.cpp | 3 + be/src/vec/exec/scan/scanner_context.cpp | 16 +- be/src/vec/exec/scan/scanner_context.h | 3 +- be/src/vec/exec/scan/vmeta_scanner.cpp | 2 +- be/src/vec/exprs/vectorized_agg_fn.cpp | 17 +- be/src/vec/exprs/vectorized_agg_fn.h | 8 +- be/src/vec/exprs/vliteral.cpp | 11 +- .../vec/functions/array/function_array_distance.h | 2 +- .../vec/functions/array/function_array_element.h | 2 +- be/src/vec/functions/array/function_array_index.h | 8 +- .../functions/array/function_array_register.cpp | 2 + .../array/function_array_with_constant.cpp | 4 +- .../vec/functions/array/varray_match_function.cpp | 145 ++ .../vec/functions/function_datetime_floor_ceil.cpp | 7 +- be/src/vec/functions/function_multi_match.cpp | 12 +- be/src/vec/functions/match.cpp | 73 +- be/src/vec/functions/match.h | 5 +- be/src/vec/json/parse2column.cpp | 2 +- be/src/vec/olap/olap_data_convertor.cpp | 7 +- be/src/vec/sink/autoinc_buffer.cpp | 4 +- be/src/vec/sink/vrow_distribution.cpp | 181 +- be/src/vec/sink/vrow_distribution.h | 9 +- be/src/vec/sink/writer/vtablet_writer.cpp | 2 +- be/src/vec/sink/writer/vtablet_writer_v2.cpp | 2 +- be/test/agent/heartbeat_server_test.cpp | 12 +- be/test/agent/mock_utils.h | 2 +- be/test/agent/task_worker_pool_test.cpp | 7 +- be/test/common/schema_util_test.cpp | 109 + be/test/http/http_client_test.cpp | 34 +- be/test/io/fs/buffered_reader_test.cpp | 108 + be/test/olap/date_bloom_filter_test.cpp | 192 ++ be/test/olap/delete_bitmap_calculator_test.cpp | 6 +- ..._deleted_term.cpp => index_compaction_test.cpp} | 626 ++---- .../index_compaction_with_deleted_term.cpp | 12 +- .../index_compaction_write_index_test.cpp | 221 ++ .../segment_v2/inverted_index/data/data1.csv | 1000 +++++++++ .../segment_v2/inverted_index/data/data2.csv | 1000 +++++++++ be/test/olap/segcompaction_test.cpp | 123 ++ be/test/olap/tablet_index_test.cpp | 108 + be/test/olap/wal/wal_manager_test.cpp | 12 +- .../runtime/routine_load_task_executor_test.cpp | 6 +- be/test/testutil/run_all_tests.cpp | 4 + be/test/util/timezone_utils_test.cpp | 144 ++ .../aggregate_functions/agg_min_max_by_test.cpp | 2 +- be/test/vec/columns/column_hash_func_test.cpp | 2 +- be/test/vec/columns/column_nullable_test.h | 2 +- be/test/vec/core/column_array_test.cpp | 47 + be/test/vec/core/column_complex_test.cpp | 87 +- be/test/vec/core/column_map_test.cpp | 155 ++ be/test/vec/core/column_string_test.cpp | 38 + be/test/vec/core/column_struct_test.cpp | 79 + be/test/vec/core/field_test.cpp | 2 +- .../data_types/serde/data_type_serde_pb_test.cpp | 6 +- .../vec/data_types/serde/data_type_serde_test.cpp | 62 + .../data_types/serde/data_type_to_string_test.cpp | 10 +- be/test/vec/exec/vfile_scanner_exception_test.cpp | 13 +- be/test/vec/exec/vwal_scanner_test.cpp | 13 +- .../vec/function/function_array_element_test.cpp | 2 +- be/test/vec/function/function_array_index_test.cpp | 4 +- be/test/vec/function/function_array_size_test.cpp | 12 +- .../vec/function/function_arrays_overlap_test.cpp | 6 +- .../function_compressed_materialization_test.cpp | 4 +- be/test/vec/function/function_string_test.cpp | 10 +- be/test/vec/function/table_function_test.cpp | 4 +- be/test/vec/jsonb/serialize_test.cpp | 2 +- build.sh | 12 +- cloud/CMakeLists.txt | 44 +- cloud/conf/doris_cloud.conf | 25 +- cloud/src/common/bvars.cpp | 5 + cloud/src/common/bvars.h | 3 + cloud/src/common/config.h | 9 +- cloud/src/meta-service/meta_service.cpp | 176 +- cloud/src/meta-service/meta_service.h | 20 + cloud/src/meta-service/meta_service_helper.h | 51 +- cloud/src/meta-service/meta_service_http.cpp | 13 + cloud/src/meta-service/meta_service_resource.cpp | 42 +- .../src/meta-service/meta_service_tablet_stats.cpp | 245 +++ cloud/src/meta-service/meta_service_tablet_stats.h | 15 + cloud/src/meta-service/meta_service_txn.cpp | 2 +- cloud/src/recycler/checker.cpp | 2 + cloud/src/resource-manager/resource_manager.cpp | 34 +- cloud/src/resource-manager/resource_manager.h | 19 + cloud/test/fdb_injection_test.cpp | 1 + cloud/test/meta_service_http_test.cpp | 85 +- cloud/test/meta_service_test.cpp | 181 +- cloud/test/schema_kv_test.cpp | 6 +- common/cpp/obj_retry_strategy.cpp | 11 +- common/cpp/sync_point.cpp | 18 + common/cpp/sync_point.h | 18 +- conf/ldap.conf | 1 + docker/runtime/doris-compose/Dockerfile | 6 +- docker/runtime/doris-compose/cluster.py | 161 +- docker/runtime/doris-compose/command.py | 102 +- .../runtime/doris-compose/resource/entrypoint.sh | 68 + docker/runtime/doris-compose/resource/init_be.sh | 2 +- .../runtime/doris-compose/resource/init_cloud.sh | 5 +- docker/runtime/doris-compose/resource/init_fe.sh | 15 +- .../scripts/create_preinstalled_scripts/run67.hql | 11 + .../scripts/create_view_scripts/create_view.hql | 6 + .../docker-compose/hive/scripts/hive-metastore.sh | 7 + .../orc/orc_tiny_stripes/output_60_3.orc | Bin 0 -> 19688 bytes .../orc/orc_tiny_stripes/output_6_1.orc | Bin 0 -> 1902 bytes .../orc/orc_tiny_stripes/random_output_60_3.orc | Bin 0 -> 24248 bytes .../orc/orc_tiny_stripes/random_output_6_1.orc | Bin 0 -> 2293 bytes .../org/apache/doris/paimon/PaimonJniScanner.java | 34 +- fe/fe-common/pom.xml | 6 + .../main/java/org/apache/doris/common/Config.java | 56 +- .../java/org/apache/doris/common/GZIPUtils.java | 48 + .../java/org/apache/doris/common/LdapConfig.java | 6 + .../java/org/apache/doris/common/ThriftUtils.java | 85 + .../authentication/AuthenticationConfig.java | 48 +- .../authentication/AuthenticationTest.java | 45 + .../org/apache/doris/alter/IndexChangeJob.java | 45 +- .../apache/doris/alter/SchemaChangeHandler.java | 28 +- .../org/apache/doris/alter/SchemaChangeJobV2.java | 8 +- .../doris/analysis/CancelAlterTableStmt.java | 4 + .../org/apache/doris/analysis/CreateTableStmt.java | 8 +- .../java/org/apache/doris/analysis/ExportStmt.java | 25 - .../apache/doris/analysis/FunctionCallExpr.java | 7 +- .../java/org/apache/doris/analysis/IndexDef.java | 16 +- .../doris/analysis/LambdaFunctionCallExpr.java | 4 +- .../apache/doris/analysis/ModifyColumnClause.java | 4 + .../apache/doris/analysis/NativeInsertStmt.java | 6 - .../org/apache/doris/analysis/NullLiteral.java | 1 + .../org/apache/doris/analysis/StringLiteral.java | 2 +- .../java/org/apache/doris/backup/AbstractJob.java | 32 +- .../org/apache/doris/backup/BackupHandler.java | 56 +- .../java/org/apache/doris/backup/BackupJob.java | 87 +- .../java/org/apache/doris/backup/RestoreJob.java | 248 ++- .../java/org/apache/doris/backup/Snapshot.java | 17 +- .../org/apache/doris/binlog/BinlogManager.java | 19 +- .../doris/catalog/BuiltinScalarFunctions.java | 4 + .../main/java/org/apache/doris/catalog/Env.java | 12 + .../java/org/apache/doris/catalog/EsTable.java | 5 +- .../doris/catalog/InternalSchemaInitializer.java | 81 + .../main/java/org/apache/doris/catalog/MTMV.java | 16 +- .../java/org/apache/doris/catalog/OlapTable.java | 12 +- .../org/apache/doris/catalog/S3StorageVault.java | 4 +- .../main/java/org/apache/doris/catalog/Table.java | 2 +- .../{load/loadv2 => catalog}/TokenManager.java | 10 +- .../apache/doris/cloud/CacheHotspotManager.java | 4 +- .../doris/cloud/analysis/UseCloudClusterStmt.java | 2 +- .../org/apache/doris/cloud/catalog/CloudEnv.java | 2 +- .../apache/doris/cloud/catalog/CloudReplica.java | 2 +- .../doris/cloud/catalog/CloudTabletRebalancer.java | 42 +- .../doris/cloud/master/CloudReportHandler.java | 89 + .../doris/cloud/qe/ComputeGroupException.java | 3 + .../apache/doris/cloud/rpc/MetaServiceClient.java | 11 + .../apache/doris/cloud/rpc/MetaServiceProxy.java | 6 + .../transaction/CloudGlobalTransactionMgr.java | 58 +- .../doris/common/proc/PartitionsProcDir.java | 31 +- .../apache/doris/common/profile/CounterNode.java | 57 - .../doris/common/profile/ExecutionProfile.java | 102 +- .../common/profile/MultiProfileTreeBuilder.java | 145 -- .../org/apache/doris/common/profile/Profile.java | 1 - .../common/{util => profile}/ProfileManager.java | 73 +- .../doris/common/profile/ProfileTreeBuilder.java | 465 ---- .../doris/common/profile/ProfileTreeNode.java | 201 -- .../doris/common/profile/ProfileTreePrinter.java | 90 - .../org/apache/doris/common/util/DocGenerator.java | 6 +- .../apache/doris/datasource/ExternalScanNode.java | 3 +- .../doris/datasource/FederationBackendPolicy.java | 2 +- .../org/apache/doris/datasource/FileSplit.java | 19 + .../doris/datasource/es/EsExternalTable.java | 16 +- .../es/EsSchemaCacheValue.java} | 36 +- .../org/apache/doris/datasource/es/EsUtil.java | 19 +- .../apache/doris/datasource/es/QueryBuilders.java | 102 +- .../doris/datasource/es/source/EsScanNode.java | 6 +- .../doris/datasource/hive/HMSExternalTable.java | 26 +- .../iceberg/source/IcebergDeleteFileFilter.java | 20 +- .../datasource/iceberg/source/IcebergScanNode.java | 7 +- .../datasource/iceberg/source/IcebergSplit.java | 6 + .../datasource/paimon/source/PaimonScanNode.java | 26 +- .../datasource/paimon/source/PaimonSplit.java | 22 + .../property/constants/S3Properties.java | 14 +- .../org/apache/doris/fs/remote/S3FileSystem.java | 11 +- .../apache/doris/fs/remote/dfs/DFSFileSystem.java | 5 + .../httpv2/controller/QueryProfileController.java | 2 +- .../org/apache/doris/httpv2/rest/LoadAction.java | 2 +- .../apache/doris/httpv2/rest/ProfileAction.java | 2 +- .../doris/httpv2/rest/manager/ClusterAction.java | 68 +- .../httpv2/rest/manager/QueryProfileAction.java | 112 +- .../apache/doris/job/extensions/mtmv/MTMVTask.java | 18 +- .../org/apache/doris/load/BrokerFileGroup.java | 19 +- .../main/java/org/apache/doris/load/ExportJob.java | 8 +- .../apache/doris/load/ExportJobStateTransfer.java | 5 +- .../org/apache/doris/load/GroupCommitManager.java | 14 +- .../apache/doris/load/loadv2/BrokerLoadJob.java | 2 +- .../apache/doris/load/loadv2/LoadLoadingTask.java | 3 + .../org/apache/doris/load/loadv2/LoadManager.java | 9 +- .../doris/load/loadv2/LoadingTaskPlanner.java | 2 +- .../apache/doris/load/loadv2/MysqlLoadManager.java | 6 +- .../doris/load/sync/canal/CanalSyncChannel.java | 2 +- .../java/org/apache/doris/master/MasterImpl.java | 4 +- .../org/apache/doris/master/ReportHandler.java | 21 +- .../doris/{spi/Split.java => mtmv/EnvInfo.java} | 49 +- .../MTMVBaseTableIf.java} | 21 +- .../java/org/apache/doris/mtmv/MTMVJobManager.java | 2 +- .../org/apache/doris/mtmv/MTMVPartitionUtil.java | 4 +- .../java/org/apache/doris/mtmv/MTMVPlanUtil.java | 27 + .../org/apache/doris/mtmv/MTMVRelatedTableIf.java | 18 +- .../doris/mysql/authenticate/ldap/LdapClient.java | 18 +- .../org/apache/doris/mysql/privilege/Auth.java | 4 +- .../apache/doris/mysql/privilege/RoleManager.java | 5 +- .../apache/doris/mysql/privilege/UserProperty.java | 45 +- .../glue/translator/ExpressionTranslator.java | 10 +- .../glue/translator/PhysicalPlanTranslator.java | 3 +- .../doris/nereids/jobs/executor/Rewriter.java | 20 +- .../doris/nereids/parser/LogicalPlanBuilder.java | 4 + .../org/apache/doris/nereids/rules/RuleType.java | 2 +- .../doris/nereids/rules/analysis/BindRelation.java | 19 +- .../expression/rules/FoldConstantRuleOnBE.java | 2 +- .../expression/rules/FoldConstantRuleOnFE.java | 4 +- .../rules/OneRangePartitionEvaluator.java | 27 +- .../rules/rewrite/PullUpJoinFromUnionAll.java | 1025 +++++---- .../doris/nereids/stats/StatsErrorEstimator.java | 2 +- .../doris/nereids/trees/expressions/Divide.java | 3 +- .../trees/expressions/ExpressionEvaluator.java | 13 - .../nereids/trees/expressions/IntegralDivide.java | 3 +- .../doris/nereids/trees/expressions/Mod.java | 3 +- .../functions/PropagateNullLiteral.java} | 15 +- .../functions/executable/NumericArithmetic.java | 34 +- .../trees/expressions/functions/scalar/Acos.java | 3 +- .../scalar/AppendTrailingCharIfAbsent.java | 3 +- .../scalar/{StContains.java => ArrayMatchAll.java} | 41 +- .../scalar/{StContains.java => ArrayMatchAny.java} | 41 +- .../trees/expressions/functions/scalar/Asin.java | 3 +- .../functions/scalar/BitmapFromArray.java | 3 +- .../functions/scalar/BitmapFromBase64.java | 3 +- .../functions/scalar/BitmapFromString.java | 3 +- .../expressions/functions/scalar/BitmapMax.java | 3 +- .../expressions/functions/scalar/BitmapMin.java | 3 +- .../functions/scalar/BitmapSubsetInRange.java | 3 +- .../functions/scalar/BitmapSubsetLimit.java | 3 +- .../expressions/functions/scalar/ConvertTz.java | 3 +- .../functions/scalar/CryptoFunction.java | 3 +- .../trees/expressions/functions/scalar/Date.java | 3 +- .../expressions/functions/scalar/DateFormat.java | 3 +- .../expressions/functions/scalar/DayCeil.java | 3 +- .../expressions/functions/scalar/DayFloor.java | 3 +- .../functions/scalar/DigitalMasking.java | 3 +- .../trees/expressions/functions/scalar/Dlog10.java | 3 +- .../trees/expressions/functions/scalar/Dsqrt.java | 3 +- .../trees/expressions/functions/scalar/Fmod.java | 3 +- .../expressions/functions/scalar/FromBase64.java | 3 +- .../expressions/functions/scalar/FromDays.java | 3 +- .../expressions/functions/scalar/FromUnixtime.java | 3 +- .../functions/scalar/GetJsonDouble.java | 3 +- .../functions/scalar/GetJsonString.java | 3 +- .../expressions/functions/scalar/HourCeil.java | 3 +- .../expressions/functions/scalar/HourFloor.java | 3 +- .../trees/expressions/functions/scalar/Ln.java | 3 +- .../trees/expressions/functions/scalar/Log10.java | 3 +- .../trees/expressions/functions/scalar/Log2.java | 3 +- .../trees/expressions/functions/scalar/Lpad.java | 3 +- .../expressions/functions/scalar/MonthCeil.java | 3 +- .../expressions/functions/scalar/MonthFloor.java | 3 +- .../expressions/functions/scalar/ParseUrl.java | 3 +- .../trees/expressions/functions/scalar/Pmod.java | 3 +- .../functions/scalar/RegexpExtract.java | 3 +- .../functions/scalar/RegexpExtractAll.java | 3 +- .../functions/scalar/RegexpReplace.java | 3 +- .../functions/scalar/RegexpReplaceOne.java | 3 +- .../trees/expressions/functions/scalar/Repeat.java | 3 +- .../trees/expressions/functions/scalar/Rpad.java | 3 +- .../expressions/functions/scalar/SecondCeil.java | 3 +- .../expressions/functions/scalar/SecondFloor.java | 3 +- .../expressions/functions/scalar/SplitPart.java | 3 +- .../trees/expressions/functions/scalar/Sqrt.java | 3 +- .../expressions/functions/scalar/StAngle.java | 3 +- .../functions/scalar/StAngleSphere.java | 3 +- .../functions/scalar/StAreaSquareKm.java | 3 +- .../functions/scalar/StAreaSquareMeters.java | 3 +- .../expressions/functions/scalar/StAstext.java | 3 +- .../expressions/functions/scalar/StContains.java | 3 +- .../functions/scalar/StDistanceSphere.java | 3 +- .../functions/scalar/StGeometryFromWKB.java | 3 +- .../functions/scalar/StGeometryfromtext.java | 3 +- .../functions/scalar/StGeomfromtext.java | 3 +- .../functions/scalar/StLinefromtext.java | 3 +- .../expressions/functions/scalar/StPoint.java | 3 +- .../functions/scalar/StPolyfromtext.java | 3 +- .../expressions/functions/scalar/StPolygon.java | 3 +- .../trees/expressions/functions/scalar/StX.java | 3 +- .../trees/expressions/functions/scalar/StY.java | 3 +- .../expressions/functions/scalar/StrToDate.java | 3 +- .../expressions/functions/scalar/SubReplace.java | 3 +- .../trees/expressions/functions/scalar/ToDate.java | 3 +- .../expressions/functions/scalar/ToDateV2.java | 3 +- .../trees/expressions/functions/udf/AliasUdf.java | 30 +- .../expressions/functions/udf/AliasUdfBuilder.java | 14 +- .../expressions/literal/StringLikeLiteral.java | 5 +- .../expressions/visitor/ScalarFunctionVisitor.java | 10 + .../trees/plans/commands/ExportCommand.java | 30 - .../trees/plans/commands/info/CreateTableInfo.java | 12 +- .../trees/plans/commands/info/IndexDefinition.java | 14 +- .../commands/insert/InsertIntoTableCommand.java | 2 +- .../insert/InsertOverwriteTableCommand.java | 33 +- .../trees/plans/commands/insert/InsertUtils.java | 2 +- .../apache/doris/persist/DropPartitionInfo.java | 4 +- .../java/org/apache/doris/persist/EditLog.java | 9 +- .../doris/persist/ModifyCommentOperationLog.java | 4 + .../org/apache/doris/persist/gson/GsonUtils.java | 30 + .../org/apache/doris/planner/ExchangeNode.java | 13 +- .../org/apache/doris/planner/OlapScanNode.java | 10 +- .../org/apache/doris/planner/PlanFragment.java | 14 +- .../java/org/apache/doris/planner/PlanNode.java | 7 + .../java/org/apache/doris/planner/ScanNode.java | 21 +- .../apache/doris/planner/SingleNodePlanner.java | 3 +- .../apache/doris/planner/StreamLoadPlanner.java | 2 +- .../java/org/apache/doris/plugin/AuditEvent.java | 7 + .../org/apache/doris/plugin/audit/AuditLoader.java | 2 +- .../java/org/apache/doris/qe/AuditLogHelper.java | 7 +- .../java/org/apache/doris/qe/ConnectContext.java | 5 - .../main/java/org/apache/doris/qe/Coordinator.java | 10 +- .../main/java/org/apache/doris/qe/DdlExecutor.java | 2 +- .../java/org/apache/doris/qe/GlobalVariable.java | 9 + .../org/apache/doris/qe/MysqlConnectProcessor.java | 1 + .../java/org/apache/doris/qe/QeProcessorImpl.java | 2 +- .../java/org/apache/doris/qe/SessionVariable.java | 192 +- .../java/org/apache/doris/qe/ShowExecutor.java | 42 +- .../java/org/apache/doris/qe/StmtExecutor.java | 57 +- .../main/java/org/apache/doris/qe/VariableMgr.java | 51 +- .../apache/doris/rewrite/FoldConstantsRule.java | 2 +- .../apache/doris/service/FrontendServiceImpl.java | 92 +- .../src/main/java/org/apache/doris/spi/Split.java | 5 + .../org/apache/doris/statistics/ColStatsData.java | 5 - .../statistics/ColumnStatisticsCacheLoader.java | 8 - .../doris/statistics/StatisticConstants.java | 2 +- .../apache/doris/statistics/StatisticsCache.java | 5 +- .../main/java/org/apache/doris/system/Backend.java | 27 +- .../org/apache/doris/system/BackendHbResponse.java | 10 +- .../java/org/apache/doris/system/HeartbeatMgr.java | 6 +- .../org/apache/doris/system/HeartbeatResponse.java | 4 +- .../org/apache/doris/system/SystemInfoService.java | 15 +- .../tablefunction/LocalTableValuedFunction.java | 2 +- .../java/org/apache/doris/task/AgentBatchTask.java | 9 + .../main/java/org/apache/doris/task/AgentTask.java | 10 + .../java/org/apache/doris/task/AgentTaskQueue.java | 2 +- .../apache/doris/transaction/TransactionEntry.java | 2 +- .../org/apache/doris/alter/IndexChangeJobTest.java | 555 +++++ .../apache/doris/analysis/CancelAlterStmtTest.java | 7 + .../org/apache/doris/analysis/IndexDefTest.java | 13 + .../catalog/InternalSchemaInitializerTest.java | 75 + .../org/apache/doris/clone/RepairVersionTest.java | 10 +- .../doris/external/elasticsearch/EsUtilTest.java | 7 +- .../external/elasticsearch/QueryBuildersTest.java | 124 +- .../doris/fs/remote/RemoteFileSystemTest.java | 158 ++ .../apache/doris/load/loadv2/TokenManagerTest.java | 1 + .../doris/nereids/parser/NereidsParserTest.java | 36 + .../rules/rewrite/PullUpJoinFromUnionTest.java | 171 ++ .../literal/StringLikeLiteralTest.java} | 47 +- .../trees/plans/commands/IndexDefinitionTest.java | 46 + .../doris/planner/FederationBackendPolicyTest.java | 95 + .../java/org/apache/doris/qe/StmtExecutorTest.java | 90 + fe/pom.xml | 2 +- gensrc/proto/cloud.proto | 14 + gensrc/thrift/FrontendService.thrift | 145 +- gensrc/thrift/HeartbeatService.thrift | 3 + gensrc/thrift/MasterService.thrift | 2 + gensrc/thrift/PaloInternalService.thrift | 6 + gensrc/thrift/PlanNodes.thrift | 17 +- gensrc/thrift/Status.thrift | 4 + .../data/auth_call/multi_load_data_1.csv | 3 + .../data/auth_call/multi_load_data_2.csv | 2 + .../routine_load_data.csv} | 0 .../data/auth_call/stream_load_data.csv | 3 + .../test_bloom_filter_drop_column.out} | 12 +- .../data/correctness_p0/test_time_round.out | 5 + .../data/datatype_p0/complex_types/test_map.out | 3 + .../data/datatype_p0/ip/test_ip_basic.out | 12 + .../import/import-way/error-data-handling.md.out | 24 + .../import/import-way/error_data_handling.csv | 3 + .../import/import-way/insert-into-manual.md.out} | 11 +- .../import/import-way/routine-load-manual.md.out | 123 ++ .../import/import-way/stream-load-manual.md.out | 154 ++ .../import/import-way/streamload_example.csv | 10 + .../import/import-way/streamload_example.json | 12 + .../data-operate/import/import-way/test_array.csv | 10 + .../data-operate/import/import-way/test_bitmap.csv | 10 + .../import/import-way/test_data_partial.csv | 3 + .../import/import-way/test_default.csv | 1 + .../import/import-way/test_enclose_and_escape.csv | 1 + .../data-operate/import/import-way/test_hll.csv | 10 + .../data-operate/import/import-way/test_map.json | 12 + .../import/import-way/test_merge_type.csv | 2 + .../import/import-way/test_rl_array.json | 3 + .../import/import-way/test_rl_bitmap.json | 3 + .../import/import-way/test_rl_column_mapping.csv | 3 + .../data-operate/import/import-way/test_rl_csv.csv | 10 + .../import/import-way/test_rl_delete.csv | 2 + .../data-operate/import/import-way/test_rl_hll.csv | 8 + .../import/import-way/test_rl_json.json | 3 + .../import/import-way/test_rl_json_path.json | 3 + .../import/import-way/test_rl_json_root.json | 3 + .../import/import-way/test_rl_map.json | 3 + .../import/import-way/test_rl_max_filter_ratio.csv | 3 + .../import/import-way/test_rl_partition.csv | 3 + .../data-operate/import/import-way/test_seq.csv | 1 + .../data/external_table_p0/es/test_es_query.out | 96 + .../data/external_table_p0/hive/test_hive_view.out | 133 ++ .../hive/test_orc_tiny_stripes.out | 2311 ++++++++++++++++++++ .../jdbc/test_mysql_jdbc_catalog.out | 18 + .../test_cloud_mow_stream_load_with_timeout.out} | 13 +- .../test_fix_tablet_stat_fault_injection.out | 13 + .../test_variant_bloom_filter.out | 40 + .../test_iot_overwrite_and_create.out | 24 + .../test_iot_overwrite_and_create_many.out | 15 + .../test_insert_nan.out} | 10 +- .../array_contains/arr_null_test_data.csv | 200 ++ .../test_array_with_inverted_index_all_type.out | 36 +- .../test_index_match_phrase_edge.out | 24 + .../inverted_index_p0/test_index_multi_match.out | 144 ++ .../documents-1000.json | 0 .../routine_load/test_routine_load_with_udf.out} | 10 +- .../load_p0/stream_load/test_stream_load_udf.csv | 3 + .../stream_load/test_stream_load_with_udf.out} | 12 +- .../broker_load/test_s3_load_properties.out | 18 + .../data/nereids_function_p0/fn_test_am.csv | 102 + .../nereids_function_p0/scalar_function/Array.out | 1320 +++++++++++ .../data/nereids_hint_tpcds_p0/shape/query11.out | 2 +- .../data/nereids_hint_tpcds_p0/shape/query14.out | 62 +- .../data/nereids_hint_tpcds_p0/shape/query4.out | 2 +- .../data/nereids_hint_tpcds_p0/shape/query71.out | 41 +- .../data/nereids_hint_tpcds_p0/shape/query74.out | 2 +- .../data/nereids_hint_tpcds_p0/shape/query76.out | 58 +- .../sql_functions/string_functions/test_concat.out | Bin 0 -> 4889 bytes .../pull_up_join_from_union.out | 306 +++ .../shape/query11.out | 2 +- .../shape/query14.out | 62 +- .../nereids_tpcds_shape_sf1000_p0/shape/query4.out | 2 +- .../shape/query71.out | 45 +- .../shape/query74.out | 2 +- .../shape/query76.out | 58 +- .../noStatsRfPrune/query11.out | 2 +- .../noStatsRfPrune/query14.out | 44 +- .../noStatsRfPrune/query4.out | 2 +- .../noStatsRfPrune/query71.out | 41 +- .../noStatsRfPrune/query74.out | 2 +- .../noStatsRfPrune/query76.out | 63 +- .../no_stats_shape/query11.out | 2 +- .../no_stats_shape/query14.out | 62 +- .../no_stats_shape/query4.out | 2 +- .../no_stats_shape/query71.out | 41 +- .../no_stats_shape/query74.out | 2 +- .../no_stats_shape/query76.out | 63 +- .../rf_prune/query11.out | 2 +- .../rf_prune/query14.out | 44 +- .../rf_prune/query4.out | 2 +- .../rf_prune/query71.out | 45 +- .../rf_prune/query74.out | 2 +- .../rf_prune/query76.out | 63 +- .../nereids_tpcds_shape_sf100_p0/shape/query11.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query14.out | 62 +- .../nereids_tpcds_shape_sf100_p0/shape/query4.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query71.out | 45 +- .../nereids_tpcds_shape_sf100_p0/shape/query74.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query76.out | 63 +- .../shape/query14.out | 62 +- .../shape/query71.out | 41 +- .../shape/query76.out | 63 +- .../tpcds_sf100/noStatsRfPrune/query11.out | 2 +- .../tpcds_sf100/noStatsRfPrune/query14.out | 44 +- .../tpcds_sf100/noStatsRfPrune/query4.out | 2 +- .../tpcds_sf100/noStatsRfPrune/query71.out | 41 +- .../tpcds_sf100/noStatsRfPrune/query74.out | 2 +- .../tpcds_sf100/noStatsRfPrune/query76.out | 63 +- .../tpcds_sf100/no_stats_shape/query11.out | 2 +- .../tpcds_sf100/no_stats_shape/query14.out | 62 +- .../tpcds_sf100/no_stats_shape/query4.out | 2 +- .../tpcds_sf100/no_stats_shape/query71.out | 41 +- .../tpcds_sf100/no_stats_shape/query74.out | 2 +- .../tpcds_sf100/no_stats_shape/query76.out | 63 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query11.out | 2 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query14.out | 44 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query4.out | 2 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query71.out | 45 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query74.out | 2 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query76.out | 63 +- .../new_shapes_p0/tpcds_sf100/shape/query11.out | 2 +- .../new_shapes_p0/tpcds_sf100/shape/query14.out | 62 +- .../new_shapes_p0/tpcds_sf100/shape/query4.out | 2 +- .../new_shapes_p0/tpcds_sf100/shape/query71.out | 45 +- .../new_shapes_p0/tpcds_sf100/shape/query74.out | 2 +- .../new_shapes_p0/tpcds_sf100/shape/query76.out | 63 +- .../new_shapes_p0/tpcds_sf1000/shape/query11.out | 2 +- .../new_shapes_p0/tpcds_sf1000/shape/query14.out | 62 +- .../new_shapes_p0/tpcds_sf1000/shape/query4.out | 2 +- .../new_shapes_p0/tpcds_sf1000/shape/query71.out | 45 +- .../new_shapes_p0/tpcds_sf1000/shape/query74.out | 2 +- .../new_shapes_p0/tpcds_sf1000/shape/query76.out | 63 +- .../data/query_p0/scan_range/test_scan_range.out | 5 + .../org/apache/doris/regression/suite/Suite.groovy | 92 +- .../doris/regression/suite/SuiteCluster.groovy | 5 + .../apache/doris/regression/suite/Syncer.groovy | 2 + .../java/org/apache/doris/udf/IntLoadTest.java | 11 +- .../java/org/apache/doris/udf/StringLoadTest.java | 14 +- .../pipeline/cloud_p0/conf/be_custom.conf | 1 + .../pipeline/cloud_p1/conf/be_custom.conf | 1 + regression-test/pipeline/p0/conf/be.conf | 1 + regression-test/pipeline/p1/conf/be.conf | 1 + .../suites/audit/test_audit_log_behavior.groovy | 11 +- .../suites/auth_call/data/multi_table_csv.csv | 2 + .../ddl/test_dml_multi_routine_load_auth_tb1.sql | 46 + .../ddl/test_dml_multi_routine_load_auth_tb2.sql | 46 + .../test_account_management_grant_auth.groovy | 100 + .../test_account_management_role_auth.groovy | 73 + .../test_account_management_user_auth.groovy | 99 + .../auth_call/test_assistant_command_auth.groovy | 96 + .../auth_call/test_cluster_management_auth.groovy | 138 ++ .../auth_call/test_database_management_auth.groovy | 143 ++ .../suites/auth_call/test_ddl_backup_auth.groovy | 126 ++ .../suites/auth_call/test_ddl_catalog_auth.groovy | 113 + .../auth_call/test_ddl_colocate_group_auth.groovy | 67 + .../suites/auth_call/test_ddl_database_auth.groovy | 103 + .../auth_call/test_ddl_encryptkey_auth.groovy | 72 + .../suites/auth_call/test_ddl_file_auth.groovy | 89 + .../suites/auth_call/test_ddl_function_auth.groovy | 87 + .../suites/auth_call/test_ddl_index_auth.groovy | 126 ++ .../suites/auth_call/test_ddl_job_auth.groovy | 99 + .../auth_call/test_ddl_mask_view_auth.groovy | 74 + .../suites/auth_call/test_ddl_mtmv_auth.groovy | 189 ++ .../suites/auth_call/test_ddl_mv_auth.groovy | 117 + .../auth_call/test_ddl_part_table_auth.groovy | 80 + .../auth_call/test_ddl_policy_storage_auth.groovy | 96 + .../auth_call/test_ddl_repository_auth.groovy | 93 + .../suites/auth_call/test_ddl_resource_auth.groovy | 97 + .../suites/auth_call/test_ddl_restore_auth.groovy | 181 ++ .../auth_call/test_ddl_row_policy_auth.groovy | 90 + .../auth_call/test_ddl_sql_block_rule_auth.groovy | 102 + .../suites/auth_call/test_ddl_table_auth.groovy | 267 +++ .../suites/auth_call/test_ddl_view_auth.groovy | 154 ++ .../auth_call/test_ddl_workload_group_auth.groovy | 86 + .../suites/auth_call/test_dml_analyze_auth.groovy | 93 + .../auth_call/test_dml_broker_load_auth.groovy | 180 ++ .../auth_call/test_dml_cancel_profile_auth.groovy | 54 + .../auth_call/test_dml_delete_table_auth.groovy | 151 ++ .../auth_call/test_dml_export_table_auth.groovy | 132 ++ .../suites/auth_call/test_dml_insert_auth.groovy | 121 + .../test_dml_multi_routine_load_auth.groovy | 160 ++ .../auth_call/test_dml_mysql_load_auth.groovy | 84 + .../suites/auth_call/test_dml_outfile_auth.groovy | 92 + .../auth_call/test_dml_routine_load_auth.groovy | 153 ++ .../auth_call/test_dml_select_udf_auth.groovy | 77 + .../auth_call/test_dml_stream_load_auth.groovy | 115 + .../auth_call/test_dml_update_table_auth.groovy | 81 + .../auth_call/test_hive_base_case_auth.groovy | 200 ++ .../suites/auth_call/test_show_backend_auth.groovy | 49 + .../suites/auth_call/test_show_broker_auth.groovy | 48 + .../suites/auth_call/test_show_charset_auth.groovy | 79 + .../test_show_convert_light_sc_auth.groovy | 48 + .../auth_call/test_show_create_table_auth.groovy | 92 + .../suites/auth_call/test_show_data_auth.groovy | 89 + .../auth_call/test_show_database_id_auth.groovy | 63 + .../auth_call/test_show_dynamic_table_auth.groovy | 67 + .../auth_call/test_show_frontend_auth.groovy | 55 + .../suites/auth_call/test_show_grant_auth.groovy | 56 + .../suites/auth_call/test_show_proc_auth.groovy | 48 + .../auth_call/test_show_query_stats_auth.groovy | 50 +- .../auth_call/test_show_repository_auth.groovy | 75 + .../suites/auth_call/test_show_tablet_auth.groovy | 91 + .../auth_call/test_show_typecast_auth.groovy | 49 +- .../suites/auth_up_down_hive_p0/load.groovy | 45 + .../test_up_down_hive_auth.groovy | 45 +- .../test_backup_restore_atomic_with_alter.groovy | 6 +- ...est_backup_restore_backup_temp_partition.groovy | 2 +- .../test_bloom_filter_drop_column.groovy | 54 + ...est_bloom_filter_hit_with_renamed_column.groovy | 2 + .../cloud_p0/auth/test_set_default_cluster.groovy | 96 + .../test_clean_tablet_when_drop_force_table.groovy | 140 ++ .../test_clean_tablet_when_rebalance.groovy | 158 ++ ...e_compaction_with_variant_inverted_index.groovy | 2 + .../suites/control_p0/test_tablet_io_error.groovy | 111 + .../suites/correctness_p0/test_time_round.groovy | 1 + .../datatype_p0/complex_types/test_map.groovy | 35 + .../suites/datatype_p0/ip/test_ip_basic.groovy | 14 + .../doc/admin-manual/data-admin/backup.md.groovy | 93 + .../admin-manual/data-admin/recyclebin.md.groovy | 49 + .../doc/admin-manual/data-admin/restore.md.groovy | 99 + .../import/import-way/client_local.csv | 6 + .../doc/data-operate/import/import-way/data.csv | 2 + .../import-way/error-data-handling.md.groovy | 322 +++ .../import-way/group-commit-manual.md.groovy | 154 ++ .../import/import-way/insert-into-manual.md.groovy | 93 + .../import/import-way/mysql-load-manual.md.groovy | 160 ++ .../import-way/routine-load-manual.md.groovy | 1019 +++++++++ .../import/import-way/stream-load-manual.md.groovy | 492 +++++ .../doc/table-design/auto-increment.md.groovy | 296 +++ .../doc/table-design/best-practice.md.groovy | 193 ++ .../table-design/data-model/aggregate.md.groovy | 89 + .../table-design/data-model/duplicate.md.groovy | 77 + .../doc/table-design/data-model/unique.md.groovy | 63 + .../doc/table-design/data-partition.md.groovy | 311 +++ .../doc/table-design/index/bloomfilter.md.groovy | 43 + .../table-design/index/inverted-index.md.groovy | 122 ++ .../index/ngram-bloomfilter-index.md.groovy | 80 + .../doc/table-design/index/prefix-index.md.groovy | 50 + .../suites/doc/table-design/row-store.md.groovy | 48 + .../doc/table-design/schema-change.md.groovy | 211 ++ .../suites/doc/table-design/test_data/test.csv | 2 + .../tiered-storage/remote-storage.md.groovy | 164 ++ .../suites/export_p0/test_export_basic.groovy | 55 +- .../external_table_p0/es/test_es_query.groovy | 10 +- .../external_table_p0/hive/test_hive_view.groovy | 66 + .../hive/test_orc_tiny_stripes.groovy | 203 ++ .../jdbc/test_clickhouse_jdbc_catalog.groovy | 2 + .../jdbc/test_mysql_jdbc_catalog.groovy | 1 + .../test_cloud_mow_broker_load_with_retry.groovy | 2 +- .../test_cloud_mow_stream_load_with_timeout.groovy | 189 ++ .../test_fix_tablet_stat_fault_injection.groovy | 159 ++ ...est_match_without_index_fault_injection.groovy} | 4 +- .../test_variant_bloom_filter.groovy | 140 ++ .../index_p0/test_ngram_bloomfilter_index.groovy | 47 + .../test_iot_overwrite_and_create.groovy | 71 + .../test_iot_overwrite_and_create_many.groovy | 64 + .../group_commit/replay_wal_restart_fe.groovy | 2 +- .../suites/insert_p0/test_insert_nan.groovy | 21 +- .../array_contains/test_add_index_for_arr.groovy | 155 ++ .../test_index_change_on_renamed_column.groovy | 2 +- .../test_index_compaction_empty_segments.groovy | 99 + .../test_index_compaction_p0.groovy | 151 ++ .../inverted_index_p0/load/test_insert.groovy | 4 + .../inverted_index_p0/load/test_stream_load.groovy | 4 + .../test_array_with_inverted_index_all_type.groovy | 4 + .../test_index_match_phrase_edge.groovy | 98 + .../test_index_multi_match.groovy | 57 + .../inverted_index_p0/test_no_index_match.groovy | 7 + ...sted_index_file_http_action_with_variant.groovy | 2 + .../test_index_compaction_p1.groovy | 151 ++ .../test_variant_index_format_v1.groovy | 3 +- .../test_json_reader_without_object.groovy | 57 + ...oad_with_user.csv => test_routine_load_udf.csv} | 0 .../routine_load/test_routine_load_with_udf.groovy | 126 ++ .../stream_load/test_stream_load_with_udf.groovy | 65 + .../load_p2/broker_load/test_broker_load.groovy | 578 +++-- .../broker_load/test_s3_load_properties.groovy | 32 + .../suites/load_p2/test_single_replica_load.groovy | 2 + .../suites/nereids_function_p0/load.groovy | 25 + .../scalar_function/Array.groovy | 42 + .../fold_constant/fold_constant_nullable.groovy | 178 ++ .../fold_constant_numeric_arithmatic.groovy | 25 + .../nereids_p0/javaudf/test_alias_function.groovy | 16 +- .../sql_functions/string_functions/test_concat.sql | 273 +++ .../partition_prune/test_date_trunc_prune.groovy | 25 + .../pull_up_join_from_union.groovy | 429 ++++ .../test_dynamic_partition.groovy | 25 +- .../query_p0/scan_range/test_scan_range.groovy | 5 + .../test_alter_table_partition.groovy | 139 ++ .../suites/show_p0/test_show_tablet.groovy | 53 +- .../suites/show_p0/test_show_variables.groovy | 38 + .../suites/statistics/analyze_stats.groovy | 4 +- regression-test/suites/stats_up_down/load.groovy | 66 + .../test_upgrade_downgrade_stats.groovy | 75 + regression-test/suites/variant_p0/load.groovy | 2 +- .../suites/variant_p0/with_index/var_index.groovy | 60 + .../alter/test_alter_use_path_style.groovy | 108 + .../vault_p0/alter/test_alter_vault_type.groovy | 78 + .../vault_p0/create/test_create_vault.groovy | 2 +- .../vault_p0/default/test_default_vault.groovy | 8 +- tools/fdb/fdb_ctl.sh | 38 +- 874 files changed, 34132 insertions(+), 6746 deletions(-) delete mode 100644 .dlc.json delete mode 100644 .github/workflows/deadlink-check.yml create mode 100644 be/src/olap/tablet_column_object_pool.cpp create mode 100644 be/src/olap/tablet_column_object_pool.h copy be/src/{util/timezone_utils.h => runtime/cluster_info.h} (51%) create mode 100644 be/src/vec/functions/array/varray_match_function.cpp create mode 100644 be/test/common/schema_util_test.cpp create mode 100644 be/test/olap/date_bloom_filter_test.cpp copy be/test/olap/rowset/segment_v2/inverted_index/compaction/{index_compaction_with_deleted_term.cpp => index_compaction_test.cpp} (59%) create mode 100644 be/test/olap/rowset/segment_v2/inverted_index/compaction/index_compaction_write_index_test.cpp create mode 100644 be/test/olap/rowset/segment_v2/inverted_index/data/data1.csv create mode 100644 be/test/olap/rowset/segment_v2/inverted_index/data/data2.csv create mode 100644 be/test/olap/tablet_index_test.cpp create mode 100644 be/test/util/timezone_utils_test.cpp create mode 100644 be/test/vec/core/column_map_test.cpp create mode 100644 be/test/vec/core/column_struct_test.cpp create mode 100644 docker/runtime/doris-compose/resource/entrypoint.sh create mode 100644 docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run67.hql create mode 100644 docker/thirdparties/docker-compose/hive/scripts/create_view_scripts/create_view.hql create mode 100644 docker/thirdparties/docker-compose/hive/scripts/preinstalled_data/orc/orc_tiny_stripes/output_60_3.orc create mode 100644 docker/thirdparties/docker-compose/hive/scripts/preinstalled_data/orc/orc_tiny_stripes/output_6_1.orc create mode 100644 docker/thirdparties/docker-compose/hive/scripts/preinstalled_data/orc/orc_tiny_stripes/random_output_60_3.orc create mode 100644 docker/thirdparties/docker-compose/hive/scripts/preinstalled_data/orc/orc_tiny_stripes/random_output_6_1.orc create mode 100644 fe/fe-common/src/main/java/org/apache/doris/common/GZIPUtils.java create mode 100644 fe/fe-common/src/main/java/org/apache/doris/common/ThriftUtils.java create mode 100644 fe/fe-common/src/test/java/org/apache/doris/common/security/authentication/AuthenticationTest.java rename fe/fe-core/src/main/java/org/apache/doris/{load/loadv2 => catalog}/TokenManager.java (95%) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/cloud/master/CloudReportHandler.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/common/profile/CounterNode.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/common/profile/MultiProfileTreeBuilder.java rename fe/fe-core/src/main/java/org/apache/doris/common/{util => profile}/ProfileManager.java (92%) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeBuilder.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeNode.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreePrinter.java copy fe/fe-core/src/main/java/org/apache/doris/{spi/Split.java => datasource/es/EsSchemaCacheValue.java} (61%) copy fe/fe-core/src/main/java/org/apache/doris/{spi/Split.java => mtmv/EnvInfo.java} (57%) copy fe/fe-core/src/main/java/org/apache/doris/{common/profile/ExecNodeNode.java => mtmv/MTMVBaseTableIf.java} (62%) copy fe/fe-core/src/main/java/org/apache/doris/{common/profile/ExecNodeNode.java => nereids/trees/expressions/functions/PropagateNullLiteral.java} (78%) copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{StContains.java => ArrayMatchAll.java} (59%) copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{StContains.java => ArrayMatchAny.java} (59%) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/alter/IndexChangeJobTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/catalog/InternalSchemaInitializerTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/fs/remote/RemoteFileSystemTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PullUpJoinFromUnionTest.java copy fe/fe-core/src/{main/java/org/apache/doris/spi/Split.java => test/java/org/apache/doris/nereids/trees/expressions/literal/StringLikeLiteralTest.java} (56%) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/IndexDefinitionTest.java create mode 100644 regression-test/data/auth_call/multi_load_data_1.csv create mode 100644 regression-test/data/auth_call/multi_load_data_2.csv copy regression-test/data/{ddl_p0/test_create_table_generated_column/gen_col_data.csv => auth_call/routine_load_data.csv} (100%) create mode 100644 regression-test/data/auth_call/stream_load_data.csv copy regression-test/data/{query_p0/scan_range/test_scan_range.out => bloom_filter_p0/test_bloom_filter_drop_column.out} (62%) create mode 100644 regression-test/data/doc/data-operate/import/import-way/error-data-handling.md.out create mode 100644 regression-test/data/doc/data-operate/import/import-way/error_data_handling.csv copy regression-test/data/{query_p0/scan_range/test_scan_range.out => doc/data-operate/import/import-way/insert-into-manual.md.out} (62%) create mode 100644 regression-test/data/doc/data-operate/import/import-way/routine-load-manual.md.out create mode 100644 regression-test/data/doc/data-operate/import/import-way/stream-load-manual.md.out create mode 100644 regression-test/data/doc/data-operate/import/import-way/streamload_example.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/streamload_example.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_array.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_bitmap.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_data_partial.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_default.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_enclose_and_escape.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_hll.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_map.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_merge_type.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_array.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_bitmap.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_column_mapping.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_csv.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_delete.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_hll.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_json.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_json_path.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_json_root.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_map.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_max_filter_ratio.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_rl_partition.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_seq.csv create mode 100644 regression-test/data/external_table_p0/hive/test_hive_view.out create mode 100644 regression-test/data/external_table_p0/hive/test_orc_tiny_stripes.out copy regression-test/data/{query_p0/scan_range/test_scan_range.out => fault_injection_p0/cloud/test_cloud_mow_stream_load_with_timeout.out} (60%) create mode 100644 regression-test/data/fault_injection_p0/test_fix_tablet_stat_fault_injection.out create mode 100644 regression-test/data/fault_injection_p0/test_variant_bloom_filter.out create mode 100644 regression-test/data/insert_overwrite_p0/test_iot_overwrite_and_create.out create mode 100644 regression-test/data/insert_overwrite_p0/test_iot_overwrite_and_create_many.out copy regression-test/data/{query_p0/scan_range/test_scan_range.out => insert_p0/test_insert_nan.out} (62%) create mode 100644 regression-test/data/inverted_index_p0/array_contains/arr_null_test_data.csv copy regression-test/data/{fault_injection_p0 => inverted_index_p2}/documents-1000.json (100%) copy regression-test/data/{query_p0/scan_range/test_scan_range.out => load_p0/routine_load/test_routine_load_with_udf.out} (54%) create mode 100644 regression-test/data/load_p0/stream_load/test_stream_load_udf.csv copy regression-test/data/{query_p0/scan_range/test_scan_range.out => load_p0/stream_load/test_stream_load_with_udf.out} (62%) create mode 100644 regression-test/data/nereids_function_p0/fn_test_am.csv create mode 100644 regression-test/data/nereids_p0/sql_functions/string_functions/test_concat.out create mode 100644 regression-test/data/nereids_rules_p0/pull_up_join_from_union/pull_up_join_from_union.out copy fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecNodeNode.java => regression-test/java-udf-src/src/main/java/org/apache/doris/udf/IntLoadTest.java (82%) copy fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecNodeNode.java => regression-test/java-udf-src/src/main/java/org/apache/doris/udf/StringLoadTest.java (80%) create mode 100644 regression-test/suites/auth_call/data/multi_table_csv.csv create mode 100644 regression-test/suites/auth_call/ddl/test_dml_multi_routine_load_auth_tb1.sql create mode 100644 regression-test/suites/auth_call/ddl/test_dml_multi_routine_load_auth_tb2.sql create mode 100644 regression-test/suites/auth_call/test_account_management_grant_auth.groovy create mode 100644 regression-test/suites/auth_call/test_account_management_role_auth.groovy create mode 100644 regression-test/suites/auth_call/test_account_management_user_auth.groovy create mode 100644 regression-test/suites/auth_call/test_assistant_command_auth.groovy create mode 100644 regression-test/suites/auth_call/test_cluster_management_auth.groovy create mode 100644 regression-test/suites/auth_call/test_database_management_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_backup_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_catalog_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_colocate_group_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_database_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_encryptkey_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_file_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_function_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_index_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_job_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_mask_view_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_mtmv_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_mv_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_part_table_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_policy_storage_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_repository_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_resource_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_restore_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_row_policy_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_sql_block_rule_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_table_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_view_auth.groovy create mode 100644 regression-test/suites/auth_call/test_ddl_workload_group_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_analyze_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_broker_load_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_cancel_profile_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_delete_table_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_export_table_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_insert_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_multi_routine_load_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_mysql_load_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_outfile_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_routine_load_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_select_udf_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_stream_load_auth.groovy create mode 100644 regression-test/suites/auth_call/test_dml_update_table_auth.groovy create mode 100644 regression-test/suites/auth_call/test_hive_base_case_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_backend_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_broker_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_charset_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_convert_light_sc_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_create_table_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_data_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_database_id_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_dynamic_table_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_frontend_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_grant_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_proc_auth.groovy copy fe/fe-core/src/main/java/org/apache/doris/spi/Split.java => regression-test/suites/auth_call/test_show_query_stats_auth.groovy (50%) create mode 100644 regression-test/suites/auth_call/test_show_repository_auth.groovy create mode 100644 regression-test/suites/auth_call/test_show_tablet_auth.groovy copy fe/fe-core/src/main/java/org/apache/doris/spi/Split.java => regression-test/suites/auth_call/test_show_typecast_auth.groovy (53%) create mode 100644 regression-test/suites/auth_up_down_hive_p0/load.groovy copy fe/fe-core/src/main/java/org/apache/doris/spi/Split.java => regression-test/suites/auth_up_down_hive_p0/test_up_down_hive_auth.groovy (54%) create mode 100644 regression-test/suites/bloom_filter_p0/test_bloom_filter_drop_column.groovy create mode 100644 regression-test/suites/cloud_p0/auth/test_set_default_cluster.groovy create mode 100644 regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_drop_force_table.groovy create mode 100644 regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_rebalance.groovy create mode 100644 regression-test/suites/control_p0/test_tablet_io_error.groovy create mode 100644 regression-test/suites/doc/admin-manual/data-admin/backup.md.groovy create mode 100644 regression-test/suites/doc/admin-manual/data-admin/recyclebin.md.groovy create mode 100644 regression-test/suites/doc/admin-manual/data-admin/restore.md.groovy create mode 100644 regression-test/suites/doc/data-operate/import/import-way/client_local.csv create mode 100644 regression-test/suites/doc/data-operate/import/import-way/data.csv create mode 100644 regression-test/suites/doc/data-operate/import/import-way/error-data-handling.md.groovy create mode 100644 regression-test/suites/doc/data-operate/import/import-way/group-commit-manual.md.groovy create mode 100644 regression-test/suites/doc/data-operate/import/import-way/insert-into-manual.md.groovy create mode 100644 regression-test/suites/doc/data-operate/import/import-way/mysql-load-manual.md.groovy create mode 100644 regression-test/suites/doc/data-operate/import/import-way/routine-load-manual.md.groovy create mode 100644 regression-test/suites/doc/data-operate/import/import-way/stream-load-manual.md.groovy create mode 100644 regression-test/suites/doc/table-design/auto-increment.md.groovy create mode 100644 regression-test/suites/doc/table-design/best-practice.md.groovy create mode 100644 regression-test/suites/doc/table-design/data-model/aggregate.md.groovy create mode 100644 regression-test/suites/doc/table-design/data-model/duplicate.md.groovy create mode 100644 regression-test/suites/doc/table-design/data-model/unique.md.groovy create mode 100644 regression-test/suites/doc/table-design/data-partition.md.groovy create mode 100644 regression-test/suites/doc/table-design/index/bloomfilter.md.groovy create mode 100644 regression-test/suites/doc/table-design/index/inverted-index.md.groovy create mode 100644 regression-test/suites/doc/table-design/index/ngram-bloomfilter-index.md.groovy create mode 100644 regression-test/suites/doc/table-design/index/prefix-index.md.groovy create mode 100644 regression-test/suites/doc/table-design/row-store.md.groovy create mode 100644 regression-test/suites/doc/table-design/schema-change.md.groovy create mode 100644 regression-test/suites/doc/table-design/test_data/test.csv create mode 100644 regression-test/suites/doc/table-design/tiered-storage/remote-storage.md.groovy create mode 100644 regression-test/suites/external_table_p0/hive/test_hive_view.groovy create mode 100644 regression-test/suites/external_table_p0/hive/test_orc_tiny_stripes.groovy create mode 100644 regression-test/suites/fault_injection_p0/cloud/test_cloud_mow_stream_load_with_timeout.groovy create mode 100644 regression-test/suites/fault_injection_p0/test_fix_tablet_stat_fault_injection.groovy rename regression-test/suites/{inverted_index_p0/test_match_without_index.groovy => fault_injection_p0/test_match_without_index_fault_injection.groovy} (96%) create mode 100644 regression-test/suites/fault_injection_p0/test_variant_bloom_filter.groovy create mode 100644 regression-test/suites/insert_overwrite_p0/test_iot_overwrite_and_create.groovy create mode 100644 regression-test/suites/insert_overwrite_p0/test_iot_overwrite_and_create_many.groovy rename fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecNodeNode.java => regression-test/suites/insert_p0/test_insert_nan.groovy (64%) create mode 100644 regression-test/suites/inverted_index_p0/array_contains/test_add_index_for_arr.groovy create mode 100644 regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_empty_segments.groovy create mode 100644 regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_p0.groovy create mode 100644 regression-test/suites/inverted_index_p1/index_compaction/test_index_compaction_p1.groovy create mode 100644 regression-test/suites/jsonb_p0/test_json_reader_without_object.groovy copy regression-test/suites/load_p0/routine_load/data/{test_routine_load_with_user.csv => test_routine_load_udf.csv} (100%) create mode 100644 regression-test/suites/load_p0/routine_load/test_routine_load_with_udf.groovy create mode 100644 regression-test/suites/load_p0/stream_load/test_stream_load_with_udf.groovy create mode 100644 regression-test/suites/nereids_p0/expression/fold_constant/fold_constant_nullable.groovy create mode 100644 regression-test/suites/nereids_p0/sql_functions/string_functions/test_concat.sql create mode 100644 regression-test/suites/nereids_rules_p0/pull_up_join_from_union/pull_up_join_from_union.groovy create mode 100644 regression-test/suites/schema_change_p0/test_alter_table_partition.groovy copy fe/fe-core/src/main/java/org/apache/doris/spi/Split.java => regression-test/suites/show_p0/test_show_tablet.groovy (55%) create mode 100644 regression-test/suites/show_p0/test_show_variables.groovy create mode 100644 regression-test/suites/stats_up_down/load.groovy create mode 100644 regression-test/suites/stats_up_down/test_upgrade_downgrade_stats.groovy create mode 100644 regression-test/suites/vault_p0/alter/test_alter_use_path_style.groovy create mode 100644 regression-test/suites/vault_p0/alter/test_alter_vault_type.groovy --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org