This is an automated email from the ASF dual-hosted git repository. mrhhsg pushed a change to branch spill_and_reserve in repository https://gitbox.apache.org/repos/asf/doris.git
discard 7041d39da9f Spill updated (#40798) omit 43c64397518 set min reserve bytes to 1MB (#40795) omit c60f093012e add reserve memory logic in scan operator (#40719) omit 19785489e87 fix bug omit f3d4f8d98b3 add low memory mode in scan operator (#40662) omit 8f27ff3e848 [fix](scanner) Fix deadlock when scanner submit failed (#40495) omit 12c06129483 [opt](scanner profile) More counter for scanner (#40144) omit c1d8332c53b [fix](exchange) fix coredump of update counter is block is null omit 05a76b686dc move paused query and handle logic to workload group manager (#40603) omit 5d18956013f [fix](memory) Not check process memory limit when thread reserve memory is sufficient (#40548) omit 2970d9121ed [improvement](spill) add counter for memory usage (#40572) omit acb837d0de4 [fix] avoid crash when no data to revoke omit 5989d3f0f2e [opt](spill) avoid query blocked when reserving failed (#40550) omit 081eca9c875 [opt](spill) Avoid occupying a large amount of memory in join build side omit 871b0d87b06 [fix](memory) Fix ThreadMemTrackerMgr consumer tracker remove reserve memory (#40380) (#40452) omit e99dc64f048 1. remove revoke memory logic in execute task 2. use memory error code when cancel query (#40426) omit bf03ec25a70 [revokememory] remove old revoke memory logic omit 75bcf63d8d1 [refactor](codestyle) use pointer as return value (#40396) omit a74129abab2 updated omit 090bef5f274 updated omit 122f486577c [feat]reserve and spill add 3615a361076 [ut](jsonb) fix expected output for cast json type (#40251) add 895dddfc1e0 [fix] (regression test) fix error test (#40098) add ee3be4fa769 [fix](index compaction)Fix core when having multiple dest segments (#40254) add a1ea5c467ab [Fix](planner) legacy planner repeat group by has grouping report error (#40281) add 1eef8a83975 [fix](ut) fix SegmentTest and SegmentMoWTest asan core (#40287) add 33ade8fe82d [fix](regression) speed up some case (#40237) add adff882f787 [Chore](inverted index) remove useless code of compound filters for inverted index (#40258) add 8bf3ff4ea2d [Enhancement](Group commit) Remove the redundant `iscloud` parameter (#40120) add ff24a4facb0 [Fix](regression-test) Fix `test_group_commit_stream_load_with_nonexist_db_and_table` regression test case (#40109) add 5ea6e4bc035 [enhance](mtmv)Optimize the speed of obtaining the last update time of Hive (#40169) add 80c65f2ce0c [fix](memory) When Load ends, check memory tracker value returns is equal to 0 (#40016) add a4956d6d378 [feature](metacache) add system table catalog_meta_cache_statistics (#40155) add b7e6d7350bc [fix](testcase) fix unstable case (#40124) add 8c6da41e24f [feature](statistics)Support enable/disable auto analyze for table. (#39532) add 000fc9e2fb7 [Bug](sink) update error status on BufferControlBlock::close (#40116) add 34754ca34df [fix](recycler) Implement missing delete_prefix() of hdfs accessor (#40215) add cdfe3c24cd3 [opt](Nereids) add show statement syntax (#40288) add f363031409a [test](inverted index) fix case for index_compaction (#40313) add ac4e1a29d17 [ut](data type) fix get_common_type_test (#40317) add 65cf3950191 [chore](errmsg) Daemon thread interruption should be info log (#40297) add fb612c3be36 [Fix](testcase) Split auto and dynamic partition into individual case (#40307) add 5e93a71f5ff [Fix](inverted index) fix profile metrics for inverted index after refactor (#40324) add d0654cd2930 [Feat](Nereids) add use mv hint (#40167) add b9331b5e7c8 [chore](recycler) Change tablet_rowset_stale_sweep_time_sec=600 to reduce possibility failed to find rowset (#40076) add da5264840d2 [chore](cloud) Improve cloud start stop script: wait for quit synchronously (#40218) add c363830612a [chore] Reduce range of nextId when calling advanceNextId() (#40160) add 4c1b9298ae8 [fix](unary function) Fix wrong result of asin, acos and sqrt when processing invalid input (#40267) add 7b860577aeb [opt](scanner) Control the degree of parallelism of scanner when only limit involved (#39927) add 33bd7545c6f [fix](memory) Fix `sum_of_all_trackers` duplicated contain reserved memory (#40304) add b2ebfe9f3b5 [Fix](inverted index) fix comparison for string after refactor compound (#40338) add 2615a039833 [Fix](Variant) sparse columns should not be added in init segment iterator (#40295) add cfee8c9c63b [fix](ga) fix scope label and add samples exclusion to shell checks (#40343) add 9a92b4c3809 [opt](function)Some geo functions incorrectly used get. (#40107) add bfbb065d0e0 [Feat](nereids) add max/min filter push down rewrite rule (#39252) add 08a3e56c2c3 [Fix](function) Refine crypto functions signature to fix wrong result (#40285) add b2ddc8d66f2 [fix](nereids) fix bug for A>n, where A.max is infinity (#39936) add dde1e0f5d7f [test](inverted index) refine test_ignore_above case, add compound query sql (#40355) add 0baa9366703 [test](mtmv) SSB mv rewrite test use little data set for test performance (#40188) add b2738496b79 [enhencement](sample) add delta and kudu samples (#40067) add 5eab877c890 [opt](in expr) Optimize the IN expression by skipping constant columns. (#39912) add f0d936ac306 [fix] (compaction) fix compaction score in time series policy (#40242) add 2fd75937b24 [fix](decimal) throw overflow exception if result is NaN of Infinit when converting from decimal to float (#40290) add 6621ae688c1 [fix](ci) add retry to avoid compile fail (#40381) add 9963ec2ddfb [chore](Nereids) do not fallback if parse failed for all case (#40331) add edb6eec2dab [fix](OrcWriter) fix be core when upgrading BE without upgrading FE (#40282) add 3856acad352 [Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked (#40322) add 9ad1f236eab [Fix](ShortCircuit) consider delete sign flag when hits row (#40300) add d177d5b0048 [chore](cloud) Support starting both meta-service and recycler within single process (#40223) add 553de2d83ab [fix](cloud) Fix missing privilege to storage vault after restarting FE (#40260) add 9f46335476e [fix](decompressor) consider the large_block_len is 0 (#40183) add 301bd04fc9a [chore](ci) tmp remove required of Clang Formatter (#40392) add ae78c52fc99 [fix](ga) try fix clang format action (#40393) add 9faecb90a27 [enhancement](show) Move storage policy info from show create table to show partitions (#39923) add 1849425cddf [fix](restore) update is_being_synced properties (#40194) add 3c5d838ef50 [Bug](sink) take turns getting data from backends to avoid dead lock (#40360) add 518a0fc018b [opt](mtmv) Support to contain select constant clause when create async materialized view (#40244) add f830527af68 [Feat](nereids) support pull up predicate from set operator (#39450) add 8e513ff78eb [fix](memory) Fix conf `max_sys_mem_available_low_water_mark_bytes` (#39962) add a2d488b7eb1 [fix](ES Catalog)Do not extract doc_values of field with ignore_above setting (#40314) add 58dbb5ffdbe [improve](cast) support ipv4 cast to ipv6 type (#39828) add 32557f8377a [fix](nestedtype)fix nested type with is_exclusive (#40398) add 68574b23ce6 [chore](thrift compatibility) Fix thrift id on master is not compatible with branch-2.1 (#40384) add 8e367a99c23 [chore](analysis) Simplify the condition of whether any partition is restoring (#40255) add 89b010d1c7a [feature](cloud) export fdb client busyness (#40369) add 06cb9580f32 [Enhancement](delete) Modify some delete cases to ajust `enable_mow_light_delete` config (#40387) add 5c253afc020 [fix] (regression test) reduce testing time (#40286) add d03f017efca [feature-WIP](query cache) cache tablets aggregate result, FE part (#39670) add 2baed5ca315 [feature](aes_encrypt) support GCM mode for aes_encrypt and aes_decrypt (#40004) add ddaec4733e0 [fix](test) fix move memtable injection test may cause other case stuck (#40356) add 1666c8334b2 [chore](error msg) More convenient error msg when function not found. (#40296) add 82c2650322d [opt](Nereids) support search from override udfs with same arity (#40432) add 009c61c5b82 [fix](tools) adjust download url (#40429) add a7b67ac7eeb [enhance](variant) throw exception when field type is not supported in cast elimation (#40388) add 271e1c4ec7d [Enhancement] support information_schema.partitions (#40153) add 4b6881f9473 [fix](memory) Fix ThreadMemTrackerMgr consumer tracker remove reserve memory (#40380) add 16e3c78dbd2 [opt](cpu-profile) enable cpu profile in BE webui (#40330) add 57978149db4 [fix](memory) Reflect Allocator memory tracker (#40439) add c051e647a4f [Compatibility](agg-state) add be_exec_version check to avoid aggregation data format changed le… (#40193) add 67531f6f91a [bugfix](hive/iceberg)align with Hive insert overwrite table functionality (#39840) add 5d41b05a084 [chore](persist) reduce truncate table operation log size (#40422) add 12b691ca821 [fix](regression-test) Sync after insert to get stable results (#40308) add 72a20b2d1f2 [enhancement](compaction) Abort compaction tasks when correspoding tablet states have been changed (#40271) add 1cde2d2a598 [fix](cloud-schema-change) Exclude merged dropped columns from projection columns (#40187) add 0013c317d04 [feature](cloud) Support multi follower fe in cloud (#38388) add 34f04eefa85 [fix](pipelinex) fix fragment instance progress reports (#40325) add ed37c8175ab [fix](multi table) do not use strlen to calculate the length of msg (#40367) add 83e31d90b9e [improve](s3) s3 load provider should be case insensitive (#39957) add c6f5fcde913 [chore] remove duplicate code in schema scanner (#40456) add 119ea59c523 [feat](Authenticator) Pluginization of Authenticator (#40113) add 27ee0b51704 [metrics](rpc) Add rpc latency metrics (#39593) add ca52591e528 [Bug](compatibility) fix stddev_samp function coredump when upgrade (#40438) add 1216425ceb0 [chore](log) Do not print profile in be log if query comes from FE (#40399) add ec6b744c156 [Enhancement](be-logger) Support custom date time format functionality in be log. (#40347) add af410303a36 [Chore](utils) remove some unused code (#40423) add 44b42b9c4b8 [Bug](map) fix wrong result on map_agg with streaming agg (#40454) add 5571a04a7b2 [refactor](pipeline) Refactor pipeline execution (#39982) add e23696cd000 [enhance](mtmv) During cache generation, no longer hold the write lock for mtmv (#40402) add 7f744ec8e56 [Fix]Fix cgroup v2 init (#39991) add 744a1ec15f0 [opt](nereids)stats derive for min()/max() agg function (#40126) add 5f7a6c10953 [Enhancement] add row count in show partition (#40455) add 775632cf39e [fix](inverted index) Handle an empty segment index file (#40337) add 922ec3a4302 [Fix](hdfs-fs)The cache expiration should explicitly release the held fs (#38610) add 25f85ce1b09 [Chore](agg-state) modify be exec version comments and catch exception on memtable insert (#40465) add 9908a182841 [fix](decompressor) impl ZstdDecompressor (#40315) add e630cc923d5 [bugfix](hms)add default value check for hms catalog with dlf (#40336) add 81fe386df7e [feature](Nereids) support query on ES table in internal catalog (#40449) add b15fd756771 [fix](jdbc catalog) Fix type recognition error when using `query` tvf to query doris (#40122) add 648c622b95c [opt](scanner profile) More counter for scanner (#40144) add f09f4a4564a [fix](logstash) fix group_commit error in logstash (#40450) add c84426f3db2 [opt](plugin) improve filebeat doris output plugin (#36355) add cd7718dba90 [enhance](cloud) Add a policy when be abnormal, tablet delay switch be (#40371) add 5db2af10b76 [ci](trigger) user 9208457 can skip buildall on branch-2.1 (#40419) add 35eb912d82d [Chore](status) Disable stack trace printing for NotFound error status (#40470) add 855140f8a35 [improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) add c45ac4fbcdf [feature](restore) support atomic restore (#40353) add 390ef1d9262 [fix](cloud) MS limit max aborted txn num for the same txn label (#40414) add 93e19b06566 [fix](regression) mem gc when load regression failure due to error msg update (#40483) add ca38271482c [Bug](compatibility) fix agg functions coredump when upgrade (#40472) add dc706ed49b5 [fix](delete_predicate) fix wrong data after upgrade from v2.0 version (#40401) add 39c5fb48630 [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) add a06625ee6b6 [feature](ES Catalog)Add FE open API for ES Catalog (#40444) add 903c05b3af3 [Improvement]Add more tip when drop workload group failed (#40468) add c38fa0272a7 [opt](function)Optimize the performance of the pad function under UTF-8. (#40162) add 84ce9451c6e [opt](memory) Refactor memory maintenance thread (#40344) add 5dd08b6b460 [opt](Nereids) fix several insert into related issues (#40467) add cd356042719 [feat](nereids)set actual row count in physical plan according to merged profile (#40361) add 9d5da50b7a4 [metrics](shuffle) Add necessary metrics (#40476) add e25560e056b [Improvement](sort) do not sort partitial when spill disabled (#40528) add 6a19a37d36f [fix](memory) Revert "[opt](memory) Refactor memory maintenance thread (#40344)" (#40545) add 5392cb13dfb [fix](mtmv)fix nested mtmv not refresh (#40433) add 0d3374bea55 [opt](nereids) tabe row count priority: user injected > BE report > analyzed (#40529) add 258ec3db385 [fix](cloud) should do check before abort transaction (#40463) add f91463c2d30 [Fix](regression-test) fix error url check in test_etl_failed case for cloud p0 (#40298) add 5f1fe8143d1 [fix](scanner) Fix deadlock when scanner submit failed (#40495) add 140008fd0e6 [opt](memory) Refactor memory maintenance thread (retry) (#40551) add ccd654a572f [fix](arrow-flight-sql) Fix regression-test cloud p0 conf (#40490) add 194b85255fd [Fix]only publish topic to alive be (#40535) add abf6897a660 [fix] (Nereids) restrict the execution of match on the scan (#40532) add b674bd12714 [fix](memory) Not check process memory limit when thread reserve memory is sufficient (#40548) add e444100523d [opt](function) Optimize ConvertTz function when using constant parameters. (#40366) add ab2c1cca92b [Chore](GHA) Add code owner for be exec version change (#40577) add e9eed642814 [test](case) add some test case for encrypt/decrypt functions (#40427) add dd8b0ee480b [Enhancement] doris to support function SESSION_USER() (#39575) add 943ac0425a0 [fix](systable) refactor code to avoid wrong usage (#40568) add 140857fa7f4 [feature](compaction) Add an http action for visibility of compaction score on each tablet (#38489) add a98a137c8aa [Fix](merge-on-write) AbstractInsertExecutor should throw exception after running out of retry times (#40436) add 4949a9e7d00 [opt](vault) Do not use latest_fs() in vault mode (#40516) add 1d0690bf0f3 [Enchancement](column) make create_always_true_column return column const (#39086) add 7fc42a5fa4f [fix](Nereids) handle continuous filter or project in plan (#40176) add 24af729933a [fix](log) fix fe doesn't print location in log when log mode is async (#40513) add baefde940f1 [fix](audit_loader) fix that old external audit loader plugin not work because of incompatibility with new audit plugin (#40565) add ebc460068ae [fix] try fix code format (#40581) add cd902b61194 [refactor](pipeline) Refactor logics (#40576) add 948fb2ab57a [improvement](mtmv) Optimize the materialized view hint info when explain (#39998) add c7f57ae1eab [bugfix](hive)Delete the temporarily created folder (#40424) add 9af79c54b26 [ci](ga) make clang format required (#40626) add c48c1ebc5a2 [Enhancement](auto-partition) Re-add deduplication to auto partition rpc (#40580) add b2eef7652fa [fix](regression-test) Fix scanner profile test failed occasionally (#40566) add b7e8d823849 [fix](metadata)Add FE metadata-related file checks (#40546) add 3f2e98a0b1b [improvement](fdb) add a tool which deploys and starts fdb (#39803) add a8d8798e960 [fix](regression) speed up some group commit case (#40624) add 42381995925 [feature](docker suite) Docker suite use independent executor (#40259) add fcceceab89f [improvement](tablet report) tablet report increase report version (#40172) add eb4673fbd52 [fix](cloud) Fix cloud auto start and add a regression case (#40027) add a01c85edb97 [chore](regression test) mv test_report_version_missing to p1 (#40184) add ee1e939e250 [fix](restore) Persist the sqlMode field of the View (#40612) add d8c34beffa5 [fix](ES Catalog)Support parse single value for array column (#40614) add 3be6ed40eae [improvement](jdbc catalog) support jdbc external catalog insert stmt in nereids (#39813) add f9a5c92db53 [regression-test](point query) add `enable_mow_light_delete=false` to related cases (#40520) add fe4efdda463 [fix](cloud) Fix possible incorrect merged tablet stats while iterating detached tablet stats (#40494) add 5e9d9430e18 [Feature-WIP](inverted index) Add boolean query functionality (#40230) add faee56f01a3 [Enhancement](inverted index) improve expr evaluate_inverted_index performace and remove useless code (#40600) add e3b81e7e3fb [feature-WIP](query cache) cache tablets aggregate result, BE part (#40171) add f21eb484448 [chore](cloud) return detail message when commit txn failed (#40584) add 8e37fb46185 [fix](cloud) fix routine load job stuck if commit transaction failed (#40539) add 36828f2994e [Fix](load) fix wrong replica number when loading to table with no partition (#40635) add 898d5254ccb [fix](Nereids) sink distribute is wrong when table is unpartitioned (#40541) add c6c112344fc [Chore](cast) remove some typeid_cast usage (#40601) add 73bf91a2480 [chore](ci) tmp remove required of Clang Formatter on branch-2.0 (#40664) add 042bfcef7b7 [opt](nereids) when table.rows > topn.limit+topn.offset, do not generate topn-filter (#40480) add 81bbd658c92 [Feat](Nereids) support string functions fold constant by fe (#40441) add 91afeec69fd [fix](brpc_client_cache) resolve hostname in DNS cache before passing to brpc (#40074) add e65d55939ea [fix](bytebuffer) fix allocate size improper in append_and_flush (#40613) add dca6b8175d7 [fix](nereids) build agg for random distributed agg table in bindRelation phase (#40181) add b37148e2cb0 [opt](join)Support MethodOneString to optimize hash join with a single string key (#40559) add cd2062ae8df [improvement](mtmv) Support to add use_for_rewrite property when create materialized view (#40332) add d736a508568 [Fix](iot) Fix unexpected error when iot auto detect get no data (#40657) add d77d74ac8a3 [fix](restore) Remove the sqlMode from the view signature (#40615) add 124ac01fecc [Fix](index tool) fix index tool compile problem (#40684) add 462c7c69ac1 [Fix](metahelper)Filename regex validation only checks if the filename is valid. (#40677) add a622ae20652 [enhancement](compaction) Abort merging rowsets when the tablet is no more active (#40478) add 25b1bc76c0a [fix](scan) Fix incorrect query results due to data race of compaction and parallel scanners building (#40552) add 1690c5563a5 [enhancement](balance) change unnecessary write locks to read locks in cloud balance (#40633) add 77fb56e0c1f [fix](planner) throw NPE when insert overwrite partition(*) (#40656) add aab6204bbb9 [chore](audit) let line comment work well (#40599) add f597d72a828 [fix](default value) Avoid default precision value of 0 for default expressions during deserialization (#40671) add d0cb1283f5f [fix](auth)show config command should check auth (#40533) add 4f1d566d7d2 [Enhancement] Add E as default value function for doris (#40209) add c8b5386fe60 [fix](cloud) fix the issue that FE's periodic reconnection to MS RPC is ineffective (#40661) add 55bde8cb48d [Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage API. (#40225) add 087048f2b2f [Fix](Job)Replace BlockingWaitStrategy with LiteTimeoutBlockingWaitStrategy to avoid deadlock issues. (#40625) add 82d6a12726a [Fix](regression-test) Fix some regression test cases (#40634) add de7d6c6ea4a [Fix](inverted index) Fix wrong need read data opt when enable_common_expr_pushdown is disabled (#40689) add de90051162d [Fix](Job)Replaying logs should not modify the original information of the job (#40474) add e20030c0da2 [chore](Nereids) remove enable_fallback_to_original_planner variable (#40416) add a912f77bd79 [fix](scanner) Fix incorrect _max_thread_num in scanner context (#40569) add 4f9aa5579d9 [fix] _cur_reader can be null in exception cases in VFileScanner (#40273) add 9caefba702f [fix](systable) fix unstable case for partitions table (#40553) add b7516c93ef3 [bugfix](loadmem) load memory is too small because flush memory is not right (#40696) add 89cf8d3f9b0 [refactor](nereids)use nereids to implement TableQueryPlanAction (#39627) add 820c9accd9d [fix](nereids)let query be able to forwarded to master (#40119) add 7fdd0be4da0 [optimize](short circuit) avoid set cacheId when non prepared execute (#40525) add fe8905abffc [opt](partial update) Remove unnecessary lock and refactor some code for partial update (#40062) add 7a34ebbccdc [feature](array-func)support array_match_all/any (#40605) add 7548ed692d1 [opt](nereids) add project under aggregate to enable extract common sub expression (#40473) add d6132bb73c2 [improvement](cloud) manage node via sql like non cloud mode (#40264) add 5eb3c608469 [Improment] Log fatal when compaction memtracker not return zero (#40667) add c71514dd406 [enhancement](cloud) clarify codes and make TTL expiration work after abnormal cache type transition (#40226) add d8f911e0746 [Fix]set workload group for load channel (#40749) add 4466541c1b4 [fix](statistics)Fix drop stats log editlog bug. Catch drop stats exception while truncate table. (#40738) add 96c299e0d8e [Fix](cases) fix test_use_mv (#40753) add 69b14b56a71 [Fix](cases) Make case success when meet expected error (#40713) add 7659de554ce [fix](Nereids) avoid bad cast when compute scale for round (#40776) add 23ccaa0f08d [fix](DCHECK) Fix the DCHECK failure when the query is canceled. (#40276) add e40275ae44a [Fix](parquet-reader) Fix parquet reader crash in set_dict(). (#40643) add 70922604242 [chore](test)Exclude Hive-related packages from `java-udf-src`. (#40757) add 61eb0de241e [fix](export) fix p2 tests of export (#40198) add 7e4bd25bff4 [bugfix](hive)Handle exceptions when submitting tasks fails to prevent dead loop (#40708) add c2ca37dcfa0 [fix](restore) Fix atomic restore with exists replicas (#40734) add e1db9450ca2 [fix](memory) Allows to enable memory tracker accuracy detection via Config (#40714) add 3e65faba3e0 [Refactor](inverted index) add analyzer for inverted index to unify analysis process (#40758) add 0ed55402087 [Bug](conv) fix conv function parser string failure return wrong result (#40530) add 6dada7fad31 [feature](array_agg) support array_agg with param is array/map/struct (#40697) add 72ce469570a [Bug](rpad) should check_chars_length avoid exceed max_string_size in pad function (#40676) add 3c8f9f7dad5 [fix](p0)Fix case timeout. (#40788) add 3af0943c392 [fix](restore) Fix clean restore with view (#40620) add 18a374f63c8 [fix](mtmv) Fix partition trace wrong when partition name is same from both side of join (#40485) add 55d6d6452f7 [Fix](nereids) set all nullable aggregate function to alwaysnullable in window expression (#40693) add 3cefe4817f4 [chore](CI)Sonar build should use JDK 17 (#40807) add ffa96082221 [Refactor](Nereids) refactor fold constant framework on fe (#40772) add 10ba034c1f6 [fix](jdbc catalog) Fixed FE memory leak by enabling weak references in HikariCP (#40773) add 5479e62f984 [test](planner) fix generated column delete cases (#40706) add 390e3333154 [Enhancement](delete) Modify some p2 delete cases' property (#37689) add f5a20378181 [Chore](function) check nested function on AggregateFunctionNullBaseInline (#40748) add b49e0256492 [feature](api) add BE HTTP /api/load_channels (#40645) add 6c995e94cb9 [fix](encrypt) wrong mode arg of encrypt and decrypt function make BE crash (#40726) add 1d2d82a08c8 [case](fix) adjust data download url about hive docker (#40805) add a953a51a143 [Fix](Column) refactor ColumnNullable to provide flags safety (#40769) add 29fc07fd589 [Enhancement](profile) Add some details timer to MergedProfile in OlapTableSinkOperator (#40832) add d4a9915ee73 [fix](regression-test) Fix failed adaptive_pipeline_task_serial_read_on_limit (#40711) add 7831e8c0cb3 [improve](percentile_approx) use pdqsort to improve percentile_approx function performance (#40096) add fed90087a93 [fix](regression) fix some group commit and txn load case (#40712) add ef2f6e1ebb9 [improve](restore) Disable alter ops during atomic restore (#40817) add e986003275a [opt](profile) Add profile_manager_gc_interval_seconds on fe.conf (#40596) add aa5492cf097 [enhancement](test) Throw exception explicitly in forward query test (#40742) add ea98b199e46 [fix](regression-test) Add varchar column as distribution key (#40791) add 19e65c3bb6f [Bug](partition_topn) fix partition_topn not reset output rows after do_partition_topn_sort (#40761) add 28950d6b5cf [fix](auth)Http api check auth (#40688) add 6a32d1e3271 [fix](cloud-mow) make cloud_txn_delete_bitmap_cache's expired time more reasonable (#40333) add 2a891a7f11a [enhancement](err-msg) Clarify error msg for forward queries (#40802) add 3bb0a2d9786 [minor](field) Occupy some field ID (#40858) add 3cd6c4cb9c5 [opt](partial update) Allow to only specify key columns in partial update (#40736) add 700f25acfcf [Fix](regression) Fix flaky case `test_delete_publish_skip_read` (#40782) add 03ff755fefb [improvement](stream load) display the time of receiving network data (#40735) add c8fb96e97f6 [fix](statistics)Fix empty table keep auto analyze bug. (#40811) add 10d9c503c7f [fix](routine load) write edit log when rescheduled job (#40728) add 99290c91cb7 [enhancement](regression-test) agg schema value modify case4 (#39404) add 9dace0ed53c [Enhancement](test) Support test DecimalV3 types in be-ut (#40830) add c226d9f46ef [fix](window_funnel) fix wrong result of fixed mode (#40459) add 01c1b15f6a9 [enhancement](regression-test) agg schema value modify case1 (#39390) add 0a2fdcee62e [PipelineX](improvement) Prepare tasks in parallel (#40844) add 2ceccace813 [Fix](cloud) Fix cluster status inconsistent with bes and add config disable auto (#40799) add ea708d9bafb [fix](cluster key) fix mow cluster key with schema change (#40372) add a49beb2c259 [Fix](regression test) Fix schema change concurrent with txn regression case (#40765) add 0f0fdc75cf7 [Enhancement](test) Add function test templates (#40462) add 0cff2257291 [Enhancement](brpc)Added enable_brpc_builtin_services parameter in be.conf (#40718) add eae9e95b1b5 [chore](enhance) Added order by create_time for show routine load (#40783) add 279e4f91076 [test](p2) print error message when assert fails (#40699) add a39d2ff4848 [fix](stream-load) catch exception when parsing CONTENT_LENGTH (#40796) add c81f73f4d32 [chore](github)Update CODEOWNERS (#40686) add 8fe3a363c51 [fix](test) fix unstable case of partitions (#40861) add 8c0789e061d [Chore](case) move slow case to p1 (#40860) add b92b63e1537 Revert "[fix](scanner) Fix incorrect _max_thread_num in scanner context" (#40804) add b58570a34a1 [fix](memory) Refactor MemCounter (#40542) add 2ae4ff86439 [opt](function) Optimize the concat(col, constant, constant, constant) function (#40670) add 8c97aa16296 [Bug](exchange) fix tablet sink shuffle without project not match the output tuple (#40299) add 5c9ac9aaf62 [improvement](statistics)User admin instead of root to run stats sql. (#40867) new ca5bf9842ab [feat]reserve and spill new cb1b116c75b updated new dfdb36cee59 updated new 28e63a74693 [refactor](codestyle) use pointer as return value (#40396) new 766664b5fb3 [revokememory] remove old revoke memory logic new 4432d5fdb23 1. remove revoke memory logic in execute task 2. use memory error code when cancel query (#40426) new 5759d685f67 [opt](spill) Avoid occupying a large amount of memory in join build side new d89033d7846 [opt](spill) avoid query blocked when reserving failed (#40550) new 92e23d94728 [fix] avoid crash when no data to revoke new 616c205bfc2 [improvement](spill) add counter for memory usage (#40572) new 0217028fad5 move paused query and handle logic to workload group manager (#40603) new efb47cb9a03 [fix](exchange) fix coredump of update counter is block is null new f3437bba0d2 add low memory mode in scan operator (#40662) new 88b4c8d13d6 fix bug new 5986b3cfc6a add reserve memory logic in scan operator (#40719) new ec4889d5511 set min reserve bytes to 1MB (#40795) new 540ccc606a1 [fix] use faststring in Block::serialize new 18a711d5115 Use SCOPED_PEAK_MEM to replace ScopedMemTracker new e1016b1f035 Avoid dead-loop when reserve failed new 736dd5580ed fix build error This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (7041d39da9f) \ N -- N -- N refs/heads/spill_and_reserve (736dd5580ed) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 20 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .asf.yaml | 1 - .github/CODEOWNERS | 2 + .github/workflows/clang-format.yml | 7 +- .github/workflows/code-checks.yml | 2 +- .github/workflows/comment-to-trigger-teamcity.yml | 10 +- .github/workflows/scope-label.yml | 2 +- .github/workflows/sonarcloud.yml | 4 +- .licenserc.yaml | 1 + LICENSE.txt | 10 +- be/src/agent/be_exec_version_manager.cpp | 92 + be/src/agent/be_exec_version_manager.h | 69 +- be/src/agent/cgroup_cpu_ctl.cpp | 67 +- be/src/agent/cgroup_cpu_ctl.h | 30 +- be/src/agent/heartbeat_server.cpp | 32 + be/src/agent/task_worker_pool.cpp | 7 +- be/src/cloud/cloud_backend_service.cpp | 14 +- be/src/cloud/cloud_meta_mgr.cpp | 29 +- be/src/cloud/cloud_meta_mgr.h | 9 +- be/src/cloud/cloud_schema_change_job.cpp | 8 +- be/src/cloud/cloud_storage_engine.cpp | 24 +- be/src/cloud/cloud_storage_engine.h | 21 +- be/src/cloud/cloud_stream_load_executor.cpp | 2 +- be/src/cloud/cloud_tablet.cpp | 10 +- be/src/cloud/cloud_tablet_mgr.cpp | 2 +- be/src/cloud/cloud_txn_delete_bitmap_cache.cpp | 42 +- be/src/cloud/cloud_txn_delete_bitmap_cache.h | 10 +- be/src/cloud/config.cpp | 18 +- be/src/cloud/config.h | 15 +- be/src/cloud/injection_point_action.cpp | 63 +- be/src/clucene | 2 +- be/src/common/config.cpp | 50 +- be/src/common/config.h | 38 +- be/src/common/daemon.cpp | 199 +- be/src/common/daemon.h | 2 +- be/src/common/logconfig.cpp | 61 +- be/src/common/status.h | 1 + be/src/exec/decompressor.cpp | 60 +- be/src/exec/decompressor.h | 32 +- be/src/exec/es/es_scroll_parser.cpp | 311 +- be/src/exec/es/es_scroll_parser.h | 2 + be/src/exec/olap_common.h | 142 +- be/src/exec/rowid_fetcher.cpp | 3 +- be/src/exec/schema_scanner.cpp | 14 + .../schema_active_queries_scanner.cpp | 39 +- .../schema_catalog_meta_cache_stats_scanner.cpp | 135 + .../schema_catalog_meta_cache_stats_scanner.h | 54 + .../schema_scanner/schema_partitions_scanner.cpp | 120 +- .../schema_scanner/schema_partitions_scanner.h | 15 +- .../exec/schema_scanner/schema_routine_scanner.cpp | 35 +- .../exec/schema_scanner/schema_scanner_helper.cpp | 71 + be/src/exec/schema_scanner/schema_scanner_helper.h | 43 + be/src/exprs/runtime_filter.cpp | 8 +- be/src/http/action/compaction_score_action.cpp | 236 ++ be/src/http/action/compaction_score_action.h | 66 + be/src/http/action/http_stream.cpp | 15 +- be/src/http/action/load_channel_action.cpp | 66 + be/src/http/action/load_channel_action.h | 43 + be/src/http/action/pprof_actions.cpp | 11 +- be/src/http/action/stream_load.cpp | 15 +- be/src/http/default_path_handlers.cpp | 30 +- be/src/index-tools/index_tool.cpp | 41 +- be/src/io/cache/block_file_cache.cpp | 49 +- be/src/io/cache/block_file_cache.h | 12 +- be/src/io/cache/file_block.cpp | 40 +- be/src/io/cache/file_block.h | 4 +- be/src/io/cache/file_cache_storage.h | 4 +- be/src/io/cache/fs_file_cache_storage.cpp | 256 +- be/src/io/cache/fs_file_cache_storage.h | 15 +- be/src/io/fs/multi_table_pipe.cpp | 2 +- be/src/io/fs/s3_obj_storage_client.cpp | 2 +- be/src/io/fs/stream_load_pipe.cpp | 2 +- be/src/olap/base_tablet.cpp | 9 + be/src/olap/base_tablet.h | 4 + be/src/olap/compaction.cpp | 6 +- .../cumulative_compaction_time_series_policy.cpp | 68 +- be/src/olap/delete_handler.cpp | 15 +- be/src/olap/delta_writer_v2.cpp | 2 +- be/src/olap/inverted_index_parser.h | 2 + be/src/olap/iterators.h | 1 - be/src/olap/lru_cache.cpp | 81 +- be/src/olap/lru_cache.h | 28 +- be/src/olap/memtable.cpp | 23 +- be/src/olap/memtable_flush_executor.cpp | 5 +- be/src/olap/memtable_memory_limiter.cpp | 5 +- be/src/olap/memtable_memory_limiter.h | 5 +- be/src/olap/memtable_writer.cpp | 13 - be/src/olap/merger.cpp | 66 +- be/src/olap/merger.h | 3 +- be/src/olap/olap_common.h | 1 + be/src/olap/olap_server.cpp | 6 + be/src/olap/page_cache.cpp | 10 +- be/src/olap/page_cache.h | 27 +- be/src/olap/parallel_scanner_builder.cpp | 66 +- be/src/olap/parallel_scanner_builder.h | 6 +- be/src/olap/partial_update_info.cpp | 21 + be/src/olap/partial_update_info.h | 7 + be/src/olap/rowset/beta_rowset_reader.cpp | 21 - be/src/olap/rowset/rowset_reader_context.h | 1 - be/src/olap/rowset/segcompaction.cpp | 4 +- be/src/olap/rowset/segment_v2/column_reader.cpp | 31 +- be/src/olap/rowset/segment_v2/column_reader.h | 7 +- .../inverted_index/analyzer/analyzer.cpp | 118 + .../segment_v2/inverted_index/analyzer/analyzer.h | 48 + .../inverted_index/query_v2/boolean_query.cpp | 91 + .../inverted_index/query_v2/boolean_query.h | 56 + .../inverted_index/query_v2/conjunction_op.cpp | 94 + .../inverted_index/query_v2/conjunction_op.h | 46 + .../inverted_index/query_v2/disjunction_op.cpp | 86 + .../inverted_index/query_v2/disjunction_op.h | 59 + .../segment_v2/inverted_index/query_v2/factory.cpp | 36 + .../segment_v2/inverted_index/query_v2/factory.h | 41 + .../inverted_index/query_v2/factory.inline.h | 47 + .../segment_v2/inverted_index/query_v2/node.h | 128 + .../segment_v2/inverted_index/query_v2/operator.h | 45 + .../inverted_index/query_v2/phrase_query.h | 38 + .../segment_v2/inverted_index/query_v2/query.h | 52 + .../inverted_index/query_v2/roaring_query.cpp | 25 + .../inverted_index/query_v2/roaring_query.h | 63 + .../inverted_index/query_v2/term_query.cpp | 39 + .../inverted_index/query_v2/term_query.h | 44 + .../rowset/segment_v2/inverted_index_cache.cpp | 6 +- .../olap/rowset/segment_v2/inverted_index_cache.h | 31 +- .../rowset/segment_v2/inverted_index_reader.cpp | 136 +- .../olap/rowset/segment_v2/inverted_index_reader.h | 16 - .../rowset/segment_v2/inverted_index_searcher.cpp | 7 +- .../rowset/segment_v2/inverted_index_writer.cpp | 107 +- be/src/olap/rowset/segment_v2/segment.cpp | 25 +- be/src/olap/rowset/segment_v2/segment.h | 3 + be/src/olap/rowset/segment_v2/segment_iterator.cpp | 155 +- be/src/olap/rowset/segment_v2/segment_iterator.h | 19 +- be/src/olap/rowset/segment_v2/segment_writer.cpp | 214 +- be/src/olap/rowset/segment_v2/segment_writer.h | 7 + .../rowset/segment_v2/vertical_segment_writer.cpp | 199 +- .../rowset/segment_v2/vertical_segment_writer.h | 7 + be/src/olap/schema_cache.h | 7 +- be/src/olap/schema_change.cpp | 4 +- be/src/olap/segment_loader.cpp | 6 +- be/src/olap/segment_loader.h | 11 +- be/src/olap/single_replica_compaction.cpp | 2 +- be/src/olap/snapshot_manager.cpp | 36 +- be/src/olap/snapshot_manager.h | 1 + be/src/olap/storage_engine.h | 8 +- be/src/olap/tablet.cpp | 14 +- be/src/olap/tablet_manager.cpp | 33 +- be/src/olap/tablet_meta.h | 9 +- be/src/olap/tablet_reader.cpp | 29 +- be/src/olap/tablet_reader.h | 12 +- be/src/olap/tablet_schema.cpp | 14 +- be/src/olap/tablet_schema.h | 5 +- be/src/olap/tablet_schema_cache.cpp | 4 +- be/src/olap/tablet_schema_cache.h | 9 +- be/src/olap/txn_manager.h | 9 +- be/src/olap/utils.cpp | 42 - be/src/olap/utils.h | 57 - be/src/pipeline/common/join_utils.h | 4 +- be/src/pipeline/dependency.h | 17 +- be/src/pipeline/exec/aggregation_sink_operator.cpp | 11 +- be/src/pipeline/exec/aggregation_sink_operator.h | 3 +- be/src/pipeline/exec/analytic_sink_operator.cpp | 11 +- be/src/pipeline/exec/analytic_sink_operator.h | 1 - be/src/pipeline/exec/analytic_source_operator.cpp | 13 +- be/src/pipeline/exec/analytic_source_operator.h | 1 - be/src/pipeline/exec/cache_sink_operator.cpp | 73 + be/src/pipeline/exec/cache_sink_operator.h | 73 + be/src/pipeline/exec/cache_source_operator.cpp | 199 + be/src/pipeline/exec/cache_source_operator.h | 104 + be/src/pipeline/exec/datagen_operator.cpp | 4 +- be/src/pipeline/exec/datagen_operator.h | 2 +- .../distinct_streaming_aggregation_operator.cpp | 22 +- .../exec/distinct_streaming_aggregation_operator.h | 1 - be/src/pipeline/exec/es_scan_operator.cpp | 4 +- be/src/pipeline/exec/es_scan_operator.h | 2 +- be/src/pipeline/exec/exchange_sink_operator.cpp | 39 +- be/src/pipeline/exec/exchange_sink_operator.h | 6 +- be/src/pipeline/exec/exchange_source_operator.cpp | 10 +- be/src/pipeline/exec/exchange_source_operator.h | 1 - be/src/pipeline/exec/file_scan_operator.cpp | 4 +- be/src/pipeline/exec/file_scan_operator.h | 2 +- .../exec/group_commit_block_sink_operator.cpp | 10 +- .../exec/group_commit_block_sink_operator.h | 2 - be/src/pipeline/exec/hashjoin_build_sink.cpp | 29 +- be/src/pipeline/exec/hashjoin_build_sink.h | 6 +- be/src/pipeline/exec/hashjoin_probe_operator.cpp | 18 +- be/src/pipeline/exec/hashjoin_probe_operator.h | 1 - be/src/pipeline/exec/hive_table_sink_operator.h | 6 +- be/src/pipeline/exec/iceberg_table_sink_operator.h | 6 +- be/src/pipeline/exec/jdbc_table_sink_operator.cpp | 9 +- be/src/pipeline/exec/jdbc_table_sink_operator.h | 1 - .../exec/join/process_hash_table_probe_impl.h | 1 + be/src/pipeline/exec/join_probe_operator.cpp | 1 + be/src/pipeline/exec/join_probe_operator.h | 10 +- .../pipeline/exec/memory_scratch_sink_operator.cpp | 11 +- .../pipeline/exec/memory_scratch_sink_operator.h | 1 - .../pipeline/exec/multi_cast_data_stream_source.h | 9 +- .../exec/nested_loop_join_build_operator.cpp | 14 +- .../exec/nested_loop_join_build_operator.h | 5 +- .../exec/nested_loop_join_probe_operator.cpp | 12 +- .../exec/nested_loop_join_probe_operator.h | 1 - be/src/pipeline/exec/olap_scan_operator.cpp | 60 +- be/src/pipeline/exec/olap_scan_operator.h | 8 +- be/src/pipeline/exec/olap_table_sink_operator.h | 7 +- be/src/pipeline/exec/olap_table_sink_v2_operator.h | 6 +- be/src/pipeline/exec/operator.cpp | 156 +- be/src/pipeline/exec/operator.h | 45 +- .../pipeline/exec/partition_sort_sink_operator.cpp | 13 +- .../pipeline/exec/partition_sort_sink_operator.h | 6 +- .../exec/partitioned_aggregation_sink_operator.cpp | 8 +- .../exec/partitioned_aggregation_sink_operator.h | 4 +- .../partitioned_aggregation_source_operator.cpp | 5 - .../exec/partitioned_aggregation_source_operator.h | 1 - .../exec/partitioned_hash_join_probe_operator.cpp | 35 +- .../exec/partitioned_hash_join_probe_operator.h | 1 - .../exec/partitioned_hash_join_sink_operator.cpp | 11 +- .../exec/partitioned_hash_join_sink_operator.h | 1 - be/src/pipeline/exec/repeat_operator.cpp | 19 +- be/src/pipeline/exec/repeat_operator.h | 1 - be/src/pipeline/exec/result_file_sink_operator.cpp | 6 +- be/src/pipeline/exec/result_file_sink_operator.h | 1 - be/src/pipeline/exec/result_sink_operator.cpp | 7 +- be/src/pipeline/exec/result_sink_operator.h | 5 +- be/src/pipeline/exec/scan_operator.cpp | 36 +- be/src/pipeline/exec/scan_operator.h | 14 - be/src/pipeline/exec/schema_scan_operator.cpp | 22 +- be/src/pipeline/exec/schema_scan_operator.h | 1 - be/src/pipeline/exec/set_probe_sink_operator.cpp | 7 +- be/src/pipeline/exec/set_probe_sink_operator.h | 4 +- be/src/pipeline/exec/set_sink_operator.cpp | 7 +- be/src/pipeline/exec/set_sink_operator.h | 4 +- be/src/pipeline/exec/sort_sink_operator.cpp | 12 +- be/src/pipeline/exec/sort_sink_operator.h | 1 - be/src/pipeline/exec/sort_source_operator.cpp | 14 +- be/src/pipeline/exec/sort_source_operator.h | 1 - be/src/pipeline/exec/spill_sort_sink_operator.cpp | 13 +- be/src/pipeline/exec/spill_sort_sink_operator.h | 3 +- .../pipeline/exec/spill_sort_source_operator.cpp | 5 - be/src/pipeline/exec/spill_sort_source_operator.h | 2 - .../exec/streaming_aggregation_operator.cpp | 15 +- .../pipeline/exec/streaming_aggregation_operator.h | 1 - be/src/pipeline/exec/table_function_operator.cpp | 14 +- be/src/pipeline/exec/table_function_operator.h | 1 - be/src/pipeline/exec/union_sink_operator.cpp | 9 +- be/src/pipeline/exec/union_sink_operator.h | 1 - be/src/pipeline/exec/union_source_operator.h | 8 +- .../local_exchange_sink_operator.cpp | 10 +- .../local_exchange/local_exchange_sink_operator.h | 2 - .../local_exchange_source_operator.h | 7 +- be/src/pipeline/pipeline.cpp | 24 +- be/src/pipeline/pipeline.h | 33 +- be/src/pipeline/pipeline_fragment_context.cpp | 343 +- be/src/pipeline/pipeline_fragment_context.h | 27 +- be/src/pipeline/pipeline_task.cpp | 2 +- be/src/pipeline/pipeline_task.h | 6 +- be/src/pipeline/query_cache/query_cache.cpp | 69 + be/src/pipeline/query_cache/query_cache.h | 151 + be/src/runtime/buffer_control_block.cpp | 11 +- be/src/runtime/buffer_control_block.h | 1 - be/src/runtime/descriptors.cpp | 16 +- be/src/runtime/descriptors.h | 14 +- be/src/runtime/exec_env.h | 16 +- be/src/runtime/exec_env_init.cpp | 14 +- be/src/runtime/fragment_mgr.cpp | 17 +- be/src/runtime/load_channel.cpp | 36 +- be/src/runtime/load_channel.h | 3 +- be/src/runtime/load_channel_mgr.cpp | 6 +- be/src/runtime/load_channel_mgr.h | 19 +- be/src/runtime/memory/cache_manager.cpp | 21 +- be/src/runtime/memory/cache_manager.h | 3 + be/src/runtime/memory/cache_policy.cpp | 8 +- be/src/runtime/memory/cache_policy.h | 31 +- be/src/runtime/memory/global_memory_arbitrator.cpp | 27 +- be/src/runtime/memory/global_memory_arbitrator.h | 36 +- be/src/runtime/memory/lru_cache_policy.h | 242 +- be/src/runtime/memory/lru_cache_value_base.h | 7 +- be/src/runtime/memory/mem_counter.h | 95 + be/src/runtime/memory/mem_tracker.cpp | 91 +- be/src/runtime/memory/mem_tracker.h | 232 +- be/src/runtime/memory/mem_tracker_limiter.cpp | 225 +- be/src/runtime/memory/mem_tracker_limiter.h | 294 +- be/src/runtime/memory/memory_reclamation.cpp | 12 - be/src/runtime/memory/thread_mem_tracker_mgr.cpp | 10 +- be/src/runtime/memory/thread_mem_tracker_mgr.h | 21 +- be/src/runtime/query_context.cpp | 8 +- .../routine_load/routine_load_task_executor.cpp | 2 +- be/src/runtime/runtime_filter_mgr.cpp | 24 +- be/src/runtime/runtime_query_statistics_mgr.cpp | 49 +- be/src/runtime/stream_load/stream_load_context.cpp | 2 + be/src/runtime/thread_context.h | 25 +- be/src/runtime/workload_group/workload_group.cpp | 19 +- .../workload_group/workload_group_manager.cpp | 39 +- be/src/service/brpc_service.cpp | 2 + be/src/service/http_service.cpp | 16 + be/src/service/internal_service.cpp | 27 +- be/src/service/internal_service.h | 2 +- be/src/service/point_query_executor.cpp | 34 +- be/src/service/point_query_executor.h | 22 +- be/src/util/brpc_client_cache.h | 35 +- be/src/util/byte_buffer.h | 8 +- be/src/util/encryption_util.cpp | 180 +- be/src/util/encryption_util.h | 17 +- be/src/util/mem_info.cpp | 12 +- be/src/util/obj_lru_cache.cpp | 6 +- be/src/util/obj_lru_cache.h | 8 +- be/src/util/pprof_utils.cpp | 4 + be/src/util/proto_util.h | 15 + be/src/util/s3_util.cpp | 3 +- be/src/util/simd/vstring_function.h | 17 + be/src/util/stack_util.cpp | 6 + be/src/util/tdigest.h | 7 +- .../aggregate_function_collect.cpp | 30 +- .../aggregate_function_collect.h | 72 +- .../aggregate_functions/aggregate_function_covar.h | 19 +- .../aggregate_function_distinct.cpp | 3 +- .../aggregate_function_foreach.cpp | 3 +- .../aggregate_functions/aggregate_function_map.h | 1 + .../aggregate_functions/aggregate_function_null.h | 1 + .../aggregate_function_percentile.h | 151 +- .../aggregate_function_simple_factory.h | 3 +- .../aggregate_function_stddev.h | 22 +- .../aggregate_function_window_funnel.h | 7 +- be/src/vec/columns/column_array.cpp | 34 +- be/src/vec/columns/column_array.h | 5 + be/src/vec/columns/column_decimal.cpp | 2 - be/src/vec/columns/column_map.cpp | 7 +- be/src/vec/columns/column_map.h | 5 + be/src/vec/columns/column_nullable.cpp | 85 +- be/src/vec/columns/column_nullable.h | 153 +- be/src/vec/columns/column_object.cpp | 10 + be/src/vec/columns/column_object.h | 2 + be/src/vec/columns/column_struct.h | 9 + be/src/vec/common/allocator.cpp | 38 +- be/src/vec/common/allocator.h | 30 +- be/src/vec/common/hash_table/hash_map_context.h | 48 +- be/src/vec/common/sort/partition_sorter.cpp | 9 +- be/src/vec/common/sort/sorter.h | 4 +- be/src/vec/common/typeid_cast.h | 23 - be/src/vec/core/block.cpp | 19 +- be/src/vec/core/types.h | 13 +- be/src/vec/data_types/data_type_agg_state.h | 15 +- be/src/vec/data_types/data_type_decimal.h | 5 +- be/src/vec/data_types/get_least_supertype.cpp | 9 + .../format/parquet/byte_array_dict_decoder.cpp | 3 + .../format/parquet/fix_length_dict_decoder.hpp | 3 + .../exec/format/table/max_compute_jni_reader.cpp | 28 +- be/src/vec/exec/scan/new_olap_scanner.cpp | 11 +- be/src/vec/exec/scan/new_olap_scanner.h | 3 - be/src/vec/exec/scan/vfile_scanner.cpp | 9 +- be/src/vec/exec/scan/vfile_scanner.h | 2 +- be/src/vec/exec/scan/vmeta_scanner.h | 1 - be/src/vec/exprs/vectorized_fn_call.cpp | 7 +- be/src/vec/exprs/vexpr.cpp | 45 +- be/src/vec/exprs/vexpr.h | 1 - be/src/vec/exprs/vin_predicate.cpp | 27 +- be/src/vec/exprs/vin_predicate.h | 4 + be/src/vec/exprs/vmatch_predicate.cpp | 16 +- .../functions/array/function_array_register.cpp | 2 + .../vec/functions/array/varray_match_function.cpp | 145 + be/src/vec/functions/function_cast.h | 11 + be/src/vec/functions/function_collection_in.h | 5 +- be/src/vec/functions/function_conv.cpp | 9 +- be/src/vec/functions/function_convert_tz.h | 128 +- be/src/vec/functions/function_encryption.cpp | 115 +- be/src/vec/functions/function_ip.h | 60 +- be/src/vec/functions/function_map.cpp | 6 +- .../functions/function_math_unary_alway_nullable.h | 94 + be/src/vec/functions/function_rpc.cpp | 5 + be/src/vec/functions/function_string.h | 238 +- be/src/vec/functions/function_struct.cpp | 6 +- be/src/vec/functions/function_tokenize.cpp | 26 +- be/src/vec/functions/function_totype.h | 1 - be/src/vec/functions/functions_comparison.h | 23 +- be/src/vec/functions/functions_geo.cpp | 68 +- be/src/vec/functions/if.cpp | 11 +- be/src/vec/functions/match.cpp | 68 +- be/src/vec/functions/match.h | 7 +- be/src/vec/functions/math.cpp | 16 +- be/src/vec/olap/block_reader.cpp | 4 +- be/src/vec/olap/vertical_block_reader.cpp | 3 +- be/src/vec/runtime/vdata_stream_mgr.cpp | 11 +- be/src/vec/runtime/vdata_stream_mgr.h | 3 +- be/src/vec/runtime/vdata_stream_recvr.cpp | 37 +- be/src/vec/runtime/vdata_stream_recvr.h | 9 +- be/src/vec/sink/load_stream_stub.cpp | 10 +- be/src/vec/sink/vdata_stream_sender.cpp | 8 + be/src/vec/sink/vrow_distribution.cpp | 6 +- be/src/vec/sink/vrow_distribution.h | 28 +- be/src/vec/sink/writer/vfile_result_writer.cpp | 5 + be/src/vec/sink/writer/vtablet_writer.cpp | 36 +- be/src/vec/sink/writer/vtablet_writer.h | 2 + be/src/vec/sink/writer/vtablet_writer_v2.cpp | 14 +- be/src/vec/utils/util.hpp | 8 +- be/test/io/cache/block_file_cache_test.cpp | 246 +- be/test/io/fs/s3_file_writer_test.cpp | 4 +- ...mulative_compaction_time_series_policy_test.cpp | 2 +- be/test/olap/lru_cache_test.cpp | 146 +- .../inverted_index/query_v2/query_test.cpp | 183 + be/test/olap/segcompaction_mow_test.cpp | 19 +- be/test/olap/segcompaction_test.cpp | 20 +- .../runtime/memory/thread_mem_tracker_mgr_test.cpp | 4 +- .../vec/aggregate_functions/agg_bitmap_test.cpp | 2 +- .../vec/aggregate_functions/agg_collect_test.cpp | 2 +- .../vec/aggregate_functions/agg_histogram_test.cpp | 2 +- .../aggregate_functions/agg_min_max_by_test.cpp | 2 +- .../vec/aggregate_functions/agg_min_max_test.cpp | 6 +- .../vec/aggregate_functions/agg_replace_test.cpp | 4 +- be/test/vec/aggregate_functions/agg_test.cpp | 4 +- .../aggregate_functions/vec_count_by_enum_test.cpp | 2 +- .../vec/aggregate_functions/vec_retention_test.cpp | 2 +- .../vec_sequence_match_test.cpp | 10 +- .../aggregate_functions/vec_window_funnel_test.cpp | 2 +- .../column_nullable_seriazlization_test.cpp | 75 +- be/test/vec/columns/column_nullable_test.cpp | 112 + be/test/vec/columns/column_nullable_test.h | 105 + be/test/vec/columns/column_resize_test.cpp | 1 - be/test/vec/core/column_string_test.cpp | 12 +- be/test/vec/core/get_common_type_test.cpp | 12 +- be/test/vec/exec/vfile_scanner_exception_test.cpp | 307 ++ be/test/vec/exec/vwal_scanner_test.cpp | 2 +- .../vec/function/function_array_element_test.cpp | 12 +- be/test/vec/function/function_array_index_test.cpp | 20 +- be/test/vec/function/function_jsonb_test.cpp | 38 +- be/test/vec/function/function_math_test.cpp | 27 +- be/test/vec/function/function_string_test.cpp | 71 +- be/test/vec/function/function_test_template.cpp | 57 + be/test/vec/function/function_test_util.cpp | 41 +- be/test/vec/function/function_test_util.h | 29 +- be/test/vec/function/table_function_test.cpp | 10 +- build.sh | 4 +- cloud/CMakeLists.txt | 2 +- cloud/script/start.sh | 31 +- cloud/script/stop.sh | 13 + cloud/src/common/bvars.cpp | 2 + cloud/src/common/bvars.h | 1 + cloud/src/common/config.h | 9 +- cloud/src/common/metric.cpp | 8 +- cloud/src/main.cpp | 60 +- cloud/src/meta-service/http_encode_key.cpp | 52 +- cloud/src/meta-service/meta_service_job.cpp | 25 +- cloud/src/meta-service/meta_service_resource.cpp | 7 +- .../src/meta-service/meta_service_tablet_stats.cpp | 75 +- cloud/src/meta-service/meta_service_tablet_stats.h | 21 +- cloud/src/meta-service/meta_service_txn.cpp | 35 +- cloud/src/meta-service/txn_kv.cpp | 4 + cloud/src/meta-service/txn_kv.h | 4 + cloud/src/recycler/checker.cpp | 2 + cloud/src/recycler/hdfs_accessor.cpp | 23 +- cloud/src/recycler/recycler.cpp | 21 +- cloud/src/recycler/recycler_service.cpp | 2 +- cloud/src/resource-manager/resource_manager.cpp | 69 +- cloud/test/http_encode_key_test.cpp | 844 ++-- cloud/test/meta_service_test.cpp | 67 + cloud/test/recycler_test.cpp | 1 + dist/LICENSE-dist.txt | 65 +- docker/runtime/doris-compose/Readme.md | 39 +- docker/runtime/doris-compose/cluster.py | 103 +- docker/runtime/doris-compose/command.py | 192 +- docker/runtime/doris-compose/database.py | 72 +- docker/runtime/doris-compose/resource/common.sh | 2 +- docker/runtime/doris-compose/resource/init_be.sh | 6 + .../runtime/doris-compose/resource/init_cloud.sh | 7 + docker/runtime/doris-compose/resource/init_fe.sh | 32 +- .../fe/resource/fe_disaggregated_entrypoint.sh | 2 + .../scripts/data/composite_type_array_bulk.json | 20 +- .../elasticsearch/scripts/data/data4_es6.json | 32 + .../elasticsearch/scripts/data/data5.json | 35 +- .../scripts/data/{data5.json => data6.json} | 0 .../elasticsearch/scripts/es_init.sh | 8 +- .../index/array_meta_composite_type_array.json | 3 +- .../scripts/index/es6_composite_type_array.json | 3 +- .../elasticsearch/scripts/index/es6_test1.json | 5 +- .../scripts/index/es7_composite_type_array.json | 3 +- .../elasticsearch/scripts/index/es7_test1.json | 5 +- .../docker-compose/hive/scripts/hive-metastore.sh | 21 +- extension/beats/doris/client.go | 391 +- extension/beats/doris/config.go | 134 +- extension/beats/doris/doris.go | 175 +- extension/beats/go.mod | 1 + extension/beats/go.sum | 2 + extension/logstash/lib/logstash/outputs/doris.rb | 26 +- extension/logstash/logstash-output-doris.gemspec | 2 +- .../apache/doris/common/jni/vec/ColumnValue.java | 12 + .../apache/doris/common/jni/vec/VectorColumn.java | 6 + fe/be-java-extensions/max-compute-scanner/pom.xml | 6 + .../doris/maxcompute/MaxComputeColumnValue.java | 121 +- .../doris/maxcompute/MaxComputeJniScanner.java | 344 +- .../doris/maxcompute/MaxComputePartitionValue.java | 137 - .../doris/maxcompute/MaxComputeTableScan.java | 114 - .../doris/maxcompute/MaxComputeJniScannerTest.java | 158 - .../org/apache/doris/catalog/PrimitiveType.java | 1 + .../main/java/org/apache/doris/catalog/Type.java | 4 - .../main/java/org/apache/doris/common/Config.java | 45 +- fe/fe-core/pom.xml | 6 + .../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 3 + .../antlr4/org/apache/doris/nereids/DorisParser.g4 | 142 +- .../antlr4/org/apache/doris/nereids/PLParser.g4 | 1 + fe/fe-core/src/main/cup/sql_parser.cup | 4 +- .../main/java/org/apache/doris/alter/Alter.java | 66 +- .../java/org/apache/doris/alter/RollupJobV2.java | 2 +- .../apache/doris/alter/SchemaChangeHandler.java | 23 +- .../org/apache/doris/alter/SchemaChangeJobV2.java | 30 +- .../java/org/apache/doris/alter/SystemHandler.java | 10 +- .../org/apache/doris/analysis/AlterRoleStmt.java | 2 +- .../org/apache/doris/analysis/AlterTableStmt.java | 3 - .../org/apache/doris/analysis/AnalyzeStmt.java | 2 +- .../java/org/apache/doris/analysis/Analyzer.java | 8 +- .../java/org/apache/doris/analysis/ColumnDef.java | 13 + .../java/org/apache/doris/analysis/CopyStmt.java | 3 +- .../org/apache/doris/analysis/CreateJobStmt.java | 40 +- .../org/apache/doris/analysis/CreateRoleStmt.java | 2 +- .../org/apache/doris/analysis/CreateTableStmt.java | 5 +- .../org/apache/doris/analysis/DateLiteral.java | 33 + .../java/org/apache/doris/analysis/DdlStmt.java | 2 +- .../apache/doris/analysis/DefaultValueExprDef.java | 6 +- .../apache/doris/analysis/DiagnoseTabletStmt.java | 2 +- .../org/apache/doris/analysis/DropUserStmt.java | 2 +- .../java/org/apache/doris/analysis/ExportStmt.java | 2 +- .../main/java/org/apache/doris/analysis/Expr.java | 25 +- .../apache/doris/analysis/FunctionCallExpr.java | 99 +- .../org/apache/doris/analysis/GroupByClause.java | 5 + .../doris/analysis/InsertOverwriteTableStmt.java | 6 +- .../java/org/apache/doris/analysis/KeysDesc.java | 67 +- .../doris/analysis/LambdaFunctionCallExpr.java | 4 +- .../java/org/apache/doris/analysis/LoadStmt.java | 3 +- .../analysis/ModifyTablePropertiesClause.java | 18 + .../org/apache/doris/analysis/OutFileClause.java | 1 + .../org/apache/doris/analysis/RestoreStmt.java | 9 + .../org/apache/doris/analysis/SchemaTableType.java | 4 +- .../java/org/apache/doris/analysis/SelectStmt.java | 2 +- .../doris/analysis/SetDefaultStorageVaultStmt.java | 2 +- .../apache/doris/analysis/SetOperationStmt.java | 2 +- .../apache/doris/analysis/SetUserPropertyStmt.java | 2 +- .../org/apache/doris/analysis/ShowAlterStmt.java | 2 +- .../org/apache/doris/analysis/ShowAnalyzeStmt.java | 2 +- .../doris/analysis/ShowAnalyzeTaskStatus.java | 2 +- .../org/apache/doris/analysis/ShowAuthorStmt.java | 2 +- .../doris/analysis/ShowAutoAnalyzeJobsStmt.java | 2 +- .../apache/doris/analysis/ShowBackendsStmt.java | 2 +- .../org/apache/doris/analysis/ShowBackupStmt.java | 2 +- .../org/apache/doris/analysis/ShowBrokerStmt.java | 2 +- .../apache/doris/analysis/ShowBuildIndexStmt.java | 2 +- .../doris/analysis/ShowCacheHotSpotStmt.java | 2 +- .../doris/analysis/ShowCatalogRecycleBinStmt.java | 2 +- .../org/apache/doris/analysis/ShowCatalogStmt.java | 2 +- .../org/apache/doris/analysis/ShowCharsetStmt.java | 2 +- .../apache/doris/analysis/ShowCloudWarmUpStmt.java | 2 +- .../org/apache/doris/analysis/ShowClusterStmt.java | 2 +- .../apache/doris/analysis/ShowCollationStmt.java | 2 +- .../apache/doris/analysis/ShowColumnHistStmt.java | 2 +- .../apache/doris/analysis/ShowColumnStatsStmt.java | 2 +- .../org/apache/doris/analysis/ShowColumnStmt.java | 2 +- .../apache/doris/analysis/ShowConvertLSCStmt.java | 2 +- .../org/apache/doris/analysis/ShowCopyStmt.java | 2 +- .../doris/analysis/ShowCreateCatalogStmt.java | 2 +- .../apache/doris/analysis/ShowCreateDbStmt.java | 2 +- .../doris/analysis/ShowCreateFunctionStmt.java | 2 +- .../apache/doris/analysis/ShowCreateMTMVStmt.java | 2 +- .../analysis/ShowCreateMaterializedViewStmt.java | 2 +- .../doris/analysis/ShowCreateRepositoryStmt.java | 2 +- .../apache/doris/analysis/ShowCreateTableStmt.java | 2 +- .../apache/doris/analysis/ShowDataSkewStmt.java | 2 +- .../org/apache/doris/analysis/ShowDataStmt.java | 2 +- .../apache/doris/analysis/ShowDataTypesStmt.java | 2 +- .../org/apache/doris/analysis/ShowDbIdStmt.java | 2 +- .../java/org/apache/doris/analysis/ShowDbStmt.java | 2 +- .../org/apache/doris/analysis/ShowDeleteStmt.java | 2 +- .../doris/analysis/ShowDynamicPartitionStmt.java | 2 +- .../apache/doris/analysis/ShowEncryptKeysStmt.java | 2 +- .../org/apache/doris/analysis/ShowEnginesStmt.java | 2 +- .../org/apache/doris/analysis/ShowEventsStmt.java | 2 +- .../org/apache/doris/analysis/ShowExportStmt.java | 2 +- .../apache/doris/analysis/ShowFrontendsStmt.java | 2 +- .../apache/doris/analysis/ShowFunctionsStmt.java | 2 +- .../org/apache/doris/analysis/ShowGrantsStmt.java | 2 +- .../org/apache/doris/analysis/ShowIndexStmt.java | 2 +- .../apache/doris/analysis/ShowLastInsertStmt.java | 2 +- .../apache/doris/analysis/ShowLoadProfileStmt.java | 2 +- .../org/apache/doris/analysis/ShowLoadStmt.java | 2 +- .../doris/analysis/ShowLoadWarningsStmt.java | 2 +- .../apache/doris/analysis/ShowOpenTableStmt.java | 2 +- .../apache/doris/analysis/ShowPartitionIdStmt.java | 2 +- .../apache/doris/analysis/ShowPartitionsStmt.java | 2 +- .../org/apache/doris/analysis/ShowPluginsStmt.java | 2 +- .../org/apache/doris/analysis/ShowPolicyStmt.java | 2 +- .../apache/doris/analysis/ShowPrivilegesStmt.java | 2 +- .../org/apache/doris/analysis/ShowProcStmt.java | 2 +- .../apache/doris/analysis/ShowProcedureStmt.java | 2 +- .../apache/doris/analysis/ShowProcesslistStmt.java | 2 +- .../doris/analysis/ShowQueryProfileStmt.java | 2 +- .../apache/doris/analysis/ShowQueryStatsStmt.java | 2 +- .../analysis/ShowReplicaDistributionStmt.java | 2 +- .../doris/analysis/ShowReplicaStatusStmt.java | 2 +- .../doris/analysis/ShowRepositoriesStmt.java | 2 +- .../apache/doris/analysis/ShowResourcesStmt.java | 2 +- .../org/apache/doris/analysis/ShowRestoreStmt.java | 2 +- .../org/apache/doris/analysis/ShowRolesStmt.java | 2 +- .../org/apache/doris/analysis/ShowRollupStmt.java | 2 +- .../apache/doris/analysis/ShowSmallFilesStmt.java | 2 +- .../apache/doris/analysis/ShowSnapshotStmt.java | 2 +- .../doris/analysis/ShowSqlBlockRuleStmt.java | 2 +- .../org/apache/doris/analysis/ShowStageStmt.java | 2 +- .../org/apache/doris/analysis/ShowStatusStmt.java | 2 +- .../java/org/apache/doris/analysis/ShowStmt.java | 2 +- .../doris/analysis/ShowStoragePolicyUsingStmt.java | 2 +- .../doris/analysis/ShowStorageVaultStmt.java | 2 +- .../apache/doris/analysis/ShowStreamLoadStmt.java | 2 +- .../org/apache/doris/analysis/ShowSyncJobStmt.java | 2 +- .../doris/analysis/ShowTableCreationStmt.java | 2 +- .../org/apache/doris/analysis/ShowTableIdStmt.java | 2 +- .../apache/doris/analysis/ShowTableStatsStmt.java | 73 +- .../apache/doris/analysis/ShowTableStatusStmt.java | 2 +- .../org/apache/doris/analysis/ShowTableStmt.java | 2 +- .../org/apache/doris/analysis/ShowTabletStmt.java | 2 +- .../analysis/ShowTabletStorageFormatStmt.java | 2 +- .../doris/analysis/ShowTabletsBelongStmt.java | 2 +- .../apache/doris/analysis/ShowTransactionStmt.java | 2 +- .../apache/doris/analysis/ShowTrashDiskStmt.java | 2 +- .../org/apache/doris/analysis/ShowTrashStmt.java | 2 +- .../apache/doris/analysis/ShowTriggersStmt.java | 2 +- .../apache/doris/analysis/ShowTypeCastStmt.java | 2 +- .../doris/analysis/ShowUserPropertyStmt.java | 2 +- .../apache/doris/analysis/ShowVariablesStmt.java | 2 +- .../org/apache/doris/analysis/ShowViewStmt.java | 2 +- .../org/apache/doris/analysis/ShowWarningStmt.java | 2 +- .../apache/doris/analysis/ShowWhiteListStmt.java | 2 +- .../doris/analysis/ShowWorkloadGroupsStmt.java | 2 +- .../java/org/apache/doris/analysis/SlotRef.java | 12 + .../org/apache/doris/analysis/StageProperties.java | 3 +- .../org/apache/doris/analysis/TransactionStmt.java | 2 +- .../org/apache/doris/backup/BackupHandler.java | 6 +- .../apache/doris/backup/RestoreFileMapping.java | 18 +- .../java/org/apache/doris/backup/RestoreJob.java | 379 +- .../doris/catalog/BuiltinScalarFunctions.java | 16 +- .../apache/doris/catalog/CloudTabletStatMgr.java | 1 + .../main/java/org/apache/doris/catalog/Env.java | 109 +- .../java/org/apache/doris/catalog/EsTable.java | 7 +- .../org/apache/doris/catalog/FunctionRegistry.java | 92 +- .../java/org/apache/doris/catalog/FunctionSet.java | 26 + .../apache/doris/catalog/FunctionSignature.java | 34 +- .../org/apache/doris/catalog/InternalSchema.java | 4 +- .../doris/catalog/InternalSchemaInitializer.java | 29 +- .../apache/doris/catalog/ListPartitionInfo.java | 8 - .../apache/doris/catalog/ListPartitionItem.java | 4 + .../main/java/org/apache/doris/catalog/MTMV.java | 60 +- .../apache/doris/catalog/MaterializedIndex.java | 10 + .../java/org/apache/doris/catalog/OlapTable.java | 171 +- .../java/org/apache/doris/catalog/Partition.java | 19 + .../org/apache/doris/catalog/PartitionInfo.java | 13 + .../org/apache/doris/catalog/PartitionItem.java | 4 + .../org/apache/doris/catalog/PartitionKey.java | 60 + .../apache/doris/catalog/RangePartitionInfo.java | 24 +- .../apache/doris/catalog/RangePartitionItem.java | 4 + .../java/org/apache/doris/catalog/Replica.java | 10 + .../java/org/apache/doris/catalog/SchemaTable.java | 11 +- .../main/java/org/apache/doris/catalog/Table.java | 5 + .../java/org/apache/doris/catalog/TableIf.java | 2 + .../org/apache/doris/catalog/TableProperty.java | 32 + .../org/apache/doris/catalog/TabletStatMgr.java | 29 +- .../main/java/org/apache/doris/catalog/View.java | 10 +- .../cloud/alter/CloudSchemaChangeHandler.java | 5 +- .../doris/cloud/analysis/UseCloudClusterStmt.java | 3 +- .../doris/cloud/catalog/CloudClusterChecker.java | 52 +- .../org/apache/doris/cloud/catalog/CloudEnv.java | 104 +- .../apache/doris/cloud/catalog/CloudReplica.java | 93 +- .../doris/cloud/catalog/CloudTabletRebalancer.java | 30 +- .../cloud/datasource/CloudInternalCatalog.java | 7 +- .../cloud/planner/CloudGroupCommitPlanner.java | 14 - .../apache/doris/cloud/rpc/MetaServiceClient.java | 5 + .../apache/doris/cloud/rpc/MetaServiceProxy.java | 14 +- .../doris/cloud/system/CloudSystemInfoService.java | 295 +- .../transaction/CloudGlobalTransactionMgr.java | 36 +- .../java/org/apache/doris/common/Log4jConfig.java | 18 +- .../doris/common/RandomIdentifierGenerator.java | 51 + .../doris/common/proc/PartitionsProcDir.java | 4 + .../doris/common/profile/ExecutionProfile.java | 1 + .../org/apache/doris/common/profile/Profile.java | 78 +- .../doris/common/profile/SummaryProfile.java | 2 - .../doris/common/publish/TopicPublisherThread.java | 5 +- .../java/org/apache/doris/common/util/Daemon.java | 2 +- .../doris/common/util/InternalDatabaseUtil.java | 3 +- .../apache/doris/common/util/ProfileManager.java | 1 + .../apache/doris/common/util/PropertyAnalyzer.java | 11 +- .../apache/doris/common/util/RuntimeProfile.java | 15 + .../java/org/apache/doris/common/util/Util.java | 7 +- .../apache/doris/datasource/ExternalCatalog.java | 16 + .../doris/datasource/ExternalMetaCacheMgr.java | 12 + .../org/apache/doris/datasource/ExternalTable.java | 11 + .../apache/doris/datasource/InternalCatalog.java | 71 +- .../apache/doris/datasource/es/EsRestClient.java | 36 +- .../apache/doris/datasource/es/MappingPhase.java | 21 + .../doris/datasource/hive/HMSExternalTable.java | 79 +- .../doris/datasource/hive/HMSTransaction.java | 46 +- .../doris/datasource/hive/HiveMetaStoreCache.java | 26 +- .../doris/datasource/hive/HiveMetadataOps.java | 13 +- .../doris/datasource/hive/source/HiveScanNode.java | 44 +- .../hudi/source/HudiCachedPartitionProcessor.java | 12 +- .../datasource/iceberg/IcebergMetadataCache.java | 15 +- .../datasource/iceberg/IcebergTransaction.java | 34 +- .../doris/datasource/jdbc/client/JdbcClient.java | 1 + .../datasource/jdbc/client/JdbcMySQLClient.java | 25 +- .../datasource/jdbc/util/JdbcFieldSchema.java | 13 + .../maxcompute/MaxComputeExternalCatalog.java | 223 +- .../maxcompute/MaxComputeExternalTable.java | 63 +- .../maxcompute/MaxComputeMetadataCache.java | 18 - .../maxcompute/source/MaxComputeScanNode.java | 442 +- .../maxcompute/source/MaxComputeSplit.java | 20 +- .../property/constants/MCProperties.java | 24 +- .../property/constants/S3Properties.java | 6 +- .../org/apache/doris/deploy/DeployManager.java | 4 +- .../doris/fs/remote/RemoteFSPhantomManager.java | 117 + .../apache/doris/fs/remote/RemoteFileSystem.java | 21 +- .../remote/RemoteFileSystemPhantomReference.java | 44 + .../org/apache/doris/fs/remote/S3FileSystem.java | 8 + .../apache/doris/fs/remote/dfs/DFSFileSystem.java | 8 + .../org/apache/doris/httpv2/rest/LoadAction.java | 11 +- .../apache/doris/httpv2/rest/MetaInfoAction.java | 1 + .../org/apache/doris/httpv2/rest/ShowAction.java | 24 +- .../doris/httpv2/rest/TableQueryPlanAction.java | 252 +- .../doris/httpv2/rest/manager/NodeAction.java | 2 +- .../doris/httpv2/restv2/ESCatalogAction.java | 108 + .../doris/httpv2/restv2/MetaInfoActionV2.java | 25 +- .../doris/insertoverwrite/InsertOverwriteUtil.java | 3 + .../apache/doris/job/disruptor/TaskDisruptor.java | 15 +- .../doris/job/executor/DispatchTaskHandler.java | 5 +- .../doris/job/executor/TimerJobSchedulerTask.java | 5 +- .../doris/job/extensions/insert/InsertJob.java | 19 +- .../doris/job/extensions/insert/InsertTask.java | 1 - .../job/manager/TaskDisruptorGroupManager.java | 22 +- .../apache/doris/job/scheduler/JobScheduler.java | 13 +- .../org/apache/doris/load/ExportTaskExecutor.java | 1 - .../org/apache/doris/load/GroupCommitManager.java | 10 +- .../org/apache/doris/load/StreamLoadRecordMgr.java | 4 +- .../org/apache/doris/load/loadv2/BulkLoadJob.java | 2 +- .../org/apache/doris/load/loadv2/LoadManager.java | 33 +- .../doris/load/routineload/RoutineLoadJob.java | 4 +- .../java/org/apache/doris/master/MasterImpl.java | 10 +- .../java/org/apache/doris/master/MetaHelper.java | 91 +- .../org/apache/doris/master/ReportHandler.java | 14 +- .../java/org/apache/doris/mtmv/MTMVPlanUtil.java | 1 - .../org/apache/doris/mtmv/MTMVPropertyUtil.java | 15 +- .../doris/mtmv/MTMVRefreshPartitionSnapshot.java | 45 +- .../org/apache/doris/mtmv/MTMVVersionSnapshot.java | 25 +- .../doris/mysql/authenticate/AuthenticateType.java | 18 + .../mysql/authenticate/AuthenticatorFactory.java | 37 + .../mysql/authenticate/AuthenticatorManager.java | 69 +- .../authenticate/DefaultAuthenticatorFactory.java | 32 + .../mysql/authenticate/ldap/LdapAuthenticator.java | 9 +- .../ldap/LdapAuthenticatorFactory.java | 37 + .../org/apache/doris/mysql/privilege/Role.java | 19 +- .../org/apache/doris/nereids/NereidsPlanner.java | 47 +- .../nereids/analyzer/UnboundJdbcTableSink.java | 84 + .../nereids/analyzer/UnboundTableSinkCreator.java | 20 +- .../glue/translator/ExpressionTranslator.java | 7 +- .../glue/translator/PhysicalPlanTranslator.java | 73 +- .../org/apache/doris/nereids/hint/UseMvHint.java | 144 + .../doris/nereids/jobs/executor/Analyzer.java | 3 - .../doris/nereids/jobs/executor/Rewriter.java | 31 +- .../doris/nereids/parser/LogicalPlanBuilder.java | 45 +- .../nereids/processor/post/PlanPostProcessors.java | 3 + .../post/ProjectAggregateExpressionsForCse.java | 150 + .../processor/post/TopnFilterPushDownVisitor.java | 8 +- .../doris/nereids/processor/post/Validator.java | 5 +- .../processor/pre/PullUpSubqueryAliasToCTE.java | 2 +- .../pre/TurnOffPageCacheForInsertIntoSelect.java | 8 + .../nereids/properties/RequestPropertyDeriver.java | 9 + .../doris/nereids/properties/SelectHintUseMv.java | 53 + .../org/apache/doris/nereids/rules/RuleSet.java | 13 +- .../org/apache/doris/nereids/rules/RuleType.java | 7 +- .../doris/nereids/rules/analysis/BindRelation.java | 136 +- .../doris/nereids/rules/analysis/BindSink.java | 165 +- .../BuildAggForRandomDistributedTable.java | 271 -- .../nereids/rules/analysis/CheckAfterRewrite.java | 20 + .../doris/nereids/rules/analysis/CheckPolicy.java | 21 +- .../rules/analysis/EliminateLogicalSelectHint.java | 52 +- .../mv/InitMaterializationContextHook.java | 11 +- .../exploration/mv/MaterializationContext.java | 2 +- .../exploration/mv/MaterializedViewUtils.java | 18 +- .../expression/rules/FoldConstantRuleOnFE.java | 13 +- ...ogicalJdbcTableSinkToPhysicalJdbcTableSink.java | 48 + .../rules/rewrite/EliminateEmptyRelation.java | 4 + .../ExtractAndNormalizeWindowExpression.java | 16 +- .../nereids/rules/rewrite/InferPredicates.java | 45 + .../rules/rewrite/MaxMinFilterPushDown.java | 133 + .../nereids/rules/rewrite/PullUpPredicates.java | 139 + .../rules/rewrite/PushProjectIntoUnion.java | 13 +- .../mv/AbstractSelectMaterializedIndexRule.java | 2 +- .../doris/nereids/stats/ExpressionEstimation.java | 21 +- .../doris/nereids/stats/FilterEstimation.java | 8 +- .../doris/nereids/stats/StatsCalculator.java | 50 +- .../org/apache/doris/nereids/trees/TreeNode.java | 17 + .../nereids/trees/expressions/ExecFunction.java | 9 - .../trees/expressions/ExpressionEvaluator.java | 190 +- .../functions/ComputePrecisionForRound.java | 4 +- .../functions/executable/DateTimeAcquire.java | 24 +- .../functions/executable/DateTimeArithmetic.java | 122 +- .../executable/DateTimeExtractAndTransform.java | 276 +- .../functions/executable/ExecutableFunctions.java | 26 +- .../functions/executable/NumericArithmetic.java | 174 +- .../functions/executable/StringArithmetic.java | 856 ++++ .../functions/executable/TimeRoundSeries.java | 288 +- .../trees/expressions/functions/scalar/Acos.java | 4 +- .../functions/scalar/AesCryptoFunction.java | 25 +- .../expressions/functions/scalar/AesDecrypt.java | 34 +- .../expressions/functions/scalar/AesDecryptV2.java | 74 - .../expressions/functions/scalar/AesEncrypt.java | 34 +- .../expressions/functions/scalar/AesEncryptV2.java | 74 - .../functions/scalar/ArrayMatchAll.java | 68 + .../functions/scalar/ArrayMatchAny.java | 68 + .../trees/expressions/functions/scalar/Asin.java | 4 +- .../trees/expressions/functions/scalar/Dsqrt.java | 4 +- .../trees/expressions/functions/scalar/Field.java | 28 +- .../trees/expressions/functions/scalar/Repeat.java | 2 + .../expressions/functions/scalar/SessionUser.java | 59 + .../expressions/functions/scalar/Sm4Decrypt.java | 15 +- .../expressions/functions/scalar/Sm4DecryptV2.java | 69 - .../expressions/functions/scalar/Sm4Encrypt.java | 15 +- .../expressions/functions/scalar/Sm4EncryptV2.java | 72 - .../trees/expressions/functions/scalar/Sqrt.java | 4 +- .../expressions/functions/table/HttpStream.java | 11 + .../trees/expressions/functions/udf/AliasUdf.java | 4 +- .../expressions/functions/udf/AliasUdfBuilder.java | 20 +- .../expressions/functions/udf/JavaUdafBuilder.java | 6 + .../expressions/functions/udf/JavaUdfBuilder.java | 6 + .../expressions/functions/udf/JavaUdtfBuilder.java | 6 + .../expressions/functions/udf/UdfBuilder.java | 3 + .../expressions/visitor/ScalarFunctionVisitor.java | 35 +- .../doris/nereids/trees/plans/AbstractPlan.java | 4 + .../nereids/trees/plans/ComputeResultSet.java | 5 +- .../apache/doris/nereids/trees/plans/PlanType.java | 3 + .../trees/plans/commands/ShowConfigCommand.java | 7 + .../trees/plans/commands/info/CreateMTMVInfo.java | 8 +- .../trees/plans/commands/info/CreateTableInfo.java | 94 +- .../trees/plans/commands/info/DMLCommandType.java | 2 + .../trees/plans/commands/info/DefaultValue.java | 9 +- .../commands/insert/AbstractInsertExecutor.java | 11 +- .../insert/BaseExternalTableInsertExecutor.java | 3 +- .../commands/insert/HiveInsertCommandContext.java | 11 + .../commands/insert/InsertIntoTableCommand.java | 27 +- .../trees/plans/commands/insert/InsertUtils.java | 9 + .../commands/insert/JdbcInsertCommandContext.java | 24 + .../plans/commands/insert/JdbcInsertExecutor.java | 113 + .../insert/OlapGroupCommitInsertExecutor.java | 2 +- .../plans/commands/insert/OlapInsertExecutor.java | 2 + .../nereids/trees/plans/logical/LogicalEsScan.java | 21 +- .../trees/plans/logical/LogicalJdbcTableSink.java | 151 + .../trees/plans/logical/LogicalOlapScan.java | 3 - .../trees/plans/logical/LogicalSelectHint.java | 30 +- .../trees/plans/physical/AbstractPhysicalJoin.java | 3 +- .../physical/PhysicalBaseExternalTableSink.java | 4 + .../trees/plans/physical/PhysicalCTEProducer.java | 1 + .../plans/physical/PhysicalEmptyRelation.java | 6 +- .../trees/plans/physical/PhysicalEsScan.java | 11 +- .../plans/physical/PhysicalHashAggregate.java | 2 +- .../plans/physical/PhysicalJdbcTableSink.java | 109 + .../plans/physical/PhysicalOlapTableSink.java | 2 +- .../plans/physical/PhysicalOneRowRelation.java | 28 +- .../trees/plans/physical/PhysicalQuickSort.java | 4 +- .../trees/plans/physical/PhysicalResultSink.java | 5 +- .../trees/plans/physical/PhysicalSqlCache.java | 2 +- .../nereids/trees/plans/physical/PhysicalTopN.java | 1 + .../trees/plans/physical/PhysicalUnion.java | 4 +- .../trees/plans/physical/PhysicalWindow.java | 3 +- .../nereids/trees/plans/visitor/SinkVisitor.java | 15 + .../apache/doris/nereids/util/ExpressionUtils.java | 29 + .../doris/nereids/util/TypeCoercionUtils.java | 4 +- .../java/org/apache/doris/nereids/util/Utils.java | 65 + .../persist/ModifyTablePropertyOperationLog.java | 24 + .../apache/doris/persist/TruncateTableInfo.java | 12 +- .../org/apache/doris/planner/AggregationNode.java | 73 + .../org/apache/doris/planner/DataStreamSink.java | 10 + .../apache/doris/planner/GroupCommitPlanner.java | 2 +- .../org/apache/doris/planner/HiveTableSink.java | 2 + .../org/apache/doris/planner/OlapScanNode.java | 122 +- .../org/apache/doris/planner/OlapTableSink.java | 6 +- .../org/apache/doris/planner/PlanFragment.java | 11 + .../java/org/apache/doris/planner/PlanNode.java | 93 + .../java/org/apache/doris/planner/ScanNode.java | 16 +- .../normalize/NormalizedPartitionPredicates.java | 56 + .../apache/doris/planner/normalize/Normalizer.java | 40 + .../PartitionRangePredicateNormalizer.java | 220 + .../doris/planner/normalize/PredicateToRange.java | 178 + .../planner/normalize/QueryCacheNormalizer.java | 230 + .../java/org/apache/doris/plugin/AuditEvent.java | 279 ++ .../java/org/apache/doris/plugin/AuditPlugin.java | 2 - .../java/org/apache/doris/plugin/PluginMgr.java | 4 +- .../org/apache/doris/plugin/audit/AuditEvent.java | 279 -- .../org/apache/doris/plugin/audit/AuditLoader.java | 243 ++ .../doris/plugin/audit/AuditLoaderPlugin.java | 242 -- .../apache/doris/plugin/audit/AuditLogBuilder.java | 5 +- .../doris/plugin/audit/AuditStreamLoader.java | 2 +- .../apache/doris/plugin/audit/LoadAuditEvent.java | 2 + .../doris/plugin/audit/StreamLoadAuditEvent.java | 2 + .../org/apache/doris/qe/AuditEventProcessor.java | 2 +- .../java/org/apache/doris/qe/AuditLogHelper.java | 16 +- .../java/org/apache/doris/qe/ConnectContext.java | 39 +- .../java/org/apache/doris/qe/ConnectProcessor.java | 53 +- .../main/java/org/apache/doris/qe/Coordinator.java | 49 +- .../main/java/org/apache/doris/qe/DdlExecutor.java | 6 +- .../java/org/apache/doris/qe/MasterOpExecutor.java | 11 +- .../org/apache/doris/qe/PointQueryExecutor.java | 5 +- .../java/org/apache/doris/qe/SessionVariable.java | 95 +- .../java/org/apache/doris/qe/ShowExecutor.java | 13 +- .../java/org/apache/doris/qe/StmtExecutor.java | 45 +- .../main/java/org/apache/doris/resource/Tag.java | 2 + .../resource/workloadgroup/WorkloadGroupMgr.java | 5 +- .../WorkloadRuntimeStatusMgr.java | 2 +- .../doris/scheduler/disruptor/TaskDisruptor.java | 10 +- .../apache/doris/service/FrontendServiceImpl.java | 8 +- .../org/apache/doris/statistics/AnalysisInfo.java | 5 - .../doris/statistics/AnalysisInfoBuilder.java | 2 +- .../apache/doris/statistics/AnalysisManager.java | 88 +- .../apache/doris/statistics/BaseAnalysisTask.java | 2 - .../apache/doris/statistics/OlapAnalysisTask.java | 2 +- .../apache/doris/statistics/StatisticRange.java | 22 +- .../org/apache/doris/statistics/Statistics.java | 35 +- .../doris/statistics/StatisticsRepository.java | 1 - .../apache/doris/statistics/TableStatsMeta.java | 39 +- .../doris/statistics/util/StatisticsUtil.java | 42 +- .../main/java/org/apache/doris/system/Backend.java | 2 +- .../java/org/apache/doris/system/HeartbeatMgr.java | 10 + .../org/apache/doris/system/SystemInfoService.java | 45 +- .../doris/tablefunction/MetadataGenerator.java | 211 +- .../java/org/apache/doris/task/SnapshotTask.java | 13 +- .../doris/transaction/DatabaseTransactionMgr.java | 3 + .../doris/transaction/JdbcTransactionManager.java | 42 + .../apache/doris/transaction/TransactionType.java | 3 +- ...e.doris.mysql.authenticate.AuthenticatorFactory | 19 + .../doris/alter/InternalSchemaAlterTest.java | 9 +- .../doris/analysis/BulkLoadDataDescTest.java | 1 - .../java/org/apache/doris/analysis/StageTest.java | 3 +- .../doris/backup/RestoreFileMappingTest.java | 6 +- .../org/apache/doris/backup/RestoreJobTest.java | 3 +- .../apache/doris/catalog/CreateFunctionTest.java | 1 - .../org/apache/doris/clone/DecommissionTest.java | 2 +- .../doris/cluster/DecommissionBackendTest.java | 8 +- .../doris/cluster/SystemInfoServiceTest.java | 2 +- .../datasource/hive/HiveDDLAndDMLPlanTest.java | 1 - .../doris/datasource/hive/HmsCommitTest.java | 22 +- .../datasource/iceberg/IcebergTransactionTest.java | 95 +- .../datasource/property/PropertyConverterTest.java | 7 +- .../apache/doris/external/hms/HmsCatalogTest.java | 1 - .../doris/http/TableQueryPlanActionTest.java | 18 + .../doris/load/routineload/RoutineLoadJobTest.java | 1 - .../org/apache/doris/master/MetaHelperTest.java | 59 + .../joinorder/joinhint/DistributeHintTest.java | 9 +- .../doris/nereids/preprocess/SelectHintTest.java | 13 +- .../nereids/rules/analysis/BindRelationTest.java | 23 + .../nereids/rules/analysis/CheckRowPolicyTest.java | 97 + .../exploration/mv/MaterializedViewUtilsTest.java | 20 + .../rewrite/ArrayContainsToArrayOverlapTest.java | 5 +- .../nereids/rules/rewrite/InferPredicatesTest.java | 131 +- .../rules/rewrite/MaxMinFilterPushDownTest.java | 115 + .../nereids/stats/ExpressionEstimationTest.java | 4 +- .../doris/nereids/stats/FilterEstimationTest.java | 35 + .../apache/doris/planner/OlapTableSinkTest.java | 6 - .../doris/planner/QueryCacheNormalizerTest.java | 515 +++ .../apache/doris/planner/StatisticDeriveTest.java | 27 - .../apache/doris/qe/AuditEventProcessorTest.java | 4 +- .../doris/statistics/util/StatisticsUtilTest.java | 62 +- .../apache/doris/utframe/TestWithFeService.java | 1 + fe/pom.xml | 2 +- gensrc/proto/cloud.proto | 3 + gensrc/proto/internal_service.proto | 2 + gensrc/script/doris_builtins_functions.py | 4 + gensrc/thrift/AgentService.thrift | 1 + gensrc/thrift/BackendService.thrift | 1 + gensrc/thrift/DataSinks.thrift | 8 + gensrc/thrift/Descriptors.thrift | 13 +- gensrc/thrift/FrontendService.thrift | 13 +- gensrc/thrift/HeartbeatService.thrift | 2 + gensrc/thrift/Normalization.thrift | 61 + gensrc/thrift/PaloInternalService.thrift | 30 +- gensrc/thrift/PlanNodes.thrift | 9 +- gensrc/thrift/Planner.thrift | 5 +- gensrc/thrift/QueryCache.thrift | 53 + gensrc/thrift/Types.thrift | 2 +- regression-test/conf/regression-conf.groovy | 12 +- .../backup_restore/test_backup_restore_atomic.out | 78 + .../correctness/test_column_nullable_cache.out | 6 + .../data/correctness_p0/test_default_e.out | 13 + .../datatype_p0/decimalv3/test_decimal256_cast.out | 10 + .../data/datatype_p0/ip/test_ip_implicit_cast.out | 25 + .../nested_types/query/sql/nested_with_join.out | 87 + .../test_delete_generated_column.out | 24 - .../data/export_p2/test_export_max_file_size.out | 2003 ++++----- .../data/external_table_p0/es/test_es_query.out | 499 ++- .../external_table_p0/es/test_es_query_nereids.out | 40 +- ...test_hive_insert_overwrite_with_empty_table.out | 18 + .../hive/write/test_hive_text_write_insert.out | 164 + ...t_iceberg_insert_overwrite_with_empty_table.out | 18 + .../jdbc/test_doris_jdbc_catalog.out | 17 + .../jdbc/test_mariadb_jdbc_catalog.out | 35 - .../jdbc/test_mysql_jdbc_catalog.out | 35 - .../jdbc/test_mysql_jdbc_catalog_nereids.out | 35 - .../jdbc/test_mysql_jdbc_driver5_catalog.out | 35 - .../mc/test_max_compute_complex_type.out | 17 - .../test_external_catalog_maxcompute.out | 138 +- .../maxcompute/test_max_compute_all_type.out | 788 ++++ .../maxcompute/test_max_compute_complex_type.out | 19 + .../cloud/test_cloud_mow_insert_timeout.out | 11 + .../group_commit/replay_wal_restart_fe.out | 4 + .../test_ignore_above_in_index.out | 3 + .../data/inverted_index_p0/test_need_read_data.out | 6 + .../data/job_p0/job_meta/job_query_test.out | 7 + .../test_move_memtable_multi_segment_index.out | 2 +- .../http_stream/test_group_commit_http_stream.out | 2 +- .../stream_load/test_group_commit_stream_load.out | 2 +- .../test_stream_load_with_data_quality.csv | 2 + .../test_stream_load_with_data_quality2.csv | 2 + .../one_row_relation_when_create.out | 4 + regression-test/data/mtmv_p0/test_es_mtmv.out | 2 + .../data/mtmv_p0/test_multi_level_mtmv.out | 11 + .../test_upgrade_downgrade_olap_mtmv.out | 9 + .../data/nereids_function_p0/fn_test_am.csv | 102 + .../nereids_function_p0/scalar_function/Array.out | 1320 ++++++ .../data/nereids_hint_tpcds_p0/shape/query14.out | 51 +- .../data/nereids_hint_tpcds_p0/shape/query64.out | 103 +- .../data/nereids_hint_tpcds_p0/shape/query66.out | 103 +- .../data/nereids_hint_tpcds_p0/shape/query8.out | 9 +- .../insert_into_table/insert_use_table_id.out | 48 - .../data/nereids_p0/insert_into_table/random.out | 3 + .../encryption_digest/test_encryption_function.out | 96 +- .../sql_functions/math_functions/test_conv.out | 3 + .../eager_aggregate/push_down_max_through_join.out | 20 +- .../push_down_expression_in_hash_join.out | 4 +- .../filter_push_down/push_filter_through.out | 6 +- .../infer_predicate/infer_intersect_except.out | 155 + .../infer_predicate/pull_up_predicate_literal.out | 1258 ++---- .../infer_predicate/pull_up_predicate_set_op.out | 563 +++ .../infer_set_operator_distinct.out | 27 +- .../max_min_filter_push_down.out | 290 ++ .../data/nereids_rules_p0/mv/ssb/mv_ssb_test.out | 4466 +++----------------- .../normalize_window_nullable_agg_test.out | 293 ++ .../data/nereids_syntax_p0/mv/agg_mv_test.dat | 1 + .../nereids_syntax_p0/mv/aggregate/agg_sync_mv.out | 179 +- .../data/nereids_syntax_p1/mv/agg_mv_test.dat | 27 + .../nereids_syntax_p1/mv/aggregate/agg_sync_mv.out | 1516 +++++++ .../bs_downgrade_shape/query8.out | 9 +- .../shape/query14.out | 51 +- .../nereids_tpcds_shape_sf1000_p0/shape/query4.out | 8 +- .../shape/query66.out | 103 +- .../nereids_tpcds_shape_sf1000_p0/shape/query8.out | 9 +- .../noStatsRfPrune/query11.out | 8 +- .../noStatsRfPrune/query14.out | 51 +- .../noStatsRfPrune/query4.out | 16 +- .../noStatsRfPrune/query66.out | 103 +- .../noStatsRfPrune/query8.out | 9 +- .../no_stats_shape/query11.out | 8 +- .../no_stats_shape/query14.out | 51 +- .../no_stats_shape/query4.out | 16 +- .../no_stats_shape/query66.out | 103 +- .../no_stats_shape/query8.out | 9 +- .../rf_prune/query14.out | 51 +- .../rf_prune/query4.out | 8 +- .../rf_prune/query66.out | 103 +- .../rf_prune/query8.out | 9 +- .../nereids_tpcds_shape_sf100_p0/shape/query14.out | 51 +- .../nereids_tpcds_shape_sf100_p0/shape/query4.out | 8 +- .../nereids_tpcds_shape_sf100_p0/shape/query66.out | 103 +- .../nereids_tpcds_shape_sf100_p0/shape/query8.out | 9 +- .../data/nereids_tpch_p0/tpch/agg_cse.out | 33 + .../tpcds_sf100/noStatsRfPrune/query11.out | 8 +- .../tpcds_sf100/noStatsRfPrune/query14.out | 51 +- .../tpcds_sf100/noStatsRfPrune/query4.out | 16 +- .../tpcds_sf100/noStatsRfPrune/query66.out | 103 +- .../tpcds_sf100/noStatsRfPrune/query8.out | 9 +- .../tpcds_sf100/no_stats_shape/query11.out | 8 +- .../tpcds_sf100/no_stats_shape/query14.out | 51 +- .../tpcds_sf100/no_stats_shape/query4.out | 16 +- .../tpcds_sf100/no_stats_shape/query66.out | 103 +- .../tpcds_sf100/no_stats_shape/query8.out | 9 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query14.out | 51 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query4.out | 8 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query66.out | 103 +- .../new_shapes_p0/tpcds_sf100/rf_prune/query8.out | 9 +- .../new_shapes_p0/tpcds_sf100/shape/query14.out | 51 +- .../new_shapes_p0/tpcds_sf100/shape/query4.out | 8 +- .../new_shapes_p0/tpcds_sf100/shape/query66.out | 103 +- .../new_shapes_p0/tpcds_sf100/shape/query8.out | 9 +- .../tpcds_sf1000/bs_downgrade_shape/query8.out | 9 +- .../new_shapes_p0/tpcds_sf1000/shape/query14.out | 51 +- .../new_shapes_p0/tpcds_sf1000/shape/query4.out | 8 +- .../new_shapes_p0/tpcds_sf1000/shape/query66.out | 103 +- .../new_shapes_p0/tpcds_sf1000/shape/query8.out | 9 +- .../auto_partition/test_auto_dynamic.out | 6 + .../test_auto_partition_behavior.out | 5 - .../data/point_query_p0/test_point_query.out | 8 + .../data/query_p0/aggregate/array_agg.out | 156 + .../aggregate/select_random_distributed_tbl.out | 14 +- .../query_p0/aggregate/test_array_agg_complex.csv | 112 + .../encryption_digest/test_encryption_function.out | 249 +- .../test_math_unary_always_nullable.out | 95 + .../sql_functions/test_template_one_arg.out | 83 + .../sql_functions/test_template_three_args.out | 471 +++ .../sql_functions/test_template_two_args.out | 279 ++ .../window_functions/test_partition_topn.out | 5 + .../query_p0/system/test_partitions_schema.out | 48 + .../data/query_p0/system/test_query_sys_tables.out | 2 - .../test_agg_schema_value_modify1.out | 11 + .../unique_with_mow_c_p0/test_schema_change_ck.out | 365 ++ .../partial_update/partial_update_delete.csv | 3 +- .../schema_change/load_with_key_column.csv | 2 +- .../partial_update/test_partial_update_delete.out | 338 +- .../test_partial_update_only_keys.out | 59 + .../org/apache/doris/regression/Config.groovy | 9 +- .../apache/doris/regression/ConfigOptions.groovy | 10 + .../apache/doris/regression/RegressionTest.groovy | 82 +- .../org/apache/doris/regression/suite/Suite.groovy | 118 +- .../doris/regression/suite/SuiteCluster.groovy | 55 +- regression-test/java-udf-src/pom.xml | 2 + .../pipeline/cloud_p0/conf/be_custom.conf | 3 +- .../cloud_p0/conf/regression-conf-custom.groovy | 1 - .../pipeline/cloud_p1/conf/be_custom.conf | 3 +- regression-test/pipeline/external/conf/be.conf | 3 +- regression-test/pipeline/p0/conf/be.conf | 3 +- .../pipeline/p0/conf/regression-conf.groovy | 1 - regression-test/pipeline/p1/conf/be.conf | 1 + .../plugins/plugin_curl_requester.groovy | 16 +- .../txn_insert_values_with_schema_change.out | 0 .../alter_p0/test_alter_muti_modify_column.groovy | 2 + .../auth_p0/test_http_meta_databases_auth.groovy | 58 + .../auth_p0/test_http_meta_tables_auth.groovy | 70 + .../test_http_meta_tables_schema_auth.groovy | 69 + .../auth_p0/test_http_table_count_auth.groovy | 69 + .../auth_p0/test_http_table_data_auth.groovy | 91 + .../test_backup_restore_atomic.groovy | 209 + .../test_backup_restore_atomic_with_alter.groovy | 241 ++ .../test_backup_restore_clean_restore.groovy | 36 +- .../test_backup_restore_with_view.groovy | 16 + .../check_before_quit/check_before_quit.groovy | 11 + .../clone_p0/test_clone_missing_version.groovy | 2 +- .../clone_p0/test_clone_no_missing_version.groovy | 2 +- .../suites/clone_p0/test_decommission_mtmv.groovy | 2 +- .../test_drop_clone_tablet_path_race.groovy | 2 +- .../cloud_p0/multi_cluster/test_auto_start.groovy | 196 + .../multi_cluster/test_multi_follower.groovy | 176 + .../cloud_p0/multi_cluster/test_rebalance.groovy | 4 +- .../suites/cloud_p0/multi_cluster/test_tvf.groovy | 4 +- .../node_mgr/test_sql_mode_node_mgr.groovy | 388 ++ .../cloud_p0/query_retry/test_retry_e-230.groovy | 2 +- .../test_schema_change_with_compaction10.groovy | 4 +- .../test_schema_change_with_compaction11.groovy | 2 +- .../test_schema_change_with_compaction5.groovy | 4 +- .../test_schema_change_with_compaction6.groovy | 4 +- .../test_schema_change_with_compaction9.groovy | 4 +- .../compaction/test_compaction_score_action.groovy | 53 + .../test_compaction_with_visible_version.groovy | 2 +- .../suites/compaction/test_full_compaction.groovy | 1 + .../test_full_compaction_by_table_id.groovy | 1 + ...e_compaction_with_variant_inverted_index.groovy | 15 +- .../control_p0/test_report_version_missing.groovy | 82 - .../control_p1/test_report_version_missing.groovy | 85 + .../correctness/test_column_nullable_cache.groovy | 57 + .../suites/correctness_p0/test_cast_decimal.groovy | 1 - .../suites/correctness_p0/test_default_e.groovy | 69 + .../test_push_conjunct_external_table.groovy | 67 - .../agg_state/max/test_agg_state_max.groovy | 2 +- .../nereids/test_agg_state_nereids.groovy | 4 - .../datatype_p0/bitmap/test_bitmap_int.groovy | 2 +- .../decimalv3/test_decimal256_cast.groovy | 71 + .../datatype_p0/decimalv3/test_decimalv3.groovy | 2 - .../datatype_p0/ip/test_ip_implicit_cast.groovy | 19 +- .../nested_types/query/sql/nested_with_join.sql | 675 +++ .../test_nestedtypes_insert_into_select.groovy | 4 +- .../test_delete_generated_column.groovy | 16 +- .../suites/demo_p0/docker_action.groovy | 10 +- regression-test/suites/demo_p0/test_action.groovy | 2 +- .../export_p2/test_export_max_file_size.groovy | 65 +- .../test_outfile_orc_max_file_size.groovy | 69 +- .../suites/export_p2/test_outfile_p2.groovy | 37 +- .../es/test_es_catalog_http_open_api.groovy | 84 + .../external_table_p0/es/test_es_query.groovy | 65 +- .../hive/ddl/test_hive_database.groovy | 54 + .../hive/ddl/test_hive_ddl.groovy | 2 +- .../hive/test_external_catalog_hive.groovy | 4 + .../hive/test_hive_statistics_p0.groovy | 65 + ...t_hive_insert_overwrite_with_empty_table.groovy | 80 + .../hive/write/test_hive_text_write_insert.groovy | 3 +- .../iceberg/test_iceberg_table_stats.groovy | 4 + ...ceberg_insert_overwrite_with_empty_table.groovy | 77 + .../jdbc/test_doris_jdbc_catalog.groovy | 3 + .../jdbc/test_mariadb_jdbc_catalog.groovy | 2 +- .../jdbc/test_mysql_jdbc_catalog.groovy | 2 +- .../jdbc/test_mysql_jdbc_catalog_nereids.groovy | 2 +- .../jdbc/test_mysql_jdbc_driver5_catalog.groovy | 2 +- .../mc/test_max_compute_complex_type.groovy | 175 - .../test_external_catalog_maxcompute.groovy | 313 +- .../maxcompute/test_max_compute_all_type.groovy | 547 +++ .../test_max_compute_complex_type.groovy | 183 + .../cloud/test_cloud_mow_insert_timeout.groovy | 88 + .../test_delete_publish_skip_read.groovy | 2 +- .../test_full_compaciton_run_status.groovy | 5 +- ...oup_commit_async_wal_msg_fault_injection.groovy | 6 +- ...test_index_empty_segment_fault_injection.groovy | 62 + .../test_mem_gc_when_load_fault_injection.groovy | 4 +- .../insert_overwrite_auto_detect.groovy | 20 +- .../group_commit/replay_wal_restart_fe.groovy | 2 +- .../test_group_commit_replay_wal.groovy | 3 +- .../insert_p0/insert_group_commit_into.groovy | 18 +- ...nsert_group_commit_into_max_filter_ratio.groovy | 15 +- .../insert_group_commit_into_unique.groovy | 9 +- ...nsert_group_commit_into_unique_sync_mode.groovy | 6 +- .../insert_group_commit_with_exception.groovy | 12 +- .../insert_group_commit_with_large_data.groovy | 1 + .../insert_group_commit_with_prepare_stmt.groovy | 2 + .../suites/insert_p0/transaction/txn_insert.groovy | 1 + .../transaction/txn_insert_inject_case.groovy | 24 +- .../transaction/txn_insert_restart_fe.groovy | 2 +- ...txn_insert_restart_fe_with_schema_change.groovy | 2 +- .../transaction/txn_insert_with_drop.groovy | 4 + ...ommit_http_stream_lineitem_schema_change.groovy | 2 + ...ommit_insert_into_lineitem_scheme_change.groovy | 2 + ...ommit_stream_load_lineitem_schema_change.groovy | 2 + .../txn_insert_concurrent_insert_ud.groovy | 1 + .../txn_insert_concurrent_insert_update.groovy | 1 + .../insert_p2/txn_insert_with_schema_change.groovy | 1 + .../test_build_index_with_clone_by_docker.groovy | 2 +- .../test_index_compaction_dup_keys.groovy | 2 + .../test_index_compaction_null.groovy | 4 + .../test_index_compaction_unique_keys.groovy | 2 + ...dex_compaction_with_multi_index_segments.groovy | 2 + .../test_ignore_above_in_index.groovy | 3 +- .../inverted_index_p0/test_need_read_data.groovy | 25 + .../suites/javaudf_p0/test_javaudtf_int.groovy | 4 +- .../suites/job_p0/job_meta/job_query_test.groovy | 28 + regression-test/suites/job_p0/job_meta/load.groovy | 50 + .../suites/job_p0/test_base_insert_job.groovy | 2 +- .../json_p0/test_json_predict_is_null.groovy | 5 - .../suites/jsonb_p0/test_jsonb_cast.groovy | 3 - .../jsonb_p0/test_jsonb_predict_is_null.groovy | 5 - .../test_jsonb_with_unescaped_string.groovy | 3 - .../test_min_load_replica_num_complicate.groovy | 2 +- .../test_move_memtable_multi_segment_index.groovy | 23 +- .../load/insert/test_publish_one_succ.groovy | 2 +- .../load/insert/test_publish_slow_not_wait.groovy | 2 +- .../load_p0/broker_load/test_etl_failed.groovy | 2 +- .../test_group_commit_http_stream.groovy | 6 +- .../insert/test_min_load_replica_num_simple.groovy | 2 +- .../test_routine_load_restart_fe.groovy | 2 +- .../stream_load/test_coordidator_be_restart.groovy | 2 +- .../test_group_commit_and_wal_back_pressure.groovy | 3 +- .../test_group_commit_stream_load.groovy | 10 +- ...t_stream_load_with_nonexist_db_and_table.groovy | 2 +- .../test_steam_load_with_data_quality.groovy | 187 + .../load_p2/broker_load/test_broker_load.groovy | 3 +- .../test_migrate_disk_with_publish_version.groovy | 2 +- .../one_row_relation_when_create.groovy | 138 + ...ble_date_non_deterministic_function_mtmv.groovy | 6 +- .../suites/mtmv_p0/test_multi_level_mtmv.groovy | 16 + .../suites/mtmv_p0/test_use_for_rewrite.groovy | 103 + .../suites/mtmv_up_down_olap_p0/load.groovy | 71 + .../test_upgrade_downgrade_olap_mtmv.groovy | 32 + .../suites/nereids_function_p0/load.groovy | 23 + .../scalar_function/Array.groovy | 42 + .../nereids_function_p0/scalar_function/S.groovy | 51 +- .../delete/delete_mow_partial_update.groovy | 3 + .../explain/test_pushdown_explain.groovy | 3 + .../fold_constant/fold_constant_by_be.groovy | 2 +- .../fold_constant_string_arithmatic.groovy | 687 +++ .../suites/nereids_p0/hint/test_use_mv.groovy | 105 + .../insert_into_table/insert_use_table_id.groovy | 107 - .../insert_into_table/no_partition.groovy | 7 - .../insert_into_table/partial_update.groovy | 1 - .../partial_update_complex.groovy | 1 - .../nereids_p0/insert_into_table/random.groovy | 11 + .../insert_into_table/unsupport_type.groovy | 4 - .../update_on_current_timestamp.groovy | 2 +- .../test_encryption_function.groovy | 105 +- .../sql_functions/math_functions/test_conv.groovy | 10 + .../suites/nereids_p0/test_timeout_fallback.groovy | 1 - .../adjust_nullable/set_operation.groovy | 4 +- .../grouping_sets/valid_grouping.groovy | 54 + .../infer_predicate/infer_intersect_except.groovy | 208 + .../pull_up_predicate_literal.groovy | 9 +- .../pull_up_predicate_set_op.groovy | 469 ++ .../infer_set_operator_distinct.groovy | 1 - .../max_min_filter_push_down.groovy | 214 + .../mv/same_name/sync_async_same_name.groovy | 2 +- .../nereids_rules_p0/mv/ssb/mv_ssb_test.groovy | 14 +- .../normalize_window_nullable_agg_test.groovy | 71 + .../nereids_syntax_p0/aggregate_strategies.groovy | 7 - .../nereids_syntax_p0/filter_to_select.groovy | 5 +- .../suites/nereids_syntax_p0/having.groovy | 3 - .../mv/aggregate/agg_sync_mv.groovy | 5 + .../null_aware_left_anti_join.groovy | 9 - .../mv/aggregate/agg_sync_mv.groovy | 552 +++ .../mv/load.groovy | 0 .../suites/nereids_tpch_p0/tpch/agg_cse.groovy | 100 + .../suites/nereids_tpch_p0/tpch/topn_filter.groovy | 40 + .../auto_partition/test_auto_dynamic.groovy | 145 + .../test_auto_partition_behavior.groovy | 139 +- ...t_dynamic_partition_mod_distribution_key.groovy | 6 +- .../multi_partition/test_multi_partition.groovy | 1 - .../multi_partition/test_range_partition.groovy | 2 - .../test_create_table_exception.groovy | 2 +- .../test_partition_create_tablet_rr.groovy | 2 +- .../suites/point_query_p0/test_point_query.groovy | 28 +- .../suites/point_query_p0/test_rowstore.groovy | 26 +- .../suites/query_p0/aggregate/array_agg.groovy | 27 + .../aggregate/select_random_distributed_tbl.groovy | 19 +- .../grouping_sets/test_grouping_sets.groovy | 2 +- .../suites/query_p0/join/test_join2.groovy | 2 +- .../limit/test_map_select_with_limit.groovy | 16 +- .../test_encryption_function.groovy | 200 +- .../test_math_unary_always_nullable.groovy | 85 + .../sql_functions/test_template_one_arg.groovy | 51 + .../sql_functions/test_template_three_args.groovy | 114 + .../sql_functions/test_template_two_args.groovy | 108 + .../window_functions/test_partition_topn.groovy | 66 + .../query_p0/system/test_partitions_schema.groovy | 201 + .../suites/query_p0/system/test_query_sys.groovy | 1 + .../query_p0/system/test_query_sys_tables.groovy | 6 +- .../query_p0/test_array_orderby_limit.groovy | 9 +- .../suites/query_p0/test_forward_qeury.groovy | 10 +- ...ptive_pipeline_task_serial_read_on_limit.groovy | 172 + .../suites/query_profile/scanner_profile.groovy | 4 + .../test_schema_change_concurrent_with_txn.groovy | 3 +- .../test_abort_txn_by_be_cloud1.groovy | 2 +- .../test_abort_txn_by_be_cloud2.groovy | 2 +- .../test_abort_txn_by_be_local5.groovy | 2 +- .../test_abort_txn_by_be_local6.groovy | 2 +- .../test_abort_txn_by_fe_cloud4.groovy | 2 +- .../test_abort_txn_by_fe_local3.groovy | 2 +- .../test_agg_schema_value_modify1.groovy | 2401 +++++++++++ .../test_agg_schema_value_modify4.groovy | 1089 +++++ .../test_merge_dropped_column.groovy | 47 + ...st_show_create_table_with_storage_policy.groovy | 97 + .../suites/statistics/analyze_stats.groovy | 2 +- .../suites/statistics/test_analyze_mtmv.groovy | 9 +- .../suites/statistics/test_analyze_mv.groovy | 86 +- .../test_auto_analyze_black_white_list.groovy | 75 + .../statistics/test_drop_stats_and_truncate.groovy | 228 + .../test_partition_default_medium.groovy | 2 +- .../test_storage_medium_has_disk.groovy | 2 +- .../suites/tpch_sf100_p2/ddl/customer.sql | 1 + .../suites/tpch_sf100_p2/ddl/lineitem.sql | 1 + .../suites/tpch_sf100_p2/ddl/nation.sql | 1 + .../suites/tpch_sf100_p2/ddl/orders.sql | 1 + regression-test/suites/tpch_sf100_p2/ddl/part.sql | 1 + .../suites/tpch_sf100_p2/ddl/partsupp.sql | 1 + .../suites/tpch_sf100_p2/ddl/region.sql | 1 + .../suites/tpch_sf100_p2/ddl/supplier.sql | 1 + .../suites/tpch_sf100_unique_p2/ddl/customer.sql | 1 + .../tpch_sf100_unique_p2/ddl/customer_sequence.sql | 1 + .../suites/tpch_sf100_unique_p2/ddl/lineitem.sql | 1 + .../tpch_sf100_unique_p2/ddl/lineitem_sequence.sql | 1 + .../suites/tpch_sf100_unique_p2/ddl/nation.sql | 1 + .../tpch_sf100_unique_p2/ddl/nation_sequence.sql | 1 + .../suites/tpch_sf100_unique_p2/ddl/orders.sql | 1 + .../tpch_sf100_unique_p2/ddl/orders_sequence.sql | 1 + .../suites/tpch_sf100_unique_p2/ddl/part.sql | 1 + .../tpch_sf100_unique_p2/ddl/part_sequence.sql | 1 + .../suites/tpch_sf100_unique_p2/ddl/partsupp.sql | 1 + .../tpch_sf100_unique_p2/ddl/partsupp_sequence.sql | 1 + .../suites/tpch_sf100_unique_p2/ddl/region.sql | 1 + .../tpch_sf100_unique_p2/ddl/region_sequence.sql | 1 + .../suites/tpch_sf100_unique_p2/ddl/supplier.sql | 1 + .../tpch_sf100_unique_p2/ddl/supplier_sequence.sql | 1 + .../tpch_sf100_unique_sql_p2/ddl/customer.sql | 1 + .../tpch_sf100_unique_sql_p2/ddl/lineitem.sql | 1 + .../suites/tpch_sf100_unique_sql_p2/ddl/nation.sql | 1 + .../suites/tpch_sf100_unique_sql_p2/ddl/orders.sql | 1 + .../suites/tpch_sf100_unique_sql_p2/ddl/part.sql | 1 + .../tpch_sf100_unique_sql_p2/ddl/partsupp.sql | 1 + .../suites/tpch_sf100_unique_sql_p2/ddl/region.sql | 1 + .../tpch_sf100_unique_sql_p2/ddl/supplier.sql | 1 + .../suites/tpch_sf10_unique_p2/ddl/customer.sql | 1 + .../tpch_sf10_unique_p2/ddl/customer_sequence.sql | 1 + .../suites/tpch_sf10_unique_p2/ddl/lineitem.sql | 1 + .../tpch_sf10_unique_p2/ddl/lineitem_sequence.sql | 1 + .../suites/tpch_sf10_unique_p2/ddl/nation.sql | 1 + .../tpch_sf10_unique_p2/ddl/nation_sequence.sql | 1 + .../suites/tpch_sf10_unique_p2/ddl/orders.sql | 1 + .../tpch_sf10_unique_p2/ddl/orders_sequence.sql | 1 + .../suites/tpch_sf10_unique_p2/ddl/part.sql | 1 + .../tpch_sf10_unique_p2/ddl/part_sequence.sql | 1 + .../suites/tpch_sf10_unique_p2/ddl/partsupp.sql | 1 + .../tpch_sf10_unique_p2/ddl/partsupp_sequence.sql | 1 + .../suites/tpch_sf10_unique_p2/ddl/region.sql | 1 + .../tpch_sf10_unique_p2/ddl/region_sequence.sql | 1 + .../suites/tpch_sf10_unique_p2/ddl/supplier.sql | 1 + .../tpch_sf10_unique_p2/ddl/supplier_sequence.sql | 1 + .../suites/tpch_sf1_p2/ddl/create_table.sql | 1 + .../suites/tpch_sf1_p2/ddl/customer.sql | 1 + .../suites/tpch_sf1_p2/ddl/lineitem.sql | 1 + regression-test/suites/tpch_sf1_p2/ddl/nation.sql | 1 + regression-test/suites/tpch_sf1_p2/ddl/orders.sql | 1 + regression-test/suites/tpch_sf1_p2/ddl/part.sql | 1 + .../suites/tpch_sf1_p2/ddl/partsupp.sql | 1 + regression-test/suites/tpch_sf1_p2/ddl/region.sql | 1 + .../suites/tpch_sf1_p2/ddl/supplier.sql | 1 + .../suites/tpch_sf1_unique_p2/ddl/customer.sql | 1 + .../suites/tpch_sf1_unique_p2/ddl/lineitem.sql | 1 + .../suites/tpch_sf1_unique_p2/ddl/nation.sql | 1 + .../suites/tpch_sf1_unique_p2/ddl/orders.sql | 1 + .../suites/tpch_sf1_unique_p2/ddl/part.sql | 1 + .../suites/tpch_sf1_unique_p2/ddl/partsupp.sql | 1 + .../suites/tpch_sf1_unique_p2/ddl/region.sql | 1 + .../suites/tpch_sf1_unique_p2/ddl/supplier.sql | 1 + regression-test/suites/trash_p0/clean_trash.groovy | 4 +- .../unique_with_mow_c_p0/test_create_table.groovy | 2 +- .../test_mow_full_clone_exception.groovy | 2 +- .../unique_with_mow_c_p0/test_schema_change.groovy | 6 +- .../test_schema_change_ck.groovy | 262 ++ .../ssb_unique_load_zstd/ddl/customer_create.sql | 1 + .../ddl/customer_sequence_create.sql | 1 + .../ssb_unique_load_zstd/ddl/date_create.sql | 1 + .../ddl/date_sequence_create.sql | 1 + .../ssb_unique_load_zstd/ddl/lineorder_create.sql | 1 + .../ddl/lineorder_sequence_create.sql | 1 + .../ssb_unique_load_zstd/ddl/part_create.sql | 1 + .../ddl/part_sequence_create.sql | 1 + .../ssb_unique_load_zstd/ddl/supplier_create.sql | 1 + .../ddl/supplier_sequence_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/customer_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/date_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/lineorder_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/part_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/supplier_create.sql | 1 + .../test_new_partial_update_delete.groovy | 3 + .../test_partial_update_after_delete.groovy | 2 + .../test_partial_update_conflict_be_restart.groovy | 2 +- .../test_partial_update_delete.groovy | 297 +- ...artial_update_insert_light_schema_change.groovy | 4 +- ...test_partial_update_insert_schema_change.groovy | 4 +- .../test_partial_update_only_keys.groovy | 62 + .../test_partial_update_schema_change.groovy | 8 +- ...t_partial_update_schema_change_row_store.groovy | 8 +- .../test_mow_full_clone_exception.groovy | 2 +- .../ssb_unique_load_zstd/ddl/customer_create.sql | 1 + .../ddl/customer_sequence_create.sql | 1 + .../ssb_unique_load_zstd/ddl/date_create.sql | 1 + .../ddl/date_sequence_create.sql | 1 + .../ssb_unique_load_zstd/ddl/lineorder_create.sql | 1 + .../ddl/lineorder_sequence_create.sql | 1 + .../ssb_unique_load_zstd/ddl/part_create.sql | 1 + .../ddl/part_sequence_create.sql | 1 + .../ssb_unique_load_zstd/ddl/supplier_create.sql | 1 + .../ddl/supplier_sequence_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/customer_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/date_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/lineorder_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/part_create.sql | 1 + .../ssb_unique_sql_zstd/ddl/supplier_create.sql | 1 + regression-test/suites/variant_p0/load.groovy | 7 +- .../suites/vaults/privilege_restart.groovy | 178 + run-regression-test.sh | 1 + samples/datalake/deltalake_and_kudu/README.md | 147 + .../customer/_delta_log/00000000000000000000.json | 7 + ...110d-44ca-aed8-93ed65c19492-c000.snappy.parquet | Bin 0 -> 392744 bytes ...a84b-4e2c-9e6f-5ac6a59ee91d-c000.snappy.parquet | Bin 0 -> 392284 bytes ...bafd-46a7-b231-400a174b520c-c000.snappy.parquet | Bin 0 -> 390594 bytes ...ef95-4fbe-95c9-2012fd9dbaed-c000.snappy.parquet | Bin 0 -> 389205 bytes samples/datalake/deltalake_and_kudu/login-doris.sh | 20 + samples/datalake/deltalake_and_kudu/login-spark.sh | 20 + .../scripts/create-delta-table.sh | 37 + .../deltalake_and_kudu/scripts/doris-sql.sql | 20 + .../deltalake_and_kudu/scripts/spark-delta.sh | 36 + .../deltalake_and_kudu/scripts/spark-delta.sql | 1 + .../deltalake_and_kudu/scripts/start_doris.sh | 28 + .../start-trinoconnector-compose.sh | 181 + .../stop-trinoconnector-compose.sh | 22 + .../deltalake_and_kudu/trinoconnector-compose.env | 22 + .../deltalake_and_kudu/trinoconnector-compose.yml | 323 ++ tools/FlameGraph/README | 13 + tools/FlameGraph/flamegraph.pl | 1302 ++++++ tools/FlameGraph/stackcollapse-perf.pl | 435 ++ tools/fdb/fdb_ctl.sh | 418 ++ tools/fdb/fdb_vars.sh | 72 + tools/tpcds-tools/bin/build-tpcds-tools.sh | 2 +- tools/tpch-tools/bin/build-tpch-dbgen.sh | 2 +- 1469 files changed, 51993 insertions(+), 18649 deletions(-) create mode 100644 be/src/agent/be_exec_version_manager.cpp create mode 100644 be/src/exec/schema_scanner/schema_catalog_meta_cache_stats_scanner.cpp create mode 100644 be/src/exec/schema_scanner/schema_catalog_meta_cache_stats_scanner.h create mode 100644 be/src/exec/schema_scanner/schema_scanner_helper.cpp create mode 100644 be/src/exec/schema_scanner/schema_scanner_helper.h create mode 100644 be/src/http/action/compaction_score_action.cpp create mode 100644 be/src/http/action/compaction_score_action.h create mode 100644 be/src/http/action/load_channel_action.cpp create mode 100644 be/src/http/action/load_channel_action.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/analyzer/analyzer.cpp create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/analyzer/analyzer.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/boolean_query.cpp create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/boolean_query.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/conjunction_op.cpp create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/conjunction_op.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/disjunction_op.cpp create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/disjunction_op.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/factory.cpp create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/factory.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/factory.inline.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/node.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/operator.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/phrase_query.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/query.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/roaring_query.cpp create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/roaring_query.h create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/term_query.cpp create mode 100644 be/src/olap/rowset/segment_v2/inverted_index/query_v2/term_query.h create mode 100644 be/src/pipeline/exec/cache_sink_operator.cpp create mode 100644 be/src/pipeline/exec/cache_sink_operator.h create mode 100644 be/src/pipeline/exec/cache_source_operator.cpp create mode 100644 be/src/pipeline/exec/cache_source_operator.h create mode 100644 be/src/pipeline/query_cache/query_cache.cpp create mode 100644 be/src/pipeline/query_cache/query_cache.h create mode 100644 be/src/runtime/memory/mem_counter.h create mode 100644 be/src/vec/functions/array/varray_match_function.cpp create mode 100644 be/src/vec/functions/function_math_unary_alway_nullable.h create mode 100644 be/test/olap/rowset/segment_v2/inverted_index/query_v2/query_test.cpp create mode 100644 be/test/vec/columns/column_nullable_test.cpp create mode 100644 be/test/vec/columns/column_nullable_test.h create mode 100644 be/test/vec/exec/vfile_scanner_exception_test.cpp create mode 100644 be/test/vec/function/function_test_template.cpp create mode 100755 docker/thirdparties/docker-compose/elasticsearch/scripts/data/data4_es6.json copy docker/thirdparties/docker-compose/elasticsearch/scripts/data/{data5.json => data6.json} (100%) delete mode 100644 fe/be-java-extensions/max-compute-scanner/src/main/java/org/apache/doris/maxcompute/MaxComputePartitionValue.java delete mode 100644 fe/be-java-extensions/max-compute-scanner/src/main/java/org/apache/doris/maxcompute/MaxComputeTableScan.java delete mode 100644 fe/be-java-extensions/max-compute-scanner/src/test/java/org/apache/doris/maxcompute/MaxComputeJniScannerTest.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/common/RandomIdentifierGenerator.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/fs/remote/RemoteFSPhantomManager.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/fs/remote/RemoteFileSystemPhantomReference.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/httpv2/restv2/ESCatalogAction.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/mysql/authenticate/AuthenticatorFactory.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/mysql/authenticate/DefaultAuthenticatorFactory.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/mysql/authenticate/ldap/LdapAuthenticatorFactory.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundJdbcTableSink.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/hint/UseMvHint.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ProjectAggregateExpressionsForCse.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/properties/SelectHintUseMv.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BuildAggForRandomDistributedTable.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalJdbcTableSinkToPhysicalJdbcTableSink.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MaxMinFilterPushDown.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/StringArithmetic.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/AesDecryptV2.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/AesEncryptV2.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayMatchAll.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayMatchAny.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/SessionUser.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sm4DecryptV2.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sm4EncryptV2.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/JdbcInsertCommandContext.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/JdbcInsertExecutor.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJdbcTableSink.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalJdbcTableSink.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/normalize/NormalizedPartitionPredicates.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/normalize/Normalizer.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/normalize/PartitionRangePredicateNormalizer.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/normalize/PredicateToRange.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/normalize/QueryCacheNormalizer.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/plugin/AuditEvent.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/plugin/audit/AuditEvent.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/plugin/audit/AuditLoader.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/plugin/audit/AuditLoaderPlugin.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/transaction/JdbcTransactionManager.java create mode 100644 fe/fe-core/src/main/resources/META-INF/services/org.apache.doris.mysql.authenticate.AuthenticatorFactory create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/MaxMinFilterPushDownTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/planner/QueryCacheNormalizerTest.java create mode 100644 gensrc/thrift/Normalization.thrift create mode 100644 gensrc/thrift/QueryCache.thrift create mode 100644 regression-test/data/backup_restore/test_backup_restore_atomic.out create mode 100644 regression-test/data/correctness/test_column_nullable_cache.out create mode 100644 regression-test/data/correctness_p0/test_default_e.out create mode 100644 regression-test/data/datatype_p0/nested_types/query/sql/nested_with_join.out create mode 100644 regression-test/data/external_table_p0/hive/write/test_hive_insert_overwrite_with_empty_table.out create mode 100644 regression-test/data/external_table_p0/iceberg/write/test_iceberg_insert_overwrite_with_empty_table.out delete mode 100644 regression-test/data/external_table_p0/mc/test_max_compute_complex_type.out create mode 100644 regression-test/data/external_table_p2/maxcompute/test_max_compute_all_type.out create mode 100644 regression-test/data/external_table_p2/maxcompute/test_max_compute_complex_type.out create mode 100644 regression-test/data/fault_injection_p0/cloud/test_cloud_mow_insert_timeout.out create mode 100644 regression-test/data/insert_p0/group_commit/replay_wal_restart_fe.out create mode 100644 regression-test/data/job_p0/job_meta/job_query_test.out create mode 100644 regression-test/data/load_p0/stream_load/test_stream_load_with_data_quality.csv create mode 100644 regression-test/data/load_p0/stream_load/test_stream_load_with_data_quality2.csv create mode 100644 regression-test/data/mtmv_p0/contain_one_row_relation/one_row_relation_when_create.out create mode 100644 regression-test/data/mtmv_up_down_olap_p0/test_upgrade_downgrade_olap_mtmv.out create mode 100644 regression-test/data/nereids_function_p0/fn_test_am.csv delete mode 100644 regression-test/data/nereids_p0/insert_into_table/insert_use_table_id.out create mode 100644 regression-test/data/nereids_rules_p0/infer_predicate/infer_intersect_except.out create mode 100644 regression-test/data/nereids_rules_p0/infer_predicate/pull_up_predicate_set_op.out create mode 100644 regression-test/data/nereids_rules_p0/max_min_filter_push_down/max_min_filter_push_down.out create mode 100644 regression-test/data/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.out create mode 100644 regression-test/data/nereids_syntax_p1/mv/agg_mv_test.dat create mode 100644 regression-test/data/nereids_syntax_p1/mv/aggregate/agg_sync_mv.out create mode 100644 regression-test/data/nereids_tpch_p0/tpch/agg_cse.out create mode 100644 regression-test/data/partition_p0/auto_partition/test_auto_dynamic.out create mode 100644 regression-test/data/query_p0/aggregate/test_array_agg_complex.csv create mode 100644 regression-test/data/query_p0/sql_functions/math_functions/test_math_unary_always_nullable.out create mode 100644 regression-test/data/query_p0/sql_functions/test_template_one_arg.out create mode 100644 regression-test/data/query_p0/sql_functions/test_template_three_args.out create mode 100644 regression-test/data/query_p0/sql_functions/test_template_two_args.out create mode 100644 regression-test/data/query_p0/sql_functions/window_functions/test_partition_topn.out create mode 100644 regression-test/data/query_p0/system/test_partitions_schema.out create mode 100644 regression-test/data/schema_change_p0/test_agg_schema_value_modify1.out create mode 100644 regression-test/data/unique_with_mow_c_p0/test_schema_change_ck.out create mode 100644 regression-test/data/unique_with_mow_p0/partial_update/test_partial_update_only_keys.out copy regression-test/{data/insert_p0/transaction => regression-test/realData/insert_p0}/txn_insert_values_with_schema_change.out (100%) create mode 100644 regression-test/suites/auth_p0/test_http_meta_databases_auth.groovy create mode 100644 regression-test/suites/auth_p0/test_http_meta_tables_auth.groovy create mode 100644 regression-test/suites/auth_p0/test_http_meta_tables_schema_auth.groovy create mode 100644 regression-test/suites/auth_p0/test_http_table_count_auth.groovy create mode 100644 regression-test/suites/auth_p0/test_http_table_data_auth.groovy create mode 100644 regression-test/suites/backup_restore/test_backup_restore_atomic.groovy create mode 100644 regression-test/suites/backup_restore/test_backup_restore_atomic_with_alter.groovy create mode 100644 regression-test/suites/cloud_p0/multi_cluster/test_auto_start.groovy create mode 100644 regression-test/suites/cloud_p0/multi_cluster/test_multi_follower.groovy create mode 100644 regression-test/suites/cloud_p0/node_mgr/test_sql_mode_node_mgr.groovy create mode 100644 regression-test/suites/compaction/test_compaction_score_action.groovy delete mode 100644 regression-test/suites/control_p0/test_report_version_missing.groovy create mode 100644 regression-test/suites/control_p1/test_report_version_missing.groovy create mode 100644 regression-test/suites/correctness/test_column_nullable_cache.groovy create mode 100644 regression-test/suites/correctness_p0/test_default_e.groovy delete mode 100644 regression-test/suites/correctness_p0/test_push_conjunct_external_table.groovy create mode 100644 regression-test/suites/datatype_p0/nested_types/query/sql/nested_with_join.sql create mode 100644 regression-test/suites/external_table_p0/es/test_es_catalog_http_open_api.groovy create mode 100644 regression-test/suites/external_table_p0/hive/ddl/test_hive_database.groovy create mode 100644 regression-test/suites/external_table_p0/hive/write/test_hive_insert_overwrite_with_empty_table.groovy create mode 100644 regression-test/suites/external_table_p0/iceberg/write/test_iceberg_insert_overwrite_with_empty_table.groovy delete mode 100644 regression-test/suites/external_table_p0/mc/test_max_compute_complex_type.groovy create mode 100644 regression-test/suites/external_table_p2/maxcompute/test_max_compute_all_type.groovy create mode 100644 regression-test/suites/external_table_p2/maxcompute/test_max_compute_complex_type.groovy create mode 100644 regression-test/suites/fault_injection_p0/cloud/test_cloud_mow_insert_timeout.groovy create mode 100644 regression-test/suites/fault_injection_p0/test_index_empty_segment_fault_injection.groovy create mode 100644 regression-test/suites/job_p0/job_meta/job_query_test.groovy create mode 100644 regression-test/suites/job_p0/job_meta/load.groovy create mode 100644 regression-test/suites/load_p0/stream_load/test_steam_load_with_data_quality.groovy create mode 100644 regression-test/suites/mtmv_p0/contain_one_row_relation/one_row_relation_when_create.groovy create mode 100644 regression-test/suites/mtmv_p0/test_use_for_rewrite.groovy create mode 100644 regression-test/suites/mtmv_up_down_olap_p0/load.groovy create mode 100644 regression-test/suites/mtmv_up_down_olap_p0/test_upgrade_downgrade_olap_mtmv.groovy create mode 100644 regression-test/suites/nereids_p0/expression/fold_constant/fold_constant_string_arithmatic.groovy create mode 100644 regression-test/suites/nereids_p0/hint/test_use_mv.groovy delete mode 100644 regression-test/suites/nereids_p0/insert_into_table/insert_use_table_id.groovy create mode 100644 regression-test/suites/nereids_rules_p0/grouping_sets/valid_grouping.groovy create mode 100644 regression-test/suites/nereids_rules_p0/infer_predicate/infer_intersect_except.groovy create mode 100644 regression-test/suites/nereids_rules_p0/infer_predicate/pull_up_predicate_set_op.groovy create mode 100644 regression-test/suites/nereids_rules_p0/max_min_filter_push_down/max_min_filter_push_down.groovy create mode 100644 regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy create mode 100644 regression-test/suites/nereids_syntax_p1/mv/aggregate/agg_sync_mv.groovy copy regression-test/suites/{nereids_syntax_p0 => nereids_syntax_p1}/mv/load.groovy (100%) create mode 100644 regression-test/suites/nereids_tpch_p0/tpch/agg_cse.groovy create mode 100644 regression-test/suites/nereids_tpch_p0/tpch/topn_filter.groovy create mode 100644 regression-test/suites/partition_p0/auto_partition/test_auto_dynamic.groovy create mode 100644 regression-test/suites/query_p0/sql_functions/math_functions/test_math_unary_always_nullable.groovy create mode 100644 regression-test/suites/query_p0/sql_functions/test_template_one_arg.groovy create mode 100644 regression-test/suites/query_p0/sql_functions/test_template_three_args.groovy create mode 100644 regression-test/suites/query_p0/sql_functions/test_template_two_args.groovy create mode 100644 regression-test/suites/query_p0/sql_functions/window_functions/test_partition_topn.groovy create mode 100644 regression-test/suites/query_p0/system/test_partitions_schema.groovy create mode 100644 regression-test/suites/query_profile/adaptive_pipeline_task_serial_read_on_limit.groovy create mode 100644 regression-test/suites/schema_change_p0/test_agg_schema_value_modify1.groovy create mode 100644 regression-test/suites/schema_change_p0/test_agg_schema_value_modify4.groovy create mode 100644 regression-test/suites/schema_change_p0/test_merge_dropped_column.groovy create mode 100644 regression-test/suites/show_p0/test_show_create_table_with_storage_policy.groovy create mode 100644 regression-test/suites/statistics/test_auto_analyze_black_white_list.groovy create mode 100644 regression-test/suites/statistics/test_drop_stats_and_truncate.groovy create mode 100644 regression-test/suites/unique_with_mow_c_p0/test_schema_change_ck.groovy create mode 100644 regression-test/suites/unique_with_mow_p0/partial_update/test_partial_update_only_keys.groovy create mode 100644 regression-test/suites/vaults/privilege_restart.groovy create mode 100644 samples/datalake/deltalake_and_kudu/README.md create mode 100644 samples/datalake/deltalake_and_kudu/data/customer/_delta_log/00000000000000000000.json create mode 100644 samples/datalake/deltalake_and_kudu/data/customer/part-00000-44ff362c-110d-44ca-aed8-93ed65c19492-c000.snappy.parquet create mode 100644 samples/datalake/deltalake_and_kudu/data/customer/part-00001-749ded2d-a84b-4e2c-9e6f-5ac6a59ee91d-c000.snappy.parquet create mode 100644 samples/datalake/deltalake_and_kudu/data/customer/part-00002-137a1b68-bafd-46a7-b231-400a174b520c-c000.snappy.parquet create mode 100644 samples/datalake/deltalake_and_kudu/data/customer/part-00003-75203d54-ef95-4fbe-95c9-2012fd9dbaed-c000.snappy.parquet create mode 100755 samples/datalake/deltalake_and_kudu/login-doris.sh create mode 100755 samples/datalake/deltalake_and_kudu/login-spark.sh create mode 100644 samples/datalake/deltalake_and_kudu/scripts/create-delta-table.sh create mode 100644 samples/datalake/deltalake_and_kudu/scripts/doris-sql.sql create mode 100755 samples/datalake/deltalake_and_kudu/scripts/spark-delta.sh create mode 100644 samples/datalake/deltalake_and_kudu/scripts/spark-delta.sql create mode 100755 samples/datalake/deltalake_and_kudu/scripts/start_doris.sh create mode 100755 samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh create mode 100755 samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh create mode 100644 samples/datalake/deltalake_and_kudu/trinoconnector-compose.env create mode 100644 samples/datalake/deltalake_and_kudu/trinoconnector-compose.yml create mode 100644 tools/FlameGraph/README create mode 100755 tools/FlameGraph/flamegraph.pl create mode 100755 tools/FlameGraph/stackcollapse-perf.pl create mode 100755 tools/fdb/fdb_ctl.sh create mode 100644 tools/fdb/fdb_vars.sh --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org