This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a change to branch refactor_rf in repository https://gitbox.apache.org/repos/asf/doris.git
discard 4a71ea94d45 refactor of RuntimeFilterSlots add 237c70251f9 [Fix](compile) Initialize the KerberosTicketEntry struct variable in kerberos ticket. (#47826) add c9519d50755 [fix] Fixed length error in compress.cpp (#48210) add 7a2cca6650e [Fix](regression) update flink connector version for test case (#48292) add 063e70cab9e [pipeline](refactor) re-construct task scheduling with spilling (#48294) add 8f0ec1cc6f1 [fix](auth)ldap template ignore partial result exception (#47858) add 3df183473ab [opt](docker) Add ranger docker component (#47697) add 87a984820db [fix](client) Do not log in thrift exception when ADDRESS_SANITIZER is defined (#48347) add 2ac6779a0fe [fix](nereids) fix merge project contains non foldable expression (#48321) add d4fca3909d7 [feat](nereids) simplify comparison for cast from smaller int type to bigger int type (#48240) add 235ffb81c32 [fix](Export) change the export logical (#48022) add 6d4ba9eb382 [fix](maxcompute)fix maxcompute partition column schema order (#48325) add fc890e590d0 [Enhancement](recycler) Add valid s3 vault config for recycler (#47723) add a5fa85bfcf0 [Fix](compaction) Fix nullptr when executing cold compaction (#48329) add 20d95aad7ae [Fix](remote-fs)Change closed Field to Instance-Level to Avoid Global Shutdown Issues in RemoteFileSystem (#48351) add a3eb715fcbf [fix](schema scan) Fix invalid pointer access (#48370) add 81c8d3d20c8 [test](regression) fix test_nonfoldable (#48386) add e0d85882bdf [fix](auth)Privatize the authentication methods in the Auth class to avoid being called incorrectly (#48033) add 939138e0411 [test](profile) More unit test for profile on FE (#47623) add 7fc46efd140 [fix](p2_test) fix Export p2 test (#48380) add 5e89a4b97a6 [case](auth)fix follower can not login (#47673) add 0ffcce85527 [fix](collect_list) fix collect_list in multi BE will ser/de failed (#48314) add a1713f0e641 [chore](profile) Remove enable_verbose_profile (#47981) add 19935277a74 [case](auth) fix export case (#48123) add ec61adfe91a [case](auth)grant usage_priv to user when in vault mode (#47316) add 031a4ab7d5f [log](regression)Add log to investigate SLEEP command error. (#48385) add 735b898d041 [fix](lock) Fix `~VDataStreamMgr` lock _receiver_map (#48057) add 4a2b55de547 [fix](nereids)Support negative partition range value and negative column default value. (#48297) add 654ec15fff5 [Fix](ecosystem) Fix kettle plugin load field correspondence problem (#48412) add 47832e3e323 [refactor](rename) Rename open function in operator (#48402) add f8ececeefd1 [sort](ut) Test sort merger with single stream (#48415) add dadbb4e5a6b [fix](nereids) Add unique id to non foldable expression (#48103) add 89e032a7fee [chore](ci) change s3Source to internal (#48427) add f79292f5927 [improve](load) improve error message in commit (#48316) add 93e8b0d1cf0 [fix](file-cache) Clear file cache when tablet meta not found in ms (#48318) add 8e2b4a10687 [fix](schema-change) Remove incorrect assertion for tablet state after committing sc job (#48219) add 3d5575e05e1 [fix](cold hot separation) Fix the issue that files on the remote storage are not deleted after triggering cold data compaction. (#48109) add e37a3a41b70 refactor of RuntimeFilterSlots add ff177645a35 some fix and add ut This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (4a71ea94d45) \ N -- N -- N refs/heads/refactor_rf (ff177645a35) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: be/src/agent/utils.cpp | 12 +- be/src/cloud/cloud_schema_change_job.cpp | 7 - be/src/cloud/cloud_tablet.cpp | 2 +- be/src/common/kerberos/kerberos_ticket_mgr.cpp | 4 +- be/src/exec/schema_scanner.cpp | 1 - be/src/olap/olap_server.cpp | 8 +- be/src/olap/tablet.cpp | 5 + be/src/pipeline/dependency.cpp | 7 + be/src/pipeline/dependency.h | 7 + be/src/pipeline/exec/aggregation_sink_operator.cpp | 4 +- be/src/pipeline/exec/aggregation_sink_operator.h | 2 +- be/src/pipeline/exec/analytic_sink_operator.cpp | 4 +- be/src/pipeline/exec/analytic_sink_operator.h | 2 +- be/src/pipeline/exec/analytic_source_operator.cpp | 4 +- be/src/pipeline/exec/analytic_source_operator.h | 2 +- be/src/pipeline/exec/cache_source_operator.h | 5 - be/src/pipeline/exec/datagen_operator.cpp | 4 +- be/src/pipeline/exec/datagen_operator.h | 2 +- .../distinct_streaming_aggregation_operator.cpp | 4 +- .../exec/distinct_streaming_aggregation_operator.h | 2 +- be/src/pipeline/exec/es_scan_operator.cpp | 4 +- be/src/pipeline/exec/es_scan_operator.h | 2 +- be/src/pipeline/exec/exchange_sink_buffer.cpp | 4 +- be/src/pipeline/exec/exchange_sink_operator.cpp | 4 +- be/src/pipeline/exec/exchange_sink_operator.h | 2 +- be/src/pipeline/exec/exchange_source_operator.cpp | 4 +- be/src/pipeline/exec/exchange_source_operator.h | 2 +- be/src/pipeline/exec/file_scan_operator.cpp | 4 +- be/src/pipeline/exec/file_scan_operator.h | 2 +- .../exec/group_commit_block_sink_operator.cpp | 4 +- .../exec/group_commit_block_sink_operator.h | 2 +- be/src/pipeline/exec/hashjoin_build_sink.cpp | 4 +- be/src/pipeline/exec/hashjoin_build_sink.h | 2 +- be/src/pipeline/exec/hashjoin_probe_operator.cpp | 4 +- be/src/pipeline/exec/hashjoin_probe_operator.h | 2 +- be/src/pipeline/exec/hive_table_sink_operator.h | 4 +- be/src/pipeline/exec/iceberg_table_sink_operator.h | 4 +- be/src/pipeline/exec/jdbc_table_sink_operator.cpp | 4 +- be/src/pipeline/exec/jdbc_table_sink_operator.h | 2 +- .../pipeline/exec/memory_scratch_sink_operator.cpp | 4 +- .../pipeline/exec/memory_scratch_sink_operator.h | 2 +- .../pipeline/exec/multi_cast_data_stream_source.h | 4 +- .../exec/nested_loop_join_build_operator.cpp | 4 +- .../exec/nested_loop_join_build_operator.h | 2 +- .../exec/nested_loop_join_probe_operator.cpp | 4 +- .../exec/nested_loop_join_probe_operator.h | 2 +- be/src/pipeline/exec/olap_table_sink_operator.h | 4 +- be/src/pipeline/exec/olap_table_sink_v2_operator.h | 4 +- be/src/pipeline/exec/operator.cpp | 6 +- be/src/pipeline/exec/operator.h | 6 +- .../pipeline/exec/partition_sort_sink_operator.cpp | 4 +- .../pipeline/exec/partition_sort_sink_operator.h | 2 +- .../exec/partitioned_aggregation_sink_operator.cpp | 4 +- .../exec/partitioned_aggregation_sink_operator.h | 2 +- .../partitioned_aggregation_source_operator.cpp | 6 +- .../exec/partitioned_aggregation_source_operator.h | 2 +- .../exec/partitioned_hash_join_probe_operator.cpp | 8 +- .../exec/partitioned_hash_join_probe_operator.h | 2 +- .../exec/partitioned_hash_join_sink_operator.cpp | 6 +- .../exec/partitioned_hash_join_sink_operator.h | 2 +- be/src/pipeline/exec/repeat_operator.cpp | 4 +- be/src/pipeline/exec/repeat_operator.h | 2 +- be/src/pipeline/exec/result_file_sink_operator.cpp | 4 +- be/src/pipeline/exec/result_file_sink_operator.h | 2 +- be/src/pipeline/exec/result_sink_operator.cpp | 4 +- be/src/pipeline/exec/result_sink_operator.h | 2 +- be/src/pipeline/exec/scan_operator.cpp | 4 +- be/src/pipeline/exec/scan_operator.h | 2 +- be/src/pipeline/exec/schema_scan_operator.cpp | 4 +- be/src/pipeline/exec/schema_scan_operator.h | 2 +- be/src/pipeline/exec/set_probe_sink_operator.cpp | 4 +- be/src/pipeline/exec/set_probe_sink_operator.h | 2 +- be/src/pipeline/exec/set_sink_operator.cpp | 4 +- be/src/pipeline/exec/set_sink_operator.h | 2 +- be/src/pipeline/exec/sort_sink_operator.cpp | 4 +- be/src/pipeline/exec/sort_sink_operator.h | 2 +- be/src/pipeline/exec/sort_source_operator.cpp | 4 +- be/src/pipeline/exec/sort_source_operator.h | 2 +- be/src/pipeline/exec/spill_sort_sink_operator.cpp | 6 +- be/src/pipeline/exec/spill_sort_sink_operator.h | 2 +- .../pipeline/exec/spill_sort_source_operator.cpp | 6 +- be/src/pipeline/exec/spill_sort_source_operator.h | 2 +- .../exec/streaming_aggregation_operator.cpp | 4 +- .../pipeline/exec/streaming_aggregation_operator.h | 2 +- be/src/pipeline/exec/table_function_operator.cpp | 4 +- be/src/pipeline/exec/table_function_operator.h | 2 +- be/src/pipeline/exec/union_sink_operator.cpp | 4 +- be/src/pipeline/exec/union_sink_operator.h | 2 +- be/src/pipeline/exec/union_source_operator.h | 4 +- .../local_exchange_sink_operator.cpp | 4 +- .../local_exchange/local_exchange_sink_operator.h | 2 +- .../local_exchange_source_operator.h | 2 +- be/src/pipeline/pipeline.cpp | 4 +- be/src/pipeline/pipeline_fragment_context.cpp | 13 - be/src/pipeline/pipeline_fragment_context.h | 2 - be/src/pipeline/pipeline_task.cpp | 164 +- be/src/pipeline/pipeline_task.h | 26 +- be/src/runtime/descriptor_helper.h | 55 + be/src/runtime/fragment_mgr.cpp | 1 - be/src/runtime/query_context.cpp | 24 +- be/src/runtime/query_context.h | 8 +- be/src/runtime/runtime_state.h | 5 - be/src/runtime_filter/runtime_filter_merger.h | 10 +- be/src/runtime_filter/runtime_filter_producer.cpp | 4 +- be/src/runtime_filter/runtime_filter_producer.h | 8 +- be/src/runtime_filter/runtime_filter_wrapper.cpp | 5 +- be/src/runtime_filter/runtime_filter_wrapper.h | 5 +- be/src/util/runtime_profile.cpp | 7 +- be/src/util/runtime_profile.h | 9 - .../aggregate_function_collect.h | 27 +- be/src/vec/core/sort_cursor.h | 5 +- be/src/vec/functions/function_compress.cpp | 14 +- be/src/vec/runtime/vdata_stream_mgr.cpp | 15 +- be/src/vec/runtime/vsorted_run_merger.cpp | 4 +- be/test/pipeline/operator/agg_operator_test.cpp | 24 +- .../operator/exchange_sink_operator_test.cpp | 2 +- be/test/pipeline/pipeline_test.cpp | 32 +- be/test/pipeline/thrift_builder.h | 245 ++- .../runtime_filter_consumer_test.cpp | 67 + .../runtime_filter/runtime_filter_merger_test.cpp | 179 ++ .../runtime_filter_producer_test.cpp | 165 ++ be/test/runtime_filter/runtime_filter_test_utils.h | 68 + .../vec/aggregate_functions/agg_collect_test.cpp | 45 +- be/test/vec/exec/vfile_scanner_exception_test.cpp | 2 +- be/test/vec/exec/vwal_scanner_test.cpp | 2 +- be/test/vec/runtime/sort_merger_test.cpp | 52 + cloud/src/common/config.h | 3 + cloud/src/meta-service/meta_service_helper.h | 64 +- cloud/src/recycler/recycler.cpp | 28 +- cloud/src/recycler/s3_accessor.cpp | 4 +- cloud/test/recycler_test.cpp | 134 ++ .../ranger/ranger-admin/ranger-entrypoint.sh | 25 +- .../ranger-mysql/my.cnf} | 4 +- .../docker-compose/ranger/ranger-solr/elevate.xml | 38 + .../ranger/ranger-solr/managed-schema | 95 + .../ranger/ranger-solr/solrconfig.xml | 1872 ++++++++++++++++++++ .../docker-compose/ranger/ranger.yaml.tpl | 87 + .../clickhouse.env => ranger/ranger_settings.env} | 4 +- .../ranger/script/install_doris_ranger_plugins.sh | 19 +- .../script/install_doris_service_def.sh} | 14 +- docker/thirdparties/run-thirdparties-docker.sh | 25 +- .../steps/dorisstreamloader/DorisStreamLoader.java | 28 +- .../load/DorisBatchStreamLoad.java | 47 +- .../steps/dorisstreamloader/load/DorisOptions.java | 2 +- .../main/java/org/apache/doris/common/Config.java | 5 - .../antlr4/org/apache/doris/nereids/DorisParser.g4 | 4 +- .../org/apache/doris/analysis/ShowClusterStmt.java | 2 +- .../ranger/doris/RangerDorisAccessController.java | 7 +- .../ranger/hive/RangerHiveAccessController.java | 7 +- .../doris/cloud/analysis/UseCloudClusterStmt.java | 2 +- .../org/apache/doris/cloud/catalog/CloudEnv.java | 2 +- .../org/apache/doris/common/profile/Profile.java | 4 +- .../doris/common/profile/ProfileManager.java | 33 +- .../doris/common/profile/RuntimeProfile.java | 42 +- .../apache/doris/datasource/InternalCatalog.java | 2 +- .../maxcompute/MaxComputeExternalTable.java | 37 +- .../maxcompute/MaxComputeSchemaCacheValue.java | 21 +- .../apache/doris/fs/remote/RemoteFileSystem.java | 2 +- .../main/java/org/apache/doris/load/ExportJob.java | 142 +- .../org/apache/doris/load/ExportTaskExecutor.java | 49 +- .../doris/mysql/authenticate/ldap/LdapClient.java | 3 + .../mysql/privilege/AccessControllerManager.java | 7 + .../org/apache/doris/mysql/privilege/Auth.java | 18 +- .../mysql/privilege/CatalogAccessController.java | 4 +- .../mysql/privilege/InternalAccessController.java | 9 +- .../apache/doris/mysql/privilege/UserProperty.java | 2 +- .../doris/nereids/parser/LogicalPlanBuilder.java | 15 +- .../processor/post/MergeProjectPostProcessor.java | 2 +- .../doris/nereids/processor/post/Validator.java | 7 - .../rules/exploration/MergeProjectsCBO.java | 1 + .../rules/SimplifyComparisonPredicate.java | 58 +- .../expression/rules/SimplifyConflictCompound.java | 32 +- .../expression/rules/SimplifySelfComparison.java | 7 +- .../rules/rewrite/DeferMaterializeTopNResult.java | 2 +- .../doris/nereids/rules/rewrite/MergeProjects.java | 5 +- .../trees/expressions/functions/scalar/Random.java | 61 +- .../expressions/functions/scalar/RandomBytes.java | 39 +- .../trees/expressions/functions/scalar/Uuid.java | 36 + .../expressions/functions/scalar/UuidNumeric.java | 36 + .../trees/expressions/literal/DecimalLiteral.java | 2 +- .../expressions/literal/DecimalV3Literal.java | 2 +- .../trees/expressions/literal/DoubleLiteral.java | 2 +- .../trees/expressions/literal/FloatLiteral.java | 2 +- .../expressions/literal/IntegerLikeLiteral.java | 2 +- .../trees/expressions/literal/LargeIntLiteral.java | 2 +- .../trees/expressions/literal/NumericLiteral.java | 2 +- .../doris/nereids/trees/plans/algebra/Project.java | 10 + .../plans/commands/call/CallExecuteStmtFunc.java | 2 +- .../plans/commands/call/CallFlushAuditLogFunc.java | 2 +- .../trees/plans/physical/PhysicalProject.java | 19 + .../org/apache/doris/nereids/util/PlanUtils.java | 24 + .../java/org/apache/doris/qe/ConnectContext.java | 10 +- .../java/org/apache/doris/qe/SessionVariable.java | 30 +- .../java/org/apache/doris/qe/ShowExecutor.java | 9 +- .../java/org/apache/doris/qe/StmtExecutor.java | 4 +- .../tablefunction/QueryTableValueFunction.java | 2 +- .../doris/transaction/DatabaseTransactionMgr.java | 6 +- .../analysis/ExportToOutfileLogicalPlanTest.java | 127 +- .../doris/common/profile/ProfileManagerTest.java | 437 +++++ .../common/profile/RuntimeProfileMergeTest.java | 296 ++++ .../doris/common/profile/RuntimeProfileTest.java | 18 +- .../apache/doris/datasource/ColumnPrivTest.java | 8 +- .../nereids/privileges/TestCheckPrivileges.java | 7 +- .../rules/expression/ExpressionRewriteTest.java | 2 +- .../rules/expression/SimplifyRangeTest.java | 24 +- .../rules/SimplifyComparisonPredicateTest.java | 29 +- .../rules/SimplifySelfComparisonTest.java | 2 +- .../functions/NonfoldableFunctionTest.java | 58 + gensrc/thrift/PaloInternalService.thrift | 2 +- gensrc/thrift/RuntimeProfile.thrift | 3 +- .../data/export_p2/test_export_max_file_size.out | Bin 206386 -> 207176 bytes .../test_external_catalog_maxcompute.out | Bin 3747 -> 4745 bytes .../push_down_filter_other_condition.out | Bin 8004 -> 7975 bytes .../filter_push_down/push_filter_through.out | Bin 10014 -> 10001 bytes .../extend_infer_equal_predicate.out | Bin 30505 -> 30317 bytes .../infer_predicate/infer_intersect_except.out | Bin 4350 -> 4358 bytes .../max_min_filter_push_down.out | Bin 5982 -> 5992 bytes .../data/nereids_rules_p0/test_nonfoldable.out | Bin 0 -> 2839 bytes .../partition_p0/test_negative_partition_value.out | Bin 0 -> 375 bytes .../data/query_p0/aggregate/array_agg.out | Bin 37278 -> 59884 bytes .../string_functions/test_compress_uncompress.out | Bin 741 -> 1321 bytes .../tpcds_sf100/noStatsRfPrune/query66.out | Bin 4417 -> 4403 bytes .../tpcds_sf100/noStatsRfPrune/query84.out | Bin 2136 -> 2132 bytes .../tpcds_sf100/no_stats_shape/query66.out | Bin 4521 -> 4507 bytes .../tpcds_sf100/no_stats_shape/query84.out | Bin 2187 -> 2183 bytes .../shape_check/tpcds_sf100/rf_prune/query66.out | Bin 4441 -> 4427 bytes .../shape_check/tpcds_sf100/rf_prune/query84.out | Bin 2192 -> 2188 bytes .../data/shape_check/tpcds_sf100/shape/query66.out | Bin 4545 -> 4531 bytes .../data/shape_check/tpcds_sf100/shape/query84.out | Bin 2192 -> 2188 bytes .../data/shape_check/tpcds_sf1000/hint/query66.out | Bin 4541 -> 4527 bytes .../data/shape_check/tpcds_sf1000/hint/query84.out | Bin 2353 -> 2349 bytes .../shape_check/tpcds_sf1000/shape/query66.out | Bin 4541 -> 4527 bytes .../shape_check/tpcds_sf1000/shape/query84.out | Bin 2195 -> 2191 bytes .../shape_check/tpcds_sf10t_orc/shape/query84.out | Bin 2193 -> 2189 bytes .../test_negative_default_column_value.out} | Bin 141 -> 143 bytes regression-test/framework/pom.xml | 2 +- .../org/apache/doris/regression/suite/Suite.groovy | 20 + .../cloud_p0/conf/regression-conf-custom.groovy | 1 + .../cloud_p1/conf/regression-conf-custom.groovy | 1 + .../pipeline/external/conf/regression-conf.groovy | 3 +- regression-test/pipeline/p0/conf/be.conf | 4 + .../pipeline/p0/conf/regression-conf.groovy | 3 +- .../pipeline/p1/conf/regression-conf.groovy | 3 +- .../vault_p0/conf/regression-conf-custom.groovy | 3 + .../suites/auth_call/test_ddl_job_auth.groovy | 12 +- .../suites/auth_call/test_ddl_mtmv_auth.groovy | 10 +- .../suites/auth_call/test_ddl_table_auth.groovy | 14 +- .../auth_call/test_dml_export_table_auth.groovy | 3 +- .../test_master_slave_consistency_auth.groovy | 40 + .../cold_data_compaction.groovy | 132 ++ .../test_external_catalog_maxcompute.groovy | 26 +- ...est_cold_data_compaction_fault_injection.groovy | 134 ++ ...group_commit_insert_into_lineitem_normal.groovy | 1 + .../nereids_rules_p0/test_nonfoldable.groovy | 77 + .../test_negative_partition_value.groovy | 58 + .../suites/query_p0/aggregate/array_agg.groovy | 14 + .../test_compress_uncompress.groovy | 20 + .../test_negative_default_column_value.groovy} | 22 +- 258 files changed, 5532 insertions(+), 1040 deletions(-) create mode 100644 be/test/runtime_filter/runtime_filter_consumer_test.cpp create mode 100644 be/test/runtime_filter/runtime_filter_merger_test.cpp create mode 100644 be/test/runtime_filter/runtime_filter_producer_test.cpp create mode 100644 be/test/runtime_filter/runtime_filter_test_utils.h copy fs_brokers/apache_hdfs_broker/conf/apache_hdfs_broker.conf => docker/thirdparties/docker-compose/ranger/ranger-admin/ranger-entrypoint.sh (72%) copy docker/thirdparties/docker-compose/{hive/hadoop-hive-2x.env.tpl => ranger/ranger-mysql/my.cnf} (94%) create mode 100644 docker/thirdparties/docker-compose/ranger/ranger-solr/elevate.xml create mode 100644 docker/thirdparties/docker-compose/ranger/ranger-solr/managed-schema create mode 100644 docker/thirdparties/docker-compose/ranger/ranger-solr/solrconfig.xml create mode 100644 docker/thirdparties/docker-compose/ranger/ranger.yaml.tpl copy docker/thirdparties/docker-compose/{clickhouse/clickhouse.env => ranger/ranger_settings.env} (90%) copy .editorconfig => docker/thirdparties/docker-compose/ranger/script/install_doris_ranger_plugins.sh (62%) mode change 100644 => 100755 copy docker/thirdparties/docker-compose/{hive/hadoop-hive-2x.env.tpl => ranger/script/install_doris_service_def.sh} (64%) mode change 100644 => 100755 create mode 100644 fe/fe-core/src/test/java/org/apache/doris/common/profile/RuntimeProfileMergeTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/NonfoldableFunctionTest.java create mode 100644 regression-test/data/nereids_rules_p0/test_nonfoldable.out create mode 100644 regression-test/data/partition_p0/test_negative_partition_value.out copy regression-test/data/{bloom_filter_p0/test_bloom_filter_drop_column.out => table_p0/test_negative_default_column_value.out} (67%) create mode 100644 regression-test/suites/cold_heat_separation/cold_data_compaction.groovy create mode 100644 regression-test/suites/fault_injection_p0/test_cold_data_compaction_fault_injection.groovy create mode 100644 regression-test/suites/nereids_rules_p0/test_nonfoldable.groovy create mode 100644 regression-test/suites/partition_p0/test_negative_partition_value.groovy copy regression-test/suites/{nereids_syntax_p0/order_by_const.groovy => table_p0/test_negative_default_column_value.groovy} (59%) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org