This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch auto-pick-44989-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git
omit 08b9811ceee [fix](regression-test) Fix potential multi FE desc table result not refresh after schema change (#44989) add fa633d751dc [pick](branch-2.1) pick #44489 #44894 (#45024) add 11c517fe1ec [enhancement][docker]update routine docker file (#45048) add 405b50b1b7a [Improvement](queue) Return value of concurrent queue should be proce… (#45032) add a98cb8c8bff branch-2.1: [fix](arrow-flight-sql) Fix query result is empty and not return query error message #45023 (#45053) add 041a1fe0958 branch-2.1: [fix](nereids) fix compare with long min for simplify comparison rule (#44922) add a67bbf3c586 branch-2.1: [Fix](job)Fix CAS competition failure leading to message publishing failure. #45018 (#45030) add 0c3bc1bcd07 [fix](auth)fix be enable http auth, some request link never return. (#44959) (#45063) add 21e21f5e3b4 [opt](exec) Use PASSTHROUGH to improve the concurrency of the ADAPTIV… (#44971) add 94746b764db branch-2.1: [fix](memory) Fix UT `ThreadMemTrackerMgrTest` #44147 (#44936) add 682cec9b9a5 [test](stream load) add cases to ensure the correctness of the document (#43016) (#44385) add f038f28682b branch-2.1: [fix](mtmv) Fixed an error in setting variables in nested materialize views #44960 (#45038) add d3c10f01e3a branch-2.1: [Chore](Job)Add scheduling logs for periodic jobs. #44843 (#45071) add 2ed306d0b16 [pick](branch-2.1) pick #44286 (#45055) add ea24410faf5 [enhancement][docker] fix kafka docker issue (#45091) add f0324e2a562 branch-2.1: [improvement](information_schema)Support show default value in information_schema. #44849 (#45080) add 53291bd040d branch-2.1: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) (#45041) add 08c8a416ea2 branch-2.1: [fix](runtime_profile) fix race condition in to_thrift #45047 (#45099) add 3018e9af4c7 branch-2.1: [fix](routine load) fix invalid EOF tasks scheduling delay #44949 (#45088) add bea9564e385 branch-2.1: [fix](Export) Fix the problem of exporting stuck #44944 (#45094) add ebb21ef0310 branch-2.1: [Fix](merge-on-write) Add defensive check before partial update #44687 (#45086) add 4f45fefea28 [enhance](mtmv)Optimize MTMV lock logic (#44967) (#45076) add d4a6fd1850f Revert #43255 & #44615 (#45096) add bd14d864ef4 branch-2.1: [fix](editlog) Fix replay BatchDropInfo #45077 (#45102) add 6ddc45b3fd9 branch-2.1: [opt](auto-inc) Allow to miss auto-increment column and other value columns in partial update #44528 (#45073) add b4e2e2cf78e branch-2.1: [fix](hudi)Add hudi catalog read partition table partition prune #44669 (#45113) add 9bb9a56a48b [opt](paimon)Remove unnecessary fields for 2.1 (#44906) (#45118) add 8f0276809a9 branch-2.1: [fix](compaction) tablet is not removed in compaction queue if dropped #45085 (#45136) add 02840a98ffa branch-2.1: [bugfix](hive)Fix split assignment leak. releated to issue (#40683). #44845 (#45144) add a44ac563521 branch-2.1: [chore](job-test)Add comments to the job's test cases #44627 (#45142) add 3714063975c branch-2.1: [feat](catalog)Replace HadoopUGI with HadoopKerberosAuthenticator to Support Kerberos Ticket Auto-Renewal #44916 (#45138) add 358bceaf577 branch-2.1: [fix](case)fix manager show proc case unstable. #45040 (#45140) add 2543af8d3ba [feature](function) support orthogonal_bitmap_expr_calculate & orthogonal_bitmap_expr_calculate_count for nereids (#44991) (#45012) add 013d724f667 [fix](regression-test) Fix potential multi FE desc table result not refresh after schema change (#44989) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (08b9811ceee) \ N -- N -- N refs/heads/auto-pick-44989-branch-2.1 (013d724f667) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: be/src/agent/task_worker_pool.cpp | 4 +- .../exec/schema_scanner/schema_columns_scanner.cpp | 14 +- be/src/http/http_handler_with_auth.cpp | 10 + be/src/olap/olap_server.cpp | 3 +- be/src/olap/rowset/segment_v2/segment_writer.cpp | 42 +- be/src/olap/rowset/segment_v2/segment_writer.h | 2 + .../rowset/segment_v2/vertical_segment_writer.cpp | 40 +- .../rowset/segment_v2/vertical_segment_writer.h | 2 + be/src/olap/tablet_manager.cpp | 20 +- be/src/olap/tablet_meta_manager.cpp | 3 +- be/src/pipeline/pipeline.h | 8 + .../pipeline_x/local_exchange/local_exchanger.h | 14 +- .../pipeline_x/pipeline_x_fragment_context.cpp | 10 +- be/src/runtime/buffer_control_block.cpp | 20 +- be/src/runtime/exec_env.h | 1 - .../arrow_flight/arrow_flight_batch_reader.cpp | 2 +- be/src/util/runtime_profile.cpp | 5 +- .../aggregate_function_orthogonal_bitmap.h | 2 +- be/src/vec/exec/format/orc/vorc_reader.cpp | 486 +++++++------------- be/src/vec/exec/format/orc/vorc_reader.h | 45 +- be/src/vec/exec/format/table/hudi_jni_reader.cpp | 15 +- be/src/vec/exec/format/table/hudi_jni_reader.h | 4 +- be/src/vec/exec/scan/scanner_context.cpp | 5 +- be/src/vec/exec/scan/vfile_scanner.cpp | 16 +- be/test/exec/test_data/orc_scanner/orders.orc | Bin 1293 -> 0 bytes be/test/http/http_client_test.cpp | 1 - .../runtime/memory/thread_mem_tracker_mgr_test.cpp | 6 +- be/test/testutil/desc_tbl_builder.cpp | 29 +- be/test/testutil/desc_tbl_builder.h | 17 +- be/test/vec/exec/orc_reader_test.cpp | 158 ------- build.sh | 4 + .../docker-compose/kafka/kafka.yaml.tpl | 21 +- docker/thirdparties/run-thirdparties-docker.sh | 16 +- fe/be-java-extensions/hadoop-hudi-scanner/pom.xml | 227 ++++++++++ .../apache/doris/hudi/HadoopHudiColumnValue.java | 219 +++++++++ .../apache/doris/hudi/HadoopHudiJniScanner.java | 271 ++++++++++++ .../src/main/resources/package.xml | 0 .../java/org/apache/doris/hudi/HudiJniScanner.java | 16 +- .../src/main/java/org/apache/doris/hudi/Utils.java | 14 +- .../org/apache/doris/hudi/BaseSplitReader.scala | 15 +- fe/be-java-extensions/pom.xml | 1 + .../common/security/authentication/HadoopUGI.java | 103 ----- .../apache/doris/analysis/NativeInsertStmt.java | 8 + .../doris/catalog/BuiltinAggregateFunctions.java | 4 + .../main/java/org/apache/doris/catalog/Env.java | 46 +- .../main/java/org/apache/doris/catalog/MTMV.java | 2 +- .../java/org/apache/doris/catalog/OlapTable.java | 20 +- .../apache/doris/datasource/FileQueryScanNode.java | 2 + .../org/apache/doris/datasource/FileScanNode.java | 1 - .../doris/datasource/hive/HMSExternalTable.java | 26 +- .../datasource/hive/HiveMetaStoreClientHelper.java | 21 +- .../apache/doris/datasource/hudi/HudiUtils.java | 52 ++- .../hudi/source/COWIncrementalRelation.java | 11 +- .../hudi/source/HudiLocalEngineContext.java | 67 ++- .../hudi/source/HudiPartitionProcessor.java | 14 +- .../doris/datasource/hudi/source/HudiScanNode.java | 142 +++--- .../doris/datasource/hudi/source/HudiSplit.java | 3 +- .../hudi/source/MORIncrementalRelation.java | 14 +- .../datasource/paimon/PaimonExternalCatalog.java | 88 ++-- .../datasource/paimon/source/PaimonScanNode.java | 37 +- .../apache/doris/job/disruptor/TaskDisruptor.java | 11 +- .../doris/job/executor/TimerJobSchedulerTask.java | 2 + .../apache/doris/job/scheduler/JobScheduler.java | 12 +- .../main/java/org/apache/doris/load/ExportMgr.java | 30 +- .../load/routineload/RoutineLoadTaskScheduler.java | 2 +- .../main/java/org/apache/doris/mtmv/MTMVCache.java | 5 +- .../org/apache/doris/mtmv/MTMVPartitionUtil.java | 16 +- .../glue/translator/PhysicalPlanTranslator.java | 4 +- .../rules/SimplifyComparisonPredicate.java | 10 +- .../rules/implementation/AggregateStrategies.java | 12 + .../functions/agg/AggregateFunction.java | 4 + .../expressions/functions/agg/AggregatePhase.java | 8 +- .../agg/OrthogonalBitmapExprCalculate.java | 79 ++++ .../agg/OrthogonalBitmapExprCalculateCount.java | 81 ++++ .../trees/plans/commands/ExplainCommand.java | 4 + .../plans/commands/info/ShowCreateMTMVInfo.java | 13 +- .../trees/plans/commands/insert/InsertUtils.java | 10 + .../trees/plans/logical/LogicalAggregate.java | 12 + .../trees/plans/logical/LogicalHudiScan.java | 2 +- .../org/apache/doris/persist/BatchDropInfo.java | 4 + .../java/org/apache/doris/persist/DropInfo.java | 2 +- .../java/org/apache/doris/persist/EditLog.java | 22 +- .../apache/doris/planner/SingleNodePlanner.java | 2 +- .../java/org/apache/doris/qe/ConnectContext.java | 41 +- .../main/java/org/apache/doris/qe/Coordinator.java | 21 +- .../java/org/apache/doris/qe/SessionVariable.java | 14 + .../java/org/apache/doris/qe/ShowExecutor.java | 12 +- .../doris/scheduler/disruptor/TaskDisruptor.java | 11 +- .../scheduler/manager/TransientTaskManager.java | 4 +- .../apache/doris/service/FrontendServiceImpl.java | 4 + .../arrowflight/DorisFlightSqlProducer.java | 73 +-- .../arrowflight/FlightSqlConnectProcessor.java | 152 ++++--- .../results/FlightSqlEndpointsLocation.java | 65 +++ fe/pom.xml | 6 +- gensrc/thrift/FrontendService.thrift | 1 + gensrc/thrift/PlanNodes.thrift | 4 +- .../data/account_p0/test_information_schema.out | 7 + .../data/arrow_flight_sql_p0/test_select.out | 8 + .../data/correctness_p0/test_mv_case.out | 3 + .../import/import-way/stream-load-manual.md.out | 154 +++++++ .../{test_rl_csv.csv => streamload_example.csv} | 0 .../import/import-way/streamload_example.json | 12 + .../data-operate/import/import-way/test_array.csv | 10 + .../data-operate/import/import-way/test_bitmap.csv | 10 + .../import/import-way/test_default.csv | 1 + .../import/import-way/test_enclose_and_escape.csv | 1 + .../data-operate/import/import-way/test_hll.csv | 10 + .../data-operate/import/import-way/test_map.json | 12 + .../import/import-way/test_merge_type.csv | 2 + .../data-operate/import/import-way/test_seq.csv | 1 + .../data/external_table_p0/hive/test_hive_orc.out | 48 -- .../hudi/test_hudi_incremental.out | 174 ++++++++ .../hudi/test_hudi_orc_tables.out | 15 + .../hudi/test_hudi_partition_prune.out | 357 +++++++++++++++ .../hudi/test_hudi_schema_evolution.out | 32 ++ .../external_table_p2/hudi/test_hudi_snapshot.out | Bin 348526 -> 696105 bytes .../external_table_p2/hudi/test_hudi_timestamp.out | 31 +- .../hudi/test_hudi_timetravel.out | 120 +++++ .../test_partial_update_auto_inc.out | 16 + regression-test/framework/pom.xml | 1 + .../org/apache/doris/regression/Config.groovy | 14 +- .../doris/regression/suite/SuiteContext.groovy | 2 +- .../account_p0/test_information_schema.groovy | 22 + .../suites/arrow_flight_sql_p0/test_select.groovy | 43 ++ .../suites/correctness_p0/test_mv_case.groovy | 69 ++- .../import/import-way/stream-load-manual.md.groovy | 492 +++++++++++++++++++++ .../external_table_p0/hive/test_hive_orc.groovy | 12 - .../hudi/test_hudi_catalog.groovy | 2 +- .../hudi/test_hudi_incremental.groovy | 16 +- ..._catalog.groovy => test_hudi_orc_tables.groovy} | 10 +- .../hudi/test_hudi_partition_prune.groovy | 333 ++++++++++++++ .../hudi/test_hudi_schema_evolution.groovy | 14 +- .../hudi/test_hudi_snapshot.groovy | 13 +- .../hudi/test_hudi_timestamp.groovy | 20 +- .../hudi/test_hudi_timetravel.groovy | 15 +- .../suites/job_p0/test_base_insert_job.groovy | 42 +- .../suites/manager/test_manager_interface_1.groovy | 122 ----- .../suites/manager/test_manager_interface_4.groovy | 146 ++++++ .../test_orthogonal_bitmap_expr_calculate.groovy | 55 +++ .../query_p0/aggregate/aggregate_count1.groovy | 2 +- .../query_p0/aggregate/select_distinct.groovy | 2 +- .../join_with_column_casesensetive.groovy | 2 +- .../suites/query_p0/cast/test_cast.groovy | 2 +- .../query_p0/except/test_query_except.groovy | 2 +- .../query_p0/group_concat/test_group_concat.groovy | 2 +- .../grouping_sets/test_grouping_sets1.groovy | 2 +- .../suites/query_p0/having/having.groovy | 2 +- .../query_p0/intersect/test_intersect.groovy | 2 +- .../suites/query_p0/join/test_join2.groovy | 2 +- .../suites/query_p0/join/test_left_join1.groovy | 2 +- .../join/test_nestedloop_outer_join.groovy | 2 +- .../join/test_partitioned_hash_join.groovy | 2 +- .../query_p0/lateral_view/lateral_view.groovy | 2 +- .../query_p0/limit/OffsetInSubqueryWithJoin.groovy | 2 +- .../query_p0/literal_view/lietral_test.groovy | 2 +- .../query_p0/operator/test_set_operator.groovy | 2 +- .../query_p0/operator/test_sort_operator.groovy | 2 +- .../session_variable/test_default_limit.groovy | 2 +- .../query_p0/show/test_show_create_table.groovy | 2 +- .../test_aggregate_all_functions.groovy | 2 +- .../case_function/test_case_function_null.groovy | 2 +- .../hash_functions/test_hash_function.groovy | 2 +- .../ip_functions/test_ip_functions.groovy | 2 +- .../json_function/test_query_json_insert.groovy | 2 +- .../json_functions/test_json_function.groovy | 2 +- .../sql_functions/math_functions/test_conv.groovy | 2 +- .../test_multi_string_search.groovy | 2 +- .../spatial_functions/test_gis_function.groovy | 2 +- .../string_functions/test_string_function.groovy | 2 +- .../table_function/explode_split.groovy | 2 +- .../sql_functions/test_alias_function.groovy | 2 +- .../query_p0/sql_functions/test_predicate.groovy | 2 +- .../test_width_bucket_function.groovy | 2 +- .../suites/query_p0/subquery/test_subquery2.groovy | 2 +- .../suites/query_p0/test_data_type_marks.groovy | 2 +- .../suites/query_p0/test_dict_with_null.groovy | 2 +- .../query_p0/test_orderby_nullliteral.groovy | 2 +- .../suites/query_p0/test_select_constant.groovy | 2 +- .../test_select_with_predicate_like.groovy | 2 +- .../test_select_with_predicate_prune.groovy | 2 +- .../query_p0/type_inference/test_largeint.groovy | 2 +- .../with/test_with_and_two_phase_agg.groovy | 2 +- .../test_partial_update_auto_inc.groovy | 41 ++ 183 files changed, 4353 insertions(+), 1445 deletions(-) delete mode 100644 be/test/exec/test_data/orc_scanner/orders.orc delete mode 100644 be/test/vec/exec/orc_reader_test.cpp create mode 100644 fe/be-java-extensions/hadoop-hudi-scanner/pom.xml create mode 100644 fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiColumnValue.java create mode 100644 fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java copy fe/be-java-extensions/{avro-scanner => hadoop-hudi-scanner}/src/main/resources/package.xml (100%) delete mode 100644 fe/fe-common/src/main/java/org/apache/doris/common/security/authentication/HadoopUGI.java copy regression-test/suites/query_p0/test_select_constant.groovy => fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AggregatePhase.java (80%) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/OrthogonalBitmapExprCalculate.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/OrthogonalBitmapExprCalculateCount.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/results/FlightSqlEndpointsLocation.java create mode 100644 regression-test/data/arrow_flight_sql_p0/test_select.out create mode 100644 regression-test/data/doc/data-operate/import/import-way/stream-load-manual.md.out copy regression-test/data/doc/data-operate/import/import-way/{test_rl_csv.csv => streamload_example.csv} (100%) create mode 100644 regression-test/data/doc/data-operate/import/import-way/streamload_example.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_array.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_bitmap.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_default.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_enclose_and_escape.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_hll.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_map.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_merge_type.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_seq.csv create mode 100644 regression-test/data/external_table_p2/hudi/test_hudi_orc_tables.out create mode 100644 regression-test/data/external_table_p2/hudi/test_hudi_partition_prune.out create mode 100644 regression-test/suites/arrow_flight_sql_p0/test_select.groovy create mode 100644 regression-test/suites/doc/data-operate/import/import-way/stream-load-manual.md.groovy copy regression-test/suites/external_table_p2/hudi/{test_hudi_catalog.groovy => test_hudi_orc_tables.groovy} (84%) create mode 100644 regression-test/suites/external_table_p2/hudi/test_hudi_partition_prune.groovy create mode 100644 regression-test/suites/manager/test_manager_interface_4.groovy create mode 100644 regression-test/suites/nereids_function_p0/agg_function/test_orthogonal_bitmap_expr_calculate.groovy --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org