This is an automated email from the ASF dual-hosted git repository. kxiao pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git
from a89edfe784 [improvement](stats) Catch exception properly (#22503) (#22896) new ca2c09e35d [stats](nereids) improve Anti join stats estimation #22444 new d0fbf443d4 [tpcds](nereids) add rule to eliminate empty relation #22203 new ae8810c7e8 [fix](planner)runtime filter shouldn't be pushed through window function node (#22501) new 0ad237d14b fix full auto analyze (#22650) new bad6e1154c [Chore](inverted index) refine log in DorisCompoundDirectory::FSIndexOutput (#22716) new 1c5435f1ba [Chore](binlog) Refactor TABLET_MISSING in ingest_binlog && set_tstatus (#22727) new f822d50e1a [fix](agg) fix coredump caused by push down count aggregation (#22699) new 5fe9aa9ab5 [refactor](Nereids) let topn runtime filter as PhysicalTopN's attr (#22745) new 8fdd91bca7 [fix](delete) Fix parsing error of delete where date statement (#22690) new 314755f6c5 [fix](Nereids) disable or expansion when pipeline engine is disable (#22719) new 01c5056693 [fix](cache) add sql cache conf `cache_result_max_data_size` (#22645) new a89df0c855 [feature](Nereids): disable join order when the join number > 63 (#22708) new c8dc7bad55 [fix](bitmap) fix wrong result of bitmap intersect functions (#22735) new 7548653bb7 [fix](cache) Fix enable sql cache lead to FE Full GC or OOM #22769 new b40a63f010 [improvement](multi-catalog)add scanner isolation class loader (#22247) new 0bff2dc09b Revert "[improvement](stats) Catch exception properly (#22503) (#22896)" new 1632da2ac2 [Fix](Sql)NPE when the Delete statement does not specify a where condition (#22766) new 8ab60325e2 [fix](nereids) parallel instance number is set to 1 incorrectly (#22748) new 441b7f16fa [enhance](multi-catalog) support multi name service when config hive catalog #21825 new 7a715be1ba [doc](stats) Add description for some new configure option in stats related docs (#22723) new d70bd94432 [opt](stats) No more sync unknown stats since cannot serialize (#22775) new 6054a4b3a5 [improvement](garbage sweep) garbage sweep sleep for a while to reduce io (#22762) new adceef2dfb [feature](agg) add aggregation function 'bitmap_agg' (#22768) new 224c83704b [feature](hive)append support for struct and map column type on textfile format of hive table (#22347) new eda85d7269 [Feature](multi-catalog) Truncate char or varchar columns if size is smaller than file columns or not found in the file column schema. (#22318) new 90d9b40722 [fix](multicatlog) fix read hive/iceberg catalog on cosn & fix read data via broker (#22087) new 8be6e7c56a [fix](tablet report) fix not add replicas when a backend re join the cluster after changing its ip or port (#22700) new 29e9b39ccc [enhance](ColdHeatSeparation) forbid change storage policy to another one with different storage resource (#22519) new 74a388523d [fix](compaction) remove check rowset overlapping in base compaction (#22599) new 037491f099 [enhance](S3FIleWriter) Add md5 check for small file not suitable for multi part upload (#22296) new 11b3fff40c [fix](iceberg) fix iceberg's filter expr to filter file (#22740) new da158fd3dd [new-feature](complex-type) support read nested parquet and orc file with complex type (#22793) new 4940d7c5ca [pipeline](exec) Support shared scan in jdbc and odbc scan node (#22826) new 9025c90fc8 [fix](executor)Fix query hang when query bitmap Orth intersect #22828 new 1fa5e44790 [enhancement](data type) add use_mysql_bigint_for_largeint config Tell Doris to use bigint when returning largeint type to mysql jdbc (#22835) new e7b39d5240 disable costmodelV2 (#22830) new 7688bba13b [log](tablet clone) add decommission replica log (#22799) new 679a0ad264 [fix](autoinc) ignore column property isAutoInc() for create table as select ... statement(#22827) new 6f35496268 [fix](bug) Fix page handle safe exit #22849 new c5cc953c45 [Dependency](fe)Upgrade dependency version (#22496) new e1ab0850c2 [improvement](create partition) create partition require quorum replicas succ (#22554) new ee9eeacc0d [Chore](RoutineLoad)Change max_batch_interval minimum limit from 5 to 1 (#22858) new 2e4dc01ff5 [Fix](multi-catalog) Fix decimal precision issue in regression test result. (#22819) new 5eb2931e5b [Improvement](log) add config to controll compression of fe log & fe audit log (#22865) new 166b34ad9a [fix](multi-catalog)fix jdbc loader (#22814) new ced3c5884c [Fix](multi-catalog) sync default catalog when forwarding query to master. (#22684) new d3051fcde2 [fix](bug) Fix BE thread safe start and stop #22560 new 3b131b3bd9 [fix](planner)shouldn't change the child type to assignmentCompatibleType if it's INVALID_TYPE (#22841) new f47d9fb57d [test](jdbc_mysql)update test_jdbc_query_mysql regression test result #22866 new 6ca9f63c08 [fix](executor)fix error when FixedContainer with null (#22850) new d0d6edc6c9 [fix](avro)Fix S3 TVF avro format reading failure (#22199) new e8a6e0f2fd [Improve](jni-avro)Reduce the volume of the avro-scanner-jar package (#22276) new 6cd9c26c15 [Fix](regression)Fix test_mysql_jdbc_catalog_nereids p2 test case (#22870) new 2c4682818a [Fix](Nereids) fix insert into table of random distribution for nereids (#22831) new e7df43bb8b [Fix](Nereids)add need forward for enable_nereids_dml and format some cases (#22888) new 1bb0e1f9bb [fix](regression)fix test_hdfs_tvf regression_test out file : decimalv3 -> decimal (#22852) new 78ed14afbb [fix](pg test) fix postgresql jdbc catalog test case (#22875) new ae6509c2a5 [Enhancement](merge-on-write) add correctness check for the calculation of delete bitmap (#22282) new 0a6bef4486 [Refactor](inverted index) refact tokenize function for inverted index (#22313) The 59 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/agent/task_worker_pool.cpp | 21 +- be/src/common/config.cpp | 6 + be/src/common/config.h | 7 + be/src/common/daemon.cpp | 7 +- be/src/exec/exec_node.cpp | 2 + be/src/exec/text_converter.cpp | 288 ++++++----------- be/src/exec/text_converter.h | 18 +- be/src/exprs/create_predicate_function.h | 4 +- be/src/exprs/hybrid_set.h | 5 +- be/src/io/fs/s3_file_write_bufferpool.h | 2 +- be/src/io/fs/s3_file_writer.cpp | 30 +- be/src/io/fs/s3_file_writer.h | 46 +-- be/src/olap/base_compaction.cpp | 14 - be/src/olap/base_compaction.h | 4 - be/src/olap/olap_server.cpp | 61 ++-- .../inverted_index_compound_directory.cpp | 8 +- be/src/olap/rowset/segment_v2/page_handle.h | 8 +- be/src/olap/rowset/segment_v2/segment_writer.cpp | 5 + be/src/olap/storage_engine.cpp | 17 +- be/src/olap/storage_engine.h | 2 +- be/src/olap/tablet.cpp | 58 ++++ be/src/olap/tablet.h | 6 + be/src/olap/tablet_manager.cpp | 100 ++++-- be/src/olap/tablet_manager.h | 14 +- be/src/olap/tablet_meta.h | 4 + be/src/pipeline/pipeline_fragment_context.cpp | 1 + be/src/runtime/exec_env.h | 2 +- be/src/runtime/memory/mem_tracker_limiter.cpp | 4 +- be/src/service/backend_service.cpp | 82 +++-- be/src/util/bitmap_intersect.h | 3 + be/src/util/jni-util.cpp | 47 +++ be/src/util/jni-util.h | 9 +- .../aggregate_function_bitmap_agg.cpp | 56 ++++ .../aggregate_function_bitmap_agg.h | 189 +++++++++++ .../aggregate_function_orthogonal_bitmap.h | 3 + .../aggregate_function_simple_factory.cpp | 2 + be/src/vec/exec/format/avro/avro_jni_reader.cpp | 31 +- be/src/vec/exec/format/avro/avro_jni_reader.h | 4 +- be/src/vec/exec/format/csv/csv_reader.cpp | 16 +- be/src/vec/exec/format/csv/csv_reader.h | 6 +- be/src/vec/exec/format/orc/vorc_reader.cpp | 4 +- .../exec/format/parquet/vparquet_column_reader.cpp | 34 +- .../exec/format/parquet/vparquet_column_reader.h | 4 + be/src/vec/exec/format/table/table_format_reader.h | 5 + be/src/vec/exec/jni_connector.cpp | 8 +- be/src/vec/exec/scan/vfile_scanner.cpp | 77 ++++- be/src/vec/exec/scan/vfile_scanner.h | 7 + be/src/vec/exec/vjdbc_connector.cpp | 3 +- be/src/vec/functions/function_tokenize.cpp | 81 +++-- be/src/vec/functions/function_tokenize.h | 7 +- be/src/vec/functions/in.h | 22 +- be/test/olap/delete_handler_test.cpp | 11 +- be/test/olap/delta_writer_test.cpp | 18 +- .../olap/engine_storage_migration_task_test.cpp | 5 +- be/test/olap/memtable_memory_limiter_test.cpp | 191 ++++++++++++ be/test/olap/remote_rowset_gc_test.cpp | 3 +- be/test/olap/segcompaction_test.cpp | 3 +- be/test/olap/tablet_cooldown_test.cpp | 5 +- be/test/olap/tablet_mgr_test.cpp | 11 +- bin/start_be.sh | 2 +- build.sh | 6 +- docs/en/docs/admin-manual/config/fe-config.md | 26 +- docs/en/docs/advanced/cache/partition-cache.md | 12 +- docs/en/docs/advanced/variables.md | 6 + docs/en/docs/lakehouse/multi-catalog/hive.md | 6 + docs/en/docs/query-acceleration/nereids.md | 5 +- .../aggregate-functions/bitmap_agg.md | 89 ++++++ .../Load/CREATE-ROUTINE-LOAD.md | 2 +- docs/zh-CN/docs/admin-manual/config/fe-config.md | 24 +- docs/zh-CN/docs/advanced/cache/partition-cache.md | 4 +- docs/zh-CN/docs/advanced/variables.md | 6 + docs/zh-CN/docs/lakehouse/multi-catalog/hive.md | 6 + docs/zh-CN/docs/query-acceleration/nereids.md | 5 +- docs/zh-CN/docs/query-acceleration/statistics.md | 37 ++- .../aggregate-functions/bitmap_agg.md | 88 ++++++ .../Load/CREATE-ROUTINE-LOAD.md | 2 +- fe/be-java-extensions/avro-scanner/pom.xml | 3 + .../java/org/apache/doris/avro/AvroJNIScanner.java | 4 +- .../java/org/apache/doris/avro/AvroProperties.java | 2 - .../java/org/apache/doris/avro/HDFSFileReader.java | 9 +- .../java/org/apache/doris/avro/S3FileReader.java | 20 +- .../main/java/org/apache/doris/avro/S3Utils.java | 109 +++++++ .../common/classloader/JniScannerClassLoader.java | 33 +- .../doris/common/classloader/ScannerLoader.java | 140 +++++++++ .../classloader/ThreadClassLoaderContext.java | 20 +- .../org/apache/doris/catalog/PrimitiveType.java | 7 + .../java/org/apache/doris/catalog/ScalarType.java | 4 +- .../main/java/org/apache/doris/common/Config.java | 31 +- fe/fe-core/pom.xml | 2 +- .../catalog/metastore/AWSGlueDecoratorBase.java | 126 ++++++++ .../java/org/apache/doris/alter/RollupJobV2.java | 11 +- .../apache/doris/alter/SchemaChangeHandler.java | 17 +- .../org/apache/doris/alter/SchemaChangeJobV2.java | 11 +- .../doris/analysis/AlterRoutineLoadStmt.java | 2 +- .../java/org/apache/doris/analysis/Analyzer.java | 2 +- .../org/apache/doris/analysis/BinaryPredicate.java | 10 +- .../doris/analysis/CreateRoutineLoadStmt.java | 4 +- .../java/org/apache/doris/analysis/DeleteStmt.java | 9 +- .../apache/doris/analysis/FunctionCallExpr.java | 26 +- .../org/apache/doris/analysis/StringLiteral.java | 4 +- .../java/org/apache/doris/backup/RestoreJob.java | 10 +- .../apache/doris/catalog/AggregateFunction.java | 2 +- .../apache/doris/catalog/CatalogRecycleBin.java | 5 +- .../java/org/apache/doris/catalog/FunctionSet.java | 13 + .../doris/catalog/HiveMetaStoreClientHelper.java | 10 +- .../java/org/apache/doris/catalog/OlapTable.java | 4 +- .../org/apache/doris/clone/TabletScheduler.java | 4 + .../java/org/apache/doris/common/FeConstants.java | 1 + .../java/org/apache/doris/common/Log4jConfig.java | 11 +- .../java/org/apache/doris/common/util/DbUtil.java | 16 +- .../apache/doris/common/util/PropertyAnalyzer.java | 26 +- .../java/org/apache/doris/common/util/S3Util.java | 7 +- .../doris/datasource/HMSExternalCatalog.java | 34 +- .../apache/doris/datasource/InternalCatalog.java | 57 ++-- .../property/constants/S3Properties.java | 2 - .../doris/external/iceberg/util/IcebergUtils.java | 27 +- .../org/apache/doris/fs/FileSystemFactory.java | 3 +- .../org/apache/doris/master/ReportHandler.java | 25 +- .../apache/doris/nereids/cost/CostCalculator.java | 10 +- .../glue/translator/PhysicalPlanTranslator.java | 4 +- .../doris/nereids/jobs/executor/Optimizer.java | 4 +- .../doris/nereids/jobs/executor/Rewriter.java | 8 +- .../doris/nereids/processor/post/TopNScanOpt.java | 34 +- .../org/apache/doris/nereids/rules/RuleType.java | 6 + .../nereids/rules/analysis/CheckAfterRewrite.java | 2 +- .../nereids/rules/exploration/OrExpansion.java | 4 +- .../rules/rewrite/EliminateEmptyRelation.java | 193 ++++++++++++ .../apache/doris/nereids/stats/JoinEstimation.java | 7 +- .../physical/PhysicalDeferMaterializeTopN.java | 14 +- .../plans/physical/PhysicalOlapTableSink.java | 37 ++- .../nereids/trees/plans/physical/PhysicalTopN.java | 52 ++-- .../org/apache/doris/planner/AggregationNode.java | 5 - .../apache/doris/planner/AssertNumRowsNode.java | 5 + .../apache/doris/planner/DistributedPlanner.java | 8 +- .../org/apache/doris/planner/EmptySetNode.java | 5 + .../java/org/apache/doris/planner/PlanNode.java | 8 +- .../java/org/apache/doris/planner/RepeatNode.java | 5 - .../org/apache/doris/planner/RuntimeFilter.java | 41 ++- .../java/org/apache/doris/planner/SelectNode.java | 5 - .../apache/doris/planner/SingleNodePlanner.java | 3 +- .../java/org/apache/doris/planner/SortNode.java | 5 - .../doris/planner/external/FileQueryScanNode.java | 16 +- .../doris/planner/external/HiveScanNode.java | 68 +++- .../planner/external/iceberg/IcebergScanNode.java | 27 +- .../doris/planner/external/jdbc/JdbcScanNode.java | 4 +- .../doris/planner/external/odbc/OdbcScanNode.java | 7 + .../java/org/apache/doris/qe/ConnectProcessor.java | 3 + .../java/org/apache/doris/qe/MasterOpExecutor.java | 1 + .../java/org/apache/doris/qe/SessionVariable.java | 52 +++- .../main/java/org/apache/doris/qe/cache/Cache.java | 13 +- .../org/apache/doris/qe/cache/PartitionCache.java | 3 + .../org/apache/doris/qe/cache/PartitionRange.java | 4 +- .../org/apache/doris/qe/cache/RowBatchBuilder.java | 12 + .../java/org/apache/doris/qe/cache/SqlCache.java | 3 + .../apache/doris/statistics/AnalysisManager.java | 27 +- .../doris/statistics/StatisticsAutoAnalyzer.java | 76 +---- .../apache/doris/statistics/StatisticsCache.java | 3 + .../doris/tablefunction/S3TableValuedFunction.java | 11 +- .../java/org/apache/doris/alter/AlterTest.java | 11 +- .../datasource/property/PropertyConverterTest.java | 4 +- .../nereids/postprocess/TopNRuntimeFilterTest.java | 9 +- .../doris/nereids/sqltest/JoinOrderJobTest.java | 19 ++ .../org/apache/doris/qe/PartitionCacheTest.java | 10 +- fe/pom.xml | 20 +- gensrc/thrift/FrontendService.thrift | 1 + gensrc/thrift/PaloInternalService.thrift | 2 + gensrc/thrift/PlanNodes.thrift | 3 +- gensrc/thrift/Status.thrift | 2 + regression-test/data/correctness/test_in_null.out | 29 ++ .../data/correctness_p0/test_select_constant.out | 3 + .../unique/test_unique_table_auto_inc.out | 155 +++++++++ .../data/empty_relation/eliminate_empty.out | 72 +++++ .../hive/test_truncate_char_or_varchar_columns.out | 85 +++++ .../external_table_p0/hive/hive_tpch_sf1_orc.out | 8 +- .../hive/hive_tpch_sf1_parquet.out | 8 +- .../jdbc/test_jdbc_query_mysql.out | 10 +- .../jdbc/test_pg_jdbc_catalog.out | 2 +- .../data/external_table_p0/tvf/test_hdfs_tvf.out | 2 +- .../hive/test_hive_text_complex_type.out | 15 + .../data/inverted_index_p0/test_array_index.out | 1 + .../data/inverted_index_p0/test_tokenize.out | 6 + .../data/nereids_p0/insert_into_table/random.out | 137 ++++++++ .../nereids_tpcds_shape_sf100_p0/shape/query11.out | 18 +- .../nereids_tpcds_shape_sf100_p0/shape/query16.out | 38 +-- .../nereids_tpcds_shape_sf100_p0/shape/query23.out | 21 +- .../nereids_tpcds_shape_sf100_p0/shape/query4.out | 27 +- .../nereids_tpcds_shape_sf100_p0/shape/query69.out | 5 +- .../nereids_tpcds_shape_sf100_p0/shape/query74.out | 18 +- .../nereids_tpcds_shape_sf100_p0/shape/query94.out | 7 +- .../aggregate/bitmap_agg.out} | 8 +- .../test_aggregate_push_down_agg.out} | 7 +- .../bitmap_functions/test_bitmap_function.out | 39 +++ .../suites/correctness/test_in_null.groovy | 56 ++++ .../test_runtimefilter_with_window.groovy | 90 ++++++ .../correctness_p0/test_select_constant.groovy | 3 + .../unique/test_unique_table_auto_inc.groovy | 346 +++++++++++++++++++++ .../suites/empty_relation/eliminate_empty.groovy | 130 ++++++++ regression-test/suites/empty_relation/load.groovy | 68 ++++ .../test_truncate_char_or_varchar_columns.groovy | 78 +++++ .../jdbc/test_mysql_jdbc_catalog_nereids.groovy | 8 +- .../hive/test_hive_text_complex_type.groovy | 46 +++ .../suites/inverted_index_p0/test_tokenize.groovy | 27 +- .../delete/delete_mow_partial_update.groovy | 19 +- .../nereids_p0/insert_into_table/aggregate.groovy | 2 +- .../insert_into_table/complex_insert.groovy | 2 +- .../nereids_p0/insert_into_table/ddl/random_t.sql | 21 ++ .../nereids_p0/insert_into_table/duplicate.groovy | 2 +- .../nereids_p0/insert_into_table/load.groovy | 2 +- .../insert_into_table/no_partition.groovy | 2 +- .../{unsupport_type.groovy => random.groovy} | 26 +- .../insert_into_table/type_cast_aggregate.groovy | 2 +- .../insert_into_table/type_cast_duplicate.groovy | 2 +- .../insert_into_table/type_cast_unique.groovy | 2 +- .../nereids_p0/insert_into_table/unique.groovy | 2 +- .../insert_into_table/unsupport_type.groovy | 2 +- regression-test/suites/nereids_p0/load.groovy | 3 +- .../suites/nereids_p0/union/or_expansion.groovy | 1 + .../suites/query_p0/aggregate/bitmap_agg.groovy | 72 +++++ .../join/test_nestedloop_semi_anti_join.groovy | 2 - .../test_aggregate_push_down_agg.groovy | 44 +++ .../bitmap_functions/test_bitmap_function.groovy | 142 +++++++++ 221 files changed, 4651 insertions(+), 1028 deletions(-) create mode 100644 be/src/vec/aggregate_functions/aggregate_function_bitmap_agg.cpp create mode 100644 be/src/vec/aggregate_functions/aggregate_function_bitmap_agg.h create mode 100644 be/test/olap/memtable_memory_limiter_test.cpp create mode 100644 docs/en/docs/sql-manual/sql-functions/aggregate-functions/bitmap_agg.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/bitmap_agg.md create mode 100644 fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/S3Utils.java copy regression-test/suites/nereids_p0/insert_into_table/unsupport_type.groovy => fe/be-java-extensions/java-common/src/main/java/org/apache/doris/common/classloader/JniScannerClassLoader.java (59%) create mode 100644 fe/be-java-extensions/java-common/src/main/java/org/apache/doris/common/classloader/ScannerLoader.java copy regression-test/suites/correctness_p0/test_select_constant.groovy => fe/be-java-extensions/java-common/src/main/java/org/apache/doris/common/classloader/ThreadClassLoaderContext.java (60%) copy regression-test/suites/correctness_p0/test_select_constant.groovy => fe/fe-core/src/main/java/org/apache/doris/common/util/DbUtil.java (64%) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateEmptyRelation.java create mode 100644 regression-test/data/correctness/test_in_null.out create mode 100644 regression-test/data/data_model_p0/unique/test_unique_table_auto_inc.out create mode 100644 regression-test/data/empty_relation/eliminate_empty.out create mode 100644 regression-test/data/external_table_emr_p2/hive/test_truncate_char_or_varchar_columns.out create mode 100644 regression-test/data/external_table_p2/hive/test_hive_text_complex_type.out create mode 100644 regression-test/data/nereids_p0/insert_into_table/random.out copy regression-test/data/{correctness_p0/test_select_constant.out => query_p0/aggregate/bitmap_agg.out} (52%) copy regression-test/data/{correctness_p0/test_select_constant.out => query_p0/sql_functions/aggregate_functions/test_aggregate_push_down_agg.out} (51%) create mode 100644 regression-test/suites/correctness/test_in_null.groovy create mode 100644 regression-test/suites/correctness_p0/test_runtimefilter_with_window.groovy create mode 100644 regression-test/suites/data_model_p0/unique/test_unique_table_auto_inc.groovy create mode 100644 regression-test/suites/empty_relation/eliminate_empty.groovy create mode 100644 regression-test/suites/empty_relation/load.groovy create mode 100644 regression-test/suites/external_table_emr_p2/hive/test_truncate_char_or_varchar_columns.groovy create mode 100644 regression-test/suites/external_table_p2/hive/test_hive_text_complex_type.groovy create mode 100644 regression-test/suites/nereids_p0/insert_into_table/ddl/random_t.sql copy regression-test/suites/nereids_p0/insert_into_table/{unsupport_type.groovy => random.groovy} (53%) create mode 100644 regression-test/suites/query_p0/aggregate/bitmap_agg.groovy create mode 100644 regression-test/suites/query_p0/sql_functions/aggregate_functions/test_aggregate_push_down_agg.groovy --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org