This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
from fb5a3e118a [feature-wip](dlf) prepare to support aliyun dlf (#13969)
add 27549564a7 [feature](table-valued-function) Support S3 tvf (#13959)
add c7b2b90504 [fix](memtracker) Fix DCHECK
!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(),
tracker)
add 7ffe88b579 [feature-array](array-type) Add array function
array_popback (#13641)
add e8d2fb6778 [feature](function)add search functions:
multi_search_all_positions & multi_match_any (#13763)
add 32fea672b0 [chore](gutil) remove some gutil macros and solve some
macro conflict with brpc (#13954)
add 3c8524b9d8 [security](fe jar) upgrade commons-codec:commons-codec to
1.13 #13951
add 7254999f02 [typo](docs) fix docs，delete redundant words #13849
add bb9182d602 [fix](repeat)remove unmaterialized expr from repeat node
(#13953)
add 0031304015 [typo](docs)fix config doc #14010
add 22b4c6af20 [feature](Nereids) support statement having aggregate
function in order by list (#13976)
add f2978fb6ff [feat](Nereids) add graph simplifier (#14007)
add 4ea1b39cb2 [enhancement](Nereids) remove unnecessary decimal cast
(#13745)
add 1c2532b9dc [Bug](udf) Make UDF's type always nullable (#14002)
add 241801ca17 [typo](doc) fix get_start doc (#14001)
add d1cbaa1de8 [fix](load) fix a bug that reduce memory work on hard limit
might be triggered twice (#13967)
add 17a4746a08 [enhancement](Nereids) support otherJoinConjuncts in
cascades join reorder (#13681)
add 95591ce49a [refactor](cv)wait on condition variable more gently
(#12620)
add 6ed443c7e8 [enhancement](profile) add instanceNum, tableIds to
profile. (#13985)
add 34f43ac781 [bug](like function)fix like '' (empty string) get wrong
result with all rows #14035
add e1654bc6ef [Enhancement](function) add to_bitmap() function with int
type (#13973)
new bcde11d82d [fix](priv) fix meta replay bug when upgrading from 1.1.x
to 1.2.x (#14046)
new 73a004c8dc [Enhancement](Dictionary-codec) update dict once on same
segment (#13936)
new df2f528781 [feature](Nereids) support query that group by use alias
generated in aggregate output (#14030)
new 944d572053 [thirdpart](lib) Add lock free queue of concurrentqueue
(#14045)
new b4857cb608 [feature-wip](multi-catalog) fix page index filter bug
(#14015)
new 7e9890ea6d [fix](Nereids) Use simple cost to calculate benefit and
avoid unuseless calculation (#14056)
new 3da8dfb658 [feature](multi-catalog) Support data on s3-compatible oss
and support aliyun DLF (#13994)
new 836021ac2e [Bug](Bitmap) fix sub_bitmap calculate wrong result to
return null (#13978)
new c3172a559b [fix](build) fix compile fail on Segment::open (#14058)
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 1 +
be/src/agent/agent_server.cpp | 6 +-
be/src/agent/agent_server.h | 2 -
be/src/agent/task_worker_pool.cpp | 85 ++---
be/src/common/signal_handler.h | 4 +-
be/src/exec/exec_node.cpp | 10 +-
be/src/exprs/bitmap_function.cpp | 30 ++
be/src/exprs/bitmap_function.h | 2 +
be/src/gutil/integral_types.h | 19 -
be/src/gutil/macros.h | 84 +----
be/src/gutil/once.cc | 2 +-
be/src/gutil/port.h | 58 +---
be/src/gutil/strings/numbers.cc | 4 +-
be/src/gutil/strings/numbers.h | 2 +-
be/src/gutil/strtoint.cc | 21 +-
be/src/olap/in_list_predicate.h | 25 +-
be/src/olap/rowset/beta_rowset.cpp | 9 +-
be/src/olap/rowset/beta_rowset_writer.cpp | 4 +-
be/src/olap/rowset/segment_v2/segment.cpp | 11 +-
be/src/olap/rowset/segment_v2/segment.h | 7 +-
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 2 +
be/src/runtime/cache/result_node.cpp | 2 +-
be/src/runtime/fragment_mgr.cpp | 10 +-
be/src/runtime/load_channel_mgr.h | 8 +-
be/src/runtime/memory/thread_mem_tracker_mgr.h | 15 +-
be/src/runtime/thread_context.cpp | 7 +-
be/src/runtime/thread_context.h | 4 +-
be/src/service/brpc.h | 4 -
be/src/service/internal_service.cpp | 74 ++++
be/src/service/internal_service.h | 5 +
be/src/util/bitmap_value.h | 89 +++--
be/src/util/blocking_priority_queue.hpp | 135 ++++----
be/src/util/blocking_queue.hpp | 58 ++--
be/src/util/threadpool.cpp | 19 +-
be/src/vec/CMakeLists.txt | 7 +-
be/src/vec/columns/column.h | 6 +
be/src/vec/columns/column_dictionary.h | 9 +
.../vdata_gen_function_inf.h} | 6 +-
.../vnumbers_tvf.cpp} | 13 +-
.../vnumbers_tvf.h} | 8 +-
be/src/vec/exec/format/csv/csv_reader.cpp | 148 +++++++-
be/src/vec/exec/format/csv/csv_reader.h | 19 +
be/src/vec/exec/format/generic_reader.h | 5 +
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 2 +-
be/src/vec/exec/scan/new_olap_scan_node.cpp | 19 +-
be/src/vec/exec/scan/scanner_context.cpp | 15 +-
...nction_scannode.cpp => vdata_gen_scan_node.cpp} | 35 +-
...d_function_scannode.h => vdata_gen_scan_node.h} | 11 +-
.../vec/functions/array/function_array_popback.cpp | 84 +++++
.../functions/array/function_array_register.cpp | 2 +
be/src/vec/functions/array/function_array_slice.h | 47 +--
.../vec/functions/array/function_array_utils.cpp | 44 +++
be/src/vec/functions/array/function_array_utils.h | 4 +
.../vec/functions/function_always_not_nullable.h | 84 +++--
be/src/vec/functions/function_bitmap.cpp | 235 ++++++++-----
.../functions/functions_multi_string_position.cpp | 236 +++++++++++++
.../functions/functions_multi_string_search.cpp | 283 +++++++++++++++
be/src/vec/functions/hll_hash.cpp | 59 ++--
be/src/vec/functions/like.cpp | 7 +-
be/src/vec/functions/regexps.h | 256 ++++++++++++++
be/src/vec/functions/simple_function_factory.h | 4 +
be/src/vec/runtime/vdata_stream_recvr.cpp | 2 +
be/test/io/cache/remote_file_cache_test.cpp | 2 +-
be/test/olap/rowset/segment_v2/segment_test.cpp | 4 +-
be/test/tools/benchmark_tool.cpp | 2 +-
be/test/util/easy_json-test.cpp | 18 -
be/test/vec/function/function_like_test.cpp | 6 +
build.sh | 17 +-
dist/licenses/LICENSE-concurrentqueue.txt | 21 ++
docs/en/docs/admin-manual/config/be-config.md | 4 +-
.../docs/ecosystem/external-table/jdbc-of-doris.md | 2 +-
.../docs/ecosystem/external-table/multi-catalog.md | 68 +++-
.../sql-functions/array-functions/array_popback.md | 57 +++
.../string-functions/search/multi_match_any.md | 54 +++
.../search/multi_search_all_positions.md | 54 +++
.../Create/CREATE-TABLE-AS-SELECT.md | 6 +
docs/zh-CN/docs/admin-manual/config/be-config.md | 4 +-
.../import/import-scenes/load-atomicity.md | 2 +-
.../docs/ecosystem/external-table/jdbc-of-doris.md | 2 +-
.../docs/ecosystem/external-table/multi-catalog.md | 71 +++-
docs/zh-CN/docs/get-starting/get-starting.md | 2 +-
.../sql-functions/array-functions/array_popback.md | 56 +++
.../string-functions/search/multi_match_any.md | 54 +++
.../search/multi_search_all_positions.md | 54 +++
.../Create/CREATE-TABLE-AS-SELECT.md | 6 +
.../org/apache/doris/analysis/CreateTableStmt.java | 2 +-
.../org/apache/doris/analysis/GroupingInfo.java | 22 ++
.../doris/analysis/TableValuedFunctionRef.java | 14 +-
.../org/apache/doris/catalog/FunctionGenTable.java | 11 +-
.../doris/catalog/HiveMetaStoreClientHelper.java | 92 +++--
.../org/apache/doris/catalog/ScalarFunction.java | 1 +
.../doris/catalog/external/HMSExternalTable.java | 46 +--
.../java/org/apache/doris/common/FeConstants.java | 10 +
.../apache/doris/common/util/ProfileManager.java | 5 +
.../java/org/apache/doris/common/util/S3URI.java | 4 +-
.../org/apache/doris/datasource/CatalogMgr.java | 2 +-
.../apache/doris/datasource/EsExternalCatalog.java | 31 +-
.../apache/doris/datasource/ExternalCatalog.java | 48 ++-
.../doris/datasource/HMSExternalCatalog.java | 83 ++---
.../apache/doris/datasource/InternalCatalog.java | 35 +-
.../datasource/PooledHiveMetaStoreClient.java | 146 ++++++++
.../apache/doris/mysql/privilege/PrivBitSet.java | 2 +-
.../doris/mysql/privilege/UserPrivTable.java | 2 +-
.../doris/nereids/jobs/batch/AnalyzeRulesJob.java | 11 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 19 +-
.../org/apache/doris/nereids/rules/RuleType.java | 9 +-
.../doris/nereids/rules/analysis/BindFunction.java | 19 +-
.../nereids/rules/analysis/BindSlotReference.java | 67 +++-
...eHaving.java => ResolveAggregateFunctions.java} | 87 ++++-
.../rules/exploration/join/InnerJoinLAsscom.java | 49 +--
.../exploration/join/InnerJoinLAsscomProject.java | 63 ++--
.../rules/exploration/join/JoinLAsscomHelper.java | 151 --------
.../rules/exploration/join/JoinReorderContext.java | 1 +
.../rules/exploration/join/OuterJoinLAsscom.java | 31 +-
.../exploration/join/OuterJoinLAsscomProject.java | 60 +++-
.../rules/exploration/join/ThreeJoinHelper.java | 76 ----
.../expression/rewrite/rules/SimplifyCastRule.java | 20 +-
.../rules/joinreorder/HyperGraphJoinReorder.java | 1 -
.../HyperGraphJoinReorderGroupPlan.java | 1 -
.../nereids/rules/joinreorder/hypergraph/Edge.java | 69 +++-
.../joinreorder/hypergraph/GraphSimplifier.java | 382 +++++++++++++++++++++
.../rules/joinreorder/hypergraph/HyperGraph.java | 84 ++++-
.../nereids/rules/joinreorder/hypergraph/Node.java | 19 +
.../rules/joinreorder/hypergraph/Receiver.java | 6 +-
.../nereids/trees/expressions/literal/Literal.java | 2 +-
.../org/apache/doris/nereids/util/JoinUtils.java | 44 +--
...dFunctionScanNode.java => DataGenScanNode.java} | 26 +-
.../apache/doris/planner/DistributedPlanner.java | 2 +-
.../apache/doris/planner/SingleNodePlanner.java | 3 +-
.../planner/external/ExternalFileScanNode.java | 63 ++--
...anProviderIf.java => HMSTableScanProvider.java} | 12 +-
.../doris/planner/external/HiveScanProvider.java | 190 +++-------
.../doris/planner/external/QueryScanProvider.java | 186 ++++++++++
.../doris/planner/external/TVFScanProvider.java | 142 ++++++++
.../main/java/org/apache/doris/qe/Coordinator.java | 13 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 6 +
.../org/apache/doris/rpc/BackendServiceClient.java | 5 +
.../org/apache/doris/rpc/BackendServiceProxy.java | 12 +
.../tablefunction/DataGenTableValuedFunction.java | 34 +-
.../ExternalFileTableValuedFunction.java | 233 +++++++++++++
.../tablefunction/NumbersTableValuedFunction.java | 29 +-
.../doris/tablefunction/S3TableValuedFunction.java | 111 ++++++
...FunctionInf.java => TableValuedFunctionIf.java} | 33 +-
.../doris/mysql/privilege/PrivEntryTest.java | 15 +
.../datasets/clickbench/AnalyzeClickBenchTest.java | 237 +++++++++++++
.../datasets/clickbench/ClickBenchTestBase.java} | 15 +-
.../datasets/clickbench/ClickBenchUtils.java | 181 ++++++++++
...est.java => ResolveAggregateFunctionsTest.java} | 236 ++++++++++---
.../join/InnerJoinLAsscomProjectTest.java | 237 ++++++++++++-
.../exploration/join/InnerJoinLAsscomTest.java | 121 +++++--
.../join/OuterJoinLAsscomProjectTest.java | 44 +++
.../exploration/join/OuterJoinLAsscomTest.java | 97 +++++-
.../expression/rewrite/ExpressionRewriteTest.java | 18 +
.../HyperGraphJoinReorderGroupPlanTest.java | 1 +
.../joinreorder/HyperGraphJoinReorderTest.java | 1 +
.../hypergraph/GraphSimplifierTest.java | 71 ++++
.../joinreorder/hypergraph/HyperGraphTest.java | 59 ++--
.../doris/nereids/util/HyperGraphBuilder.java | 101 ++++++
.../doris/nereids/util/LogicalPlanBuilder.java | 10 +-
.../org/apache/doris/nereids/util/PlanChecker.java | 8 +
fe/pom.xml | 2 +-
fs_brokers/apache_hdfs_broker/pom.xml | 2 +-
gensrc/proto/internal_service.proto | 12 +
gensrc/script/doris_builtins_functions.py | 36 +-
gensrc/thrift/PlanNodes.thrift | 14 +-
.../data/correctness_p0/test_subquery_grouping.out | 5 +
.../data/javaudf_p0/test_javaudf_null.out | 26 ++
.../test_string_function_like_pushdown.out | 26 ++
.../array_functions/test_array_functions.out | 36 ++
.../bitmap_functions/test_bitmap_function.out | 12 +
.../test_multi_string_position.out | 25 ++
.../search_functions/test_multi_string_search.out | 45 +++
.../string_functions/test_string_function_like.out | 24 ++
.../main/java/org/apache/doris/udf/NullTest.java | 12 +-
.../correctness_p0/test_subquery_grouping.groovy | 94 +++++
.../suites/javaudf_p0/test_javaudf_null.groovy | 70 ++++
.../test_string_function_like_pushdown.groovy | 5 +
.../array_functions/test_array_functions.groovy | 4 +
.../bitmap_functions/test_bitmap_function.groovy | 7 +-
.../test_multi_string_position.groovy | 31 ++
.../test_multi_string_search.groovy | 41 +++
.../test_string_function_like.groovy | 6 +-
thirdparty/build-thirdparty.sh | 8 +
thirdparty/vars.sh | 7 +
184 files changed, 6252 insertions(+), 1711 deletions(-)
rename be/src/vec/exec/{tablefunction/vtable_valued_function_inf.h =>
data_gen_functions/vdata_gen_function_inf.h} (90%)
rename be/src/vec/exec/{tablefunction/vnumbers_tbf.cpp =>
data_gen_functions/vnumbers_tvf.cpp} (87%)
rename be/src/vec/exec/{tablefunction/vnumbers_tbf.h =>
data_gen_functions/vnumbers_tvf.h} (87%)
rename be/src/vec/exec/{vtable_valued_function_scannode.cpp =>
vdata_gen_scan_node.cpp} (69%)
rename be/src/vec/exec/{vtable_valued_function_scannode.h =>
vdata_gen_scan_node.h} (84%)
create mode 100644 be/src/vec/functions/array/function_array_popback.cpp
create mode 100644 be/src/vec/functions/functions_multi_string_position.cpp
create mode 100644 be/src/vec/functions/functions_multi_string_search.cpp
create mode 100644 be/src/vec/functions/regexps.h
create mode 100644 dist/licenses/LICENSE-concurrentqueue.txt
create mode 100644
docs/en/docs/sql-manual/sql-functions/array-functions/array_popback.md
create mode 100644
docs/en/docs/sql-manual/sql-functions/string-functions/search/multi_match_any.md
create mode 100644
docs/en/docs/sql-manual/sql-functions/string-functions/search/multi_search_all_positions.md
create mode 100644
docs/zh-CN/docs/sql-manual/sql-functions/array-functions/array_popback.md
create mode 100644
docs/zh-CN/docs/sql-manual/sql-functions/string-functions/search/multi_match_any.md
create mode 100644
docs/zh-CN/docs/sql-manual/sql-functions/string-functions/search/multi_search_all_positions.md
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/PooledHiveMetaStoreClient.java
rename
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/{ResolveHaving.java
=> ResolveAggregateFunctions.java} (63%)
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinLAsscomHelper.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/ThreeJoinHelper.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifier.java
rename
fe/fe-core/src/main/java/org/apache/doris/planner/{TableValuedFunctionScanNode.java
=> DataGenScanNode.java} (78%)
rename
fe/fe-core/src/main/java/org/apache/doris/planner/external/{HMSTableScanProviderIf.java
=> HMSTableScanProvider.java} (75%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/external/QueryScanProvider.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/external/TVFScanProvider.java
rename be/src/service/brpc_conflict.h =>
fe/fe-core/src/main/java/org/apache/doris/tablefunction/DataGenTableValuedFunction.java
(56%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/tablefunction/S3TableValuedFunction.java
rename
fe/fe-core/src/main/java/org/apache/doris/tablefunction/{TableValuedFunctionInf.java
=> TableValuedFunctionIf.java} (52%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java
copy fe/fe-core/src/{main/java/org/apache/doris/catalog/FunctionGenTable.java
=>
test/java/org/apache/doris/nereids/datasets/clickbench/ClickBenchTestBase.java}
(67%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/ClickBenchUtils.java
rename
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/{HavingClauseTest.java
=> ResolveAggregateFunctionsTest.java} (60%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifierTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/util/HyperGraphBuilder.java
create mode 100644
regression-test/data/correctness_p0/test_subquery_grouping.out
create mode 100644 regression-test/data/javaudf_p0/test_javaudf_null.out
create mode 100644
regression-test/data/query_p0/sql_functions/search_functions/test_multi_string_position.out
create mode 100644
regression-test/data/query_p0/sql_functions/search_functions/test_multi_string_search.out
copy fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionGenTable.java
=>
regression-test/java-udf-src/src/main/java/org/apache/doris/udf/NullTest.java
(76%)
create mode 100644
regression-test/suites/correctness_p0/test_subquery_grouping.groovy
create mode 100644 regression-test/suites/javaudf_p0/test_javaudf_null.groovy
create mode 100644
regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_position.groovy
create mode 100644
regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_search.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
