This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a change to branch high-priority-column
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 4dbd91fe989 High priority queue and map. (#31509)
discard 44dab1abe46 Collect high priority columns. (#31235)
add 059468973f6 [fix](cloud) disable pipelinex temporarily (#31649)
add 5e30a00b2a5 [Fix](nereids) fix leading with left join without leftside
condition (#31387)
add ee9ae6ef4fb [fix](merge-cloud) fix meta server test conf name (#31658)
add 8a1bc8b82f9 [fix](Nereids) check slot from children may produce wrong
format err (#31638)
add 172007f23b7 [opt](nereids) Composite predicate supports range
predicate when rewritting by materialzied view (#31538)
add ec295c779d2 Revert "[test](regression) add regression test for schange
change of complex …" (#31660)
add 5515a4aac06 [opt](plsql) Fix plsql exception and doris exception
compatibility (#31647)
add 1eb6abf79bb [feat](nereids) support nullSafeEqual estimation (#31616)
add 1d8b5c442cb [feature](merge-cloud) Implement cloud compaction (#31215)
add db9713855bf [fix](profile)Fix load profile print by default (#31677)
add 4a6d9887b78 move 3 cases to p2, reducing regression pipeline runtime
(#31663)
add 5e6b7716c81 [ci](jdk17) adjust fe.conf (#31683)
add 72cca294210 [Update](cloud) add inverted index tmp dir support (#31484)
add a6c3ec64fa1 [fix](cloud) disable pipelinex in cloud p1 temporialy
(#31678)
add 9466e3f96d8 [improvement](statistics)Log one bdbje record for one load
transaction. #31619
add 5d99dffe6f1 [opt](scan) read scan ranges in the order of partitions
(#31630)
add 97b8c85b4c2 [fix](pipelineX)error output_block settings result in
excessive memory usage. #31680
add aa7998cd326 [regression test]Test the unique model by modify a key
type from BOOLEAN to other type (#31540)
add 31644412c85 [Bug](regression) fix bug of null in topn opt (#31669)
add 09609dd2db8 [feature](merge-cloud) support MoW table in cloud mode
(#31317)
add 45645333b4d [Refactor](rf) Refactor the rf code interface to remove
update filter v1 (#31643)
add 21f7881e168 [fix](arrow-flight)Fix the prompt that be
arrow_flight_sql_port is not set (#31668)
add 65c2dd85fa0 [Chore](dependency)Upgrade some dependencies of FE (#31667)
add 7d4e855631b [Bug](coredump) fix regresstion test coredump in multi
thread access map (#31664)
new eafbcb51507 Collect high priority columns. (#31235)
new 1e71f334bdf High priority queue and map. (#31509)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4dbd91fe989)
\
N -- N -- N refs/heads/high-priority-column (1e71f334bdf)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/agent/agent_server.cpp | 5 +-
be/src/agent/task_worker_pool.cpp | 32 ++
be/src/agent/task_worker_pool.h | 2 +
be/src/cloud/cloud_base_compaction.cpp | 388 ++++++++++++++++
.../cloud_base_compaction.h} | 43 +-
be/src/cloud/cloud_compaction_action.cpp | 338 ++++++++++++++
.../action => cloud}/cloud_compaction_action.h | 41 +-
be/src/cloud/cloud_cumulative_compaction.cpp | 477 +++++++++++++++++++
be/src/cloud/cloud_cumulative_compaction.h | 61 +++
.../cloud/cloud_cumulative_compaction_policy.cpp | 206 ++++++++
be/src/cloud/cloud_cumulative_compaction_policy.h | 73 +++
be/src/cloud/cloud_delete_task.cpp | 10 +-
be/src/cloud/cloud_delta_writer.cpp | 4 +
be/src/cloud/cloud_delta_writer.h | 2 +
.../cloud/cloud_engine_calc_delete_bitmap_task.cpp | 168 +++++++
.../cloud/cloud_engine_calc_delete_bitmap_task.h | 72 +++
be/src/cloud/cloud_full_compaction.cpp | 356 ++++++++++++++
be/src/cloud/cloud_full_compaction.h | 64 +++
be/src/cloud/cloud_rowset_builder.cpp | 30 +-
be/src/cloud/cloud_rowset_builder.h | 2 +
be/src/cloud/cloud_rowset_writer.cpp | 5 -
be/src/cloud/cloud_rowset_writer.h | 5 +-
be/src/cloud/cloud_storage_engine.cpp | 517 ++++++++++++++++++++-
be/src/cloud/cloud_storage_engine.h | 53 +++
be/src/cloud/cloud_tablet.cpp | 100 ++++
be/src/cloud/cloud_tablet.h | 96 ++++
be/src/cloud/cloud_tablets_channel.cpp | 11 +-
be/src/cloud/cloud_txn_delete_bitmap_cache.cpp | 190 ++++++++
be/src/cloud/cloud_txn_delete_bitmap_cache.h | 95 ++++
be/src/cloud/cloud_vertical_rowset_writer.cpp | 71 +++
.../cloud_vertical_rowset_writer.h} | 23 +-
be/src/cloud/config.cpp | 10 +
be/src/cloud/config.h | 10 +
be/src/common/config.cpp | 6 +
be/src/common/config.h | 7 +
be/src/exprs/bitmapfilter_predicate.h | 2 +-
be/src/exprs/bloom_filter_func.h | 65 ++-
be/src/exprs/hybrid_set.h | 16 +-
be/src/exprs/minmax_predicate.h | 10 +-
be/src/exprs/runtime_filter.cpp | 5 +
be/src/exprs/runtime_filter.h | 4 +-
be/src/http/action/cloud_compaction_action.cpp | 41 --
be/src/olap/accept_null_predicate.h | 56 ++-
be/src/olap/base_tablet.cpp | 201 ++++++++
be/src/olap/base_tablet.h | 18 +
be/src/olap/compaction.cpp | 81 ++++
be/src/olap/compaction.h | 32 ++
be/src/olap/comparison_predicate.h | 44 +-
be/src/olap/data_dir.cpp | 2 +-
be/src/olap/olap_common.h | 28 +-
be/src/olap/rowset/beta_rowset_writer.cpp | 2 +-
be/src/olap/rowset/beta_rowset_writer.h | 5 +-
be/src/olap/rowset/rowset.cpp | 4 +
be/src/olap/rowset/rowset.h | 1 +
be/src/olap/rowset/rowset_factory.cpp | 9 +-
.../inverted_index_compound_directory.cpp | 3 +-
.../rowset/segment_v2/inverted_index_writer.cpp | 15 +-
.../olap/rowset/segment_v2/inverted_index_writer.h | 36 ++
be/src/olap/rowset/segment_v2/segment_writer.cpp | 5 -
.../rowset/segment_v2/vertical_segment_writer.cpp | 9 -
be/src/olap/rowset/vertical_beta_rowset_writer.cpp | 174 +------
be/src/olap/rowset/vertical_beta_rowset_writer.h | 11 +-
....cpp => vertical_beta_rowset_writer_helper.cpp} | 116 +++--
...iter.h => vertical_beta_rowset_writer_helper.h} | 48 +-
be/src/olap/rowset_builder.cpp | 2 +-
be/src/olap/rowset_builder.h | 4 +-
be/src/olap/snapshot_manager.cpp | 2 +-
be/src/olap/storage_engine.h | 4 +-
be/src/olap/tablet.cpp | 215 +--------
be/src/olap/tablet.h | 17 +-
be/src/olap/tablet_fwd.h | 2 +
be/src/pipeline/exec/file_scan_operator.cpp | 55 ++-
be/src/pipeline/exec/union_source_operator.cpp | 15 +-
be/src/pipeline/exec/union_source_operator.h | 4 +-
be/src/pipeline/pipeline_fragment_context.cpp | 2 +-
.../pipeline_x/pipeline_x_fragment_context.cpp | 2 +-
be/src/runtime/exec_env.h | 5 +
be/src/runtime/exec_env_init.cpp | 2 +
be/src/runtime/memory/cache_policy.h | 3 +
be/src/runtime/plan_fragment_executor.cpp | 2 +-
be/src/service/http_service.cpp | 6 +-
be/src/util/doris_metrics.cpp | 6 +
be/src/util/doris_metrics.h | 2 +
be/src/vec/columns/column_nullable.h | 2 +-
be/src/vec/exec/scan/new_file_scan_node.cpp | 55 ++-
cloud/test/meta_server_test.cpp | 2 +-
.../hive/scripts/create_preinstalled_table.hql | 14 -
.../org/apache/doris/analysis/CreateTableStmt.java | 5 -
.../main/java/org/apache/doris/catalog/Table.java | 30 +-
.../transaction/CloudGlobalTransactionMgr.java | 278 ++++++++++-
.../org/apache/doris/common/InternalErrorCode.java | 5 +-
.../apache/doris/common/util/MetaLockUtils.java | 24 +
.../apache/doris/common/util/PropertyAnalyzer.java | 2 +-
.../org/apache/doris/datasource/ExternalTable.java | 2 +-
.../apache/doris/datasource/InternalCatalog.java | 5 +-
.../org/apache/doris/journal/JournalEntity.java | 12 +
.../java/org/apache/doris/master/MasterImpl.java | 31 +-
.../nereids/rules/analysis/CheckAfterRewrite.java | 12 +-
.../rules/analysis/CollectJoinConstraint.java | 6 +
.../mv/AbstractMaterializedViewRule.java | 3 +-
.../exploration/mv/MaterializedViewUtils.java | 3 +
.../nereids/rules/exploration/mv/Predicates.java | 44 +-
.../rules/exploration/mv/PredicatesSplitter.java | 12 +
.../rules/expression/rules/SimplifyRange.java | 9 +-
.../doris/nereids/stats/FilterEstimation.java | 15 +-
.../trees/plans/commands/info/CreateTableInfo.java | 6 -
.../apache/doris/nereids/util/ExpressionUtils.java | 4 +
.../java/org/apache/doris/persist/EditLog.java | 20 +-
.../main/java/org/apache/doris/plsql/Console.java | 8 +-
.../src/main/java/org/apache/doris/plsql/Exec.java | 44 +-
.../src/main/java/org/apache/doris/plsql/Stmt.java | 8 +-
.../doris/plsql/executor/PlSqlOperation.java | 46 +-
.../apache/doris/plsql/executor/PlsqlResult.java | 35 +-
.../main/java/org/apache/doris/qe/Coordinator.java | 2 +-
.../org/apache/doris/qe/MysqlConnectProcessor.java | 2 +-
.../apache/doris/statistics/AnalysisManager.java | 62 ++-
.../doris/statistics/NewPartitionLoadedEvent.java | 22 +-
.../org/apache/doris/statistics/Statistics.java | 6 +-
.../apache/doris/statistics/UpdateRowsEvent.java | 27 +-
.../java/org/apache/doris/task/AgentBatchTask.java | 10 +
.../apache/doris/task/CalcDeleteBitmapTask.java | 119 +++++
.../doris/transaction/DatabaseTransactionMgr.java | 6 +-
.../rules/expression/SimplifyRangeTest.java | 134 ++++++
.../doris/nereids/stats/FilterEstimationTest.java | 36 ++
fe/pom.xml | 39 +-
gensrc/proto/internal_service.proto | 3 +
.../data/nereids_p0/hint/fix_leading.out | 202 +++++++-
.../cloud_p0/conf/regression-conf-custom.groovy | 6 +-
.../pipeline/cloud_p0/conf/session_variables.sql | 1 +
.../pipeline/cloud_p1/conf/session_variables.sql | 2 +-
regression-test/pipeline/external/conf/fe.conf | 3 +
regression-test/pipeline/p0/conf/fe.conf | 2 +-
regression-test/pipeline/p1/conf/fe.conf | 5 +
.../plugins/plugin_get_table_properties.groovy | 10 +-
.../hive/test_hive_parquet.groovy | 10 -
.../test_build_index_fault.groovy | 2 +-
.../test_segcompaction_fault_injection.groovy | 2 +-
.../test_too_many_segments_fault_injection.groovy | 2 +-
.../suites/nereids_p0/aggregate/aggregate.groovy | 2 +-
.../suites/nereids_p0/hint/fix_leading.groovy | 114 ++++-
.../filter_equal_or_notequal.groovy | 104 +++--
.../test_unique_model_schema_key_change.groovy | 137 ++++++
...est_update_rows_and_partition_first_load.groovy | 137 ++++++
143 files changed, 5990 insertions(+), 927 deletions(-)
create mode 100644 be/src/cloud/cloud_base_compaction.cpp
copy be/src/{http/action/cloud_compaction_action.h =>
cloud/cloud_base_compaction.h} (50%)
create mode 100644 be/src/cloud/cloud_compaction_action.cpp
rename be/src/{http/action => cloud}/cloud_compaction_action.h (56%)
create mode 100644 be/src/cloud/cloud_cumulative_compaction.cpp
create mode 100644 be/src/cloud/cloud_cumulative_compaction.h
create mode 100644 be/src/cloud/cloud_cumulative_compaction_policy.cpp
create mode 100644 be/src/cloud/cloud_cumulative_compaction_policy.h
create mode 100644 be/src/cloud/cloud_engine_calc_delete_bitmap_task.cpp
create mode 100644 be/src/cloud/cloud_engine_calc_delete_bitmap_task.h
create mode 100644 be/src/cloud/cloud_full_compaction.cpp
create mode 100644 be/src/cloud/cloud_full_compaction.h
create mode 100644 be/src/cloud/cloud_txn_delete_bitmap_cache.cpp
create mode 100644 be/src/cloud/cloud_txn_delete_bitmap_cache.h
create mode 100644 be/src/cloud/cloud_vertical_rowset_writer.cpp
copy be/src/{olap/rowset/vertical_beta_rowset_writer.h =>
cloud/cloud_vertical_rowset_writer.h} (68%)
delete mode 100644 be/src/http/action/cloud_compaction_action.cpp
copy be/src/olap/rowset/{vertical_beta_rowset_writer.cpp =>
vertical_beta_rowset_writer_helper.cpp} (58%)
copy be/src/olap/rowset/{vertical_beta_rowset_writer.h =>
vertical_beta_rowset_writer_helper.h} (54%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/task/CalcDeleteBitmapTask.java
create mode 100644
regression-test/suites/statistics/test_update_rows_and_partition_first_load.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
