This is an automated email from the ASF dual-hosted git repository.
mrhhsg pushed a change to branch spill_repartition
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 02e0e1e654a fix compile
discard 2c56f06224f fix compile error
discard a535b886989 code format
discard 693a37df8d5 refactor code and add unit test
discard a98d9a8b1cc add force spill logic in join sink operator
discard cbb941ff1e4 update dir and file meta realtime
discard 2a0710ab575 fix agg profile bug
discard 6a15945270f change spill file to shared ptr
discard 7754b38a9b0 fix probe revokeable memory size bug
discard 104d07306e9 fix compile bug
discard 4db0fb2cd98 simplify code
discard 51f9e1e3534 simplify agg code
discard 79048a6a864 simplify probe code enhancement probe operator
discard 87cabe52bf3 refactor spill file interface
discard ddd18159320 some tiny fix
discard f27324e312c fix agg revocable mem size
discard d20dc99b6a7 disbale distinct streaming agg when spill enabled
discard a3ba913ee19 Using spill_buffer_size_bytes as read limit when
recovering data
discard 4b4724d7a7c [pipeline] Proactively pause query for spill under memory
pressure in PipelineTask
discard 0307ef9b8c5 Make spill stream RAII
discard 16874c892b7 Clear revoked data in agg & avoid null pointer in join
discard 3ea6c8ba8db Make repartitioner level-aware
discard d61e3d6f3be fix compile
discard a5dd009afa4 fix compile
discard b7601f05add remove some codes
discard 770425ecc22 f
discard 8b8d60fffc7 f
discard 63f31ff8259 f
discard 8ab3f5fa63c f
discard 215984b592c repartitioner
add 6a38371759d [chore](cloud) add build fdb script (#60877)
add 28e3145181a [improvement](iceberg) Improve VIcebergSortWriter code
quality (#60978)
add 453879e0a2d [fix](test) add more log to debug (#61009)
add f974fe7c003 [fix](cache) invalidate sorted partition cache after
replacing temp partition in cloud mode (#60888)
add 1d9f11fba80 repartitioner
add aa7ea559946 refactor: use DorisVector in byte array dict decoder
add 80bcaf484b6 refactor: use DorisVector in fixed length dict decoder
add bb7de29f99d fix UT & use DorisVector
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (02e0e1e654a)
\
N -- N -- N refs/heads/spill_repartition (bb7de29f99d)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/src/pipeline/dependency.h | 4 +-
be/src/pipeline/exec/operator.h | 6 +-
.../exec/partitioned_aggregation_sink_operator.cpp | 19 +-
.../partitioned_aggregation_source_operator.cpp | 18 +-
.../exec/partitioned_hash_join_probe_operator.cpp | 172 ++++-----
.../exec/partitioned_hash_join_probe_operator.h | 8 +-
.../exec/spill_iceberg_table_sink_operator.cpp | 36 +-
.../exec/spill_iceberg_table_sink_operator.h | 5 +-
be/src/pipeline/exec/spill_sort_sink_operator.cpp | 2 +-
be/src/pipeline/pipeline_task.cpp | 26 +-
.../format/parquet/byte_array_dict_decoder.cpp | 5 +-
.../exec/format/parquet/byte_array_dict_decoder.h | 4 +-
.../format/parquet/fix_length_dict_decoder.hpp | 2 +-
be/src/vec/exec/scan/split_source_connector.h | 15 +-
.../sink/writer/iceberg/viceberg_sort_writer.cpp | 382 ++++++++++++++++++++
.../vec/sink/writer/iceberg/viceberg_sort_writer.h | 345 +++++-------------
.../sink/writer/iceberg/viceberg_table_writer.h | 8 +-
be/src/vec/spill/spill_file_manager.h | 4 +-
be/src/vec/spill/spill_file_reader.cpp | 4 +
be/src/vec/spill/spill_file_writer.cpp | 3 +-
.../exec/multi_cast_data_streamer_test.cpp | 7 +-
.../partitioned_aggregation_sink_operator_test.cpp | 12 +-
...artitioned_aggregation_source_operator_test.cpp | 2 -
.../partitioned_aggregation_test_helper.cpp | 4 +
.../partitioned_hash_join_probe_operator_test.cpp | 124 +++----
.../partitioned_hash_join_sink_operator_test.cpp | 4 +-
.../operator/partitioned_hash_join_test_helper.cpp | 5 +
.../operator/spillable_operator_test_helper.cpp | 13 +-
be/test/vec/spill/spill_file_test.cpp | 5 +-
cloud/script/build_fdb.sh | 384 +++++++++++++++++++++
.../main/java/org/apache/doris/catalog/Env.java | 14 +
.../cache/NereidsSortedPartitionsCacheManager.java | 5 +
.../hive/write/test_hive_ctas_to_doris.groovy | 2 +
.../cache/clear_sorted_partition_cache.groovy | 55 +++
34 files changed, 1214 insertions(+), 490 deletions(-)
create mode 100644 be/src/vec/sink/writer/iceberg/viceberg_sort_writer.cpp
create mode 100644 cloud/script/build_fdb.sh
create mode 100644
regression-test/suites/query_p0/cache/clear_sorted_partition_cache.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
