This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 98877cfc878 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
omit d56c8f356c3 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
omit ca840c9dbea [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
omit 876bbbd5bcc [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
omit 76485a6362d [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
omit 9a093765115 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
omit d5abf3d44f4 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
omit 4e0006f8842 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
omit 91130e6daf7 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
omit fa2da3682ac [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
omit 6608e977e0f [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit a1dbf68c9af [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit ed6f1caecf0 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit 0c97e207c89 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit 40c95c079d5 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit e2818c76e5b [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit 9df89f774cb [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit 287cfd18da0 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit ff643d4bc4d [refactor](catalog) Catalog spi 11 hive (#65473)
omit 16013fe2a63 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit 16965f7cda7 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit a0214b93a54 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit 10bba885d96 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit 239d6d441cb [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit 6e8608658ef [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit 1e16c5f6f1c [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit 3ed8be6edc2 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit e9e99eaae05 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit 6974f595d81 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 3d946f09fb3 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit 1d20d3aba20 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit 504397ad88a [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit c5d997ee8cc [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit 890014ad706 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit 665edee8c83 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit 5b7fe17dd6f [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit 1001d755516 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit b47e2ed24a7 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit 72d4c47dc58 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit c68e5e47b86 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit 6bab6000122 update P5 handoff and fix compile issue
omit 046cb643379 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit 47bfc11d554 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit 3be0bcf2ed1 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit e0f0c79fa3a [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit 5ca1d15d6c6 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit b8983bb19dc [doc](connector) add project tracking system for catalog
SPI migration
add 9ec7f76d15e [bench](parquet) Add native reader microbenchmark matrix
(#65921)
add 4016780fca4 [fix](paimon) Handle special characters in partition
values (#65904)
add 0c2a7b5a7cf [fix](be) Backpressure async group commit by table WAL
count (#65362)
add cc9491a751d [fix](regression) Tolerate dynamic partition scheduler
race (#65902)
add f2f2a5701f0 [fix](be) Fix waiter accounting in blocking queue (#65827)
add c6617278765 [doc](connector) add project tracking system for catalog
SPI migration
add 8c7f0a4c718 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add 637a4f624e9 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add be8f0e622de [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add 884a95f1b48 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 8b5cb0ddc89 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add 7878a5dc29f update P5 handoff and fix compile issue
add 8a94692baa0 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add 35bbba558c2 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add 8d007607460 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add 7875720ed41 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add 481f54e94f1 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add a0460f200a0 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add 7abe8fe70f4 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add 75702575c5d [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add 901ff505862 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 369b8a68cc1 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 7e775663587 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add 0590c0da1ef [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add 6dfce469d94 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add d8609b90ff1 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add 6cfce0cf880 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add 2be4f515239 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 6d8f0f2e8e0 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 3bdde9431ab [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add abf3ca1795d [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add bbb3cc0d14a [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 7f1d90ace8b [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add eb9ade27810 [refactor](catalog) Catalog spi 11 hive (#65473)
add 9ba7af0969f [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add 1cd38df75c3 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add f1d3096ffa2 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add fdde0cbced4 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add 397aaafe85b [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add e04c2490721 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add 215f1981e0f [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add 9920780dca8 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add 21eaf91db7a [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add 9820b0c10ba [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add a680703c747 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add ff890431947 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add 5101a6c1c36 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add 0fbd57b3891 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
add 15165e9e064 [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
add 942a4dd041d [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
add d1e03fb7682 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
add 8079812f33d [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
add 6c9f53c789f [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (98877cfc878)
\
N -- N -- N refs/heads/branch-catalog-spi (6c9f53c789f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/benchmark/benchmark_main.cpp | 39 ++
be/benchmark/parquet/AGENTS.md | 296 +++++++++++++
be/benchmark/parquet/README.md | 63 +++
be/benchmark/parquet/benchmark_parquet_decoder.hpp | 479 +++++++++++++++++++++
be/benchmark/parquet/benchmark_parquet_reader.hpp | 448 +++++++++++++++++++
be/benchmark/parquet/parquet_benchmark_scenarios.h | 250 +++++++++++
be/src/common/config.cpp | 3 +
be/src/common/config.h | 2 +
.../operator/group_commit_block_sink_operator.cpp | 9 +-
be/src/load/group_commit/group_commit_mgr.cpp | 56 ++-
be/src/load/group_commit/group_commit_mgr.h | 5 +-
be/src/load/group_commit/wal/wal_manager.cpp | 13 +-
be/src/load/group_commit/wal/wal_manager.h | 1 +
be/src/load/group_commit/wal/wal_table.cpp | 14 +
be/src/load/group_commit/wal/wal_table.h | 4 +-
be/src/util/blocking_queue.hpp | 69 ++-
be/test/format/wal/wal_manager_test.cpp | 32 ++
.../parquet/parquet_benchmark_scenarios_test.cpp | 129 ++++++
be/test/util/blocking_queue_test.cpp | 316 ++++++++++++++
.../connector/paimon/PaimonConnectorMetadata.java | 57 ++-
.../PaimonConnectorMetadataPartitionTest.java | 130 ++++++
.../test_group_commit_wal_num_backpressure.groovy | 99 +++++
.../auto_partition/test_auto_dynamic.groovy | 62 +--
23 files changed, 2485 insertions(+), 91 deletions(-)
create mode 100644 be/benchmark/parquet/AGENTS.md
create mode 100644 be/benchmark/parquet/README.md
create mode 100644 be/benchmark/parquet/benchmark_parquet_decoder.hpp
create mode 100644 be/benchmark/parquet/benchmark_parquet_reader.hpp
create mode 100644 be/benchmark/parquet/parquet_benchmark_scenarios.h
create mode 100644
be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp
create mode 100644
regression-test/suites/insert_p0/group_commit/test_group_commit_wal_num_backpressure.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
