merlimat opened a new pull request, #25633:
URL: https://github.com/apache/pulsar/pull/25633
## Summary
Add a JVM-wide multi-broker test cluster mirroring `SharedPulsarCluster`,
plus
V5 cross-broker coverage for scalable topics. Three brokers share a single
bookie + a real Oxia metadata store (the in-memory store collapses ephemeral
sessions across instances, so leader election doesn't work correctly across
multiple brokers in the same JVM).
### Infrastructure
- `SharedMultiBrokerPulsarCluster` — JVM-wide singleton with `NUM_BROKERS =
3`,
lazy start, shutdown-hook teardown. Backed by `OxiaContainer`
(testcontainers).
Exposes per-broker `PulsarService` / `PulsarAdmin` / `PulsarClient` lists
plus
broker-0 conveniences.
- `SharedMultiBrokerPulsarBaseTest` — v4 base wrapping the cluster with a
fresh
namespace per test method.
- `V5MultiBrokerClientBaseTest` — V5 layer on top, with one V5 `PulsarClient`
per broker.
### Tests (`V5MultiBrokerScalableTopicTest`)
- `testProduceConsumeAcrossBrokers` — produce on broker 0's V5 client,
consume
on the last broker's; lookups must route to segment owners.
- `testControllerLeadershipDistributesAcrossBrokers` — cycle which broker
materializes each new topic's controller; assert leadership lands on every
broker.
- `testStreamConsumerControllerCoordinationAcrossBrokers` — alice + bob
attached
via two non-leader brokers must reach the controller via the lookup →
controller-URL path; segments split across the group.
- `testCheckpointConsumerControllerCoordinationAcrossBrokers` — same with
`consumerGroup(...)`.
- `testSplitFromNonLeaderBrokerRedirectsToOwner` — split request to a
non-leader
broker is 307-redirected to the controller leader.
- `testMergeFromNonLeaderBrokerRedirectsToOwner` — same for merge.
### Broker fixes (uncovered while building the tests)
- `DagWatchSession.buildDagProto` was setting `controllerBrokerUrl` on the
response object but never propagating it into the `ScalableTopicDAG` proto.
Without this V5 clients always saw a null controller URL and fell back to
their configured service URL — on multi-broker that's rarely the controller
leader, so consumer subscribe lands on the wrong broker. Single-broker
tests
didn't catch it because the fallback URL was always the controller.
- `ScalableTopicController` previously discovered subscriptions for split /
merge
by querying segment topics through `admin.topics().getSubscriptionsAsync(
segment://...)`, which the v4 Topics REST endpoint doesn't route. Replaced
with `resources.listSubscriptionsAsync(topicName)` — the scalable-topic
metadata is the single source of truth for subscriptions; segments are
storage
layout. Single-broker tests didn't catch it because the segment topic was
always locally cached and short-circuited the admin path.
### Matching PR(s) in forked repositories
- area/test
- area/broker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
