merlimat opened a new pull request, #25573:
URL: https://github.com/apache/pulsar/pull/25573
> **Stacked on top of #25565** (which is on top of #25564, which is on top of
> #25559). Opened as a draft; will be rebased on master once the upstream PRs
> land. Reviewers: the new code in this PR is the single commit titled
> "PIP-468: Implement V5 client with scalable topic producer and consumer
> types" — everything else comes from #25559 / #25564 / #25565.
## Summary
Fourth (and final for this series) PR in PIP-468. Adds the
`pulsar-client-v5` module that implements the V5 client API (PIP-466) for
scalable topics.
### Core pieces
- **`PulsarClientV5`** + **`PulsarClientBuilderV5`** — wraps the existing
v4 `PulsarClientImpl` and adds the scalable-topic routing layer on top.
- **`PulsarClientProviderV5`** — ServiceLoader SPI implementation for the
V5 API.
- **`ScalableTopicProducer`** — key-hashing segment routing, round-robin
for unkeyed messages, per-segment underlying v4 producers.
- **`ScalableStreamConsumer`** — ordered, cumulative-ack consumer that
receives segment assignments from the controller leader and drives v4
per-segment consumers.
- **`ScalableQueueConsumer`** — controller-bypassing consumer that
attaches to every segment and drains them with the receive-loop fix
from the earlier iterations.
- **`ScalableCheckpointConsumer`** — unmanaged connector-style consumer
that can resume from a caller-supplied `Checkpoint` across all
segments.
- **`DagWatchClient`** — client-side handle for the DAG watch session;
implements the `DagWatchSession` callback from the protocol PR.
- **`MessageIdV5`** — V5 message-id wrapping a v4 id with a per-segment
position vector for cumulative ack.
- **`CheckpointV5`** — caller-durable checkpoint including
`EARLIEST` / `LATEST` sentinels and timestamp variants.
- **`ClientSegmentLayout`** — client-side view of the DAG built from a
`ScalableTopicDAG` protobuf.
- **`SegmentRouter`** — 16-bit Murmur3 key-hash routing and round-robin
for empty keys.
- **`AuthenticationAdapter`**, **`CryptoKeyReaderAdapter`**,
**`SchemaAdapter`** — bridge V5 API types to the underlying v4
implementations.
- **`V5Utils`** — small helpers (e.g. `asScalableTopicName` which rewraps
a parsed name with the `topic://` domain).
### Supporting v4 change
- `PulsarClientImpl.createSegmentProducerAsync` — internal entry point
for the V5 client to create producers on `segment://` topics,
bypassing the scalable-domain guard that's meant for user-facing
producer creation.
## Tests
### New unit test classes (48 tests)
- **`V5UtilsTest`** (5 tests) — `asScalableTopicName` on bare name /
short form / persistent:// / topic:// inputs; also pins the
caching behavior for the topic:// fast path.
- **`SegmentRouterTest`** (10 tests) — key-routing determinism and
correctness (hash lands inside the picked segment's range),
single-segment, empty-segments error, uncovered-hash error,
round-robin cycling + wrap, hash 16-bit bound, string/bytes agree.
- **`MessageIdV5Test`** (15 tests) — constructor + null-rejection,
immutable position vector, equals/hashCode (position vector is
diagnostic, not identity), compareTo by segment then v4 id, foreign-
type rejection, byte-array roundtrip with and without position
vector, malformed input handling, EARLIEST/LATEST sentinels,
`toString`.
- **`ClientSegmentLayoutTest`** (7 tests) — `fromProto` filters active
segments, sorts by hash-range start, builds segment topic names,
collects broker URLs, propagates the controller URL (plus TLS
variant), and the returned collections are immutable.
- **`CheckpointV5Test`** (11 tests) — regular checkpoint roundtrip
with multi-segment positions, empty positions, EARLIEST/LATEST
sentinel roundtrip (sentinel identity preserved, 1-byte compact
wire), timestamp variant roundtrip + wire shape, null / empty /
unknown-type input rejection, immutable positions map.
### Already exercised by prior PRs in this stack
- `PulsarClientImplTest.createSegmentProducerAsync*` — on #25565 (the
small v4 change we added here is also exercised there).
- Scalable tests for the broker side — 93 tests in #25565 / #25564 /
#25559.
- Admin-API data-class tests — 24 tests in #25565.
- Protocol-command roundtrip tests — 8 tests in #25564.
## Test plan
- [x] `./gradlew :pulsar-client-v5:test --tests
"org.apache.pulsar.client.impl.v5.*"` — 48/48 new tests pass.
- [x] Full compile of all impacted modules
(`pulsar-client-v5`, `pulsar-client-original`, `pulsar-broker`,
`pulsar-common`, `pulsar-client-admin-api`).
- [x] Checkstyle clean on all modules touched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
