merlimat opened a new pull request, #25516: URL: https://github.com/apache/pulsar/pull/25516
## Summary Adds PIP-468, a sub-PIP of [PIP-460: Scalable Topics](https://github.com/apache/pulsar/blob/master/pip/pip-460.md), describing the broker-side **Scalable Topic Controller** — the component responsible for segment lifecycle management, split/merge coordination, consumer assignment, leader election, and DAG watch sessions for scalable topics. The document covers: - **Architecture** — the four-layer design (Admin API → ScalableTopicService → ScalableTopicController → ScalableTopicResources) and the three broker roles (any broker for DAG watch sessions, controller leader for layout mutations and consumer assignment, segment-owning brokers for produce/consume). - **Split and merge protocols** — the step-by-step create-first / update-metadata-last ordering that guarantees no messages are lost and all subscription cursors exist before producers are redirected, plus cross-broker coordination via the Segments admin API. - **Consumer assignment** — StreamConsumer / CheckpointConsumer receive coordinated assignments from the controller leader; QueueConsumer attaches to all segments with no controller-side assignment. Includes a detailed **Consumer Session Lifecycle** section clarifying what is persisted (the `ConsumerRegistration`) vs. what is in-memory on the leader (the keep-alive state and grace-period timer), and how controller leader failover restarts grace-period timers from zero. - **Leader election, failover, and metadata store schema** — including the new `SubscriptionMetadata` and `ConsumerRegistration` paths. - **Public-facing changes** — new binary protocol commands (`CommandScalableTopicLookup` / `Update` / `Close`), new Admin REST endpoints for scalable topics and segments, new `ScalableTopics` admin client interface, and the `scalableTopicEnabled` broker configuration. This PIP only changes documentation under `pip/`. Implementation will land in subsequent PRs against the parent PIP-460 tracking issue. ## Test plan - [ ] Review the document on the PIP mailing list thread (TBD link to be added once thread is open). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
