merlimat opened a new pull request, #26008: URL: https://github.com/apache/pulsar/pull/26008
Follow-up to [#25980](https://github.com/apache/pulsar/pull/25980), completing [PIP-483](https://github.com/apache/pulsar/blob/master/pip/pip-483.md): the auto split/merge policy can now be overridden per namespace and per topic, resolved most-specific-wins on top of the broker defaults. This also addresses the review question on #25980 about controlling `maxSegments` / `minSegments` / `maxDagDepth` per scalable topic — application is lazy: the controller picks up override changes on its next evaluation and converges using the load stats. ## Modifications - **`AutoScalePolicyOverride`** — all-optional override carrying the same knobs as the broker config (caps, cooldowns, merge window, the eight rate thresholds, `enabled`). Unset fields fall through to the next layer; `enabled = false` opts a namespace or topic out entirely. - **Storage** — `Policies.scalableTopicAutoScalePolicy` (namespace level, following the `autoTopicCreationOverride` pattern) and `ScalableTopicMetadata.autoScalePolicy` (topic level, broker-internal + admin wire shapes). `SegmentLayout.toMetadata` now takes the original record and carries over all non-layout fields — without this, every split/merge CAS would have silently dropped the per-topic override. - **Resolution** — `AutoScaleConfig.resolve(conf, nsOverride, topicOverride)` layers the overrides and runs the existing invariant validation on the combined result, so an override that is only invalid in combination (e.g. a merge threshold raised above the default split threshold) is rejected with a 412 at set time. - **Controller** — `evaluateAndAct` resolves the effective policy per evaluation from the metadata-cache-backed namespace policies + topic metadata, so override changes take effect on the next tick with no controller restart or leadership cycle. - **Admin API** — `admin.scalableTopics().set/get/removeAutoScalePolicy(topic)` and `admin.namespaces().set/get/removeScalableTopicAutoScalePolicy(namespace)`, with REST endpoints (`POST/GET/DELETE .../autoScalePolicy` and `.../scalableTopicAutoScalePolicy`) guarded by the new `PolicyName.SCALABLE_TOPIC_AUTO_SCALE`. ## Verifying this change - `AutoScaleConfigTest` — override layering (topic wins over namespace over broker), null-overrides identity, invalid-combination rejection. - `ScalableTopicControllerAutoScaleTest` — namespace `enabled=false` suppresses splits; topic override wins over namespace; per-topic `maxSegments` caps splits; the override survives a split's metadata rewrite. - `ScalableTopicAutoScalePolicyTest` — end-to-end admin round-trips at both levels through the real HTTP path, including 412 on invariant violation and 404 on a missing topic. - `SegmentLayoutTest` — `toMetadata` round-trips all non-layout fields. - Full `org.apache.pulsar.broker.service.scalable.*` suite + checkstyle across the five touched modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
