codelipenghui opened a new pull request, #25574:
URL: https://github.com/apache/pulsar/pull/25574
PIP: #TBD (see `pip/pip-470.md` in this PR)
### Motivation
Deployments with very large numbers of mostly-idle topics suffer broker
memory pressure and per-topic metrics cardinality, even when they explicitly do
**not** want the topic data deleted. Today the only mechanisms are:
1. `brokerDeleteInactiveTopicsEnabled` — detects inactivity the right way,
but deletes the data. Not acceptable when data must be retained.
2. Manual `pulsar-admin topics unload` driven by an external cron —
reimplements the broker's inactivity detection outside the broker.
This PR adds a first-class broker option that reuses the existing inactivity
detection and **closes (unloads)** the inactive topic instead of deleting it.
The topic's BookKeeper ledgers, subscriptions, cursors and policies are all
preserved; only the in-memory `PersistentTopic`/`NonPersistentTopic` and its
broker-cache entry are released. The next producer/consumer reconnect
transparently reloads the topic.
### Modifications
- **`ServiceConfiguration`**: new dynamic config
`brokerCloseInactiveTopicsEnabled` (default `false`).
- **`PulsarService.start()`**: fail-fast validation —
`brokerDeleteInactiveTopicsEnabled` and `brokerCloseInactiveTopicsEnabled` are
mutually exclusive.
- **`BrokerService.startInactivityMonitor()`**: schedule `checkGC` when
**either** flag is on.
- **`AbstractTopic.isCloseWhileInactive()`**: broker-level helper reading
the new flag.
- **`PersistentTopic.checkGC()`** / **`NonPersistentTopic.checkGC()`**: when
close mode is active (and delete is not), after the same inactivity and
replication-producer checks as today, call `close(true, false)`. That path
already disconnects clients, closes the managed ledger without deleting
ledgers, and removes the topic from the broker cache via `disposeTopic` →
`removeTopicFromCache`. The retention-window guard is bypassed in the close
branch because it exists only to prevent data loss, which is moot when nothing
is deleted.
- Partitioned-topic metadata cleanup is not run in close mode (nothing is
deleted).
- Added **`pip/pip-470.md`** describing motivation, design, and interactions.
Detection inputs are unchanged and reused: `brokerDeleteInactiveTopicsMode`,
`brokerDeleteInactiveTopicsFrequencySeconds`,
`brokerDeleteInactiveTopicsMaxInactiveDurationSeconds`.
### Verifying this change
This change added tests and can be verified as follows:
- **New** `InactiveTopicCloseTest`:
- `testCloseInactiveTopicKeepsDataAndEvictsFromCache` — produces a
message, drops the subscription, waits for the inactivity window, asserts the
topic is evicted from the broker cache (`getTopicReference` empty) but still
listed in metadata, and a fresh consumer on a new subscription reads the
original message from earliest — proving data is preserved.
- `testActiveTopicIsNotClosed` — topic with an active subscription must
not be closed.
- `testMutualExclusionWithDeleteInactive` — broker startup fails when both
flags are enabled.
- **Existing** `InactiveTopicDeleteTest` (20 tests) — all pass; delete-path
behavior is unchanged.
### Does this pull request potentially affect one of the following parts:
- [x] The default values of configurations — a new configuration
`brokerCloseInactiveTopicsEnabled` is introduced with default `false`; no
existing configuration default is changed.
### Documentation
- [x] `doc-required` — when this feature merges, the
inactive-topic-management docs should be updated to describe the
close-without-delete mode alongside the existing delete mode.
### Matching PR in forked repository
PR in forked repository:
https://github.com/codelipenghui/incubator-pulsar/tree/penghui/jolly-nightingale-e2e980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
