lhotari opened a new pull request, #25979: URL: https://github.com/apache/pulsar/pull/25979
### Motivation `managedLedgerMaxReadsInFlightSizeInMB` controls the `InflightReadsLimiter`, which bounds the memory retained by data read from BookKeeper (or the cache) until it has been delivered to the consumer's Netty channel. It is the mechanism that provides [backpressure](https://github.com/apache/pulsar/blob/master/ARCHITECTURE.md#backpressure) for BookKeeper reads and tiered storage reads. The current default is `0`, which **disables** the limiter. With no limit in place there is no backpressure on the read path: under many concurrent reads to many consumers — especially with large entries, catch-up reads, or tiered storage reads — the broker can accumulate unbounded read buffers and run into direct-memory OOM and broker stability issues. The next release is the **Pulsar 5.0 LTS**, which makes it the right time to change this default. Shipping a sane, bounded default in an LTS gives operators safe out-of-the-box read backpressure instead of an unbounded read path. A conservative fraction of direct memory keeps the limiter coordinated with the existing `managedLedgerCacheSizeMB` default (20% of direct memory) and the broker's overall direct-memory budget. ### Modifications - `managedLedgerMaxReadsInFlightSizeInMB` now defaults to **unset** (`null` in `ServiceConfiguration`, empty value in `conf/broker.conf` / `conf/standalone.conf`). When unset, the broker uses **15% of the available JVM direct memory** (`DirectMemoryUtils.jvmMaxDirectMemory()`). - An explicit `0` still **disables** the feature; an explicit value `> 0` is used as the buffer size in MB (unchanged semantics). - The field type changed from `long` to `Long` so the unset state can be represented; the resolution from the configured value to bytes is done in `ManagedLedgerClientFactory`. The only in-tree caller of the getter is updated accordingly. - Documentation for the setting is made consistent across `ServiceConfiguration.java`, `conf/broker.conf` and `conf/standalone.conf`; `standalone.conf` gains the previously-missing entry. - The existing "resolved value is below `dispatcherMaxReadSizeBytes`" warning now reports the resolved MB value instead of the raw config value (which can now be `null`). ### Verifying this change This change is already covered by existing tests, such as the managed-ledger / Key_Shared broker cache tests (`KeySharedSubscriptionBrokerCacheTest`, `KeySharedSubscriptionDisabledBrokerCacheTest`), which exercise the limiter with explicit values. Those two tests are updated to pass `long` literals (`100L` / `0L`) to the now-`Long` setter. The default-resolution path (`null` → 15% of direct memory, `0` → disabled, `> 0` → MB) is covered by the broker startup wiring in `ManagedLedgerClientFactory`. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [x] The default values of configurations - [x] Anything that affects deployment The default of `managedLedgerMaxReadsInFlightSizeInMB` changes from `0` (disabled) to unset = 15% of JVM direct memory, enabling read backpressure by default. Operators who want the previous behavior can set `managedLedgerMaxReadsInFlightSizeInMB=0` explicitly. The `ServiceConfiguration` getter/setter type changes from `long` to `Long`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
