wolfstudy commented on PR #25580: URL: https://github.com/apache/pulsar/pull/25580#issuecomment-4326602244
> > Now that the BookKeeper client exposes these TCP keep-alive tuning knobs, the Pulsar Broker should also expose matching > > configuration entries, so that operators can tune the TCP keep-alive behavior of the broker's > > BookKeeper client without having to patch code or rely on OS-wide defaults. > > To solve the issues where connections stall, it's usually necessary to configure TCP keep-alive on both sides: the client and the server. Keep-alive is enabled by default for BookKeeper client and server, but they do rely on OS defaults. > > It will anyways be necessary to tune OS defaults since the BookKeeper server doesn't have a way to configure the keep-alive parameters. In cloud managed k8s nodes, the settings have reasonable values by default at least in GCP. > > ``` > # sysctl -a |grep keepalive > net.ipv4.tcp_keepalive_intvl = 60 > net.ipv4.tcp_keepalive_probes = 5 > net.ipv4.tcp_keepalive_time = 300 > ``` Thanks for the context! A quick clarification on a couple of points: 1. BookKeeper server now exposes keep-alive tuning. Since BookKeeper 4.17.3 (via apache/bookkeeper#4683), both the BK client and the bookie server expose tcpKeepIdle, tcpKeepIntvl, and tcpKeepCnt as configuration options. Pulsar already pulls in BookKeeper 4.17.3, so tuning can be done at the application level on both ends — we no longer have to rely on OS defaults for the bookie server side. 2. Relying on OS defaults is not always viable. The typical Linux kernel defaults are tcp_keepalive_time=7200s (2h), tcp_keepalive_intvl=75s, tcp_keepalive_probes=9, which means a broken connection can go undetected for ~2h 11min. That's too long for a messaging system where broker↔bookie liveness matters. While GCP GKE ships with saner defaults (time=300, intvl=60, probes=5), this is not guaranteed across other environments: - EKS / AKS / on-prem clusters often keep the 7200s default. - Tuning net.ipv4.tcp_keepalive_* via sysctl requires privileged pods or node-level DaemonSets, which many operators either cannot or don't want to deploy. - OS-level settings are global and affect every TCP socket on the node, whereas the BookKeeper-level settings are scoped to BK connections only. 3. What this PR actually does. It doesn't add new Pulsar-specific config; it just documents how to forward these three BK client options through the existing bookkeeper_ prefix mechanism in broker.conf, so operators have a discoverable, Pulsar-idiomatic way to tune broker↔bookie keep-alive without needing node-level privileges or changing OS defaults. The defaults remain -1 (fall back to OS), so behavior is unchanged unless explicitly opted in. So the motivation isn't "OS defaults aren't enough"; it's "give operators a per-application, documented knob that works uniformly across environments, now that BookKeeper 4.17.3 makes this possible." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
