merlimat opened a new pull request, #25522: URL: https://github.com/apache/pulsar/pull/25522
### Motivation This upgrade brings Pulsar onto the Netty 4.2 line. The direct motivation is to unblock the async-http-client 3.x upgrade (#25023), which transitively depends on Netty 4.2 and cannot be landed while Pulsar force-pins Netty 4.1 — AHC 3.x bytecode is compiled against Netty 4.2 and would silently be downgraded to Pulsar's Netty 4.1 at resolution time, with `NoSuchMethodError` / `NoClassDefFoundError` surfacing in integration tests (which is what #25023's CI is hitting). Netty 4.1 and 4.2 cannot co-exist on the classpath (same `io.netty.*` package namespace), so the upgrade has to be done in a single step. The Netty team asserts source/binary forward compatibility from 4.1 to 4.2 for regular API users: - Announcement: https://netty.io/news/2025/04/03/4-2-0.html - Migration guide: https://github.com/netty/netty/wiki/Netty-4.2-Migration-Guide ### Modifications **`gradle/libs.versions.toml`** - Bump `netty` from `4.1.132.Final` to `4.2.12.Final`. - Drop the separate `netty-iouring = "0.0.26.Final"` version. io_uring has graduated from incubator (`io.netty.incubator:netty-incubator-transport-*-io_uring`) to a first-class Netty artifact (`io.netty:netty-transport-{classes,native}-io_uring`) and is now pinned to the same Netty version. **`pulsar-common/build.gradle.kts`** - Point the io_uring consumer at the renamed aliases. **`pulsar-common/.../EventLoopUtil.java`** - Netty 4.2 removed the dedicated `IOUringEventLoopGroup` class. io_uring now uses the generic `MultiThreadIoEventLoopGroup` + `IoUringIoHandler.newFactory()` pattern, which makes io_uring groups indistinguishable from any other `MultiThreadIoEventLoopGroup` by type — breaking the existing `instanceof`-based channel class dispatch in `getClientSocketChannelClass`, `getServerSocketChannelClass`, `getDatagramChannelClass`. Fix: introduce a private marker subclass `IoUringMultiThreadIoEventLoopGroup` used at construction so `instanceof` keeps discriminating. - Repoint imports from `io.netty.incubator.channel.uring.*` to `io.netty.channel.uring.*` and adjust class names (`IOUring` → `IoUring`). **`build-logic/conventions/.../pulsar.java-conventions.gradle.kts`** - Exclude `io.netty.incubator` from all configurations. BookKeeper 4.17.3 (`bookkeeper-common` and `stream-storage-java-client`) still declares a transitive dependency on the `0.0.26.Final` incubator io_uring jars. Those jars are compiled against Netty 4.1 internals and are not safe to leave on the 4.2 classpath. Pulsar uses the core io_uring API via `EventLoopUtil`; BK stream-storage is an optional feature that Pulsar does not expose in its default surface. **`distribution/{server,shell}/src/assemble/LICENSE.bin.txt`** - Bump all `4.1.132.Final` entries to `4.2.12.Final`. - Replace the monolithic `netty-codec-*.jar` with its 4.2 split-out sub-modules `netty-codec-base` and `netty-codec-compression` (`netty-codec` is now an aggregator POM that ships no classes). - Rename the incubator io_uring entries to the core io_uring artifacts. - The jar set was cross-checked against the output of `:distribution:pulsar-server-distribution:serverDistTar` and `:distribution:pulsar-shell-distribution:shellDistTar`. **`pulsar-common/.../BitSetRecyclableRecyclableTest` and `ConcurrentBitSetRecyclableTest`** - Relax the `testRecycle` assertion. Netty 4.2's `io.netty.util.Recycler` (itself deprecated in 4.2) no longer guarantees same-thread immediate reuse, so we only assert functional behavior: any recycled instance must come back cleared, and distinct `create()` calls must return distinct objects. ### Verification - `./gradlew compileJava compileTestJava` — clean across the entire project. Only deprecation warnings for the 4.2 compat shims (`NioEventLoopGroup`, `EpollEventLoopGroup`, `DefaultEventLoopGroup`, `ChannelOption.RCVBUF_ALLOCATOR`, `EpollMode`, `Recycler`, `PlatformDependent.threadLocalRandom`). These are still functional in 4.2; cleanup can follow in a separate PR. - `:pulsar-common:test` — passes. - `:pulsar-broker:test --tests BrokerServiceTest` — passes (broker startup, producer/consumer flow, Netty transport end-to-end). - `:pulsar-proxy:test --tests ProxyServiceTlsStarterTest` — passes (proxy, TLS handshake, tcnative-boringssl integration). - `:distribution:pulsar-server-distribution:serverDistTar` and `:distribution:pulsar-shell-distribution:shellDistTar` — both build and the Netty jar set inside each tarball matches `LICENSE.bin.txt` exactly. ### Out of scope (follow-ups) - **TLS endpoint identification default change.** Netty 4.2 changes `SslContextBuilder.endpointIdentificationAlgorithm` default from `null` to `"HTTPS"`. Pulsar's TLS client sites need to be audited and explicitly configured. This is deliberately not in this PR because the audit touches many modules (`pulsar-client`, `pulsar-broker`, `pulsar-proxy`, `pulsar-broker-auth-oidc`, admin) and should land as its own reviewable change. - **Default allocator change.** Netty 4.2 changes the default `ByteBufAllocator` from pooled to adaptive. Pulsar is insulated for its main allocator path because `PulsarByteBufAllocator` explicitly constructs a `PooledByteBufAllocator.DEFAULT` via BK's `ByteBufAllocatorBuilder` — so Pulsar's own allocation does not go through Netty's `ByteBufAllocator.DEFAULT`. If CI soak tests surface regressions in Netty-internal allocation paths, `-Dio.netty.allocator.type=pooled` can be added to the launch scripts as a follow-up. - **Deprecation cleanups.** The 4.2 compat shims (`NioEventLoopGroup`, `EpollEventLoopGroup`, `RCVBUF_ALLOCATOR`, etc.) can be migrated to the new APIs in a follow-up. They all still work in 4.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
