pjfanning opened a new issue, #997: URL: https://github.com/apache/pekko-http/issues/997
## Feature request: Plug-in support for custom compression (e.g., zstd) in Pekko HTTP coding subsystem ### Context Currently, Pekko HTTP provides built-in support for gzip (and deflate) compression via the `Coder` abstraction in `org.apache.pekko.http.scaladsl.coding`. This is used via `encodeResponse`, `decodeRequest`, and related directives, and the negotiation machinery is well-established. There is frequent interest (e.g., for modern formats like zstd) in supporting custom content encoding algorithms. Users may want to plug in their own encoders/decoders without modifying Pekko HTTP core code. ### Analysis After reviewing the codebase ([scaladsl/coding](https://github.com/apache/pekko-http/tree/main/http/src/main/scala/org/apache/pekko/http/scaladsl/coding/)), here's what enables and what currently blocks custom compressions: #### What works: - `Coder` is a marker trait (`trait Coder extends Encoder with Decoder`). Users can implement it today to support new encodings, e.g., zstd. - `HttpEncoding.custom(...)` allows defining new encoding tokens, such as `zstd`. - Directives (`encodeResponseWith`, `decodeRequestWith`) accept arbitrary `Encoder`/`Decoder`s, so user-provided implementations can already be injected explicitly per-route. #### Current barriers: 1. **Compressor and StreamDecoder are Internal/Deprecated:** - `Compressor` (needed for encoding) and `StreamDecoder` (for decoding) are `@InternalApi` and `@deprecated`. Implementing `Encoder`/`Decoder` needs usage of these, but they're not public API and may change or disappear, so user-providers can't rely on them. 2. **Java API is closed:** - The Java-side equivalent (`org.apache.pekko.http.javadsl.coding.Coder`) is a closed `enum`—Java users cannot add custom encoders at all. 3. **Default coders are fixed:** - `Coders.DefaultCoders` is always `[Gzip, Deflate, NoCoding]`. There is no mechanism for registering a custom coder with the negotiation machinery globally. So, while you can use your custom coder per-route with the `With` directives, it won't be automatically used with default directives. #### Example: How it could work (after stabilising APIs) ```scala object ZstdCoder extends Coder with StreamDecoder { val encoding: HttpEncoding = HttpEncoding.custom("zstd") def messageFilter = Encoder.DefaultFilter private[this] def newCompressor = new ZstdCompressor() // if Compressor is public def newDecompressorStage(maxBytesPerChunk: Int) = () => new ZstdDecompressorStage(maxBytesPerChunk) } // Usage: encodeResponseWith(ZstdCoder, Coders.Gzip, Coders.Deflate, Coders.NoCoding) { ... } decodeRequestWith(ZstdCoder, Coders.Gzip, Coders.Deflate) { ... } ``` ### Proposed Enhancements 1. **Promote Compressor and StreamDecoder to Public API:** - Remove `@InternalApi` and `@deprecated` from these types - Alternatively, provide a new public API for plugging in custom encoder/decoder flows 2. **Open up Java API for Custom Coders:** - Replace or supplement the hardcoded `Coder` enum with an interface/factory-based mechanism in the Java DSL 3. **Make DefaultCoders Configurable or Injectable:** - Allow applications to register additional coders (e.g., via ActorSystem config or a registration method) for use by the default directives/negotiator 4. **(Optional) Add zstd as a built-in or contrib module:** - To showcase extensibility and provide instant value 5. **(Docs) Add a guide for implementing and registering custom coders** ### Impact - This would unlock modern and/or domain-specific compression for Pekko HTTP users - Immediate benefit for projects that need zstd or other algorithms (Brotli, LZ4, etc.) - Makes Pekko HTTP encoding/decoding as extensible as its routing layers **If maintainers agree, I'm happy to help with API stabilisation PRs or further design discussion!** --- Origin: https://github.com/apache/pekko-http (discussion/analysis by pjfanning) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
