This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 86e9e64c31c [cleanup] PIP-462: Remove Etcd metadata store backend
(#25329)
86e9e64c31c is described below
commit 86e9e64c31cbc159cf08de8e4f94bb153962bcdc
Author: Matteo Merli <[email protected]>
AuthorDate: Sat Mar 21 09:16:04 2026 -0700
[cleanup] PIP-462: Remove Etcd metadata store backend (#25329)
---
pip/pip-462.md | 117 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 117 insertions(+)
diff --git a/pip/pip-462.md b/pip/pip-462.md
new file mode 100644
index 00000000000..439ff8adb4d
--- /dev/null
+++ b/pip/pip-462.md
@@ -0,0 +1,117 @@
+# PIP-462: Remove Etcd metadata store backend
+
+## Motivation
+
+Apache Pulsar supports multiple metadata store backends through the pluggable
`MetadataStore` interface introduced in PIP-45. One of these backends is Etcd,
implemented via the `jetcd` client library. While the Etcd backend has been
available since the metadata store abstraction was introduced, it has seen very
limited adoption in production environments.
+
+Maintaining Etcd support imposes several costs on the project:
+
+1. **Dependency burden**: The `jetcd-core` library pulls in a large transitive
dependency tree (gRPC, Netty, Vert.x, and others) that conflicts with
dependencies used elsewhere in Pulsar. This required creating a dedicated
shading module (`jetcd-core-shaded`) with relocations for `io.vertx` and
`io.netty` packages.
+
+2. **Distribution size**: The shaded jetcd JAR and its transitive dependencies
increase both the Pulsar tarball and Docker image size significantly, even for
users who never use Etcd.
+
+3. **Library maintenance concerns**: The `jetcd` library is not actively
maintained, making it increasingly risky to depend on for a production metadata
backend. Security patches and compatibility updates may lag behind.
+
+4. **Maintenance overhead**: The `EtcdMetadataStore` implementation,
`EtcdSessionWatcher`, and associated test infrastructure must be maintained,
tested, and considered in every metadata store interface change.
+
+5. **Strategic direction**: Both ZooKeeper and Oxia are fully supported and
battle-tested metadata store backends. Oxia is a purpose-built metadata store
designed for Pulsar with superior operational characteristics, and the
community's future efforts are focused there. Maintaining a third, unused
backend dilutes this focus.
+
+Given that Etcd is not used in production deployments, adds significant
distribution size, relies on a poorly maintained client library, and competes
for attention with the strategically preferred Oxia backend, removing it
simplifies the project and reduces the maintenance surface area.
+
+## Goals
+
+### In Scope
+
+- Remove the `EtcdMetadataStore` and `EtcdSessionWatcher` implementation
classes from `pulsar-metadata`
+- Remove the `EtcdMetadataStoreProvider` registration from
`MetadataStoreFactoryImpl`
+- Remove the `jetcd-core-shaded` shading module entirely
+- Remove `jetcd-core` and `jetcd-test` dependency declarations from the root
`pom.xml`
+- Remove Etcd-specific tests (`EtcdMetadataStoreTest`)
+- Remove or update any documentation references to Etcd as a supported backend
+- Provide a clear error message if a user configures an `etcd:` metadata store
URL, guiding them to use ZooKeeper or Oxia instead
+
+### Out of Scope
+
+- Changes to ZooKeeper or Oxia metadata store implementations
+- Changes to the `MetadataStore` interface or plugin system
+
+## High Level Design
+
+The removal is a straightforward deletion of code and dependencies:
+
+1. **Implementation removal**: Delete `EtcdMetadataStore.java` and
`EtcdSessionWatcher.java` from `pulsar-metadata`.
+
+2. **Provider deregistration**: Remove the Etcd provider from
`MetadataStoreFactoryImpl`. Add a check that produces a clear error message
when an `etcd:` URL is configured, informing operators that Etcd is no longer
supported and recommending ZooKeeper or Oxia.
+
+3. **Module removal**: Delete the `jetcd-core-shaded` module and remove it
from the root `pom.xml` modules list.
+
+4. **Dependency cleanup**: Remove `jetcd-core`, `jetcd-test`, and
`jetcd-core-shaded` dependency declarations from the root `pom.xml` dependency
management section. Remove these dependencies from `pulsar-metadata/pom.xml`.
+
+5. **Test removal**: Delete `EtcdMetadataStoreTest.java`.
+
+## Detailed Design
+
+### Implementation Classes
+
+The following source files will be deleted:
+-
`pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/EtcdMetadataStore.java`
(includes inner `EtcdMetadataStoreProvider` class)
+-
`pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/EtcdSessionWatcher.java`
+
+### Factory Update
+
+In `MetadataStoreFactoryImpl`, the Etcd provider registration will be removed
from `loadProviders()`. A runtime check will be added so that if a user
configures `metadataStoreUrl=etcd:...`, the broker fails fast with a clear
error:
+
+```
+Etcd metadata store backend has been removed in Pulsar 5.0.
+Please use ZooKeeper (zk:) or Oxia (oxia:) as your metadata store.
+See PIP-462 for details.
+```
+
+### Module Removal
+
+The `jetcd-core-shaded/` directory will be deleted entirely, and the
`<module>jetcd-core-shaded</module>` entry will be removed from the root
`pom.xml`.
+
+### Dependency Removal
+
+The following entries will be removed from the root `pom.xml`:
+
+From `<dependencyManagement>`:
+- `io.etcd:jetcd-core`
+- `io.etcd:jetcd-test`
+- `org.apache.pulsar:jetcd-core-shaded`
+
+From `pulsar-metadata/pom.xml`:
+- `org.apache.pulsar:jetcd-core-shaded` (compile dependency)
+- `io.etcd:jetcd-test` (test dependency)
+
+### Test Removal
+
+The following test file will be deleted:
+-
`pulsar-metadata/src/test/java/org/apache/pulsar/metadata/impl/EtcdMetadataStoreTest.java`
+
+## Public-facing Changes
+
+### Configuration
+
+The `metadataStoreUrl` and `configurationMetadataStoreUrl` configuration keys
will no longer accept `etcd:` URLs. Attempting to start a broker, proxy, or
other Pulsar component with an Etcd metadata URL will result in a clear error
message at startup.
+
+### Binary Protocol
+
+No changes.
+
+### CLI
+
+No changes. CLI tools that accept metadata store URLs (e.g., `pulsar
initialize-cluster-metadata`) will reject `etcd:` URLs with a descriptive error.
+
+### Metrics
+
+No changes.
+
+## General Notes
+
+The `MetadataStore` plugin system (PIP-45) remains fully intact. If there is
community interest in continuing Etcd support, it can be maintained as an
external plugin outside the core Pulsar repository, without any changes to the
Pulsar codebase.
+
+### Related PIPs
+
+- PIP-45: Pluggable metadata interface (introduced the MetadataStore
abstraction and Etcd backend)
+- PIP-454: Metadata Store Migration Framework (provides migration path for
users needing to switch backends)
