vaquar khan created KAFKA-20436:
-----------------------------------
Summary: Implement KIP-1318: Model Context Protocol (MCP) Server
for Apache Kafka
Key: KAFKA-20436
URL: https://issues.apache.org/jira/browse/KAFKA-20436
Project: Kafka
Issue Type: New Feature
Reporter: vaquar khan
{panel}
*Background & Motivation:* The Model Context Protocol (MCP) is an open standard
that allows AI agents to interact with tools and data sources. Currently,
interacting with Apache Kafka programmatically requires writing Java/Python
code, using CLI tools, or relying on Connect REST APIs none of which are
accessible to AI agents via MCP.
While community MCP implementations exist, they lack critical capabilities such
as ACL management, transactional operations, and support for vanilla Apache
Kafka (many are tightly coupled to Confluent Cloud).
This KIP proposes adding a first-party, Apache-licensed MCP server for Kafka.
It will wrap Kafka's native Java client APIs directly, require zero external
dependencies, and allow AI agents to manage topics, read/write messages, handle
consumer groups, and manage ACLs via natural language.
*Scope & Implementation Details:*
* *New Standalone Module:* Introduce a new standalone module under
{{tools/mcp-server}} that packages a JSON-RPC 2.0 server. This runs outside the
broker process.
* *No Protocol Changes:* This introduces no changes to the Kafka protocol,
public APIs, or client behaviors. All native standard properties
({{{}security.protocol{}}}, {{{}sasl.*{}}}, {{{}ssl.*{}}}) will simply be
passed through to the underlying {{{}Admin{}}}, {{{}KafkaProducer{}}}, and
{{KafkaConsumer}} instances.
* *Transport Modes:* Supports both {{stdio}} (local execution) and {{http}}
(remote deployment).
* *MCP Tools (State-Changing Actions):*
** _Topic Management:_ {{{}create_topic{}}}, {{{}delete_topic{}}},
{{{}alter_topic_config{}}}, {{{}create_partitions{}}}, etc.
** _Message Ops:_ {{{}produce_message{}}}, {{{}produce_batch{}}},
{{{}produce_transactional{}}}, {{{}consume_messages{}}}.
** _Group Management:_ {{{}delete_consumer_group{}}},
{{{}alter_consumer_group_offsets{}}}, etc.
** _ACL Management:_ {{{}create_acls{}}}, {{{}delete_acls{}}}.
** _Cluster/Connect Ops:_ Manage connectors, alter broker configs, trigger
leader elections.
* *MCP Resources (Read-Only Data):* Use an internal {{kafka://}} scheme to
expose resources like {{{}kafka://topics/\{name}{}}},
{{{}kafka://groups/\{id}/lag{}}}, {{{}kafka://cluster{}}}, etc.
*Phased Rollout Strategy:*
* *Phase 1 (Core):* Topics, messages, consumer groups, offsets, and cluster
basics.
* *Phase 2 (Security + Connect):* ACL management and Kafka Connect tools.
* *Phase 3 (Advanced):* Transactional produce/abort, share/streams groups,
leader elections.
h3. *Acceptance Criteria*
# *Module Creation:* {{tools/mcp-server}} module is initialized with
standalone packaging.
# *Configuration:* The server accepts required config properties
({{{}bootstrap.servers{}}}, {{{}mcp.transport{}}}, etc.) and correctly passes
standard Kafka security/client properties down to the Java clients.
# *Phase 1 Tools & Resources:* Core topic, message, and consumer group
functionalities are implemented and accessible via MCP JSON-RPC messages.
# *Graceful Shutdown:* The server correctly halts new requests, flushes
pending producer records, and closes out clients on
{{{}SIGTERM{}}}/{{{}SIGINT{}}}.
# *Testing:* * Unit tests validate tool creation/logic using mocked
{{{}Admin{}}}, {{{}KafkaProducer{}}}, and {{KafkaConsumer}} instances.
** Integration tests are added using {{kafka.test.ClusterTestExtensions}} to
verify end-to-end interactions via the MCP server on an embedded cluster.
# *Documentation:* Dedicated MCP server documentation is added to the Kafka
Tools site, including configuration, transport setup, and usage examples.
h3. *Links & References*
* *KIP-1318 Wiki:*
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-1318%3A+Model+Context+Protocol+%28MCP%29+Server+for+Apache+Kafka]
{panel}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
