Hi everyone, I’ve published KIP-1310: General Transaction Session and would like to open the floor for discussion.
Historically, Kafka’s transaction logic has been locked inside KafkaProducer. This worked well for simple "write-and-commit" patterns, but as the ecosystem matures, this monolithic design has become a hurdle. We now have multiple entities e.g. Flink/External Coordinators, Share Group Consumers (KIP-1289), Kafka Connect—that need to participate in or complete transactions without needing the heavy baggage of a full KafkaProducer (record batching, serializers, sender threads, etc.). KIP-1310 extracts transaction identity and lifecycle management into a first-class TransactionSession client. Proposal: - New TransactionSession Class: A lightweight, thread-safe object for identity (producerId/epoch) and lifecycle (initialize, beginTransaction, commit). - Reflection-Free Recovery: Replaces reflection workarounds with a public TransactionSession.resume() API for external coordinators. - Decoupled Architecture: Allows KafkaProducer and KafkaShareConsumer to share a single transaction identity for atomic "Exactly-Once" Kafka-to-Kafka pipelines (KIP-1302). - Automatic Heartbeats: Provides a dedicated home for the background transaction heartbeat (KAFKA-20381), independent of the producer's data-sending loop. This change is fully backward compatible and introduces no new wire protocol changes—it simply aligns the client API with what the Kafka protocol already supports. I’m looking forward to your feedback on the proposed interfaces and the refactoring of TransactionManager. KIP Link: https://cwiki.apache.org/confluence/x/nJY8G Best regards,Shekharhttps://github.com/Shekharrajak
