Hi team,
To support works around transactions - I believe we need General Transaction
Session - Please have a look & review :
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1310%3A+General+Transaction+Session
Regards,
Shekhar
On Saturday 28 March 2026 at 01:30:05 pm GMT+5:30, Shekhar Prasad Rajak via
dev <[email protected]> wrote:
Hi Andrew,
Thank you for the detailed breakdown, especially the distinction between making
the ShareConsumer transactional (Option A) vs. the Producer-assisted model
(Option B).
I agree that Option B is the superior path for architectural consistency. By
having the Producer "assist" the ShareConsumer, we can leverage the existing
mechanisms.
I am looking at implementing a prepareAcknowledgementsForTransaction() method
on the KafkaShareConsumer. This would return a ShareGroupTxnContext containing
the MemberID and the specific record acquisitions. The KafkaProducer would then
use this context to trigger the AddShareAcksToTxn RPC.
Regarding your points on KIP-1191 discussion thread, this Option B approach
also allows us to solve the "Atomic DLQ" problem. If we include the DLQ write
in the Producer's transaction, the Share Coordinator can wait for the
WriteTxnMarker before finalising the move to the ARCHIVED state. This ensures
the 1:1 audit trail remains intact even during broker failures.
I have updated the KIP and will keep improving/rephrasing it to clarify
implementation: https://cwiki.apache.org/confluence/x/J448G
I am also starting to explore the broader challenge of Atomicity for
cross-cluster scenarios. While standard transactions are intra-cluster, I am
looking into "Acknowledgment Tokens" that could be recorded on a target cluster
and mirrored back to the source coordinator to finalize state. I’d be
interested to hear if the community sees value in laying the groundwork for
that in this KIP.
Regards,Shekhar Prasad Rajak,
On Sunday 1 March 2026 at 11:09:59 pm GMT+5:30, Shekhar Prasad Rajak via
dev <[email protected]> wrote:
Thank you Andrew for the review.
It took sometime to analyse the review comments and understand Kafka Producer
2PC implementation in depth.All the points are strongly valid and I have
updated the KIP to adopt the RPC already exists and leverage the same methods
for , CTP pattern (Consume-Transform-Produce) for kafka as sink or no producer
usecase.
Please have a look:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1289+Support+Hierarchical+Transactional+Acknowledgments+for+Share+Groups
Regards,Shekhar Prasad Rajak,
On Thursday 26 February 2026 at 03:32:36 am GMT+5:30, Andrew Schofield
<[email protected]> wrote:
Hi Shekhar,
Thanks for the KIP. Adding transactional support for acknowledgements is one of
the future features I'd like to see for share groups, so I'm glad you've
started this.
Here are some comments from an initial read.
AS1: Previously, Kafka transactions have always been a producer concept. You
set a configuration called `transactional.id` in the producer configuration
which gives authorization checking and helps with fencing. Then you use
KafkaProducer methods like beginTransaction and commitTransaction to mark the
start and end of the transaction. Only the producer is currently enabled for
transactional operation, and the Kafka RPCs it uses include transactional ID as
necessary to indicate operations which are transactional.
There will be differences for sure with share groups, but the overall model
should probably still hold. Over the years, we've gradually tightened up the
transaction protocol by adding features such as the producer epoch and
two-phase commit. I would rather not re-invent everything. For example,
WriteTxnMarkers is used by the transaction coordinator to tell the group
coordinator to complete transactional operations on the __consumer_offsets
topic. I would expect a similar kind of interaction with the share coordinator
on the __share_group_state topic.
I suggest looking at how to adapt the existing Kafka protocol RPCs rather than
defining new ones.
AS2: Why is there an acknowledge(ConsumerRecord<,>, AcknowledgeType, String)
method? I would have expected that the transactional ID is a configuration, not
a parameter of this method.
AS3: Have you considered how to write an application which uses a share group
and producer, and coordinates their operations in the same transaction?
Thanks,
Andrew
On 2026/02/24 08:54:36 Shekhar Prasad Rajak via dev wrote:
> Hello team,
> In this discussion thread we will finalise the design for the KIP-1289
> Support Hierarchical Transactional Acknowledgments for Share Groups.
>
> Share Groups currently support only immediate acknowledgement modes
> (IMPLICIT/EXPLICIT). This causes data loss in distributed streaming
> frameworks:
>
> 1. Worker acknowledges records → Records removed from Kafka
> 2. Checkpoint fails before sink write
> 3. Records lost (acknowledged but never persisted)
>
> Goal: Enable exactly-once read semantics via transactional acknowledgements.
> We already have similar pattern in Kafka producer transaction
>
> KIP: KIP-1289 Support Hierarchical Transactional Acknowledgments for Share
> Groups - Apache Kafka - Apache Software Foundation
>
> Looking forward to community's feedback!
>
>
> Regards,Shekhar Prasad Rajak
>
>
