The GitHub Actions job "Flink Connector Kafka CI" on 
flink-connector-kafka.git/FLINK-33484 has failed.
Run started by GitHub user dalelane (triggered by dalelane).

Head commit for run:
9164260b9a74ac537ecab97ea5308a8aa395e628 / Dale Lane <[email protected]>
[FLINK-33484][Connectors/Kafka] Correct offsets committed to Kafka

Offsets that the Kafka source commits back to Kafka can lag the
consumer's true position, because entries that the consumer reads but
never delivers as records (e.g. transaction commit / abort markers, and
records of aborted transactions) were not accounted for. The result is
that the committed offset was stopping at the last delivered record,
while the consumer had advanced past that.

This wasn't impacting Flink processing guarantees, as restarts restore
from checkpointed offsets, not offsets committed to Kafka. External
tooling that uses the Kafka committed offset for monitoring would give
the appearance of consumer lag. On low-traffic or bursty transactional
topics, this would report lag that does not exist until new events are
received. A partition holding only aborted transactions, read from the
earliest offset, never had an offset committed at all.

This is now changed to reconcile the offset to commit with the
consumer's position:

- KafkaPartitionSplitReader now tracks, per partition, the offset of
  the last record handed to the reader, and the consumer position
  observed at the end of each fetch()
- At checkpoint commit, reconcileOffsetsToCommit advances the offset to
  commit to the consumer position when the reader is caught up to what
  it delivered, so the committed offset will reflect the non-record
  entries that the consumer read past. Reconciliation is skipped for
  partitions that aren't assigned and for running bounded splits
- Splits that have not emitted a record still hold a starting offset
  placeholder such as EARLIEST_OFFSET, so KafkaSourceReader has no
  offset to commit for them. Their partitions are passed to the split
  reader separately, which commits the consumer position for them
  unless records have been fetched that the checkpoint may not include
- The split reader remembers the offset it last committed for each
  assigned partition, and leaves a partition out of a commit rather
  than move its committed offset backwards. This can otherwise happen
  when a record is fetched between a checkpoint and its commit, so the
  offset can no longer be advanced. The offset is recorded when the
  commit is sent, so that a later commit cannot overtake one that is
  still in flight
- A finished bounded split is committed at its stopping offset, rather
  than after its last emitted record, as it only finishes once the
  consumer position has reached the stopping offset. This only applies
  once the split holds a real offset: a split that is still at a
  starting offset placeholder may have been empty when it was assigned,
  with the consumer already past its stopping offset, so committing the
  stopping offset could move the offset backwards
- Per-partition tracking state is cleaned up when partitions are
  unassigned
- KafkaSourceReader records the committed-offset metric from the
  offsets that are committed (rather than those requested) so Kafka and
  Flink's committedOffsets metrics agree.

Generated-by: Claude Code (Sonnet 5)
Signed-off-by: Dale Lane <[email protected]>

Report URL: 
https://github.com/apache/flink-connector-kafka/actions/runs/36937151384

With regards,
GitHub Actions via GitBox

Reply via email to