0AyanamiRei commented on PR #67869:
URL: https://github.com/apache/doris/pull/67869#issuecomment-5758008869
Absorbed into the Kinesis shard topology change, closing this draft.
The fix landed in the working branch (`wip/kinesis-shard-topology-20260916`):
- `data_consumer.{h,cpp}`: `KinesisQueueItem`, no progress update in
`_process_records`, `Status::OK()` on a batch-boundary queue close, the ordered
`end_of_shard` marker, and removal of `_committed_sequence_numbers` /
`_closed_shard_ids`.
- `data_consumer_group.{h,cpp}`: `_candidate_sequence_numbers` is updated
only after a successful pipe append, `_closed_shard_ids` is filled from the
marker, and `_on_finish` publishes both.
- `be/test/runtime/kinesis_batch_progress_test.cpp` is byte-identical to
this PR's version.
- `KinesisQueueItem` additionally carries `child_shard_parent_ids`, a
superset of this PR's struct, so the split/merge lineage travels behind the
same ordered marker.
The commit boundary introduced here is the BE half of the new state machine:
the FE reads `closedShardIds` as "the end marker was consumed" and only
completes the parent after the transaction visibility barrier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
