0AyanamiRei opened a new pull request, #68357:
URL: https://github.com/apache/doris/pull/68357
### What problem does this PR solve?
Issue Number: None
Related PR: #67678, #67696, #67869, #67874, #67879
Problem Summary: Kinesis shards can split and merge while a routine load job
is running. Maintaining only open/closed shard lists and sequence progress
cannot express parent-child lineage or the COMMITTED-to-VISIBLE completion
barrier. That can schedule children too early, lose lineage, or rediscover
completed parents. This PR adds a durable shard topology state machine,
concrete initial-position persistence, idempotent journal/image replay, and
Kinesis BE progress handling for EOF and ChildShards metadata. It also adds
FE/BE tests and deterministic upstream mock coverage.
### Release note
Add parent-child shard topology tracking and visibility-gated scheduling for
Kinesis Routine Load.
### Check List (For Author)
- Test: Unit Test
- `./run-be-ut.sh --run --filter='*Kinesis*' -j48`
- Result: 18 Kinesis tests passed, including the two deterministic
upstream mock tests.
- FE unit tests and real AWS regression tests were not run in this
environment.
- Behavior changed: Yes (Kinesis Routine Load now schedules shards through
the topology state machine.)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
