0AyanamiRei commented on PR #67678: URL: https://github.com/apache/doris/pull/67678#issuecomment-5758545814
Absorbed into the Kinesis shard topology change, closing this draft. Production code from this PR is in the working branch (`wip/kinesis-shard-topology-20260916`): - FE: `Config.kinesis_latest_sequence_timeout_second`, `KinesisUtil.getLatestSequenceNumbersAsync`, `KinesisLatestPositionOperation`, `OP_KINESIS_LATEST_POSITION` + `EditLog`/`JournalEntity` wiring, the `prepare()` scan lifecycle (submit, deadline, cancel, all-or-nothing publish), `unprotectUpdateState` cancellation, and the `divideRoutineLoadJob` gate. - BE: `kinesis_latest_sequence_scan_threads` / `kinesis_latest_sequence_request_timeout_ms`, the dedicated `kinesis_latest_scan` pool, `KinesisLatestSequenceBatch`, `get_latest_sequence_number`, and the direct dispatch in `PInternalService::get_info`. - `be/test/runtime/kinesis_latest_sequence_test.cpp` is byte-identical to this PR's version. Deliberate differences in the absorbed form: - `PKinesisMetaProxyResult.shard_latest_sequences` is field **2**, not 3: the deprecated `shard_ids` field was removed rather than kept, since this is a new feature with no mixed-version requirement. `PShardInfo`/`KinesisMetaProxyResult` numbering therefore differs from this PR. - `prepare()` only resolves once `KinesisShardTopology.initialSnapshotFinalized` is set, and the unresolved set comes from `shardTopology.getUnresolvedLatestShardIds()` instead of the open-shard list, so a child that is still `PENDING_PARENT` at snapshot time is resolved before the first task as well. - `replayLatestPosition()` resolves the positions into the topology nodes and `KinesisProgress` in the same job write lock, and never overwrites an already concrete committed position. - "An older BE returns no position map" is now a plain error; there is no fallback path. The regression suite is added verbatim as `regression-test/suites/load_p0/kinesis_routine_load/test_kinesis_routine_load_latest_restart.groovy` plus its `.out` file. It has never been executed: it needs AWS credentials and a docker cluster and self-skips without them, which is why this PR left its regression-test checkbox unchecked. Its first real run will be in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
