0AyanamiRei commented on PR #67879: URL: https://github.com/apache/doris/pull/67879#issuecomment-5758010734
Superseded by the Kinesis shard topology change, closing this draft without merging it as-is. The problem this PR fixes is real and is covered, but the marker it introduces would duplicate topology state: - This PR adds a job-level `kinesisInitialPositionSet` flag so that an empty progress map is no longer read as "first setup". - The topology change replaces that signal entirely: `KinesisShardTopology.initialSnapshotFinalized` freezes the initial shard snapshot, and each node keeps its own `initialStartPosition`. A shard discovered after the snapshot gets `TRIM_HORIZON`; the initial snapshot keeps the configured default position. Neither decision consults the progress map, so "progress became empty because the consumed parent was dropped" can no longer change the outcome. - The child-lineage path from `GetRecords.ChildShards` resolves to the same `TRIM_HORIZON`. The regression is covered by the new tests: - `KinesisShardTopologyTest.testShardDiscoveredAfterInitialSnapshotUsesTrimHorizon` (parent consumed, children discovered afterwards, configured default is `LATEST`), - `KinesisShardTopologyTest.testChildShardLineageUsesTrimHorizonAfterParentCompletion` (lineage path, still waiting for the parent's visibility barrier), - `KinesisRoutineLoadJobTest.testShardDiscoveredAfterInitialSnapshotUsesTrimHorizonInProgress` (the same scenario at job level, asserting the position that reaches progress). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
