yujun777 opened a new pull request, #70242: URL: https://github.com/apache/doris/pull/70242
### Background OLAP table streams track consumption separately for each base-table partition. Each partition can carry a consumed offset, its last consumption time, and a historical snapshot TSO until initial rows are consumed. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Refactor Problem Summary: These values currently live in three maps keyed by the same partition ID, requiring repeated lookups and coordinated updates and cleanup. Replace them with one `Map<Long, PartitionStreamState>` whose immutable values hold the complete partition state. Update commit validation and tests that inspect the old fields. Query and transaction semantics are preserved; the unreleased stream image format changes without migration. ### Details - **Change map:** 150 changed production lines in the state holder and commit validation, and 189 changed test lines adapting fixtures and covering persistence and offset validation. No generated code or documentation changes. - **Moves vs. logic changes:** Consolidates the three maps and replaces each partition state in one operation at commit. Cleanup traverses and removes entries from one map. Both consumption display paths share the existing snapshot conversion. - **Behavior change:** None for reads, conflict detection, or consumption commits. History-only state still has no consumed offset, and the empty-partition sentinel `-1` still does not count as consumed data. - **Compatibility:** Stream image metadata uses the new aggregate state format. No compatibility handling for the old three-map image format is added because table streams have not been released. Transaction and cleanup journal formats are unchanged. - **Performance impact:** State lookup and updates remain O(1); stale partition collection traverses one map rather than three. No benchmark was run. - **Rollback:** Revert before persisting streams in the new image format; old code cannot read this new unreleased stream metadata. ### Testing Locally passed 40 FE unit tests across: - `OlapTableStreamUpdateTest`: historical and incremental conflicts, missing previous offsets, new partitions, image round-trip, consumption transition, and pruning. - `OlapTableStreamConsumptionTest` and `OlapTableStreamSnapshotReadTest`: lock release during selector evaluation and snapshot read semantics. - `TableStreamManagerCleanupTest`: cleanup, races, and journal replay. - `TableStreamOffsetTransactionTest` and `InsertIntoTableCommandTableStreamTest`: transactional offset validation and updates and planner integration. FE Checkstyle validation passed with `mvn -f fe/pom.xml -pl fe-core -am validate -DskipTests`. ### Release note None ### Check List (For Author) - Behavior changed: No (unreleased image format change described above) - Does this need documentation: No - Pre-submission status: Ready -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
