morningman opened a new issue, #65418:
URL: https://github.com/apache/doris/issues/65418
### Description
This issue tracks the end-to-end incremental computation capability in
Apache Doris.
The implementation is being built in three layers:
```text
OLAP table DML
-> row binlog: durable row-level changes, before images, operation type,
LSN/TSO
-> Table Stream: change-query semantics, per-partition consumption
offsets, snapshots and reset
-> MTMV incremental maintenance: derive and transactionally apply only
the affected result changes
```
This tracker replaces the stale umbrella issue #57921 and incorporates the
row-binlog foundation originally tracked by #61956.
### Goals
- Provide a durable row-level change log for Doris internal tables.
- Expose deterministic incremental consumption through Table Stream.
- Provide snapshot reads aligned with stream consumption points for joins
and recovery.
- Integrate Table Stream with MTMV planning and refresh execution for
row-level incremental maintenance.
- Preserve correctness across replicas, compaction, schema changes,
concurrent consumers, failures, retries, and restart/replay.
- Add complete unit, regression, compatibility, and failure-recovery
coverage.
- Document supported table models, SQL syntax, semantics, limitations, and
operational guidance.
### Scope boundary
This tracker covers the internal row-binlog and Table Stream stack used by
incremental computation, plus its MTMV integration. External CDC Streaming Jobs
and general MTMV partition refresh or query-rewrite changes are out of scope
unless they directly modify this stack.
The initial repository review covered 2026-04-09 through 2026-07-09. PR
#61382 predates that window but is included because it is the Table Stream
metadata and DDL foundation.
### Progress
#### Row binlog foundation
- [x] #62058 — define row-binlog metadata and schema.
- [x] #63110 — implement row-binlog read and write paths.
- [x] #63643 — implement replica clone, scheduling, and compaction.
- [x] #64249 — initialize cumulative policy safely during binlog scheduling.
- [x] #64545 — fix incomplete binlog metadata in the grouped rowset writer
path.
- [x] #65032 — prevent a null cumulative policy when binlog compaction runs
first.
- [ ] #64133 — allocate LSNs at the sink and keep LSN/TSO semantics
consistent across replicas.
- [ ] #64674 — factor row-binlog derivation and historical-value lookup into
the segment-writer transform chain.
- [ ] #65076 — include hidden key columns while excluding hidden non-key
internal columns.
#### Table Stream foundation
- [x] #61382 — add stream metadata, DDL, and information-schema visibility.
- [x] #62453 — add stream query and transactional consumption infrastructure.
- [x] #63850 — connect OLAP Table Stream to row binlog and add DETAIL,
APPEND_ONLY, and MIN_DELTA scan modes.
- [x] #64776 — add stream snapshot/reset reads, table incremental reads, and
time-travel snapshot support.
- [ ] #65274 — expand Table Stream negative and exception-path regression
coverage.
#### MTMV incremental maintenance
No PR implementing the row-level MTMV integration was identified as of
2026-07-09. The existing `PartitionIncrementMaintainer` supports
partition-aware MTMV refresh analysis; it is not the row-level
incremental-computation runtime tracked here.
- [ ] Define supported MTMV query shapes and eligibility/fallback rules.
- [ ] Derive delta plans from Table Streams, including snapshot sides for
joins.
- [ ] Define incremental semantics for projection, filter, join, union,
aggregation, and supported expressions.
- [ ] Integrate stream creation and lifecycle with MTMV creation,
alteration, pause/resume, and drop.
- [ ] Execute result changes and stream-offset advancement atomically.
- [ ] Support full-refresh bootstrap/reset and fallback when incremental
maintenance is not valid.
- [ ] Define retry, idempotency, concurrent refresh, failover, and EditLog
replay behavior.
- [ ] Add observability for refresh mode, consumed offsets, lag, fallback
reason, and failures.
- [ ] Add FE and BE unit tests plus end-to-end regression tests for
correctness and recovery.
- [ ] Add user and operator documentation.
### Correctness and compatibility checklist
- [ ] DUP and supported UNIQUE KEY MOW table semantics are documented and
tested.
- [ ] Full update, partial update, delete, sequence column, hidden key, and
schema-change paths are covered.
- [ ] LSN/TSO ordering is deterministic across replicas and remains valid
after clone and compaction.
- [ ] Concurrent stream consumption either commits data and offsets together
or fails without advancing offsets.
- [ ] Snapshot, reset, and incremental reads agree at partition boundaries
and after restart.
- [ ] Binlog retention cannot silently invalidate an active stream or MTMV;
stale state is observable and recoverable.
- [ ] Unsupported plans fail clearly or use a documented full-refresh
fallback.
- [ ] Cloud-mode support or exclusion is explicit.
- [ ] Upgrade, downgrade, rolling restart, backup/restore, and replication
behavior is defined before general availability.
### Overlapping or superseded PRs found during review
These PRs are relevant to the same row-binlog failure paths but are not
separate remaining deliverables:
- #64137 — closed; its compaction-policy crash path is covered by #64249.
- #64484 and #64495 — overlap with the metadata and nullable-column fixes
merged in #64545.
### Exit criteria
- Row binlog and Table Stream semantics are stable and documented.
- At least one supported MTMV query class refreshes incrementally end to end
with atomic offset advancement.
- Correctness, failure recovery, replay, compaction, and schema evolution
tests pass in the supported deployment modes.
- Unsupported cases have explicit diagnostics and a documented fallback.
- Remaining limitations and follow-up work are tracked by linked issues or
PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
