Doris-Breakwater commented on issue #70278: URL: https://github.com/apache/doris/issues/70278#issuecomment-6654214362
Breakwater-GitHub-Analysis-Slot: slot_37086109f6df **Assessment:** The reported partial-update failure is supported by the source at `72e91896bbdd8760f8bd4e488e5672406b40ce52`. This is a high-confidence correctness issue in the MoW DETAIL before-image contract, with potential silent errors for consumers that retract old values. **This investigation did not execute the SQL reproduction or a Flink job**, so the displayed result and downstream impact remain predictions from the verified code path. Scope is the cited revision; affected release versions have not been established. The issue currently has no labels, assignee, or milestone; suggested routing is changelog/Table Stream maintainers, with correctness priority. **Verified code path** 1. Both admission checks restrict historical-value requirements to MIN_DELTA: [`BindRelation.checkChangeScanCondition()`](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java#L755-L795) and [`OlapTable.checkAsTableStreamBaseTable()`](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java#L4351-L4372). DETAIL passes these checks without history. [`generateTableChangelogSchema()`](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java#L2554-L2582) creates value-column before images only when historical values are enabled. 2. Disabling before images does **not** prevent partial updates from becoming UPDATE records. [`changelog_needs_historical_lookup()`](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/transform/changelog_derive.cpp#L303-L314) still selects historical lookup for direct partial updates. For an existing live key, [`retrieve_historical_row()`](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/segment/historical_row_retriever.cpp#L112-L135) records UPDATE. The fixed-partial-update derivation [reconstructs the complete after image](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/transform/changelog_derive.cpp#L406-L435), while [`emit_changelog_block()`](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/transform/changelog_derive.cpp#L257-L297) persists before values only if `write_before` is true. SQL UPDATE can ent er this path when its [partial-update rewrite conditions](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/UpdateCommand.java#L187-L195) hold; this is not a claim about every UPDATE statement. 3. The [DETAIL reader](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/iterator/block_reader.cpp#L370-L417) unconditionally expands stored UPDATE into UPDATE_BEFORE and UPDATE_AFTER, and uses the before mapping for DELETE. [`_append_change_row()`](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/iterator/block_reader.cpp#L194-L209) copies through that mapping; [`_init_before_column_ordinals()`](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/schema.cpp#L197-L216) falls back to the same column when its before column is absent. Thus, for the supplied partial update, the code predicts both emitted update rows contain `(1,20,10,10)`, instead of a before row `(1,10,10,10)`. For DELETE, the same mechanism returns the stored delete payload; exact non-key values depend on the deletion path and need a separate reproduction. **Coverage gap and recommended direction** Existing tests explicitly accept [DETAIL stream creation without history](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/regression-test/suites/table_stream_p0/test_create_append_only_stream_without_historical_value.groovy#L49-L63). The [no-history incremental-read test](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/regression-test/suites/table_stream_p0/test_min_delta_stream.groovy#L890-L940) uses a full-row upsert and expects INSERT output. That exercises a different write path and does not validate partial-update before images. These are inspected test expectations, not tests run during this investigation. The least ambiguous fix direction is to reject MoW DETAIL without historical values at both direct `@incr` analysis and stream creation, and cover reads through already-created detail streams. Preserve Duplicate Key append semantics and APPEND_ONLY behavior. Maintainers should explicitly resolve the existing acceptance tests and document the resulting contract. An alternative after-image-only mode needs a defined consumer contract, including DELETE handling; merely suppressing UPDATE_BEFORE does not provide old values for retractions. Do not remove the schema's identity fallback globally: unchanged key and metadata columns legitimately use it. **Evidence still needed / next steps** - Run the supplied SQL on the cited build, record the actual timestamp bounds within changelog retention, and capture base-table rows plus DETAIL output including OP, TSO, LSN, and all values after both writes are visible. The decisive check is whether the update's `op=2` and `op=3` rows both have `v1=20`. A corresponding table created with history enabled should instead yield `v1=10` and `v1=20`. - Repeat through a detail stream created before the writes with `show_initial_rows=false`. Add separate cases for SQL UPDATE that takes the partial-update rewrite, flexible partial updates, and DELETE against an existing row. Keep full-row upsert and Duplicate Key controls to avoid conflating their semantics. - For a consumer-level claim, supply connector/Flink versions, the source configuration, emitted RowKinds, and a small aggregation result demonstrating the incorrect retraction. A Doris Profile or broad production log dump is not required for the initial SQL correctness check; provide query errors/plans if the reproduction behaves differently. Until the contract is fixed, avoid using no-history MoW DETAIL as a source of old-row retractions. A replacement table intended for that use should be created with `changelog.need_historical_value=true` and given a fresh, consistent consumer baseline. At this revision, [`ChangelogConfig.alterFromProperties()`](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/catalog/ChangelogConfig.java#L93-L113) rejects changing this setting, so enabling it in place is not an available workaround; previously unrecorded before images cannot be recovered from these changelog rows. Switching to MIN_DELTA on the same no-history table is also rejected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
