Doris-Breakwater commented on issue #70540: URL: https://github.com/apache/doris/issues/70540#issuecomment-6840997784
Breakwater-GitHub-Analysis-Slot: slot_f86f340bc8b3 **Triage:** This warrants high-priority data-correctness handling in streaming-job target validation and commit/offset integration. The supplied reproduction is sufficient for initial triage. Independent inspection of the exact commits cited in the report supports the proposed mechanism; this investigation did **not** reproduce it against a running Doris/Lakekeeper deployment. The issue currently has no labels. **Verified in source** - In `917b20be838` (the reported 4.1.5 revision), the OLAP executor registers the streaming task's `InsertLoadJob` and sets the transaction callback to its parent job. The Iceberg executor instead inherits the external executor's unregistered-load-job construction and commits through the catalog transaction manager. That manager commits the external transaction and removes its bookkeeping entries, without invoking the Doris load-transaction callbacks. See [executor selection](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L452-L502), [external commit](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BaseExternalTableInsertExecutor.java#L62-L131), and [external transaction manager](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf 7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/transaction/AbstractExternalTransactionManager.java#L41-L55). - The missing callbacks supply both the cancellation check and the committed offset/statistics attachment. `afterCommitted()` advances the offset, and `replayOnCommitted()` restores it. Simply calling these callbacks from the external executor would be insufficient: `beforeCommitted()` requires a registered load job and otherwise throws `load job not found`. Successful-task counters can still increase independently of committed load statistics. See [callbacks](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java#L1331-L1397). - A successful external insert returns OK; task success creates the next task without advancing the source offset. With `currentOffset == null`, S3 batch selection starts from the beginning and `hasMoreDataToConsume()` remains true. This explains the reported repeated first batch and append snapshots. The rapid successful-task chain also means `max_interval=10` is not a mandatory ten-second gap between these tasks. See [task-success scheduling](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java#L882-L929) and [S3 offset provider](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3SourceOffsetProvider.java#L58-L200). - Streaming-job creation initializes the INSERT plan without a streaming-specific target-type restriction. At master commit `172120cbaa4`, the connector executor still inherits the catalog commit path, so the architectural gap remains visible in source. This is **not** a runtime confirmation on master. See [creation validation](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateJobInfo.java#L325-L349) and [master external commit](https://github.com/apache/doris/blob/172120cbaa46b0a2eff6c37d86fb438c3a307375/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BaseExternalTableInsertExecutor.java#L100-L140). **Scope and remaining uncertainty** The evidence supports an accepted-but-unsupported target/commit integration defect, regardless of whether external streaming targets were intended to be supported. It does not establish a Lakekeeper fault. Other external sinks are candidates for the same missing-offset problem, but their write and cancellation semantics differ; they need individual checks before declaring them reproduced. The missing cancellation fence explains how a pause can race with an already-finishing write; the exact observed pause/commit ordering still needs timestamps. **Recommended maintainer action** 1. For containment, restrict streaming INSERT targets to internal OLAP targets with the required transaction integration. Apply the check at CREATE, SQL-changing ALTER, and before task writes so persisted jobs are covered. Use a clear, non-retryable unsupported-target failure. ALTER already prevents changing the qualified target name; revalidation must still check the resolved target's capability. 2. Add coverage for external-target rejection, an existing persisted external-target job after upgrade, ALTER validation, and continued internal-table offset/replay correctness. If external support is implemented, additionally test multi-batch uniqueness, pause near commit, and FE failure immediately after remote commit but before local offset persistence. 3. Treat Iceberg support as a separate commit/recovery design. Advancing an in-memory offset or journaling it after remote commit leaves a replay/duplicate window. An atomic Iceberg commit marker plus recovery and retry deduplication is a possible direction, but snapshot summaries alone do not establish exactly-once behavior; job identity, concurrent writers, snapshot retention/rollback, ambiguous commit results, and cancellation ordering need defined behavior. **Useful follow-up evidence, not a blocker for triage:** full FE/BE build identifiers and deployment mode; sanitized job/task rows with job, task and query IDs; FE logs spanning two consecutive successful tasks and the pause; Iceberg snapshot IDs, parent IDs, timestamps and summaries for that interval. Correlate the repeated `RunningOffset` with distinct committed snapshots. A runtime check on the cited master revision would establish its affected status. No query profile is needed for the current correctness diagnosis. **Immediate mitigation:** keep the affected external-target job paused or drop it, and verify that in-flight writes have settled before assessing cleanup. Do not resume it expecting a different interval or database context to repair the offset. Preserve snapshot lineage and counts first; rollback to a prior snapshot also removes legitimate intervening writes and needs an explicit cleanup decision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
