morningman opened a new pull request, #68405: URL: https://github.com/apache/doris/pull/68405
Cherry-picked from #67347, #68193, #68336, #68170, #68180 Batch pick of every merged PR carrying the `incremental-computation` label that `branch-incremental-computation` does not have yet (no `incremental-computation-picked` label), in the order they landed on master (`git log --first-parent`). One commit per PR, each ending with `(cherry picked from commit <master sha>)`. Follows the same convention as #67830, #68017, #68073, #68151, #68236 and #68303. | # | Master commit | PR | Title | |---|---|---|---| | 1 | a7c0a7843a0 | #67347 | [fix](cloud) bind the packed slice location lifetime to its writer | | 2 | addf0c82974 | #68193 | [feature](ivm) Add the per-partition refresh state and its journal channel | | 3 | e885b4634df | #68336 | [refactor](ivm) Rename IvmInfo.refreshVersion to sequencePrefix | | 4 | 01efbca40a5 | #68170 | [fix](ivm) Fall back to complete refresh when the IVM stream is unusable | | 5 | d6c1a2b5f26 | #68180 | [fix](ivm) Choose IVM baseline rebuild partitions from the MV partition mapping | Not included on purpose: - The 29 labelled PRs that already carry `incremental-computation-picked` (every other closed PR with the label). This batch closes the selection query: 34 closed labelled PRs = 29 already picked + these 5. ### Prerequisite check - **The IVM series (#68193, #68336, #68170, #68180)** — one series by the same author, merged 2026-09-21/22, all tracing to issue #65418. #68193 says in its own description that it "adds the state that the following PRs need" — `MTMVPartitionState` plus its journal channel — and #68336 is the rename of the field sitting next to it. The PRs that consume that state are in this batch too, so the series is carried whole and in master merge order. Its earlier PRs (#67802, #67837, #67814, #68138, #67646, #67669, #67575) were picked in the previous rounds. - **#67347** is standalone (cloud packed-file slice lifetime). Every file that uses the APIs it changes is in the pick; `be/src/io/fs/packed_file_system.cpp`, which also reads the global slice index but is not modified, compiles unchanged against the picked headers. - **No unpicked master commit is required by any of the five.** Verified by compiling, not by inspection alone: the whole FE main + test tree compiles and the touched unit tests pass (below), and every BE/cloud file the picks touch passes a `-fsyntax-only` compile with the real build's flags. - The unlabelled master commits touching the same files (#67186 Hive partition batching, #66530 external scan task reuse, #66761 TIMESTAMP_NS, #67545 DLF, #67067 eager-agg) are *not* prerequisites — see the drift check: none of the picks' own added lines reference what they introduce. ### Drift check against master Per pick, `git show` of the branch commit against the master squash, with `index`/`@@` lines stripped: #67347, #68193 and #68336 are identical. Two differ, both mechanically: - **#68170 / `MTMVTask.java`** — master's context around `executePartitionBasedRefresh` carries #67186's `try` block and snapshot preload; this branch does not have #67186, so the auto-merge kept the branch's block. The pick's own added and removed lines are identical to master's. - **#68180 / `MTMV.java`** — master's squash also adds `import org.apache.doris.datasource.mvcc.MvccSnapshot;`, for #67186's `pinnedSnapshots` overloads of `calculatePartitionMappings` / `getEffectiveQueryUsedBaseTablePartitionMap`. Those overloads do not exist here and the pick's new code only calls the single-argument form, which exists on both sides, so the import is not needed and not carried. For every file this batch touches, `git diff upstream-apache/master -- <file>` was taken and each master-side line attributed to the unpicked commit that added it. All 42 files resolve: 37 have no master-side difference at all, and the 5 that do are fully owned by | file | master-side lines | owner | |---|---|---| | `MTMV.java` | 20 | #67186 | | `MTMVTask.java` | 46 | #67186, #66530 | | `MTMVTaskTest.java` | 44 | #67186 | | `MTMVPartitionUtil.java` | 65 | #67186, #67545 | | `MTMVRelatedPartitionDescSyncLimitGenerator.java` | 3 | #66761 | i.e. nothing belonging to the picks is missing, and no unlabelled commit has to come along. ### Verification - FE: `run-fe-ut.sh --run` on this branch (regenerates thrift/protobuf, compiles fe-core main 4480 files + test 1485 files) over the 11 touched test classes — `AlterMTMVTest` 25, `MTMVTest` 23, `MTMVTaskTest` 48, `MetaLockUtilsTest` 6, `IvmBaselineRebuildTest` 37, `IvmAggDeltaHandlerTest` 33, `IvmDeltaRewriteStateTest` 10, `IvmFailureReasonTest` 1, `IvmInfoTest` 6, `IvmSequenceCalculatorTest` 4, `DatabaseTransactionMgrTest` 20 — **213 tests, 0 failures, 0 errors, 0 skipped, BUILD SUCCESS**. - FE checkstyle on fe-core: 0 violations. - BE/cloud: `-fsyntax-only` with the Release flags of the real build and its own compiler (`/opt/homebrew/opt/llvm@20/bin/clang++`) on `be/src/io/fs/packed_file_manager.cpp`, `be/src/io/fs/packed_file_writer.cpp`, `be/src/cloud/cloud_rowset_writer.cpp` and `be/test/io/fs/packed_file_manager_test.cpp` (the last with `-DBE_TEST -fno-access-control`): no errors. - The three new regression suites parse (`test_ivm_baseline_marker_scope`, `test_ivm_chained_stream_scope`, `test_ivm_partitions_fallback_stream_unusable`); their `.out` files are the upstream ones, unmodified. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
