liaoxin01 opened a new pull request, #70301: URL: https://github.com/apache/doris/pull/70301
### Background Group commit producers build blocks using the insert plan on their FE. The shared load queue and WAL header use the internal consumer plan generated by the master FE. Blocks already contain the computed synchronous materialized-view columns when they reach the queue. ### What problem does this PR solve? Issue Number: None Related PR: #69660 (alternative fix for group commit column layout), #70160 (upstream WAL replay compatibility for duplicate column IDs) PR Type: Bugfix Problem Summary: Different FEs can enumerate synchronous materialized-view columns in different orders. A producer block such as `[base, mv_int, mv_string]` can therefore reach a consumer expecting `[base, mv_string, mv_int]`, causing a bad cast or silently swapped same-type values. This PR aligns each producer's block with the consumer schema before queue insertion and WAL serialization. Most of the diff adds layout and replay coverage. ### Details - **Change map:** 104 changed lines in four production files, 402 lines in three test files, 25 generated regression-result lines, and no documentation changes. The sink changes (33 lines) cache and apply the permutation; queue changes (71 lines) retain the consumer schema and match columns. - **Frequency:** Conditional on producer and consumer plans having different column orders; multiple synchronous materialized views and inserts through another FE expose this. - **Root cause:** FE metadata reconstruction can change index iteration order and hence the materialized-view tail of the full schema. Equal schema versions and index counts do not imply equal block layouts. The consumer and WAL header assume the consumer's order. - **Why existing tests missed it:** Existing coverage did not combine differing FE column layouts, same-type MV columns, and WAL replay with repeated index-local column IDs. - **Fix:** Store the consumer slot descriptors before publishing the queue. Match producer columns by name and non-nullable type, consuming each match once, and cache the resulting permutation per producer. Reorder column references before buffering, enqueueing, and writing WAL; identity layouts skip reordering. Name/type/count mismatches return an explicit data-quality error. Column IDs cannot identify columns here because MV IDs are index-local. - **Other paths and branches:** This targets master and reuses #70160 for duplicate-ID WAL replay; there is no production WAL Reader change in this PR. FE planning, wire protocols, and the WAL format are unchanged. Release-branch backports need the corresponding Reader compatibility fix. - **Behavior change:** Group commit accepts differing producer/consumer column orders without changing values. Incompatible schemas fail explicitly instead of being interpreted positionally. ### Testing Local checks on the rebased branch: - 19 ASAN tests passed with `./run-be-ut.sh --run --filter='GroupCommitBlockSinkLayoutTest.*:WalReaderV2Test.*:WriterBacked/WalReaderV2ProjectionTest.*' -j 32`. - Coverage includes mixed FE layouts, same-type value swaps, shared MV expressions with different index-local IDs, shadow-column type matching, incompatible schemas, and replay of two WAL blocks with repeated IDs and differing MV tails. - clang-format 16, build hygiene, changed-file clang-tidy, and whitespace checks passed. - The local UT linker used an uncommitted glibc compatibility adapter for the installed toolchain; production linker settings were unchanged. Integration checks passed on the pre-rebase revision in an isolated two-FE/one-BE ASAN cluster: - The existing six-MV `multiple_ssb` scenario passed through both master and follower. EXPLAIN confirmed identical base columns but different MV column orders. - Base-only and the new three-MV WAL replay regressions passed. Failed original transactions remained aborted; replay restored five rows and correct results in all three views. - Preserved WAL files with repeated column IDs initially failed under the strict Reader, then replayed successfully after the compatibility fix, with file hashes unchanged. The rebase replaces that local Reader fix with upstream #70160. Current-source unit tests passed afterward; live-cluster integration has not been rerun after the rebase. CI is pending. ### Release note Fix group commit failures and incorrect column values when inserts and internal loads use different synchronous materialized-view column orders across FEs. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: current-source ASAN unit and static checks passed; two-FE and WAL replay integration passed before rebase and has not been rerun afterward; CI pending. ### AI Disclosure - Tool: Codex - Generated: Code, tests, and description - Human review: Pending author review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
