englefly opened a new pull request, #69308:
URL: https://github.com/apache/doris/pull/69308
### What problem does this PR solve?
Issue Number: DORIS-29796
Related PR: #62828 (source of the pick), #61146 (source of the dependency
fix), #68211
Problem Summary: backport the CTE empty/cascading pruning fix #62828 to
branch-4.2, plus the
`ColumnPruning` CTE protection that #62828 implicitly depends on. Without it
the three branch-4.2
RQG cases tracked by DORIS-29796 (6540158413 / DORIS-25384, 6540160970 /
DORIS-25408,
6540160107 / DORIS-25409) keep failing.
#### Why #62828 alone is not enough on branch-4.2
`ColumnPruning` on branch-4.2 only protects `LogicalCTEProducer` with
`skipPruneThis` and has no
`visitLogicalCTEAnchor`. When the required slots of an anchor do not include
the producer outputs
(the normal case, because consumers reference the producer slots directly),
the producer subtree is
pruned with an empty required set and `LogicalProject` materializes the
empty projection as the
dummy `1 AS 1` projection. The rewrite plan therefore becomes
```
LogicalCteAnchor
|--LogicalProject ( projects=[1 AS 1] )
| +--LogicalCteProducer
+--...
```
which (a) breaks the `anchor.left() instanceof LogicalCTEProducer` invariant
that #62828's
`normalizeCtePlan`/`refreshCteContext` relies on (`ClassCastException`), and
(b) is an invalid CTE
structure: the materialized columns no longer match the slot mapping the
consumers were built from,
which is what surfaces as `VSlotRef ... have invalid slot id` in case
6540160107.
Master has had the protection since #61146 (`visitLogicalCTEAnchor` +
`skipPruneThisAndFirstLevelChildren`); this PR ports exactly that hunk.
#### Hunk audit of the #62828 pick (source commit `c75eba5488f`)
| Source file | Disposition |
| --- | --- |
| `nereids/jobs/executor/Optimizer.java` | Ported verbatim
(`normalizeCtePlan`, `refreshCteContext`, `containsCte`,
`generateFilterPushedDownInlinedPlan`, method rename). |
| `nereids/rules/rewrite/CTEInliner.java` | Ported verbatim
(`inlineByCurrentConsumerCount`, `InlineResult`, zero-consumer anchor removal,
recursive anchor trimming). |
| `nereids/StatementContext.java` | Not applied: branch-4.2 already has
`clearCteEnvironment()` from the multi-mode CTE inline work (#68211), and its
variant also clears `consumerIdToLimitRows`. |
| `nereids/rules/rewrite/ClearContextStatus.java` | Not applied: 4.2 keeps
the explicit per-map clears, which are a superset of what the picked code
needs. |
| `qe/SessionVariable.java` | Not applied: 4.2 keeps
`@VariableMgr.VarAttr(..., alias = "cbo_cte_inline_mode")`; the source commit
only dropped the alias. |
| `.../CTEInlineTest.java` | Ported verbatim, including the new
`refreshCteConsumersAfterNormalizeEliminatesEmptyBranch`. |
| `regression-test/suites/query_p0/cte/test_cbo_cte_inline_prune.groovy` |
Ported verbatim, path adapted to the 4.2 layout (`suites/nereids_p0/cte/`;
`suites/query_p0/cte` does not exist on 4.2). |
| `regression-test/data/query_p0/...` baselines | Paths adapted to the 4.2
layout (`data/nereids_p0/repeat/`, `data/nereids_rules_p0/unique_function/`);
content unchanged. |
The dependency fix is a separate commit, ported from `5de1ed88d4b` (#61146)
and limited to the
`ColumnPruning` hunk (12 added lines): CTE anchors and producers are no
longer column pruned and
keep their first level children's full output requirement.
No BE changes.
### Release note
None
### Check List (For Author)
- Test
- [x] FE unit test: `./run-fe-ut.sh --run
org.apache.doris.nereids.rules.rewrite.CTEInlineTest`
→ `Tests run: 2, Failures: 0, Errors: 0` (includes the new
`refreshCteConsumersAfterNormalizeEliminatesEmptyBranch`).
- [x] FE build: `./build.sh --fe` → `BUILD SUCCESS` on this branch
(checkstyle included).
- [x] Original RQG cases replayed against a local 4.2 cluster (FE+BE
built from this branch,
BE alive), using the full DDL/INSERT/session settings from the
DORIS-29796 attachment
`repro-cte-empty-42.sql` (database renamed to a hyphen-free name so
branch-4.2's parser
accepts it; no other change):
| case | expected | branch-4.2 before | this branch |
| --- | --- | --- | --- |
| 6540158413 | 0 rows | `size of sources must be greater than 0` | 0
rows, no error |
| 6540160970 | 0 rows | `size of sources must be greater than 0` | 0
rows, no error |
| 6540160107 | 0 rows | `VSlotRef col_int_undef_signed have invalid
slot id: 15` | 0 rows, no error |
With only the #62828 commit replayed, cases 6540158413/6540160970
return 0 rows and
6540160107 still fails (`ClassCastException: LogicalProject cannot be
cast to
LogicalCTEProducer`), which is why the `ColumnPruning` commit is part
of this PR.
- [x] Candidate 6540166225 from the DORIS-29796 evidence could not be
replayed: the evidence
attachment only carries its error text, not its SQL, so there is no
input to verify locally.
- [x] Regression (local 4.2 cluster, JDBC `127.0.0.1:29731`):
- `-d nereids_p0/cte` → 9 suites, failed 0 (includes the new
`test_cbo_cte_inline_prune`)
- `-s test_repeat_output_slot` → pass (baseline updated by the pick)
- `-s agg_with_unique_function` → pass (baseline updated by the pick)
- `-d nereids_rules_p0/cte` → 3 suites, failed 0
- `-d nereids_rules_p0/cte_limit_pushdown` → 1 suite, failed 0
- `-d nereids_rules_p0/column_pruning` → 9 suites, failed 0
- Introspective analysis
- [x] `normalizeCtePlan` iterates to a fixpoint instead of comparing
`Plan.equals()`, because
`LogicalCTEAnchor`/`LogicalSubQueryAlias` ignore children in
`equals()`.
- [x] Zero-consumer anchors are removed before memoization; anchors with
an empty producer or
consumers below `inline_cte_referenced_threshold` are inlined.
- [x] CTE environment (`cteIdToProducer`, `cteIdToConsumers`, rewritten
caches) is refreshed
from the normalized plan so memo construction does not see stale CTE
state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
