englefly opened a new pull request, #68340:
URL: https://github.com/apache/doris/pull/68340
### What problem does this PR solve?
Problem Summary:
A query that references a materialized CTE more than once and whose
references need different row
ranges silently loses rows when runtime filters are enabled: the runtime
filters generated for one
reference are pushed into the shared CTE producer scan and prune the rows
that the other references
still need. The result is empty or shorter than expected.
Reproduction on internal OLAP tables (no external table needed):
CREATE TABLE f (k INT) ... ; INSERT INTO f VALUES (0), (1), (5), (10);
CREATE TABLE b (x INT) ... ; INSERT INTO b VALUES (3);
SET enable_cte_materialize=true;
SET inline_cte_referenced_threshold=0;
SET runtime_filter_type='MIN_MAX';
SET runtime_filter_mode='GLOBAL';
WITH t AS (SELECT k, ABS(k) AS v FROM f)
SELECT c2.k AS lo, c1.k AS hi, c2.v AS lo_v, c1.v AS hi_v, b.x
FROM t c2 CROSS JOIN t c1 CROSS JOIN b
WHERE c1.k > b.x AND c2.k < b.x
ORDER BY lo, hi;
The query returns no row at all, while `runtime_filter_mode='OFF'` returns
the four rows
`(0,5) (0,10) (1,5) (1,10)`. The plan shows both consumer-side filters
installed on the shared
producer scan:
0:VOlapScanNode ... runtime filters: RF003[min] -> k, RF004[max] -> k
Root cause in code:
`RuntimeFilterGenerator` moved the runtime filters of the CTE consumers into
their shared producer
whenever the consumers' `srcExpr` sets intersect, and removed them from the
consumers. The code that
moved them only checked that the filters map to the same producer target
expression, not that they are
the same filter. Here `c1.k > b.x` generates a
`MIN_MAX/MIN` filter (k >= min(x)) while `c2.k < b.x` generates a
`MIN_MAX/MAX` filter (k <=
max(x)) on the same producer column, so both were installed on the shared
scan and their
conjunction (`k >= 3 AND k <= 3`) pruned every row.
Fix:
Decide per filter identity instead of per source expression:
`pushRuntimeFiltersIntoCTEProducer()`
collects the runtime filters of the CTE consumers, and
`selectPushableRuntimeFilters()` selects the
groups that may be pushed. The filters of one source expression
are grouped by identity -- same `TRuntimeFilterType`, same
`TMinMaxRuntimeFilterType` and same
target expression on the producer, that is the same predicate on the
producer's rows -- and each
group is pushed on its own when every consumer of the CTE applies a filter
of that identity. Only
filters that all consumers would apply anyway are applied on the shared
producer, which keeps the
pushdown equivalent to the per-consumer filters:
- `t c1 where c1.k > b.x` and `t c2 where c2.k < b.x`: the MIN group is
applied by one consumer and
the MAX group by the other, so neither group is pushed and the bug is
fixed;
- every consumer applying the same MIN_MAX and IN_OR_BLOOM filters, which is
the common shape when
`runtime_filter_type` enables both (the default value 12): both groups are
still pushed, so the
optimization is kept and TPC-DS q95 keeps the runtime filters on the inner
scan of its CTE body;
- a filter that only some consumers apply is skipped while the groups the
other consumers apply
are still pushed.
### Release note
Fixed a bug where a query referencing a materialized CTE more than once
could silently return
fewer rows than expected (or no rows) when runtime filters were enabled, for
example when each
reference joins the CTE with an opposite range condition.
### Check List (For Author)
- Test: FE UT
(`RuntimeFilterTest#testPushSharedCteRuntimeFiltersWhichEveryConsumerApplies`,
`#testDoNotPushSharedCteRuntimeFiltersWhichOtherConsumersDoNotApply`) and
the new regression test
`nereids_rules_p0/cte/test_cte_shared_producer_min_max_runtime_filter`,
which was run against an
unpatched FE (fails: `Check tag
'shared_cte_min_max_rf_opposite_directions' failed`) and against
the patched FE (passes). Manual test on a local cluster with internal OLAP
tables confirmed that
`runtime_filter_mode=GLOBAL/LOCAL` with `MIN_MAX` and a materialized CTE
now returns the same rows
as `OFF`. TPC-DS q95 was compared with and without this change
(`runtime_filter_type=12`, same
data and settings): the runtime filters on the inner scan of the CTE body
are unchanged and the
result is identical.
- Behavior changed: Yes. Only runtime filters that every consumer of a
materialized CTE applies are
pushed into the shared producer; such queries now return correct results
instead of empty ones.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
