englefly opened a new pull request, #70194:
URL: https://github.com/apache/doris/pull/70194
### What problem does this PR solve?
Issue Number: N/A
Related PR: #70100
Problem Summary:
A TopN runtime filter above `UNION ALL` can reach scans below operators that
synthesize NULLs. Another UNION branch can publish a bound before those scans
finish, causing incorrect results:
- **Grouping sets:** rows rejected on their original grouping key can still
contribute to a grouping set that replaces that key with NULL. Filtering them
before Repeat produces incomplete aggregate counts or loses the grand-total
row. For one million input rows, `ORDER BY k NULLS FIRST LIMIT 1` can return
`(NULL, 1024)` instead of `(NULL, 1000000)`.
- **Scalar subqueries:** filtering before AssertNumRows can remove the
scalar's only row, making it synthesize a NULL row instead. A scalar result of
`20` unioned with `10` can consequently become NULL and win `ASC NULLS FIRST`;
`COALESCE(..., -1)` also fails with `NULLS LAST`. Filtering can additionally
remove multiple rows and suppress a required cardinality error.
Only push a TopN filter through Repeat when every probe input slot is common
to all grouping sets, using the existing common-grouping-expression helper.
Stop pushdown at AssertNumRows to preserve its input cardinality and
empty-input behavior. Keep filtering on common grouping keys and unaffected
UNION branches.
### Release note
Fix incorrect TopN results for grouping sets and scalar subqueries, and
preserve scalar-subquery cardinality errors when TopN runtime filters are
enabled.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test
- [ ] No need to test or manual test
Validation:
- `./build.sh --fe -j16` — successful; zero Checkstyle violations.
- `./run-fe-ut.sh --run
org.apache.doris.nereids.postprocess.TopNRuntimeFilterTest` — 12 tests passed.
The new Repeat test covers 36 grouping-set/expression/sort/NULL-order
combinations. The Assert test covers eight combinations and directly visits the
assertion, with a positive control on its child, so it does not depend on CBO's
scalar-join orientation.
- `./run-regression-test.sh --run -d nereids_p0 -s
topn_filter_grouping_sets,topn_filter_assert_num_rows` — two suites passed,
with 35 result checks and eight expected-cardinality-error checks. Covers
GROUPING SETS/ROLLUP/CUBE, common grouping keys, scalar one-row/empty/multi-row
inputs, COALESCE, and both NULL orders. Expected output was generated by the
regression runner.
- The new Assert regression plan assertion fails on the unpatched FE: the
filter has two scan targets instead of only the unaffected UNION scan.
- Manual reproduction confirmed incomplete grouping-set counts, synthetic
scalar NULLs, and suppressed scalar-cardinality errors before the corresponding
fixes.
Validation used a single-BE native-OLAP cluster. FE was built from this
branch; BE reused an existing ASAN master build (`7acfd9bfc27`).
- Behavior changed:
- [ ] No.
- [x] Yes. Prevent unsafe scan filtering across Repeat and AssertNumRows
while retaining safe targets.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
