morningman opened a new pull request, #69024:
URL: https://github.com/apache/doris/pull/69024
### What problem does this PR solve?
Issue Number: None
Related PR: #68360
Problem Summary:
**In short.** This ports branch-4.2 #68360 to master. When a nested column's
name is not all lower case, Nereids decided that the whole-column access path
did not cover a predicate's sub-field path, and sent both to BE. The check now
compares path components case-insensitively, so FE sends only the whole-column
path.
**Background.** For a scan of a nested column, `NestedColumnPruning` sends
BE two lists: the *all* access paths (what the scan must produce) and the
*predicate* access paths (what the predicates read first). Paths are collected
with lower-cased components, while `buildColumnAccessPaths` rebuilds a
whole-column path from the column name as the catalog stores it.
`addPredicatePathsToFinalAllAccessPaths` then keeps the all paths a superset of
the predicate paths: a predicate path that no all path covers is appended.
**The problem, and what it cost.** `isPrefixPath` compared the components
case-sensitively, so for a column named `S` the whole-column path `[S]` did not
cover `[s, city]`:
| `SELECT S FROM t WHERE struct_element(S, 'City') = 'beijing'` | master |
with this PR |
|---|---|---|
| all access paths | `[S, s.city]` | `[S]` |
| predicate access paths | `[s.city]` | `[s.city]` |
| map column `M`, `WHERE array_contains(map_values(M), 1)` | `[M, m.VALUES]`
| `[M]` |
On branch-4.x the extra path made BE read only `city` and return the struct
with its other fields empty, which is what #68360 fixed there. The BE on master
reads every child for a whole-column path, so the query results on master are
already correct (checked on a local cluster); what remains is that FE sends a
sub-field path next to the whole column it already covers, which the BE has to
tolerate.
**How this PR fixes it.** `isPrefixPath` becomes `isPrefixPathIgnoreCase`
and compares components with `equalsIgnoreCase`, as #68360 does on branch-4.2.
All collected components are already lower case, so in practice this only
changes the comparison of the root, the one component that keeps the catalog
spelling.
**Results.** The table above is the EXPLAIN output on a local cluster before
and after this change. The FE UT of #68360 passes with this change and fails
without it (`expected: <[DATA:S]> but was: <[DATA:S, DATA:s.city]>`); the other
68 `PruneNestedColumnTest` cases still pass. The regression case checks the
access paths and the returned rows, and its `.out` file was generated by the
script.
Original author: @hubgeter
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
-
`nereids_rules_p0/column_pruning/upper_case_nested_column_pruning`: the access
paths of a struct and a map column whose names are upper case, and the rows
returned; `.out` generated by the script.
- [x] Unit Test
-
`PruneNestedColumnTest.testWholeUpperCaseColumnOutputKeepsSubFieldPredicatePath`
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- When a nested column's name has upper-case letters, FE no longer
adds a predicate's sub-field path next to the whole-column access path.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
