jayzhan211 opened a new pull request, #25999:
URL: https://github.com/apache/datafusion/pull/25999
## Which issue does this PR close?
- Closes #25998.
## Rationale for this change
`ORDER BY <list or struct column> LIMIT n` can return wrong rows when the
sort key values contain NULL elements, because the TopK dynamic filter compares
nested values with a NULL ordering that differs from the sort's. See #25998 for
details and a reproducer.
## What changes are included in this PR?
- `TopK::build_filter_expression` stops at the first sort key whose
threshold value is nested (`DataType::is_nested`, which also covers
dictionaries of nested values).
- If that is the first key, no filter is built and the dynamic filter
stays `true`.
- Otherwise the filter keeps the terms for the earlier flat keys and adds
"all earlier keys equal their thresholds" as the last disjunct, so rows tied on
the flat prefix are kept.
- Filters for flat sort keys are unchanged. Nested sort keys only lose
dynamic filter pruning; the TopK result is still computed with the row format,
which follows the sort's NULL ordering.
- `compare_op_for_nested` and the semantics of `<` / `>` on nested types are
not changed (the `PiecewiseMergeJoinExec` sibling is #25957).
## What is the testing strategy for this PR?
New `sqllogictest` cases in `topk.slt` cover a list key (`ASC`, `DESC`,
`NULLS FIRST`), a struct key, a nested key after a flat key (`ORDER BY a, l
LIMIT 1` with ties on `a`), and a nested key followed by a flat key (`ORDER BY
l, b LIMIT 1`). All of them except the `NULLS FIRST` case fail on `main`;
`NULLS FIRST` happens to match the default nested ordering and is kept as a
regression check. Existing `EXPLAIN ANALYZE` tests that show TopK dynamic
filters on flat keys (e.g. `dynamic_row_group_pruning.slt`,
`push_down_filter_parquet.slt`) pass unchanged.
## Are there any user-facing changes?
Queries with `ORDER BY` on list/struct columns and `LIMIT` return correct
results. No API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
