Dandandan opened a new pull request, #24787:
URL: https://github.com/apache/datafusion/pull/24787
## Which issue does this PR close?
- Closes #.
## Rationale for this change
`Unnest::try_new` copies its input's functional dependencies unchanged:
```rust
// We can use the existing functional dependencies:
let deps = input_schema.functional_dependencies().clone();
```
That is not true for a list unnest, which turns one input row into several. A
determinant that occurred once in the input can occur many times in the
output,
so it is no longer a unique key.
Optimizer rules that read those dependencies then produce wrong results:
```sql
CREATE TABLE t_list (k INT, vals INT[], PRIMARY KEY (k))
AS VALUES (1, [10, 20, 30]), (2, [40]);
CREATE TABLE t_join (k INT) AS VALUES (1), (2);
SELECT u.k, u.v FROM (SELECT k, unnest(vals) AS v FROM t_list) u
JOIN t_join j ON u.k = j.k;
```
returns 2 rows. The correct answer is 4, and removing the `PRIMARY KEY` gives
4. `eliminate_join` sees `u` as unique on `k`, rewrites the inner join into a
semi join, and the repeated rows are dropped.
## What changes are included in this PR?
The dependency still holds in the weaker sense: every row produced from one
input row carries the same determinant, so it determines the same columns. It
is downgraded to `Dependency::Multi` rather than dropped, which keeps it
useful
for other purposes and stops it being read as a uniqueness guarantee.
Unnesting a struct produces one row per input row, so those dependencies are
left as they are.
## Are these changes tested?
Yes. `functional_dependencies.slt` gains the query above, which returns 4
rows,
and a plan assertion that a struct unnest still keeps its input's
dependencies.
The full sqllogictest suite (504 files) passes.
## Are there any user-facing changes?
Queries that unnest a list from a table with a declared key and then join or
aggregate on that key now return correct results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
