rgbuilds opened a new issue, #26172:
URL: https://github.com/apache/datafusion/issues/26172
### Describe the bug
When inferring a schema across multiple Parquet files, a required nested
field remains non-nullable even when it is absent from one file.
For example, both files contain struct `s`, but only one contains its
required child `s.y`. The inferred `s.y` remains non-nullable, causing two
problems:
- Selecting `s.y` fails because the missing required child cannot be filled
with `NULL`.
- `WHERE s.y IS NULL` incorrectly returns no rows because the inferred
non-nullability allows the optimizer to simplify the predicate to false.
### To Reproduce
Create two Parquet files with these schemas and values:
```text
File A:
Schema: required id: Int32,
required s: struct<x: required Int32, y: required Int32>
Row: id = 1, s = {x: 1, y: 10}
File B:
Schema: required id: Int32,
required s: struct<x: required Int32>
Row: id = 2, s = {x: 2}
```
Register the directory as table `t` using inferred schema, then run:
```sql
SELECT id, s.y FROM t ORDER BY id;
SELECT id FROM t WHERE s.y IS NULL ORDER BY id;
```
The first query fails, and the second returns no rows.
### Expected behavior
The inferred nested field `s.y` should be nullable.
The first query should return:
```text
id | s.y
1 | 10
2 | NULL
```
The second query should return:
```text
id
2
```
### Additional context
This was identified while reviewing #26104, which fixes missing top-level
columns. The author confirmed that the nested-field case should be tracked
separately:
https://github.com/apache/datafusion/pull/26104#discussion_r4227414300
Reproduced locally on both commits:
- Base: `102a1628592121d96bfe3d09d80e63180eef288a`
- Reviewed PR head: `bc92f5bf512410c46e75262a4c6e03d040830dfb`
Environment: Apple Silicon Mac Mini, macOS.
This is a preexisting limitation, not a regression introduced by #26104.
### Willingness to contribute
I would like to contribute a fix for this bug, but request guidance from the
DataFusion community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
