sovsparrow opened a new issue, #24381:
URL: https://github.com/apache/datafusion/issues/24381
### Describe the bug
When a Parquet file contains two columns with the same name and type,
DataFusion reads it successfully but only returns the first column. The
second column is missing from the result.
I reproduced this with DataFusion 54.0.0 and PyArrow 25.0.0.
### To Reproduce
This example joins two Arrow tables that both have a column named `value`.
PyArrow keeps both columns in the joined table and writes them to Parquet.
```python
from pathlib import Path
from tempfile import TemporaryDirectory
import pyarrow as pa
import pyarrow.parquet as pq
from datafusion import SessionContext
left = pa.table({
"id": [1, 2, 3],
"value": [10, 20, 30],
})
right = pa.table({
"id": [1, 2, 3],
"value": [100, 200, 300],
})
joined = left.join(right, keys="id")
with TemporaryDirectory() as directory:
path = Path(directory) / "joined.parquet"
pq.write_table(joined, path)
context = SessionContext()
result = pa.Table.from_batches(
context.read_parquet(str(path)).collect()
)
print("Written:")
print(joined.column_names)
print([column.to_pylist() for column in joined.columns])
print("DataFusion:")
print(result.column_names)
print([column.to_pylist() for column in result.columns])
```
Output:
```text
Written:
['id', 'value', 'value']
[[1, 2, 3], [10, 20, 30], [100, 200, 300]]
DataFusion:
['id', 'value']
[[1, 2, 3], [10, 20, 30]]
```
### Expected behavior
DataFusion should either preserve all three physical columns, using a
deterministic way to distinguish the duplicate names, or reject the file
with a clear duplicate-column error.
It shouldn't read successfully after dropping one of the columns.
### Additional context
DuckDB reads the same file and keeps all three columns, renaming the second
`value` column to `value_1`.
PyArrow's low-level `ParquetFile.read()` also preserves all three physical
columns. Its higher-level `pq.read_table()` and dataset reader reject the
ambiguous name, and Polars raises a duplicate-column error.
DataFusion is the only reader in this comparison that returns success while
omitting one of the physical columns.
PyArrow's `Table.join` allows colliding names when `left_suffix` and
`right_suffix` are not supplied, and `pq.write_table` writes the result.
DataFusion's current Parquet schema inference passes the fetched schemas
through
[`Schema::try_merge`](https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/file_format.rs#L303-L378).
That appears to merge the two same-name, same-type fields before the file is
scanned.
A related report for duplicate CSV headers is present:
[#12852](https://github.com/apache/datafusion/issues/12852). That issue asks
for duplicate columns to be renamed or rejected clearly. This is the Parquet
version.
Apache Arrow also tracks duplicate-column handling in
[apache/arrow#24407](https://github.com/apache/arrow/issues/24407).
Environment:
- DataFusion 54.0.0
- PyArrow 25.0.0
- Python 3.12.13
- macOS arm64
I found this while comparing Parquet readers with
[Parquity 0.2.0](https://github.com/sovsparrow/parquity).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
