hanke580 opened a new issue, #26492:
URL: https://github.com/apache/datafusion/issues/26492
### Describe the bug
`COPY … TO 'dir/' PARTITIONED BY (p0)` names the directory `p0=<value>`. The
value is not escaped with Hive's
escaping rules. It goes through `object_store::path::Path::join`, which
percent-encodes only the characters that
object stores advise against (`/`, `%`, `#`, `?`, `*`, `\`, `"`, `[`, `]`,
`{`, `}`, `^`, `|`, `<`, `>`, `~`, `` ` ``, control
and non-ASCII characters). The key/value separator `=` is written as is, and
so are `:` and `'`, which Hive's
`escapePathName` escapes, and spaces, which pyarrow, DuckDB and Polars
escape.
A value `x=y` gives the directory `p0=x=y`.
DataFusion and pyarrow split at the first `=` and read the dataset back.
DuckDB and Polars do not:
- next to other partitions, DuckDB fails with `Hive partition mismatch … key
"p0" not found`, and Polars fails as well;
- when `p0=x=y` is the only partition, both return the rows **without the
`p0` column**, with no error.
For the same table, pyarrow, DuckDB and Polars write `p0=x%3Dy` (and
`p0=a%20b`), which every reader, including
DataFusion, reads back.
The path is built in `compute_hive_style_file_path`
(`datafusion/datasource/src/write/demux.rs:636`):
`file_path.join(format!("{}={}", partition_by[j].0, part_key[j]))`. Escaping
the value with the Hive rules (at least
`=` and `%`, which `ListingTable` already decodes on read since #23226)
would make the output portable.
### To Reproduce
```python
import pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({'rid': [1, 2, 3, 4], 'p0': ['x=y', 'a b', 'é',
'a/b']}), 'src.parquet')
```
```sql
CREATE EXTERNAL TABLE src STORED AS PARQUET LOCATION 'src.parquet';
COPY (SELECT * FROM src) TO 'd/' STORED AS PARQUET PARTITIONED BY (p0);
```
```
$ ls d
'p0=%C3%A9' 'p0=a b' 'p0=a%2Fb' 'p0=x=y'
```
```python
import duckdb, polars as pl
duckdb.sql("SELECT rid, p0 FROM read_parquet('d/**/*.parquet',
hive_partitioning = true)")
# Binder Error: Hive partition mismatch between file "d/p0=%C3%A9/….parquet"
and "d/p0=x=y/….parquet": key "p0" not found
pl.scan_parquet('d/**/*.parquet', hive_partitioning=True, hive_schema={'p0':
pl.String}).collect()
# could not create a new DataFrame: height of column 'p0' (3) does not match
DataFrame height (4)
```
Only the `x=y` partition:
```sql
COPY (SELECT 1 AS rid, 'x=y' AS p0) TO 'one/' STORED AS PARQUET PARTITIONED
BY (p0); -- one/p0=x=y/…
```
```python
duckdb.sql("SELECT * FROM read_parquet('one/**/*.parquet', hive_partitioning
= true)").columns # ['rid'] — p0 is gone
pl.scan_parquet('one/**/*.parquet', hive_partitioning=True).collect()
# only column rid
```
### Expected behavior
Partition values are escaped as Hive, pyarrow, DuckDB and Polars do
(`p0=x%3Dy`; the last three also write
`p0=a%20b`), so that DuckDB, Polars, Spark and Hive read the
dataset back with the right `p0` values.
### Additional context
- DataFusion main 290ad15 (2026-10-18), built from source; also 91cd7ffb
(2026-10-15).
- Related (read side, merged): #23226 "Decode Hive partition values in
listing tables". Open: #26310 (Hive's NULL
placeholder), #9269 (panic with `'` in partition values).
- Values with `=` are common: key-value strings, URL query strings, base64
with padding.
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
