manuzhang opened a new pull request, #7106:
URL: https://github.com/apache/datafusion-comet/pull/7106
## Which issue does this PR close?
Part of #7032 (the nested values) and #4764. Split out of #6246.
## Rationale for this change
When a Parquet BINARY column without the UTF8 annotation is read as a
string, through a read schema or `spark.sql.parquet.binaryAsString=true`, the
native scan converts nested values (struct fields, list elements, map keys and
values) in `parquet_convert_array`. That fell through to Arrow's safe cast,
which turns invalid UTF-8 into NULL, so those values silently became NULL where
Spark keeps the bytes.
This decodes them the way `CAST(binary AS string)` and native shuffle
already do (#4764), replacing each ill-formed sequence with U+FFFD. Collected
values now match Spark. Byte-level results such as `hex` still differ, as
described under "Strings with non-UTF-8 bytes" in the compatibility guide.
It is split out of #6246, which also changes the top-level path and is
waiting on a decision about filter pushdown. This part doesn't affect pushdown,
doesn't depend on `spark.sql.parquet.binaryAsString`, and applies to
`branch-1.1` as well.
## What changes are included in this PR?
- `parquet_convert_array_impl` converts `Binary` to `Utf8` and `LargeBinary`
to `LargeUtf8` with Spark's cast instead of Arrow's safe cast.
- The behavior also changes with `spark.sql.parquet.binaryAsString=false`,
when a read schema declares a string over a BINARY column: a malformed nested
value used to become NULL and now becomes U+FFFD.
- Docs: the #7032 entry under the Parquet known issues, and "Strings with
non-UTF-8 bytes" in the compatibility guide.
Top-level columns are unchanged. DataFusion's file-schema coercion decodes
them straight into `Utf8` without validation, so they never reach this
conversion. That is the rest of #7032.
## How are these changes tested?
- A Rust unit test converts a `List<Binary>` holding `0xFF` to `List<Utf8>`.
- A new `ParquetReadSuite` test writes a raw Parquet file with unannotated
BINARY in a list, a struct and a map, holding `41 FF 42`. It reads the file
with `binaryAsString=true` and, separately, with a read schema, and checks:
- the inferred schema,
- Spark answer parity with a `CometNativeScanExec`, which fails without
this change because the nested values come back NULL,
- that `hex` of each nested value is `41EFBFBD42`, which shows the value
was decoded rather than passed through unchecked.
- The `hex` check also guards the DataFusion 56 upgrade.
apache/datafusion#25193 extends DataFusion's file-schema coercion into struct
and list children, which would send these values around this conversion and
into unchecked `Utf8` arrays.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
