andygrove opened a new issue, #7032:
URL: https://github.com/apache/datafusion-comet/issues/7032
### Describe the bug
When a Parquet BINARY column without the UTF8 annotation is read as STRING,
which a read schema that declares `string` or
`spark.sql.parquet.binaryAsString=true` does, the native Parquet scan returns
NULL for any nested value that is not valid UTF-8. Spark returns the bytes
unchanged. Nothing fails, so the NULLs are silent. Top-level values take a
different path that skips UTF-8 validation entirely.
- **Nested values** (a struct field, list element or map value) are
converted by `parquet_convert_array_impl`, which falls back to Arrow's
`cast_with_options` with `PARQUET_OPTIONS`
(`native/core/src/parquet/parquet_support.rs` lines 74-79 and 333-336). Those
options set `safe: true`, so a value that is not valid UTF-8 becomes NULL.
- **Top-level values** are decoded by arrow-rs straight into a `Utf8` array.
arrow-rs validates UTF-8 only for columns annotated UTF8 (`parquet` 59.3.0
`arrow/array_reader/byte_array.rs` line 192), so in a release build the invalid
bytes reach a `Utf8` array unchecked, the unsound state #4764 describes. A
debug build panics instead, in `OffsetBuffer::into_array`, which validates only
under `debug_assertions` (`arrow/buffer/offset_buffer.rs` line 141).
### Steps to reproduce
In a `CometTestBase` suite:
```scala
spark.sql("SELECT X'41FF42' AS t, named_struct('b', X'41FF42') AS s,
array(X'41FF42') AS l")
.coalesce(1).write.parquet(path)
spark.read.schema("t string, s struct<b:string>, l
array<string>").parquet(path)
.selectExpr("hex(s.b)", "hex(l[0])", "s.b IS NULL")
.collect()
```
On `main` at 66c63bd307, the same on Spark 3.4.3, 3.5.9 and 4.1.3:
| Query | Spark | Comet native scan |
| --- | --- | --- |
| `hex(s.b), hex(l[0]), s.b IS NULL` | `[41FF42,41FF42,false]` |
`[null,null,true]` |
### Expected behavior
The native scan does not turn a nested value into NULL because its bytes are
not valid UTF-8, and the top-level path does not admit invalid UTF-8 into a
`Utf8` array. #4764 proposes decoding at the ingress boundary with
`decode_utf8_spark_lossy`, the decoder `CAST(binary AS string)` and native
shuffle already share, which would cover both paths.
### Additional context
Found in a nested-type audit of the native scans. This is another ingress
point for the policy #4764 tracks. #4121 covers UTF8-annotated STRING columns,
which the native reader rejects with `encountered non UTF-8 data`. The
BINARY-read-as-STRING path is not covered there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
