manuzhang commented on code in PR #6246:
URL: https://github.com/apache/datafusion-comet/pull/6246#discussion_r4315293290
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -330,6 +330,24 @@ fn parquet_convert_array_impl(
Ok(Arc::new(string_array))
}
+ // Parquet's unannotated BINARY is read as Spark StringType when
+ // spark.sql.parquet.binaryAsString is enabled or a read schema
declares a string.
+ // Arrow's cast below is safe, so it would turn malformed UTF-8 into
NULL. Use Spark's
+ // cast so the scan's top-level projection and nested values replace
malformed input
+ // with U+FFFD, as the JVM's `new String(bytes, UTF_8)` does.
+ (Binary, Utf8) | (LargeBinary, LargeUtf8) => {
Review Comment:
I've split this out to #7106 while holding off the rest for
[apache/datafusion#25802](https://github.com/apache/datafusion/pull/25802),
which adds an option to turn off the coercion from a file's BINARY to the
table's Utf8. Rebuilt on that support,
- The table schema stays Utf8 and filters are pushed unchanged. Annotated
files prune as they do today, with no double cast.
- Only an unannotated column is decoded, per file, by the existing adapter's
Spark cast. Pruning is lost only for that column in that file. Spark doesn't
push `=` or `IN` there either; `IS NULL` is the one case to follow up on.
- There's no skipped filter pushdown, so no need to document.
cc @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
