Fokko commented on code in PR #13445:
URL: https://github.com/apache/iceberg/pull/13445#discussion_r2296774701
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/data/ParquetWithSparkSchemaVisitor.java:
##########
@@ -182,18 +186,18 @@ private static <T> T visitField(
private static <T> List<T> visitFields(
StructType struct, GroupType group, ParquetWithSparkSchemaVisitor<T>
visitor) {
- StructField[] sFields = struct.fields();
- Preconditions.checkArgument(
- sFields.length == group.getFieldCount(), "Structs do not match: %s and
%s", struct, group);
+ Map<String, StructField> sFieldsByName =
Review Comment:
@rdblue I'm not crazy about this change, but I don't think there are other
options. Previously we could assume and enforce that the Spark struct, and the
Parquet group had the same number of fields. With the UnknownType, the Spark
Dataframe will have a NullType, but not corresponding field in Parquet.
Since the Spark dataframe does not carry field-IDs, we have to map it by
name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
