manuzhang commented on code in PR #18121:
URL: https://github.com/apache/iceberg/pull/18121#discussion_r4162058744
##########
arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java:
##########
@@ -148,11 +154,55 @@ protected VectorizedReader<?>
vectorizedReader(List<VectorizedReader<?>> reorder
@Override
public VectorizedReader<?> struct(
Types.StructType expected, GroupType groupType,
List<VectorizedReader<?>> fieldReaders) {
- if (expected != null) {
- throw new UnsupportedOperationException(
- "Vectorized reads are not supported yet for struct fields");
+ if (expected == null) {
+ return null;
+ }
+
+ // no field ID / no matching Iceberg field: fall back like primitive()
+ if (groupType.getId() == null) {
+ return null;
}
- return null;
+
+ Types.NestedField structField =
icebergSchema.findField(groupType.getId().intValue());
+ if (structField == null) {
+ return null;
+ }
+
+ List<VectorizedReader<?>> reorderedFields =
+ reorderFields(expected.fields(), groupType.getFields(), fieldReaders);
+
+ int structDefinitionLevel =
parquetSchema.getMaxDefinitionLevel(currentPath());
+
+ VectorizedArrowReader presenceReader = null;
+ // must agree with ParquetSchemaUtil.PresenceColumnSelector, which retains
this presence column
+ if (structDefinitionLevel > 0 && !hasFileBackedLeaf(reorderedFields)) {
+ ColumnDescriptor presence =
+ ParquetSchemaUtil.selectPresenceColumn(parquetSchema, currentPath());
+ // list/map are not vectorized, so the presence leaf is always rep level
0 (iterator
Review Comment:
Not projecting lists/maps does not guarantee that the presence leaf has
repetition level 0.
For example, an old file can contain `s: optional struct<items:
list<long>>`, while the read projection contains only a newly added `s.added:
int`. The only stored leaf available to determine whether `s` is null is inside
`items`, so its maximum repetition level is 1.
This condition skips it, leaving `structNulls` unset and returning `{added:
NULL}` for a stored `s = NULL`.
We need to handle repeated presence leaves or fall back to the row reader
instead of silently losing parent nullness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
