syun64 commented on code in PR #921:
URL: https://github.com/apache/iceberg-python/pull/921#discussion_r1676885325
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1450,14 +1451,17 @@ def field_partner(self, partner_struct:
Optional[pa.Array], field_id: int, _: st
except ValueError:
return None
- if isinstance(partner_struct, pa.StructArray):
- return partner_struct.field(name)
- elif isinstance(partner_struct, pa.Table):
- return partner_struct.column(name).combine_chunks()
- elif isinstance(partner_struct, pa.RecordBatch):
- return partner_struct.column(name)
- else:
- raise ValueError(f"Cannot find {name} in expected
partner_struct type {type(partner_struct)}")
+ try:
+ if isinstance(partner_struct, pa.StructArray):
+ return partner_struct.field(name)
+ elif isinstance(partner_struct, pa.Table):
+ return partner_struct.column(name).combine_chunks()
+ elif isinstance(partner_struct, pa.RecordBatch):
+ return partner_struct.column(name)
+ else:
+ raise ValueError(f"Cannot find {name} in expected
partner_struct type {type(partner_struct)}")
+ except KeyError:
Review Comment:
Yes, that's right - the `ArrowProjectionVisitor` is responsible for
detecting that the `field_partner` is `None` and then checking if the table
field is also optional before filling it in with a null array. This change is
necessary so that the `ArrowAccessor` doesn't throw an exception if the field
can't be found in the arrow component, and enables `ArrowProjectionVisitor` to
make use of a code pathway it wasn't able to make use of before:
https://github.com/apache/iceberg-python/blob/b11cdb54b1a05cce0ade34af4ce81a94c34b2650/pyiceberg/io/pyarrow.py#L1388-L1395
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
