Fokko commented on code in PR #6346:
URL: https://github.com/apache/iceberg/pull/6346#discussion_r1038381242
##########
python/pyiceberg/io/pyarrow.py:
##########
@@ -387,6 +388,12 @@ def _(_: BinaryType) -> pa.DataType:
return pa.binary()
+def _convert_scalar(value: Any, iceberg_type: IcebergType) -> pa.scalar:
+ if not isinstance(iceberg_type, PrimitiveType):
Review Comment:
This is because there is some extra logic when passing in the values other
than an PyArrow iterable:
https://github.com/apache/arrow/blob/8a8999e94038aa9a60d3ac15741cf9c7abad0433/python/pyarrow/_compute.pyx#L2431-L2450
When constructing the array, it will infer the type:
https://github.com/apache/arrow/blob/8a8999e94038aa9a60d3ac15741cf9c7abad0433/python/pyarrow/array.pxi#L117-L129
Which will go to the visitor:
https://github.com/apache/arrow/blob/8a8999e94038aa9a60d3ac15741cf9c7abad0433/python/pyarrow/src/arrow/python/inference.cc#L352-L422
To determine the type. Instead, we can also pass in the type when we
manually construct the array, which is nicer. I've updated the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
