kevinjqliu commented on code in PR #1188:
URL: https://github.com/apache/iceberg-python/pull/1188#discussion_r1768984958
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1068,8 +1068,17 @@ def primitive(self, primitive: pa.DataType) ->
PrimitiveType:
return StringType()
elif pa.types.is_date32(primitive):
return DateType()
- elif isinstance(primitive, pa.Time64Type) and primitive.unit == "us":
+ elif isinstance(primitive, pa.Time64Type) and (primitive.unit == "us"
or primitive.unit == "ns"):
+ primitive = cast(pa.TimestampType, primitive)
Review Comment:
`Time64Type` and `TimestampType` and different, dont need to cast here
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1068,8 +1068,17 @@ def primitive(self, primitive: pa.DataType) ->
PrimitiveType:
return StringType()
elif pa.types.is_date32(primitive):
return DateType()
- elif isinstance(primitive, pa.Time64Type) and primitive.unit == "us":
+ elif isinstance(primitive, pa.Time64Type) and (primitive.unit == "us"
or primitive.unit == "ns"):
Review Comment:
i think we can get rid fo the us/ns check here, since Time64Type only
supports those 2
https://arrow.apache.org/docs/python/generated/pyarrow.Time64Type.html
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1068,8 +1068,17 @@ def primitive(self, primitive: pa.DataType) ->
PrimitiveType:
return StringType()
elif pa.types.is_date32(primitive):
return DateType()
- elif isinstance(primitive, pa.Time64Type) and primitive.unit == "us":
+ elif isinstance(primitive, pa.Time64Type) and (primitive.unit == "us"
or primitive.unit == "ns"):
+ primitive = cast(pa.TimestampType, primitive)
+ if primitive.unit == "ns":
+ if self._downcast_ns_timestamp_to_us:
+ logger.warning("Iceberg does not yet support 'ns'
timestamp precision. Downcasting to 'us'.")
Review Comment:
lets add a test to verify that when `_downcast_ns_timestamp_to_us` is True,
the downcasting happens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
