rahulsmahadev commented on code in PR #4409:
URL: https://github.com/apache/iceberg-python/pull/4409#discussion_r4617208234
##########
pyiceberg/conversions.py:
##########
@@ -352,14 +356,28 @@ def _(_: PrimitiveType, b: bytes) -> int:
@from_bytes.register(TimeType)
-@from_bytes.register(TimestampType)
@from_bytes.register(TimestamptzType)
-@from_bytes.register(TimestampNanoType)
@from_bytes.register(TimestamptzNanoType)
def _(_: PrimitiveType, b: bytes) -> int:
return _LONG_STRUCT.unpack(b)[0]
+@from_bytes.register(TimestampType)
+def _(_: TimestampType, b: bytes) -> int:
+ if len(b) < 8:
+ # If the length is 4 bytes, it is a date promoted to timestamp, stored
as days from the epoch
+ return _INT_STRUCT.unpack(b)[0] * _MICROS_PER_DAY
Review Comment:
Happy to look into it. One constraint: the spec says readers must infer the
write-time type from the bound's length, because manifests do not store the
bound type and promotion does not rewrite bounds ("Schema Evolution" in
`format/spec.md`). The manifest's `schema` metadata is the schema when the
manifest was written, which can be newer than the data file when manifests are
rewritten, so it cannot replace the length check everywhere. I'll dig into
whether there is a reliable source for the write-time type, and open a
follow-up PR if there is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]