jqin61 commented on PR #453:
URL: https://github.com/apache/iceberg-python/pull/453#issuecomment-1959848004
> @jqin61 I wanted to do a second round, but I think you forgot to push? :)
Hi Fokko sorry for the delayed push of the fixes. It took a little time to
think through how to use the literal function. I made the changes according to
your last comment except for the literal one. I think the literal function and
Literal Class might not solve the issue I am encountering in the PartitionKey
class - I need some utilities to convert a python datetime to micros and
convert python date to days. The literal() function could only take in python
primitive types and it seems wrong to extend it to take datetime/date to return
TimestampLiteral and DateLiteral.
Also thanks for pointing out that timestamp in iceberg corresponds to
timestamp_ntz in spark. I added tests for it and discovered some new
discrepancies:
1.CAST('2023-01-01 12:00:00' AS TIMESTAMP_NTZ) becomes 2023-01-01T12:00 in
the hive partition path when spark writes it while iceberg writes as
2023-01-01T12:00:00
2.CAST('2023-01-01 12:00:01.000999+03:00' AS TIMESTAMP) becomes
2023-01-01T09:00:01.000999Z in the hive partition path when spark writes it
while iceberg writes as 2023-01-01T09:00:01.000999+00:00
I am not sure whether the path generation behavior is part of iceberg's spec
and we should work towards making them exactly the same - it seems as long as
the partition in the manifest entry is correct the query planning could
leverage the partition info to do pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
