Re: [PR] Issue 1045 - Allow downcasting of parquet timestamp[ns] to timestamp[us] for Table.add_files() [iceberg-python]

[via GitHub]

lloyd-EA commented on PR #1569:
URL: https://github.com/apache/iceberg-python/pull/1569#issuecomment-2619838067

   @sungwy I updated the PR there. It seems that Spark is reads in the column 
as `us` and doesn't check the underlying data file's precision being `ns`, so 
when it is converted to a `datetime` the value is wrong.
   
   expected: `2021-03-17 07:54:47.249846175`
   
![image](https://github.com/user-attachments/assets/9e6056b9-ce1e-4e89-a16e-c48bde478934)
   
   To me, this is a problem with Spark not being able to handle `ns` timestamps 
properly, and since I won't be using Spark for my application, I'll continue to 
use my forked repo for now as it's sufficient for my application.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org