sungwy commented on issue #1045: URL: https://github.com/apache/iceberg-python/issues/1045#issuecomment-2617788267
Hi @fusion2222 and @lloyd-EA - thank you both for jumping on the issue and contributing the PRs! I agree with @kevinjqliu here that it would be good to consolidate our efforts on a single PR before duplicating our efforts further. I apologize that the purpose of the error message in the `_ConvertToIceberg` visitor isn't making much sense without additional context. Here's my attempt at explaining its purpose. The original PR to introduce `ns` downcasting only supports downcasting for `write` operations, because we are able to set the parquet type to timestamp logical type[1] with the desired downcasted `us` precision when we rewrite the data files and hence seemed more straight forward to implement. `add_files` on the other hand does not rewrite the data file, so if we support adding data files with `ns` precision, we aren't actually rewriting the data files with `us` precision, but instead we are keeping the data files with `ns` precision. Hence we will need to focus on verifying that we are able to read the parquet files that are added with `ns` precision when the IcebergSchema is of `us` precision. `Iceberg does not yet support 'ns' timestamp precision. Use 'downcast-ns-timestamp-to-us-on-write' configuration property to automatically downcast 'ns' to 'us' on write.` This assertion is thrown within the `_ConvertToIceberg` visitor that visits a pyarrow schema and converts it to Iceberg schema. It should be silenced when `_downcast_ns_timestamp_to_us` attribute is set to `True` on the visitor. [1] (https://github.com/apache/parquet-format/blob/master/LogicalTypes.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
