kevinjqliu commented on code in PR #2173:
URL: https://github.com/apache/iceberg-python/pull/2173#discussion_r2217383692
##########
pyiceberg/utils/schema_conversion.py:
##########
@@ -69,8 +69,10 @@
LOGICAL_FIELD_TYPE_MAPPING: Dict[Tuple[str, str], PrimitiveType] = {
("date", "int"): DateType(),
("time-micros", "long"): TimeType(),
+ ("timestamp-millis", "int"): TimestampType(),
Review Comment:
> My initial use case was, that I want to use the Schema conversion
functions to create an Iceberg table based of an Avro Schema. At the moment I
use AvroSchemaConversion.avro_to_iceberg().as_arrow() to create the Arrow table
that goes eventually into Iceberg.
I see, thanks for the context! I can see that its also used
[here](https://gist.github.com/matthias-Q/87632a18301324e4bc3d02dd2c396210#file-avro_msg_iceberg_upsert-py-L98-L101)
It seems like there's a nuance in how
`AvroSchemaConversion().avro_to_iceberg` is used in the project versus for your
use case.
In the pyiceberg project, `AvroSchemaConversion().avro_to_iceberg` is meant
to be used for reading the iceberg metadata files and follows the [spec for
data type mapping](https://iceberg.apache.org/spec/#avro). Because of this, we
cannot support `timestamp-millis` here because it is not part of the spec.
For your use case, we would want to support data mapping from
`timestamp-millis` to an iceberg data type, likely `timestamp-micros`. This
logic should probably live outside of `AvroSchemaConversion().avro_to_iceberg`.
Once the avro schema is converted to iceberg schema, i believe the data itself
will also be converted to the correct type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
