syun64 commented on PR #473: URL: https://github.com/apache/iceberg-python/pull/473#issuecomment-1967538763
So it looks like using a custom @field_serializer isn't working in the [current IcebergBaseModel definition](https://github.com/apache/iceberg-python/blob/pyiceberg-0.6.x/pyiceberg/typedef.py#L128) with "None" values, because we are setting `exclude_none=True` by default for `model_dump_json`. Unfortunately the model exclusion logic runs before the custom field_serializer, meaning that if we store the current_snapshot_id internally as `None`, we won't be able to use custom pydantic serializers to cast `None` to `-1` only on output. There is a [recent discussion on this specific issue in the Pydantic community](https://github.com/pydantic/pydantic/discussions/5461), and people had to resort to other hacky workarounds to support field_serialization on None values when exclude_none is set to True. If we want to make Iceberg tables created by PyIceberg compatible with older versions of Java, I think we'll just have to store current_snapshot_id as `-1` in the TableMetadata, or refactor the way we identify fields that we want to exclude if they are None in the IcebergBaseModel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
