mxm commented on PR #18517: URL: https://github.com/apache/iceberg/pull/18517#issuecomment-6232039538
Yes, those are the possible scenarios. However, I don't think it is possible to distinguish the former from the latter case. The reason is that once a field gets dropped, it is impossible to tell whether a DynamicRecord with a schema whose diff consists of only the removed field is old or new. Remember, we only match by name/type. The field either belongs to the old schema or is a request to re-create the field. All of this is true for schema changes done by dynamic sink. When we factor in external changes, what you suggest can make sense but I'm still not convinced. Consider this scenario: A field "myfield" with type string gets removed. It is replaced externally by a "myfield" with type double. When an old record arrives, do we really want to fail the job because we try to change the type from double to string, or would we rather write with the old schema? I prefer the latter. IMHO sorting schemas by last-created solves the issue that @adityaksolves faces. In this case the re-created field has the same type as the original field. The new field will have precedence. Also, old records will automatically use the new schema because the types are compatible. The solution is simple to implement and lenient towards old records in case the type of the field changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
