pvary commented on code in PR #18517: URL: https://github.com/apache/iceberg/pull/18517#discussion_r4454757901
########## docs/docs/flink-writes.md: ########## @@ -524,6 +524,8 @@ The dynamic sink tries to match the schema provided in `DynamicRecord` with the - If there is no direct match, DynamicSink tries to adapt the provided schema such that it matches one of table schemas. For example, if there is an additional optional column in the table schema, a null value will be added to the RowData provided via DynamicRecord. - Otherwise, we evolve the table schema to match the input schema, within the constraints described below. +Schemas are matched by column name. The dynamic sink prefers the current table schema: it writes with the current schema whenever the record fits it, converting values if needed (for example `Integer` to `Long`), and falls back to an older table schema only when the record does not fit the current one. Older table schemas which use a column name that belongs to a different column in the current schema (for example after a column was dropped and added again, or renamed and its name reused) are tried last, because data written with them is stored under column IDs which queries on the current schema do not read. Values written with an older table schema may therefore not be visible in columns that have changed since. Review Comment: ```suggestion Schemas are matched by column name. The table schemas are checked starting with the current schema, then from newest to oldest. A table schema which matches the record exactly is preferred over one which needs a conversion (for example `Integer` to `Long`), so a record which has the shape of an older table schema is written with that schema, even if it would also fit the current one. The exceptions are older table schemas which use a column name that belongs to a different column in the current schema, for example after a column was dropped and added again, or renamed and its name reused. Data written with such a schema is stored under column IDs which queries on the current schema do not read, so these schemas are only used when the record fits no other table schema, not even with a conversion. In that case the affected values are not visible through the current schema. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
