binarybrain-009 commented on issue #17555:
URL: https://github.com/apache/iceberg/issues/17555#issuecomment-6492878707
I couldn't reproduce this on main, and the auto-create code hasn't changed
since 1.11.0. With `iceberg.tables.schema-force-optional=true`,
`SchemaUtils.toIcebergType` makes struct fields, list elements and map values
optional at every nesting level. I ran your scenario end to end with the
bundled `DebeziumTransform` and an in-memory catalog. The auto-created table
has `_cdc`, `_cdc.key` and `_cdc.key.account_id` all optional. After a PK
change (key `{account_id}` → `{uuid}`), the sink adds `_cdc.key.uuid` and keeps
writing.
The NPE does reproduce when the table was created with the flag off, or
outside the sink. In that case `_cdc.key.account_id` is required, and turning
the flag on later doesn't loosen it. Evolution only makes an existing column
optional when the record's schema marks it optional. Were the affected tables
created before the flag was enabled?
Changing evolution to loosen every required column under this flag, as the
stale #17556 did, wouldn't cover your case: a key field that has already left
the records is never visited. It would also break tables with identifier
fields. The sink would try to make the identifier column optional on every
record, and the schema commit fails with `Cannot add field ... as an identifier
field: not a required field`. For tables that already have required nested
fields, a one-time `ALTER COLUMN ... DROP NOT NULL` is still the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
