Wyatt-Hawes opened a new pull request, #16853: URL: https://github.com/apache/iceberg/pull/16853
## What `IcebergToGlueConverter.setTableInputInformation` builds a `name → comment` map from the existing Glue table via `Collectors.toMap(Column::name, Column::comment)`, which throws `IllegalStateException` on duplicate keys. **Intended Fix:** Add an existing-entry-wins merge function so duplicates do not throw. ## How this happens The following `RENAME COLUMN` cycle (`id → id__tmp → Id`): (Used SPARK with Glue Data Catalog) Note how the First and Last column names are identical if lowercased 1. By design, `toColumns` walks `metadata.schemas()` and emits both historical and current names; case-sensitive dedupe lets `Id` and `id` through as distinct entries `columns=[Id, id, id__tmp]`. 2. AWS Glue lowercases `Column.name` on persist, collapsing them into two `name="id"` rows that both carry a non-null comment `columns=[id, id, id__tmp]`. This is [expected AWS Behavior.](https://docs.aws.amazon.com/athena/latest/ug/tables-databases-columns-names.html) 3. The next commit hits `Collectors.toMap` → duplicate-key throws `IllegalStateException`. 4. The throw is swallowed by the outer `catch (RuntimeException e)`, leaving `tableInputBuilder.storageDescriptor` unset. 5. The engine calls`glue.updateTable()` with a null `StorageDescriptor` which deletes the tables StorageDescriptor. The name collision itself isn't the bug trigger. Without comments, the `column.comment() != null` filter strips the colliding entries before `toMap` sees them. ## Fix First-seen-wins: `toColumns` writes current-schema entries before historical ones, so the current entry's comment is preserved. Glue still receives duplicate-name `Columns[]` after a case-only rename, and `addColumnWithDedupe` is untouched. ## Test Adds `testSetTableInputInformationWithDuplicateExistingColumnComments`: builds an `existingTable` with two `name="id"` entries (each with a comment) and asserts a non-null `StorageDescriptor` with the first-seen comment. Pre-fix fails on null `storageDescriptor()`; Post-fix passes. All 13 tests in `TestIcebergToGlueConverter` pass ---- *PR description drafted with AI* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
