ShivamSharma43 opened a new issue, #2625:
URL: https://github.com/apache/iceberg-go/issues/2625

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   `UpdateSchema.AddColumn` accepts a column name that already exists. `Apply` 
then returns a schema with two columns of the same name, and `Commit` either 
fails with a confusing visitor error or, in case-insensitive mode, commits a 
schema whose names only differ in case.
   
   Reproduced on main (5803266):
   
   ```go
   sc := iceberg.NewSchema(0,
        iceberg.NestedField{ID: 1, Name: "id", Type: 
iceberg.PrimitiveTypes.Int32, Required: true},
        iceberg.NestedField{ID: 2, Name: "name", Type: 
iceberg.PrimitiveTypes.String},
   )
   md, _ := table.NewMetadata(sc, nil, table.UnsortedSortOrder, "", nil)
   txn := table.New([]string{"t"}, md, "", nil, nil).NewTransaction()
   
   s, err := table.NewUpdateSchema(txn, true, false).
        AddColumn([]string{"name"}, iceberg.PrimitiveTypes.String, "", false, 
nil).
        Apply()
   fmt.Println(err, s)
   
   err = table.NewUpdateSchema(txn, true, false).
        AddColumn([]string{"name"}, iceberg.PrimitiveTypes.String, "", false, 
nil).
        Commit()
   fmt.Println(err)
   ```
   
   Output:
   
   ```
   <nil> table {
        1: id: required int
        2: name: optional string
        3: name: optional string
   }
   invalid schema: error encountered during schema visitor: invalid schema: 
multiple fields for name name: 2 and 3
   ```
   
   The same happens for:
   
   - a nested column (`AddColumn([]string{"address", "city"}, ...)` when 
`address.city` exists)
   - a name that another column is renamed to in the same update 
(`RenameColumn(["age"], "years")` then `AddColumn(["years"], ...)`), including 
through `UnionByNameWith`
   - case-insensitive mode: `NewUpdateSchema(txn, false, 
false).AddColumn([]string{"NAME"}, ...)` when `name` exists. Here `Commit` 
succeeds and the table ends up with both `name` and `NAME`. Adding `x` and then 
`X` in the same update does the same.
   
   Expected: `AddColumn` fails with `cannot add column, name already exists`.
   
   **Cause:** the duplicate check in `addColumn` (and the copy in 
`unionAddColumn`) in `table/update_schema.go` only fails when the existing 
column is found *and* some pending update in the same struct has that name. 
With no pending update it lets the add through, and it never checks a name that 
another column is being renamed to.
   
   **Java:** `SchemaUpdate.internalAddColumn` rejects the add unless the 
existing column is being deleted:
   
https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/core/src/main/java/org/apache/iceberg/SchemaUpdate.java#L141-L156
   It looks the name up with `caseInsensitiveFindField` in case-insensitive 
mode. PyIceberg's `add_column` does the same:
   
https://github.com/apache/iceberg-python/blob/639a3bf505b6932945bba3f661395255bac872c2/pyiceberg/table/update/schema.py#L214-L221
   
   I'd like to work on this. I have a fix and a unit test ready.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to