mzzz-zzm opened a new issue, #2393:
URL: https://github.com/apache/iceberg-go/issues/2393
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
With field IDs left unset (all 0), `CreateTable` succeeds but usually binds
the partition to the wrong column:
```go
sc := iceberg.NewSchema(0,
iceberg.NestedField{Name: "created_at", Type:
iceberg.PrimitiveTypes.TimestampTz, Required: true},
iceberg.NestedField{Name: "event", Type: iceberg.PrimitiveTypes.String,
Required: true},
)
spec, _ := iceberg.NewPartitionSpecOpts(
iceberg.AddPartitionFieldByName("created_at", "created_day",
iceberg.DayTransform{}, sc, nil))
tbl, _ := cat.CreateTable(ctx, ident, sc, catalog.WithPartitionSpec(&spec))
// tbl.Spec() is day(event), not day(created_at), about 7 times in 8
```
`Append` also succeeds and writes null partition values. The first error
appears only when querying `event = 'login'`: `cannot apply day transform for
type string`.
Repro (fails every run):
```go
package poc
import (
"context"
"fmt"
"strings"
"testing"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/catalog"
"github.com/apache/iceberg-go/catalog/hadoop"
"github.com/apache/iceberg-go/table"
"github.com/stretchr/testify/require"
)
func TestUnsetFieldIDs(t *testing.T) {
ctx := context.Background()
cat, err := hadoop.NewCatalog(ctx, "local", t.TempDir(), nil)
require.NoError(t, err)
require.NoError(t, cat.CreateNamespace(ctx, catalog.ToIdentifier("db"),
nil))
reproduce(t, cat, "db")
}
// reproduce fails while the bug exists. It might pass once CreateTable
returns an error for this schema
// which column gets the partition is random (due to Go's map iteration
order),
// and it is the wrong one about 7 out of 8 in my case
// so the test makes up to 50 tables until one comes out wrong
// the odds that all 50 come out right are next to nil (maybe 1e-45?)
func reproduce(t *testing.T, cat catalog.Catalog, namespace string) {
ctx := context.Background()
for i := range 50 {
sc := iceberg.NewSchema(0, // no NestedField.ID set, so both
are 0
iceberg.NestedField{Name: "created_at", Type:
iceberg.PrimitiveTypes.TimestampTz, Required: true},
iceberg.NestedField{Name: "event", Type:
iceberg.PrimitiveTypes.String, Required: true},
)
spec, err := iceberg.NewPartitionSpecOpts(
iceberg.AddPartitionFieldByName("created_at",
"created_day", iceberg.DayTransform{}, sc, nil))
require.NoError(t, err)
ident := catalog.ToIdentifier(namespace,
fmt.Sprintf("unset_ids_%d", i))
tbl, err := cat.CreateTable(ctx, ident, sc,
catalog.WithPartitionSpec(&spec))
if err != nil {
t.Logf("CreateTable rejected the schema: %v", err)
return
}
t.Cleanup(func() { _ = cat.PurgeTable(context.Background(),
ident) })
got := tbl.Spec()
col, _ :=
tbl.Schema().FindColumnName(got.Field(0).FirstSourceID())
if col == "created_at" {
continue
}
tbl, err = tbl.Append(ctx, oneRow(t, tbl), nil)
require.NoError(t, err)
files, err := tbl.Scan().PlanFiles(ctx)
require.NoError(t, err)
_, queryErr := tbl.Scan(table.WithRowFilter(
iceberg.EqualTo(iceberg.Reference("event"),
"login"))).ToArrowTable(ctx)
t.Fatalf("CreateTable and Append succeeded, but the table is
partitioned by day(%s), not day(created_at)\n"+
"partition values written: %v\nquery event = 'login':
%v",
col, files[0].File.Partition(), queryErr)
}
t.Log("every table kept day(created_at)")
}
func oneRow(t *testing.T, tbl *table.Table) array.RecordReader {
sc, err := table.SchemaToArrowSchema(tbl.Schema(), nil, true, false)
require.NoError(t, err)
rec, _, err := array.RecordFromJSON(memory.DefaultAllocator, sc,
strings.NewReader(`[{"created_at": "2026-11-02T00:00:00Z",
"event": "login"}]`))
require.NoError(t, err)
rdr, err := array.NewRecordReader(sc, []arrow.RecordBatch{rec})
require.NoError(t, err)
return rdr
}
```
**Cause:** `checkDuplicateFieldIDs` [skips IDs ≤
0](https://github.com/apache/iceberg-go/blob/b354505/schema.go#L104-L106)
(#879), so `reassignIDs`
[resolves](https://github.com/apache/iceberg-go/blob/b354505/table/metadata.go#L3740)
source ID 0 to a name, and [map
order](https://github.com/apache/iceberg-go/blob/b354505/schema.go#L1129-L1133)
picks one of the two names it has. Every catalog except REST takes this path.
**Suggestion:** have `CreateTable` reject duplicate field IDs, including 0
but single 0 should stay valid (#761).
It seems Java version looks fine since its `types/IndexByName.java`'s
`byId()` method uses `ImmutableMap` to avoid the duplicated IDs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
