EthanBlackburn opened a new issue, #398:
URL: https://github.com/apache/iceberg-go/issues/398
### Apache Iceberg version
None
### Please describe the bug 🐞
My schema contains a _nullable_ struct with _required_ fields. When I call
`AppendTable` with the data, I get the panic
```
error encountered during schema visitor invalid: field says not-nullable,
child #5 has nulls
stack trace: goroutine 103 [running]:
runtime/debug.Stack()
/Users/ethanblackburn/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.darwin-arm64/src/runtime/debug/stack.go:26
+0x64
github.com/apache/iceberg-go.VisitSchemaWithPartner[...].func1()
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/schema.go:1323
+0x80
panic({0x107544fc0?, 0x140012a6d60?})
/Users/ethanblackburn/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.darwin-arm64/src/runtime/panic.go:792
+0x124
github.com/apache/iceberg-go/table.retOrPanic[...](...)
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/table/arrow_utils.go:683
github.com/apache/iceberg-go/table.(*arrowProjectionVisitor).Struct(0x140012a63a0,
{{0x14000f1b888?, 0x109922ac0?, 0x107b2cbe0?}}, {0x107b60fe0, 0x14000e78f00},
{0x14000f57c00, 0x8, 0x14001678f00?})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/table/arrow_utils.go:820
+0x4e0
github.com/apache/iceberg-go.visitStructWithPartner[...]({{0x14000f1b888,
0x0?, 0x0?}}, {0x107b60fe0?, 0x14000e78f00}, {0x107b44d50, 0x140012a63a0},
{0x107b40220, 0x14001678fc0})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/schema.go:1366
+0x360
github.com/apache/iceberg-go.visitTypeWithPartner[...]({0x107b2c9a0?,
0x14000522168?}, {0x107b60fe0?, 0x14000e78f00?}, {0x107b44d50?,
0x140012a63a0?}, {0x107b40220?, 0x14001678fc0?})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/schema.go:1435
+0x98
github.com/apache/iceberg-go.visitStructWithPartner[...]({{0x14000e42e08,
0x0?, 0x0?}}, {0x107b60fe0?, 0x14000e78ec0}, {0x107b44d50, 0x140012a63a0},
{0x107b40220, 0x14001678fc0})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/schema.go:1359
+0x234
github.com/apache/iceberg-go.visitTypeWithPartner[...]({0x107b2c9a0?,
0x14000522138?}, {0x107b60fe0?, 0x14000e78ec0?}, {0x107b44d50?,
0x140012a63a0?}, {0x107b40220?, 0x14001678fc0?})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/schema.go:1435
+0x98
github.com/apache/iceberg-go.visitStructWithPartner[...]({{0x14001090008,
0x14000e78bc0?, 0x0?}}, {0x107b60fe0?, 0x14000e78bc0}, {0x107b44d50,
0x140012a63a0}, {0x107b40220, 0x14001678fc0})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/schema.go:1359
+0x234
github.com/apache/iceberg-go.VisitSchemaWithPartner[...](0x14000b91f10,
{0x107b60fe0, 0x14000e78bc0}, {0x107b44d50, 0x140012a63a0}, {0x107b40220,
0x14001678fc0})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/schema.go:1335
+0xf4
github.com/apache/iceberg-go/table.ToRequestedSchema({0x107b35350,
0x109922ac0}, 0x14000b91f10, 0x14001678fc0, {0x107b5b670?, 0x14000e71e00?},
0x0, 0x1, 0x0)
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/table/arrow_utils.go:893
+0x110
github.com/apache/iceberg-go/table.(*writer).writeFile(0x140009fce10,
{0x107b35350, 0x109922ac0}, {{0x7a, 0x10, 0x34, 0xae, 0x69, 0x34, 0x44, ...},
...})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/table/writer.go:65
+0x120
github.com/apache/iceberg-go/table.writeFiles.func3({{0x7a, 0x10, 0x34,
0xae, 0x69, 0x34, 0x44, 0x95, 0x94, 0x94, ...}, ...})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/table/writer.go:126
+0x48
github.com/apache/iceberg-go/table/internal.MapExec[...].func1()
/Users/ethanblackburn/go/pkg/mod/github.com/apache/iceberg-go@v0.2.1-0.20250420133620-1dbffabc82f2/table/internal/utils.go:501
+0xd0
golang.org/x/sync/errgroup.(*Group).Go.func1()
/Users/ethanblackburn/go/pkg/mod/golang.org/x/sync@v0.13.0/errgroup/errgroup.go:79
+0x54
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
/Users/ethanblackburn/go/pkg/mod/golang.org/x/sync@v0.13.0/errgroup/errgroup.go:76
+0x94
```
## Schema
```
schema:
fields: 34
...
- finding_info: type=struct<analytic: struct<category: utf8, desc: utf8,
name: utf8, related_analytics: list<item: struct<category: utf8, desc: utf8,
name: utf8, type: utf8, type_id: int32, uid: utf8, version: utf8>, nullable>,
type: utf8, type_id: int32, uid: utf8, version: utf8>, uid: utf8>
metadata: ["iceberg.field_id": "17"]
...
```
### Sample data
```
...
finding_info (nullable=false) len=100 nulls=0 type=struct<analytic:
struct<category: utf8, desc: utf8, name: utf8, related_analytics: list<item:
struct<category: utf8, desc: utf8, name: utf8, type: utf8, type_id: int32, uid:
utf8, version: utf8>, nullable>, type: utf8, type_id: int32, uid: utf8,
version: utf8>, uid: utf8>
analytic (nullable=true) len=100 nulls=100 type=struct<category:
utf8, desc: utf8, name: utf8, related_analytics: list<item: struct<category:
utf8, desc: utf8, name: utf8, type: utf8, type_id: int32, uid: utf8, version:
utf8>, nullable>, type: utf8, type_id: int32, uid: utf8, version: utf8>
category (nullable=true) len=100 nulls=100 type=utf8
desc (nullable=true) len=100 nulls=100 type=utf8
name (nullable=true) len=100 nulls=100 type=utf8
related_analytics (nullable=true) len=100 nulls=100 type=list<item:
struct<category: utf8, desc: utf8, name: utf8, type: utf8, type_id: int32, uid:
utf8, version: utf8>, nullable>
related_analytics.element (nullable=true) len=0 nulls=0
type=struct<category: utf8, desc: utf8, name: utf8, type: utf8, type_id: int32,
uid: utf8, version: utf8>
category (nullable=true) len=0 nulls=0 type=utf8
desc (nullable=true) len=0 nulls=0 type=utf8
name (nullable=true) len=0 nulls=0 type=utf8
type (nullable=true) len=0 nulls=0 type=utf8
type_id (nullable=false) len=0 nulls=0 type=int32
uid (nullable=true) len=0 nulls=0 type=utf8
version (nullable=true) len=0 nulls=0 type=utf8
type (nullable=true) len=100 nulls=100 type=utf8
type_id (nullable=false) len=100 nulls=100 type=int32 <--- This
line triggers the panic
uid (nullable=true) len=100 nulls=100 type=utf8
...
```
Iceberg allows nullable structs with required fields, so I assume there's a
bug in the validation. I think its coming from `NewStructArrayWithFields` but
not positive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
