paveon opened a new pull request, #2391:
URL: https://github.com/apache/iceberg-go/pull/2391
## What
Check a data file's schema against the table's without formatting a report
row for every field, and convert the file's Arrow schema to an Iceberg schema
once instead of twice.
## Why
`fileToDataFile` runs `checkSchemaCompat` for every file added with
`AddFiles`/`AddDataFilesByPath`. The visitor formatted a `[]string{status,
lhs.String(), rhs.String()}` row for every field even when the schemas were
compatible, although the rows are only used in the mismatch error. It also
cloned each provided field with `FindFieldByID`. For a file with field IDs, the
Arrow schema was converted a second time for the path-to-ID mapping.
In a service adding about 100K files an hour to tables of up to 250 columns,
the schema check was 13% of CPU and 19% of allocated bytes.
## Implementation
- `checkSchemaCompat` visits once without a report. Only when that finds a
mismatch does it visit again to build the report, so the error text is
unchanged except for one detail: the row of a nested field that is present
shows the provided field, as the rows of other fields do, instead of only its
type.
- `isFieldCompat` looks up the provided field with `FindFieldByIDRef`, which
does not clone it.
- `fileToDataFile` converts the file's schema once and reuses it for the
path-to-ID mapping when the file has field IDs.
- `visitStructPreOrder` takes the address of each field, so its closures no
longer copy every `NestedField` to the heap.
## Benchmark
`BenchmarkCheckSchemaCompat`, a compatible 250-column schema, Apple M1 Pro:
- 180 µs → 26 µs per check
- 106 KB → 26 KB, 2,263 → 503 allocations
## Test plan
- `TestCheckSchemaCompat` in `table/arrow_utils_internal_test.go` covers a
compatible schema (promotion, missing optional field) and the exact mismatch
report of a nested schema. `TestAddFilesFailsSchemaMismatch` still checks the
flat report.
- `go test ./ ./table/...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
