Gabriel39 opened a new pull request, #68859:
URL: https://github.com/apache/doris/pull/68859
### What problem does this PR solve?
A `VARBINARY(16)` column becomes an unspecified-length `VARBINARY` after a
Block serialization round trip. Iceberg FIXED/UUID writers then reject it with
`Iceberg fixed width does not match Doris VARBINARY length: expected 16, got
-1`, including when inserting omitted columns with write defaults.
`PTypeDesc` already preserves the declared length, but Blocks use
`PColumnMeta`, which had no length field. The receiver consequently
reconstructed every VARBINARY column with the default length of -1.
Add an optional `varbinary_length` field to `PColumnMeta`, serialize it from
`DataTypeVarbinary`, and restore it in `DataTypeFactory`. The field defaults to
-1 so metadata from older senders retains its existing behavior. Nested and
nullable types reuse the same metadata path. Iceberg width validation remains
unchanged.
### Release note
Preserve VARBINARY declared lengths across Block serialization to prevent
Iceberg FIXED/UUID write failures.
### Check List (For Author)
- Test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Serialized Blocks retain VARBINARY lengths, including nested
fields.
- Does this need documentation?
- [x] No.
### Validation
New tests cover:
- Full compressed Block and protobuf wire round trips for lengths -1, 16,
and 32, with nullable and non-nullable columns, NULL values, and embedded zero
bytes.
- Nested array/map/struct metadata retaining independent binary lengths.
- Legacy metadata without the new field restoring length -1.
Before the fix, the Block and nested metadata tests fail with lengths
restored as -1; the legacy test passes.
After the fix, all 17 `DataTypeVarbinaryTest.*` tests pass in an ASAN test
binary. The focused build regenerated `data.proto` and rebuilt the affected
type/Block implementations and test source, reusing existing dependency objects.
Also verified old/new protobuf wire compatibility and passed clang-format 16
checks on all changed C++ files.
Full distributed Iceberg regression remains for CI. Old senders still omit
lengths, so the fix does not recover information already lost by an older BE.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
