nssalian opened a new pull request, #17520:
URL: https://github.com/apache/iceberg/pull/17520
### Rationale for the change
Kafka Connect and other generic `Record` writers can't shred variant columns
`GenericFormatModels` registers the `Record` Parquet model with no analyzer, so
`write.parquet.shred-variants=true` silently does nothing.
### Changes
- New `RecordVariantShreddingAnalyzer`: stateless, resolves variant columns
by position
in `Schema.columns()`.
- `GenericFormatModels` registers it for the `Record` Parquet model.
- `ParquetFormatModel`: opt-in `shouldDeriveEngineSchema()` hook derives the
engine schema from the Iceberg schema when none is set (order-independent).
- `RegistryBasedFileWriterFactory`: sets `engineSchema` only when input
schema is non-null.
- Docs: note per-file shred layout and the tunable inference buffer size.
Note this continues forward @soumilshah1995's
[#16370](https://github.com/apache/iceberg/pull/16370) whilst incorporating the
feedback on it. Thanks Soumil for the initial work on this.
### Testing
- New `TestRecordVariantShreddingAnalyzer`: tests for column resolution,
null/non-Variant handling, Kafka connect round-trip via
`GenericFileWriterFactory`, and the buffer-flush boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
