rdblue opened a new pull request, #12060: URL: https://github.com/apache/iceberg/pull/12060
This adds `InternalData` with read and write builder interfaces that can be used with Avro and Parquet by passing a `FileFormat`. Formats are registered by calling `InternalData.register` with callbacks to create format-specific builders. The class is `InternalData` because registered builders are expected to use the internal object model that is used for Iceberg metadata files. Using a specific object model avoids needing to register callbacks to create value readers and writers that produce the format needed by the caller. To demonstrate the new interfaces, this PR implements them using both Avro and Parquet. Parquet can't be used because it would fail at runtime until #11904 is committed (it is also missing custom type support). Avro is working. To demonstrate that the builders can be used for metadata, this updates `ManifestWriter`, `ManifestReader`, and `ManifestListWriter` to use `InternalData` builders. It was also necessary to migrate the metadata classes to extend `StructLike` for the internal writers instead of `IndexedRecord`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
