qzyu999 opened a new issue, #17748: URL: https://github.com/apache/iceberg/issues/17748
### Problem Iceberg's write path (`FileAppender<Record>`, `TaskWriter<Record>`) accepts only row objects. Projects with Arrow-columnar data (Fluss, Flink, PyIceberg) must materialize `GenericRecord` instances to write Iceberg tables, even when the source data is already in columnar form. The vectorized read path exists (`iceberg-arrow` module) but there is no write-side equivalent. ### Proposal Add a `FileAppender<VectorSchemaRoot>` implementation (likely in `iceberg-arrow` or a new `iceberg-arrow-writer` module) that: 1. Accepts Arrow `VectorSchemaRoot` batches 2. Writes to Parquet using Iceberg's existing `ParquetWriter` infrastructure (or `parquet-java`'s Arrow writer once [parquet-java#3733](https://github.com/apache/parquet-java/issues/3733) lands) 3. Collects Iceberg `Metrics` (column sizes, null counts, min/max bounds) from the Arrow vectors 4. Integrates with existing `OutputFileFactory`, `TaskWriter`, and file rolling logic ### Motivation - **Apache Fluss** — Arrow-native streaming storage that tiers to Iceberg. Currently materializes `GenericRecord` per row ([fluss#4047](https://github.com/apache/fluss/issues/4047)). - **Apache Flink** — Flink's Iceberg connector serializes internal rows to `Record` objects at the write boundary. - **PyIceberg / iceberg-rust** — Arrow-native clients (Polars, DataFusion, DuckDB) writing to Iceberg. ### Dependencies - Can start today using Iceberg's internal `ColumnWriteStore` access (Iceberg's `ParquetWriter` already manages this directly, not through `parquet-java`'s public `ParquetWriter.Builder`) - Full zero-copy encoding depends on [parquet-java#3733](https://github.com/apache/parquet-java/issues/3733), but an initial implementation iterating Arrow vectors per-value is still a significant improvement over row-object construction ### Scope - New module or extension in `iceberg-arrow` - Existing `FileAppender<Record>` and `TaskWriter<Record>` APIs unchanged - Arrow dependency remains optional -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
