ZENOTME commented on issue #34: URL: https://github.com/apache/iceberg-rust/issues/34#issuecomment-1684831408
> We can expose a similiar trait as FileIO and keep opendal underhood as an implementation. I think the next thing is to decide the FileIo interface and we can work on our SpecificFormatWriter.(Or call it FileWriter, FileAppender). Here is some info may useful for FileIo interface design: 1. For [parquet writer](https://docs.rs/parquet/45.0.0/parquet/arrow/async_writer/struct.AsyncArrowWriter.html), what we need to pass in is a [AsynWriter](https://docs.rs/tokio/1.29.1/tokio/io/trait.AsyncWrite.html). 2. For [avro writer](https://docs.rs/apache-avro/latest/apache_avro/struct.Writer.html), it only support sync [writer](https://doc.rust-lang.org/nightly/std/io/trait.Write.html). In icelake, to support async write, what we do is write it to Vec<u8> first and then write this `Vec<u8>` using [another async write method in opendal::Operator](https://github.com/icelake-io/icelake/blob/393d000f2e952bd32045da87a0d06d66047278f0/icelake/src/types/on_disk/manifest_list.rs#L215). And as suggestion, interface may need to have the following function look like: ``` trait FileIo { fn writer() -> impl FileWriter } trait FileWriter { // This interface can be used by parquet writer. fn aysnc_writer() -> impl AsynWriter; // These interfaces can be used by avro writer. async fn write(); async fn close(); } ``` cc @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
