Xuanwo commented on code in PR #176:
URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1470676713
##########
crates/iceberg/src/io.rs:
##########
@@ -278,7 +288,7 @@ impl OutputFile {
}
/// Creates output file for writing.
- pub async fn writer(&self) -> Result<impl FileWrite> {
+ pub async fn writer(&self) -> Result<Writer> {
Ok(self.op.writer(&self.path[self.relative_path_pos..]).await?)
Review Comment:
> Yes, we can open a discussion about it. IMO, the concurrent writer is used
in the case that writes multiple files concurrently. So concurrent writer may
should be a higher-level writer.
Yes, there are two levels of concurrency here. We can write different files
concurrently, and we can also write different parts of the same file
concurrently. The second case is particularly relevant when writing a large
parquet file, such as 10GiB or even 1TiB.
However, we don't need to delve into such details at the current stage. I
just want to note that there are potential optimizations possible. Should we
leave a todo here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
