wenym1 commented on PR #760: URL: https://github.com/apache/iceberg-rust/pull/760#issuecomment-2801244755
> > I haven't done a benchmark on iceberg-rust yet. In RisingWave, we used to have a similar PR [risingwavelabs/risingwave#4182](https://github.com/risingwavelabs/risingwave/pull/4182). In RisingWave's LSM SST iterator, we used to have to create a `BoxedFuture` for every row in SST due to the previous use of `async_trait`. After using static-typed future, we've observed at most 20% improvement in the benchmark. > > Thank you for sharing this. However, the difference between having loops inside a `Box<dyn Iterator>` and `impl Iterator` can be quite significant compared to the difference between a `Box<dyn Future>` and `impl Future`, especially when I/O is involved. > > I suggest having a simple benchmark for us to evaluate in these cases. Just to clarify, the mentioned usage is more like `Box<dyn Stream>` vs `impl Stream`, because the LSM SST iterator also involves I/O, and this is the ultimate reason for using async. And for `Box<dyn Future>`, it means only some, rather than all, of the calls involve I/O. For example, in a concrete `FileWrite` implementation, we may have a large write buffer, and in every call on write, we append new data to the write buffer, and do I/O to send the buffered data only when the buffer is full. In this scenario, only the few writes that make the buffer full involve I/O. In the worse case, when we write data byte by byte, we will have to repeatedly create a `Box<dyn Future>` for simply writing a byte to the write buffer for each byte. Anyway, I will have some simple benchmarks later to see the effects later when I have time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
