Fokko commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-1912130317
Thanks @tustvold for raising this and please don't hesitate to open an issue or PR. > For example Spark has had a very hard time getting a performant S3 integration, with proper vectored IO only being added to OSS Spark https://github.com/apache/arrow-datafusion/issues/2205#issuecomment-1100069800. This is why the Iceberg Java implementation ships with its own vectorized parquet reader :) It looks to me that `object_store` and FileIO aim to solve the same problem. Iceberg is designed to work on object stores from the start, and not on filesystems. Similar to object_store the FileIO concept is very opinionated. Since many people are still on HDFS, this is also supported since Filesystems offer stronger guarantees than object stores. If you want to learn more about the FileIO concept, [this](https://tabular.io/blog/iceberg-fileio-cloud-native-tables/) is a good primer on the concept. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
