Fokko commented on issue #310: URL: https://github.com/apache/iceberg-python/issues/310#issuecomment-1913318831
What would be your proposal? The [FileIO is an abstraction](https://tabular.io/blog/iceberg-fileio-cloud-native-tables/) layer to use different implementations for your needs. For example, fsspec is lightweight compared to Arrow and might be preferred if you are inside of a lambda/cloud function or in an orchestration engine like Apache Airflow. As you mentioned, Arrow is more equipped to read tables. Next to that, PyIceberg is designed to be used as a library as part of a query engine. If that query engine prefers a different implementation to fetch the data from an object store, the FileIO abstraction layer allows for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
