kevinjqliu commented on issue #1317: URL: https://github.com/apache/iceberg-python/issues/1317#issuecomment-2549081163
> From the posted meesage, I see pyarrow is an expected dependency but s3fs is the problematic one Oh you're right. Sorry for the confusion. From the original message "`I would have expected that I would just need pyarrow if that was the FileIO implementation`". Looking into `s3fs` usage, its imported with `fsspec` https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3fs+language%3APython+path%3A%2F%5Epyiceberg%5C%2Fio%5C%2F%2F&type=code This means that depending on which FileIO is chosen, the required library is different. * `fsspec` requires `s3fs` to read from s3 * `pyarrow` requires only `pyarrow` and not `s3fs` Your comment above verified the `pyarrow` path, without `s3fs`. And I would assume `fsspec` doesn't use `pyarrow` at all (we can verify this). I think we can close this investigation based on the above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
