kevinjqliu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324750207
Thanks for taking a look at this @TiansuYu > why we are implementing a custom I think custom scheme parsing avoids picking one library over another (`fsspec` vs `pyarrow`). `fsspec.core.get_fs_token_paths` seems like an interesting replacement when using `fsspec`. > (On a side note: this looks a bit confusing to me, as why for gs, FSSPEC_FILE_IO is not added as a viable io method as we have added gcfs into extras.) Good catch, `fsspec` should be included since GCS is supported https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc37eb3/pyiceberg/io/fsspec.py#L161-L176 > (Also I am not sure why pyarrow is not using fsspec as io layer but implement things on their own.) `pyarrow` is using its own native fs implementations https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc37eb3/pyiceberg/io/pyarrow.py#L348-L403 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
