kevinjqliu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324758344
> I dont think fixing SqlCatalog alone is the proper answer to this bug. The io layer seems to me ill written and has to be fixed somewhere in the uppper level (e.g. FsspecInputFile or InputFile). yea, the main issue is the assumption that the same `io` (and `fs` implementation) is used for reading both data and metadata files. The example you pointed to pass in the `io` parameter https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc37eb3/pyiceberg/table/__init__.py#L655-L657 Instead, we would want to recreate `io`/`fs` based on the file currently being processed. Here's another example of passing in the `io` parameter on the write path https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc37eb3/pyiceberg/table/__init__.py#L530-L532 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
