c-thiel commented on PR #741: URL: https://github.com/apache/iceberg-python/pull/741#issuecomment-2127322592
@Fokko I added docs and the constant. The constant is a very good idea - I hope we will be able to use remote signing with FileIO as well eventually. Right now only the fsspec impl. respects it. I saw that tabular.io actually provides explicit S3 credentials (on top of remote signing), presumably via AWS STS, if "vended-credentials" are requested (https://github.com/apache/iceberg/blob/b3c25fb7608934d975a054b353823ca001ca3742/open-api/rest-catalog-open-api.yaml#L1495). This is something that can only ever work for AWS S3 and is noticeably slower than using remote signing. As remote signing works also with on-prem deployments, I really hope this is going to become the default for all clients and not vended-credentials. tabular does this only for pyiceberg. Spark requests remote-signing so there is no need to go the extra mile and generate S3 creds. Right now unfortunately in pyiceberg, "vended-credentials" is hardcoded https://github.com/apache/iceberg-python/blob/42afc439d362ef1b3dcff03a1ffd959bc0a399ca/pyiceberg/catalog/rest.py#L501 , even though "remote-signing" is actually supported via fsspec. If the server decides to just ignore what the client requests and push remote signing anyway together with: "rest.sigv4-enabled": "true", "py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO", it works like a charm. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
