guitcastro commented on PR #741: URL: https://github.com/apache/iceberg-python/pull/741#issuecomment-2274893639
> @Fokko thanks a lot for your feedback - I added docs and the constant. The constant is a very good idea - I hope we will be able to use remote signing with FileIO as well eventually. Right now only the fsspec impl. respects it. > > I saw that tabular.io actually provides explicit S3 credentials (on top of remote signing), presumably via AWS STS, if "vended-credentials" are requested (https://github.com/apache/iceberg/blob/b3c25fb7608934d975a054b353823ca001ca3742/open-api/rest-catalog-open-api.yaml#L1495). This is something that can only ever work for AWS S3 and is noticeably slower than using remote signing. As remote signing works also with on-prem deployments, I really hope this is going to become the default for all clients and not vended-credentials. tabular does this only for pyiceberg. Spark requests remote-signing so there is no need to go the extra mile and generate S3 creds. > > Right now unfortunately in pyiceberg, "vended-credentials" is hardcoded > > https://github.com/apache/iceberg-python/blob/42afc439d362ef1b3dcff03a1ffd959bc0a399ca/pyiceberg/catalog/rest.py#L501 > > , even though "remote-signing" is actually supported via fsspec. If the server decides to just ignore what the client requests and push remote signing anyway together with: > > ``` > "rest.sigv4-enabled": "true", > "py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO", > ``` > > it works like a charm. Unfortunately, for nessie, when using `X-Iceberg-Access-Delegation: vended-credentials` does not work. The endpoint does not return the `s3.signer.uri`. When the head is set to `remote-signing` it does return the correct value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
