David-Banquet commented on issue #2657: URL: https://github.com/apache/iceberg-python/issues/2657#issuecomment-6146064385
Still reproducible on main (b32392dc) against S3 Tables, with a single account. I build the `botocore_session` from a valid profile and set bogus `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` in the environment. `create_namespace` and `create_table` go through the REST signer and succeed, then `append` fails in `CreateMultipartUpload` with `INVALID_ACCESS_KEY_ID`: the FileIO used the environment credentials. `botocore_session` ends up in `table.io.properties`, but neither `PyArrowFileIO` (`_initialize_s3_fs`) nor `FsspecFileIO` (`_s3`) reads it. They only build credentials from the string properties (`s3.*`, `client.*`) or the default chain. That's the same for the Glue and DynamoDB catalogs, where the session is only passed to the boto3 client. #2071 added it for SigV4 signing, and the docs don't mention it. Two ways I can see to fix it: 1. When `botocore_session` is set and no `s3.access-key-id` is given, have the FileIO take the session's credentials. The catch is that pyarrow's `S3FileSystem` only takes static keys, so the credentials would be frozen at FileIO creation and expire for SSO or assumed-role sessions in long-running processes. 2. Document that `botocore_session` only applies to the catalog client, and that the FileIO needs `client.*`/`s3.*` credentials or the default chain. Is 1 something you'd accept, given the expiry? I can open a PR for either one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
