lemon0333 opened a new pull request, #3838:
URL: https://github.com/apache/iceberg-python/pull/3838
<!-- Closes #2747 -->
Closes #2747
# Rationale for this change
At the moment, assume-role (`client.role-arn` / `client.role-session-name`,
via STS) is only honored by the S3 FileIO (`pyiceberg/io/pyarrow.py`). The AWS
catalogs (Glue, DynamoDB) build their own `boto3.Session` and ignore these
properties, so users who authenticate to AWS by assuming a role cannot use it
to reach the catalog — even though the other `client.*` properties (region,
access key, etc.) are already shared between S3 FileIO and the catalogs.
This PR extends the same assume-role behavior to the `GlueCatalog` and
`DynamoDbCatalog` boto3 sessions:
- A small shared helper `_get_aws_session_with_assumed_role()` (in
`pyiceberg/catalog/__init__.py`) performs the STS `AssumeRole` from the base
session and returns a new session using the returned temporary credentials.
- Each catalog resolves the role config with the **same precedence already
used for the other `client.*` properties**: a service-prefixed override falls
back to the unified property:
- `glue.role-arn` / `dynamodb.role-arn` → `client.role-arn`
- `glue.role-session-name` / `dynamodb.role-session-name` →
`client.role-session-name`
- When no role ARN is configured, behavior is **unchanged**.
I've kept the new service-prefixed property names consistent with the
existing `glue.*` / `dynamodb.*` → `client.*` convention, but I'm happy to
adjust the naming (e.g. unified-only) to match maintainer preference — feedback
welcome.
# Are these changes tested?
Yes. New `moto`-based tests (mock STS + Glue + DynamoDB) in
`tests/catalog/test_glue.py` and `tests/catalog/test_dynamodb.py` cover:
- assume-role is triggered via the unified `client.role-arn` /
`client.role-session-name`;
- service-prefixed properties (`glue.role-arn` / `dynamodb.role-arn`) take
precedence over the unified ones;
- no assume-role call is made when no role ARN is set (default behavior
unchanged);
- the helper returns a session backed by temporary STS credentials (session
token present).
```
uv run python -m pytest tests/catalog/test_glue.py
tests/catalog/test_dynamodb.py -k "role or assume or session_properties"
# 11 passed
uv run python -m pytest tests/catalog/test_glue.py
tests/catalog/test_dynamodb.py
# 98 passed
uv run prek run --files pyiceberg/catalog/__init__.py
pyiceberg/catalog/glue.py pyiceberg/catalog/dynamodb.py \
tests/catalog/test_glue.py tests/catalog/test_dynamodb.py
# ruff / ruff-format / mypy / pydocstyle / codespell: all Passed
```
# Are there any user-facing changes?
Yes — new opt-in configuration properties for assuming an IAM role when
connecting to the Glue/DynamoDB catalogs: `client.role-arn` /
`client.role-session-name` (now honored by the catalogs too), plus
service-specific `glue.role-arn` / `glue.role-session-name` and
`dynamodb.role-arn` / `dynamodb.role-session-name`. Documented in
`mkdocs/docs/configuration.md`. No change to default behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
