c-thiel opened a new issue, #11608:
URL: https://github.com/apache/iceberg/issues/11608
### Apache Iceberg version
1.7.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Currently when configuring two REST catalogs in spark, the `s3.signer.uri`
of the first catalog is used also for the second catalog.
During initial connect to the REST catalog, the catalog may return a
`s3.signer.uri` attribute as part of the `overrides` of the `/v1/config`
endpoint. This property seems to be set globally for the spark session.
Whichever catalog I use first, the sign request for the second catalog is sent
to the sign endpoint of the first. Using each catalogs separately works
perfectly fine.
I tested with one Lakekeeper where we use different sign endpoints for each
warehouse as well as with two Nessies. Warehouses share the same bucket but use
different path prefixes in my tests.
My spark configuration looks like this:
```python
"spark.sql.catalog.catalog1": "org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.catalog1.type": "rest",
"spark.sql.catalog.catalog1.uri": CATALOG_1_URL,
"spark.sql.catalog.catalog1.warehouse": "warehouse_1",
"spark.sql.catalog.catalog1.io-impl":
"org.apache.iceberg.aws.s3.S3FileIO",
"spark.sql.catalog.catalog1.s3.remote-signing-enabled": "true",
"spark.sql.catalog.catalog2": "org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.catalog2.type": "rest",
"spark.sql.catalog.catalog2.uri": CATALOG_2_URL,
"spark.sql.catalog.catalog2.warehouse": "warehouse_2",
"spark.sql.catalog.catalog2.io-impl":
"org.apache.iceberg.aws.s3.S3FileIO",
"spark.sql.catalog.catalog1.s3.remote-signing-enabled": "true",
```
If required, I can add a docker compose example as well.
If someone could point me into the right direction, I might be able to
create a fix PR.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [X] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
