VivekMalipatel commented on issue #3896:
URL:
https://github.com/apache/iceberg-python/issues/3896#issuecomment-5700354916
We hit this in production against a remote-signing REST catalog (Lakekeeper)
with an S3-compatible store, and tracked it to a specific interaction between
two caches. Reads are affected as well as writes. Sharing the mechanism, the
evidence, and — more usefully — why the obvious one-line fix is **not** safe in
general.
Versions: `pyiceberg 0.12.0`, `botocore 1.43.75`, `s3fs 2026.7.0`, `fsspec
2026.7.0`, `aiobotocore 3.9.1`, Python 3.11.
### Mechanism
`pyiceberg/io/fsspec.py::_s3` installs the signer by
unregister-then-register, and sets `signature_version=UNSIGNED`:
```python
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function,
unique_id=1925)
```
Two caches with different scopes collide:
* `FsspecFileIO.get_fs` is memoised **per thread** (`threading.local` +
`lru_cache`), so every worker in the scan pool runs `_s3` itself.
* fsspec's `_Cached` metaclass caches `S3FileSystem` **per process**, so all
of those workers get the *same* instance and mutate the *same*
`fs.s3.meta.events` emitter.
Between the `unregister` and the `register_last` there is no
`before-sign.s3` handler. Because signing is `UNSIGNED`, a request that
serialises in that gap is not signed by anything and goes out **with no
`Authorization` header at all** — it is not a bad signature, it is an
unauthenticated request. The store then answers with whatever it does for
anonymous access; for us that is `403 AccessDenied`, which is why this reads as
a permissions problem and sends you looking at the catalog, the credentials,
and clock skew.
Short-lived jobs are hit hardest: the registration storm and the first
object requests are the same event, so the **first** file of a scan was the
victim nearly every time, while a long-lived process that warms its pool once
never failed.
### Evidence
Wrapping `AioEndpoint._send` to pair each request with its response:
* failing request carried only `User-Agent`, `amz-sdk-invocation-id`,
`amz-sdk-request`, and the signer had never seen its URL
* a request that succeeded seconds earlier on the same connection carried
`Authorization`, `x-amz-date`, `x-amz-content-sha256`, and the signer had seen
its URL
A packet capture confirmed the failing request's response body is
byte-identical to that of a deliberately unauthenticated request, and distinct
from both a duplicated-header and a wrong-secret response.
Interleaved A/B on one scan, n=50: **9/25 failures with 6 workers, 0/25 with
1 worker**, and every failure carried at least one unsigned request while no
success did. In production, across short-lived jobs, 23 of 47 attempts (49%)
failed before and 0 of 10 after.
### Workaround
Setting max workers to 1 for short-lived readers removes it entirely. We
deliberately did not add a 403 retry, since that would mask this class of bug.
### Why deleting the `unregister` is tempting but unsafe
`botocore/hooks.py::_register_section` early-returns for a `unique_id` it
already holds:
```python
if unique_id in self._unique_id_handlers:
# We've already registered a handler using this unique_id
# so we don't need to register it again.
...
return
```
So re-registering the same `unique_id` is a no-op, and it is natural to
conclude the `unregister` is redundant and can simply be dropped — closing the
window while keeping concurrency.
**That breaks multi-table processes.** With remote signing, the signer is
**per table**, not per catalog: `s3.signer.endpoint` embeds the table id, and
`S3V4RestSigner.__call__` reads `self.properties`, so each instance is bound to
the table it was constructed for. Three tables in one of our processes give
three distinct endpoints:
```
v1/signer/<warehouse-id>/tabular-id/<table-A-id>/v1/aws/s3/sign
v1/signer/<warehouse-id>/tabular-id/<table-B-id>/v1/aws/s3/sign
v1/signer/<warehouse-id>/tabular-id/<table-C-id>/v1/aws/s3/sign
```
A job that reads table A and then table B shares one process-cached
`S3FileSystem`. Drop the `unregister` and botocore's early-return keeps **table
A's** signer installed and silently skips table B's registration — so table B's
object requests are signed against table A's signer path. Against a catalog
that authorises per table location that is either a 403 for a new reason or,
worse, an authorisation that should not have been granted. The `unregister` is
not vestigial; it is what swaps the per-table signer on a shared client.
So the root cause is narrower than "the unregister": *a per-table signer
installed on a process-cached client and mutated under concurrency*.
### Suggested direction
Make multiple signers able to coexist instead of swapping one in place — for
example scope each signer to its own table's object prefix so it no-ops for
requests outside it, and register it under a `unique_id` derived from the table
id, never unregistering. No gap, no swap, concurrency preserved. Giving each
table its own `S3FileSystem` instance (bypassing the fsspec instance cache)
would also work, at the cost of more clients per process.
Whatever the fix, a regression test that **reads two different tables
concurrently in one process and asserts each request was signed against its own
table's endpoint** would be worth having: a test that only checks "no 403 under
concurrency" passes for the unsafe patch above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
