kszucs opened a new pull request, #3890:
URL: https://github.com/apache/iceberg-python/pull/3890
Supersedes #3609, which the stale bot closed and GitHub won't let me reopen.
Rebased on current `main`, with the review feedback from that round folded in
(see below).
# Rationale for this change
HuggingFace Hub (`hf://`) storage already works via fsspec's `HfFileSystem`,
but there's no way to select a revision (branch/tag/commit) the way
iceberg-rust's `hf.revision` does. `HfFileSystem` resolves `revision` per call
rather than as a filesystem-wide default (its constructor only takes
`endpoint`/`token`/`block_size`/`expand_info`), so `hf.revision` is threaded
through as a keyword argument on the fsspec input file, output file and delete
calls.
Semantics match iceberg-rust's `HF_REVISION` ("default git revision for all
paths that don't specify one"): the property applies to reads, writes and
deletes, and a revision embedded in the location —
`hf://datasets/user/repo@revision/path` — takes precedence. That precedence
isn't cosmetic: `huggingface_hub` raises `ValueError: Revision specified in
path (...) and in 'revision' argument (...) are not the same` when both are
present and differ, so passing the kwarg unconditionally would break any table
whose locations pin a revision themselves.
Two smaller things fall out of this:
- `FsspecInputFile.exists()` went through `lexists()`, which doesn't forward
`**kwargs` (fsspec's `AbstractFileSystem.lexists` calls `self.exists(path)`,
dropping them). It now calls `exists()` directly when there are kwargs to
forward — that path forwards to `info()` with the same broad exception handling
`lexists()` would have provided, and `LocalFileSystem`'s symlink-aware
`lexists()` override is still used when there is nothing to forward.
- `FsspecOutputFile.to_input_file()` carries the kwargs over, so a read-back
after a write stays on the same revision.
## Are these changes tested?
Yes: unit tests covering revision forwarding on reads (`len()`, `.exists()`,
`.open()`), on writes and deletes, path-embedded revisions taking precedence,
the no-revision-set default, and non-`hf` schemes being unaffected. Plus an
`HF_TOKEN`-gated integration test against a real temporary dataset repo that
demonstrates the concrete problem: without `hf.revision`, reads follow the
moving default branch, so a file read at one commit silently returns different
content once someone pushes to the same path.
## Are there any user-facing changes?
Yes: a new `hf.revision` catalog/table property, documented in
`configuration.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
