ajiteshsingh opened a new pull request, #2180: URL: https://github.com/apache/iceberg-go/pull/2180
Fixes #2159 ## Problem On ADLS Gen2 with a hierarchical namespace (OneLake), every directory is also stored as a 0-byte blob named after it, with no trailing slash. `blobfs.FileIO.WalkDir` walked gocloud's `fs.FS`, which treats a key as a file whenever an object with that exact key exists. So: - the walk root opened as a file, and the walk never descended - subdirectories appeared twice, once as a 0-byte file and once as a directory, and listing them failed with `readdir: not implemented` As a result, `PlanOrphanFiles` / `clean-orphan-files` reported the table location itself as the only orphan (0 bytes). ## Fix `WalkDir` now walks a small adapter over the bucket (`walkFS`), which implements `fs.StatFS` and `fs.ReadDirFS`: - `Stat` treats any key with objects under it as a directory, checked with a 1-key `ListPage` - `ReadDir` lists one level and hides a 0-byte object named after a child directory Trailing-slash markers on flat stores (S3/GCS) walk exactly as before. The walk also makes fewer requests: one listing per directory, instead of an `Exists` check plus a listing. ## Caveats - **Applies on every backend.** The rule infers "directory blob" from size and name, because listings don't return Azure's `hdi_isfolder` metadata. So on S3 or GCS, a real empty file named exactly like a sibling directory (object `x` next to keys under `x/`) is no longer reported. Iceberg doesn't write such files. - **Empty hierarchical-namespace directories are still reported as 0-byte files.** With no children, they can only be identified through `hdi_isfolder`, which costs an `Attributes` call per 0-byte object. Table locations are never empty, and deleting an empty directory's blob is harmless. This is noted in a code comment. ## Testing - New `TestBlobFileIOWalkDirHierarchicalNamespace` (in-memory bucket). It fails before this change and passes after. - `go test ./io/... ./catalog/hadoop/... ./table/maintenance/... ./cmd/iceberg/...` passes. - Not tested against a real OneLake account. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
