zeroshade opened a new issue, #2185:
URL: https://github.com/apache/iceberg-go/issues/2185
On flat object stores (S3, GCS, in-memory), `blobfs.FileIO.WalkDir` reports
a trailing-slash marker for a **subdirectory** as a file named after that
directory.
Repro (memblob bucket):
```
keys: tbl/, tbl/data/, tbl/data/a.parquet, tbl/metadata/v1.json
WalkDir("s3://mybucket/tbl"):
D s3://mybucket/tbl
D s3://mybucket/tbl/data
F s3://mybucket/tbl/data/a.parquet
F s3://mybucket/tbl/data/data <- the tbl/data/ marker
D s3://mybucket/tbl/metadata
F s3://mybucket/tbl/metadata/v1.json
```
`isDirectoryMarker` only skips the marker of the walk root. A marker one
level down is listed under the prefix `tbl/data/` as the object `tbl/data/`,
named by `path.Base` as `data`, and joined onto the parent path as
`tbl/data/data`, a path that doesn't exist. Orphan cleanup (`PlanOrphanFiles` /
`clean-orphan-files`) would report it as an orphan and try to delete a
nonexistent key.
This is the behavior on `main` and is unchanged by #2180, whose `walkFS`
deliberately names entries the way gocloud's `fs.FS` does. Fix: skip any listed
object whose key equals the directory prefix being read (i.e. `obj.Key ==
prefix`) in `walkFS.ReadDir`, and add a subdirectory-marker case to the WalkDir
tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
