danielcweeks commented on code in PR #15630: URL: https://github.com/apache/iceberg/pull/15630#discussion_r3261859399
########## format/spec.md: ########## @@ -168,6 +185,46 @@ All columns must be written to data files even if they introduce redundancy with Writers are not allowed to commit files with a partition spec that contains a field with an unknown transform. +### Paths in Metadata + +Path strings stored in Iceberg metadata location fields are classified as one of two types: + +* **Absolute path** -- A path string that includes a [URI scheme](https://datatracker.ietf.org/doc/html/rfc3986#section-3.1) (e.g., `s3:`, `gs:`, `hdfs:`, `file:`). Absolute paths are used as-is without modification. +* **Relative path** -- A path string that does not include a URI scheme. Relative paths must be resolved against the table's base location before use. + +Prior to v4, all path fields must contain fully-qualified paths. Starting with v4, path fields may contain either absolute or relative paths. [Relative resolution within a URI](https://datatracker.ietf.org/doc/html/rfc3986#section-5.2) (e.g. `.` and `..`) and other file system navigation conventions are not supported in relative paths. + +#### Path Resolution + +Path resolution is the process of producing an absolute path from a relative path by combining it with the table's base location: + +* If the path contains a URI scheme, it is absolute and is used without modification. +* If the path does not contain a URI scheme, the resolved path is the table location followed by the relative path joined by the URI separator character `/`. + +Paths used as prefixes should not end in a path separator. The relative portion is joined to the prefix without consideration of any additional separator characters. Review Comment: I don't agree with these comments and it's not consistent with what we've discussed. We're trying to treat the location fields as opaque as possible and still support the maximum number of scenarios. The `should` requirement here is intentional in that we still want a way to represent a `//` in a path since it is technically valid in some storage solutions. The "without consideration" is also intentional in that the strings should not be interpreted to try to normalize the path values. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
