rdblue commented on code in PR #17918: URL: https://github.com/apache/iceberg/pull/17918#discussion_r4222159702
########## format/spec.md: ########## @@ -322,6 +325,36 @@ For `geography` types, an additional parameter A specifies an algorithm for inte * `andoyer`: Thomas, Paul D. Mathematical models for navigation systems. US Naval Oceanographic Office, 1965. * `karney`: [Karney, Charles FF. "Algorithms for geodesics." Journal of Geodesy 87 (2013): 43-55](https://link.springer.com/content/pdf/10.1007/s00190-012-0578-z.pdf), and [GeographicLib](https://geographiclib.sourceforge.io/) +#### File Type + +A **`file`** represents a range of bytes that may be stored inline as a value or a reference to an external file. The `file` type and its value semantics are defined by the `FILE` logical type in the [Parquet project](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file). + +A `file` value has a fixed set of sub-fields. The sub-fields are implicit: they are not represented in the Iceberg schema and cannot be added, removed, reordered, or promoted. Their names, types, and field-ID offsets are: + +| Sub-field | ID offset | Type | +|----------------|-----------|----------| +| `uri` | +1 | `string` | +| `offset` | +2 | `long` | +| `size` | +3 | `long` | +| `content_type` | +4 | `string` | +| `checksum` | +5 | `string` | +| `inline` | +6 | `binary` | + +Adding a `file` field reserves the root field's ID plus six consecutive IDs for its sub-fields, assigned by the offsets above. Writers must advance `last-column-id` past all seven IDs and must not assign these IDs to any other field. + +The `uri` field may contain absolute or relative references. Relative resolution within a URI (e.g. `.` and `..`) and other file system navigation conventions are not supported. Implementations that receive a relative path should resolve the path against the table location (see [Path Resolution](#path-resolution)). + +A `file` value has no whole-value statistics. Each sub-field's statistics are tracked in `content_stats` under the sub-field's reserved ID, as for any field of the sub-field's type. Writers should produce statistics for `uri`, `content_type`, and `inline` fields; other fields may be omitted. + +A `file` column is subject to the following restrictions: + +* Non-null values for `initial-default` or `write-default` are invalid. +* No type promotion to or from `file` is defined. +* Whole-value equality, ordering, and hashing are not defined. +* A `file` column cannot be an identifier field or a source for partition or sort transforms. +* A `file` is not interchangeable with a `struct`; a `struct` with the same sub-fields is not equivalent to a `file`. +* The behavior for a `file` entry that references a non-existent file is not defined. Review Comment: 😬 I think we need to expand this more. It isn't that the behavior isn't defined, it is delegated to engines, right? Iceberg doesn't guarantee that the reference is valid, which we should state. But the behavior is context- and engine-dependent. An engine could have a `is_valid_file` function that relies directly on whether the reference can be resolved, for example. And an engine may fail if a value is required but return `null` if the reference is no longer valid. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
