singhpk234 commented on code in PR #17918: URL: https://github.com/apache/iceberg/pull/17918#discussion_r4235666543
########## format/spec.md: ########## @@ -322,6 +325,35 @@ For `geography` types, an additional parameter A specifies an algorithm for inte * `andoyer`: Thomas, Paul D. Mathematical models for navigation systems. US Naval Oceanographic Office, 1965. * `karney`: [Karney, Charles FF. "Algorithms for geodesics." Journal of Geodesy 87 (2013): 43-55](https://link.springer.com/content/pdf/10.1007/s00190-012-0578-z.pdf), and [GeographicLib](https://geographiclib.sourceforge.io/) +#### File Type + +A **`file`** represents a range of bytes that may be stored inline as a value or as a reference to an external file. The `file` type and its value semantics are defined by the `FILE` logical type in the [Parquet project](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file). + +A `file` value has a fixed set of sub-fields. The sub-fields are implicit: they are not represented in the Iceberg schema and cannot be added, removed, reordered, or promoted. Their names, types, and field-ID offsets are: + +| Sub-field | ID offset | Type | +|----------------|-----------|----------| +| `uri` | +1 | `string` | +| `offset` | +2 | `long` | +| `size` | +3 | `long` | +| `content_type` | +4 | `string` | +| `checksum` | +5 | `string` | +| `inline` | +6 | `binary` | + +A `file` field reserves the root field's ID plus six consecutive IDs for its sub-fields, assigned by the offsets above. Adding a `file` field must advance `last-column-id` to account for all 7 IDs. Review Comment: in case of struct containing `file` type we would have to adjust the id assignment as top level fields are assigned first then nested ones ... not an issue from spec POV but i wonder if we should add this an implementation note ? ########## format/spec.md: ########## @@ -322,6 +325,35 @@ For `geography` types, an additional parameter A specifies an algorithm for inte * `andoyer`: Thomas, Paul D. Mathematical models for navigation systems. US Naval Oceanographic Office, 1965. * `karney`: [Karney, Charles FF. "Algorithms for geodesics." Journal of Geodesy 87 (2013): 43-55](https://link.springer.com/content/pdf/10.1007/s00190-012-0578-z.pdf), and [GeographicLib](https://geographiclib.sourceforge.io/) +#### File Type + +A **`file`** represents a range of bytes that may be stored inline as a value or as a reference to an external file. The `file` type and its value semantics are defined by the `FILE` logical type in the [Parquet project](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file). + +A `file` value has a fixed set of sub-fields. The sub-fields are implicit: they are not represented in the Iceberg schema and cannot be added, removed, reordered, or promoted. Their names, types, and field-ID offsets are: + +| Sub-field | ID offset | Type | +|----------------|-----------|----------| +| `uri` | +1 | `string` | +| `offset` | +2 | `long` | +| `size` | +3 | `long` | +| `content_type` | +4 | `string` | +| `checksum` | +5 | `string` | +| `inline` | +6 | `binary` | + +A `file` field reserves the root field's ID plus six consecutive IDs for its sub-fields, assigned by the offsets above. Adding a `file` field must advance `last-column-id` to account for all 7 IDs. + +The `uri` field may contain absolute or relative references. Implementations that receive a relative path should resolve the path against the table location (see [Path Resolution](#path-resolution)). + +Statistics for `file` are tracked using separate field stats for each sub-field. Writers should produce statistics for `uri`, `content_type`, and `inline` fields; other fields may be omitted. Review Comment: [doubt] what was thought process to include only uri / content_type / inline and not size ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
