This is an automated email from the ASF dual-hosted git repository.
danielcweeks pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new 59ccccb19a SPEC: Add File Type to v4 spec (#17918)
59ccccb19a is described below
commit 59ccccb19ac12c77c237b3f4a35485506c3728fe
Author: Daniel Weeks <[email protected]>
AuthorDate: Mon Oct 19 16:07:11 2026 -0700
SPEC: Add File Type to v4 spec (#17918)
* [SPEC] Add File Type to v4 spec
Co-authored-by: Daniel Weeks <[email protected]>
Co-authored-by: Russell Spitzer <[email protected]>
Co-authored-by: Talat Uyarer <[email protected]>
Co-authored-by: Claude <[email protected]>
* Address comments
---------
Co-authored-by: Daniel Weeks <[email protected]>
Co-authored-by: Russell Spitzer <[email protected]>
Co-authored-by: Talat Uyarer <[email protected]>
Co-authored-by: Claude <[email protected]>
---
format/spec.md | 43 +++++++++++++++++++++++++++++++++++++++++--
1 file changed, 41 insertions(+), 2 deletions(-)
diff --git a/format/spec.md b/format/spec.md
index 3e8ba7232f..b265dd91e1 100644
--- a/format/spec.md
+++ b/format/spec.md
@@ -65,6 +65,7 @@ Version 4 of the Iceberg spec restructures metadata for
improved performance and
* Data files and their corresponding deletion vector are stored in a combined
entry, removing the need for two-phase planning
* Support for [relative locations](#file-locations-in-metadata) in metadata
fields
* Writing new [equality deletes](#equality-delete-files) is no longer allowed
+* New data type: `file`
The full set of changes are listed in [Appendix E](#version-4).
@@ -235,7 +236,7 @@ When the `location` field is present in table metadata, it
is used directly as t
### Schemas and Data Types
-A table's **schema** is a list of named columns. Data types are primitive,
nested, or semi-structured. Nested types are maps, lists, or structs. A table
schema is also a struct type.
+A table's **schema** is a list of named columns. Data types are primitive,
nested, or semi-structured. Nested types are maps, lists, structs, or files. A
table schema is also a struct type.
For the representations of these types in Avro, ORC, and Parquet file formats,
see Appendix A.
@@ -247,6 +248,8 @@ A **`list`** is a collection of values with some element
type. The element field
A **`map`** is a collection of key-value pairs with a key type and a value
type. Both the key field and value field each have an integer id that is unique
in the table schema. Map keys are required and map values can be either
optional or required. Both map keys and map values may be any type, including
nested types.
+A **`file`** stores a binary value, either inline or as a reference to an
external file. The structure of this type is a fixed set of [nested
fields](#file-type). These fields are referenced by offset from the field ID
assigned to the encoding type, not represented directly in the schema.
+
#### Semi-structured Types
A **`variant`** is a value that stores semi-structured data. The structure and
data types in a variant are not necessarily consistent across rows in a table
or data file. The variant type and binary encoding are defined in the [Parquet
project](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md),
with support currently available for V1. Support for Variant is added in
Iceberg v3.
@@ -328,6 +331,35 @@ For `geography` types, an additional parameter A specifies
an algorithm for inte
* `andoyer`: Thomas, Paul D. Mathematical models for navigation systems. US
Naval Oceanographic Office, 1965.
* `karney`: [Karney, Charles FF. "Algorithms for geodesics." Journal of
Geodesy 87 (2013):
43-55](https://link.springer.com/content/pdf/10.1007/s00190-012-0578-z.pdf),
and [GeographicLib](https://geographiclib.sourceforge.io/)
+#### File Type
+
+A **`file`** represents a range of bytes that may be stored inline as a value
or as a reference to an external file. The `file` type and its value semantics
are defined by the `FILE` logical type in the [Parquet
project](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file).
+
+A `file` value has a fixed set of sub-fields. The sub-fields are implicit:
they are not represented in the Iceberg schema and cannot be added, removed,
reordered, or promoted. Their names, types, and field ID offsets are:
+
+| Sub-field | ID offset | Type |
+|----------------|-----------|----------|
+| `uri` | 1 | `string` |
+| `offset` | 2 | `long` |
+| `size` | 3 | `long` |
+| `content_type` | 4 | `string` |
+| `checksum` | 5 | `string` |
+| `inline` | 6 | `binary` |
+
+A `file` field reserves the root field's ID plus six consecutive IDs for its
sub-fields, assigned by the offsets above. Adding a `file` field must advance
`last-column-id` to account for all 7 IDs.
+
+The `uri` field may contain absolute or relative references. Implementations
that receive a relative path should resolve the path against the table location
(see [Path Resolution](#path-resolution)).
+
+Statistics for `file` are tracked using separate field stats for each
sub-field. Writers should produce statistics for `uri`, `content_type`, and
`inline` fields; other fields may be omitted.
+
+A `file` column is subject to the following restrictions:
+
+* Equality, ordering, and hashing are not defined for `file` objects
+* A `file` column's `field-id` cannot be referenced as an identifier field or
a source for partition or sort transforms.
+* A `file` is not interchangeable with a `struct`; a `struct` with the same
sub-fields is not equivalent to a `file`.
+* A `file` column cannot be promoted from `struct` with the same sub-fields.
+* A `file` entry that references a `uri` does not guarantee its existence.
+
#### Default values
Default values can be tracked for struct fields (both nested structs and the
top-level schema's struct). There can be two defaults with a field:
@@ -339,7 +371,7 @@ The `initial-default` is set only when a field is added to
an existing schema. T
The `initial-default` and `write-default` produce SQL default value behavior,
without rewriting data files. SQL default value behavior when a field is added
handles all existing rows as though the rows were written with the new field's
default value. Default value changes may only affect future records and all
known fields are written into data files. Omitting a known field when writing a
data file is never allowed. The write default for a field must be written if a
field is not supplied [...]
-All columns of `unknown`, `variant`, `geometry`, and `geography` types must
default to null. Non-null values for `initial-default` or `write-default` are
invalid.
+All columns of `unknown`, `variant`, `geometry`, `geography`, and `file` types
must default to null. Non-null values for `initial-default` or `write-default`
are invalid.
Default values for the fields of a struct are tracked as `initial-default` and
`write-default` at the field level. Default values for fields that are nested
structs must not contain default values for the struct's fields (sub-fields).
Sub-field defaults are tracked in sub-field's metadata. As a result, the
default stored for a nested struct may be either null or a non-null struct with
no field values. The effective default value is produced by setting each
fields' default in a new struct.
@@ -1692,6 +1724,7 @@ Maps with non-string keys must use an array
representation with the `map` logica
|**`variant`**|`record` with `metadata` and `value` fields. `metadata` and
`value` must not be assigned field IDs and the fields are accessed through
names. |Shredding is not supported in Avro.|
|**`geometry`**|`bytes`|WKB format, see [Appendix
G](#appendix-g-geospatial-notes)|
|**`geography`**|`bytes`|WKB format, see [Appendix
G](#appendix-g-geospatial-notes)|
+|**`file`**|`record` with the `file` sub-fields. Sub-fields must be assigned
field IDs. |See [File Type](#file-type). Avro has no `FILE` logical type; type
identity comes from the Iceberg schema.|
Notes:
@@ -1747,6 +1780,7 @@ Lists must use the [3-level
representation](https://github.com/apache/parquet-fo
| **`variant`** | `group` with `metadata` and `value` fields. `metadata`
and `value` must not be assigned field IDs and the fields are accessed through
names. | `VARIANT` | See Parquet docs for
[Variant
encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md)
and [Variant shredding
encoding](https://github.com/apache/parquet-format/blob/master/VariantShredding.md).
|
| **`geometry`** | `binary`
| `GEOMETRY` | WKB format, see [Appendix
G](#appendix-g-geospatial-notes). |
| **`geography`** | `binary`
| `GEOGRAPHY` | WKB format, see [Appendix
G](#appendix-g-geospatial-notes). |
+| **`file`** | `group` with the `file` sub-fields. Sub-fields must be
assigned field IDs.
| `FILE` | See Parquet docs for the
[`FILE`
type](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file)
and [File Type](#file-type). |
When reading an `unknown` column, any corresponding column must be ignored and
replaced with `null` values.
@@ -1779,6 +1813,7 @@ When reading an `unknown` column, any corresponding
column must be ignored and r
| **`variant`** | `struct` with `metadata` and `value` fields. `metadata`
and `value` must not be assigned field IDs. | `iceberg.struct-type`=`VARIANT`
| Shredding is not supported in ORC.
|
| **`geometry`** | `binary` | `iceberg.binary-type`=`GEOMETRY`
| WKB format, see [Appendix
G](#appendix-g-geospatial-notes).
|
| **`geography`** | `binary` | `iceberg.binary-type`=`GEOGRAPHY`
| WKB format, see [Appendix
G](#appendix-g-geospatial-notes).
|
+| **`file`** | `struct` with the `file` sub-fields. Sub-fields must be
assigned and accessed by field IDs.| `iceberg.struct-type`=`FILE`| See [File
Type](#file-type). |
Notes:
@@ -1875,6 +1910,7 @@ Types are serialized according to this table:
| **`variant`**| `JSON string: "variant"`|`"variant"`|
| **`geometry(C)`** |`JSON string: "geometry(<C>)"`|`"geometry(srid:4326)"`|
| **`geography(C, A)`** |`JSON string: "geography(<C>,
<A>)"`|`"geography(srid:4326, spherical)"`|
+| **`file`**| `JSON string: "file"`|`"file"`|
The schema JSON type strings in this table are the canonical serialized forms.
Readers should accept optional whitespace around parameters and separators in
parameterized type strings.
@@ -2034,6 +2070,7 @@ This serialization scheme is for storing single values as
individual binary valu
| **`variant`** | Not supported
|
| **`geometry`** | WKB format, see [Appendix
G](#appendix-g-geospatial-notes)
|
| **`geography`** | WKB format, see [Appendix
G](#appendix-g-geospatial-notes)
|
+| **`file`** | Not supported
|
### Bound serialization
@@ -2077,6 +2114,8 @@ The binary single-value serialization can be used to
store the lower and upper b
### Version 4
+The `file` type is added in v4.
+
Relative path support is added in v4.
Reading v3 or prior metadata for v4:
