RussellSpitzer commented on code in PR #16025: URL: https://github.com/apache/iceberg/pull/16025#discussion_r4160399836
########## format/spec.md: ########## @@ -656,15 +662,33 @@ A data or delete file is associated with a sort order by the sort order's id wit ### Manifests -A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. +A table's metadata tree is composed of manifests. A manifest is an immutable file that tracks a subset of a table metadata for a given [snapshot](#snapshots). Leaf manifests are the lowest level of the metadata tree and track data or delete files, along with each file's partition data, metrics, and tracking information. The root level of the metadata tree tracks leaf manifests; in v4 and later the root is also a manifest and can also track data files. A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. -A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata. +Each manifest type contains the following content: -A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest. +| Manifest type | Contents | +|----------------|----------| +| v1-v3 data manifest | Data files | +| v2-v3 delete manifest | Delete files | +| v4 root manifest | Data files, data manifests, delete manifests | +| v4 data manifest | Data files and their colocated deletion vectors | -A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata: +In v2-v3, a manifest may store either data files or delete files, but not both; whether a manifest is a data manifest or a delete manifest is stored in manifest metadata. + +**Partition Spec Binding:** + +- v1-v3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. Review Comment: New files are always written to a new manifest, I don't think you have a way around that. I'm not sure what the intent of the statement "When a table's partition spec changes ..." is. Maybe ```suggestion - v1-v3: All entries in a manifest use the same partition spec, because the manifest's partition struct schema is derived from that spec. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
