amogh-jahagirdar commented on code in PR #16025: URL: https://github.com/apache/iceberg/pull/16025#discussion_r4160808369
########## format/spec.md: ########## @@ -656,15 +662,33 @@ A data or delete file is associated with a sort order by the sort order's id wit ### Manifests -A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. +A table's metadata tree is composed of manifests. A manifest is an immutable file that tracks a subset of a table metadata for a given [snapshot](#snapshots). Leaf manifests are the lowest level of the metadata tree and track data or delete files, along with each file's partition data, metrics, and tracking information. The root level of the metadata tree tracks leaf manifests; in v4 and later the root is also a manifest and can also track data files. A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. -A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata. +Each manifest type contains the following content: -A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest. +| Manifest type | Contents | +|----------------|----------| +| v1-v3 data manifest | Data files | +| v2-v3 delete manifest | Delete files | +| v4 root manifest | Data files, data manifests, delete manifests | +| v4 data manifest | Data files and their colocated deletion vectors | -A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata: +In v2-v3, a manifest may store either data files or delete files, but not both; whether a manifest is a data manifest or a delete manifest is stored in manifest metadata. + +**Partition Spec Binding:** + +- v1-v3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. Review Comment: I can change this, but this was the existing spec wording for v1- v3 so I didn't want to mess with it. FWIW though I agree that we should be able to simplify this as you did, I'm just reluctant to mess with prior wording. If you feel strongly about it, can change it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
