szehon-ho commented on code in PR #8579:
URL: https://github.com/apache/iceberg/pull/8579#discussion_r1452983761
##########
format/spec.md:
##########
@@ -1119,21 +1156,30 @@ Partition specs are serialized as a JSON object with
the following fields:
Each partition field in the fields list is stored as an object. See the table
for more detail:
-|Transform or Field|JSON representation|Example|
-|--- |--- |--- |
-|**`identity`**|`JSON string: "identity"`|`"identity"`|
-|**`bucket[N]`**|`JSON string: "bucket[<N>]"`|`"bucket[16]"`|
-|**`truncate[W]`**|`JSON string: "truncate[<W>]"`|`"truncate[20]"`|
-|**`year`**|`JSON string: "year"`|`"year"`|
-|**`month`**|`JSON string: "month"`|`"month"`|
-|**`day`**|`JSON string: "day"`|`"day"`|
-|**`hour`**|`JSON string: "hour"`|`"hour"`|
-|**`Partition Field`**|`JSON object: {`<br /> `"source-id": <id
int>,`<br /> `"field-id": <field id int>,`<br /> `"name":
<name string>,`<br /> `"transform": <transform JSON>`<br
/>`}`|`{`<br /> `"source-id": 1,`<br /> `"field-id":
1000,`<br /> `"name": "id_bucket",`<br /> `"transform":
"bucket[16]"`<br />`}`|
+| Transform or Field | JSON representation
| Example
|
+|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **`identity`** | `JSON string:
"identity"`
|
`"identity"`
|
+| **`bucket[N]`** | `JSON string:
"bucket[<N>]"`
|
`"bucket[16]"`
|
+| **`bucket[N]`** (multi-arg bucket [1]) | `JSON string:
"bucketV2[<N>]"`
|
`"bucketV2[16]"`
|
+| **`truncate[W]`** | `JSON string:
"truncate[<W>]"`
|
`"truncate[20]"`
|
+| **`year`** | `JSON string: "year"`
| `"year"`
|
+| **`month`** | `JSON string: "month"`
| `"month"`
|
+| **`day`** | `JSON string: "day"`
| `"day"`
|
+| **`hour`** | `JSON string: "hour"`
| `"hour"`
|
+| **`Partition Field`** | `JSON object: {`<br
/> `"source-id": <id int>,`<br /> `"field-id": <field id
int>,`<br /> `"name": <name string>,`<br /> `"transform":
<transform JSON>`<br />`}` | `{`<br
/> `"source-id": 1,`<br /> `"field-id": 1000,`<br
/> `"name": "id_bucket",`<br /> `"transform":
"bucket[16]"`<br />`}` |
+| **`Partition Field with multi-arg transform`** [2] | `JSON object: {`<br
/> `"source-id": -1,`<br /> `"source-ids": <list of
ids>,`<br /> `"field-id": <field id int>,`<br /> `"name":
<name string>,`<br /> `"transform": <transform JSON>`<br />`}` |
`{`<br /> `"source-id": -1,`<br /> `"source-ids":
[1,2],`<br /> `"field-id": 1000,`<br /> `"name":
"id_type_bucket",`<br /> `"transform": "bucketV2[16]"`<br />`}` |
In some cases partition specs are stored using only the field list instead of
the object format that includes the spec ID, like the deprecated
`partition-spec` field in table metadata. The object format should be used
unless otherwise noted in this spec.
The `field-id` property was added for each partition field in v2. In v1, the
reference implementation assigned field ids sequentially in each spec starting
at 1,000. See Partition Evolution for more details.
+Notes:
+
+1. For multi-arg bucket, the serialized form is `bucketV2[N]` instead of
`bucket[N]` to distinguish it from the single-arg bucket transform. Therefore,
old readers/writers will identify this transform as an unknown transform, old
writer will stop writing the table if it encounters this transform, but old
readers would still be able to read the table by scanning all the partitions.
Review Comment:
I actually think we can remove this note. I feel there is nothing here that
is specific to bucketv2 transform, but rather this is just the behavior of
existing readers/writers to ignore unknown transforms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
