szehon-ho commented on code in PR #9661:
URL: https://github.com/apache/iceberg/pull/9661#discussion_r1484905812
##########
format/spec.md:
##########
@@ -1117,7 +1121,17 @@ Partition specs are serialized as a JSON object with the
following fields:
|**`spec-id`**|`JSON int`|`0`|
|**`fields`**|`JSON list: [`<br /> `<partition field JSON>,`<br
/> `...`<br />`]`|`[ {`<br /> `"source-id": 4,`<br
/> `"field-id": 1000,`<br /> `"name": "ts_day",`<br
/> `"transform": "day"`<br />`}, {`<br /> `"source-id":
1,`<br /> `"field-id": 1001,`<br /> `"name":
"id_bucket",`<br /> `"transform": "bucket[16]"`<br />`} ]`|
-Each partition field in the fields list is stored as an object. See the table
for more detail:
+Each partition field in the `fields` is stored as a JSON object with the
following properties.
+
+| V1 | V2 | V3 | Field | JSON representation |
Example |
+|----------|----------|----------|------------------|---------------------|--------------|
+| required | required | required | **`source-id`** | `JSON int` | 1
|
Review Comment:
Yes, so for backward compatibility, here is the flow for newer readers:
1. if table is V3 (or V1/V2 table with a special flag), read from source-ids.
2. If table is V1/V2 without special flag, read from source-id.
Then when they write back to the table:
1. if table is V3 (or V1/V2 table with a special flag), write to source-ids.
2. If table is V1/V2 without special flag, write source-id.
For forward compatibility, an old reader will not be able to read the new
tables with `source-ids` field. They will have an error de-serializing the
JSON object for partition/sort fields.
The -1 idea is just to make it so they get further, but they will still
error when they realize it is not a valid field id.
The special table config `compatibility.multi-arg-transform.enabled` will
advertise, it is the user's responsibility to ensure that all readers/writers
are new , ie support multi-arg transform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
