minleejae opened a new pull request, #16927: URL: https://github.com/apache/iceberg/pull/16927
## Why `AddSchemaUpdate.last-column-id` is deprecated and optional in the REST OpenAPI spec. Java already ignores this field when parsing add-schema updates and computes the table-level last column ID internally. However, Java still serializes the deprecated field using `schema.highestFieldId()`. After dropping the column with the highest field ID, this value can decrease even though table metadata `last-column-id` must not decrease. REST catalog implementations that validate the deprecated request field can reject otherwise valid schema evolution updates, as reported in #13850. This does not remove table metadata `last-column-id`; it only stops sending the deprecated optional field in `AddSchemaUpdate`. Field-id conflict validation is still carried by `AssertLastAssignedFieldId` requirements. ## Compatibility notes - The REST OpenAPI spec marks `AddSchemaUpdate.last-column-id` as deprecated and optional, and says that when it is omitted it will be computed on the server side: https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L3116-L3135 - Apache Iceberg's REST server path validates update requirements before applying metadata updates, and `TableMetadata.Builder.addSchema` preserves the table-level last column ID with `Math.max(lastColumnId, schema.highestFieldId())`: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java#L613-L659 and https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1136-L1138 - Nessie already models this field as optional (`OptionalInt`) and its changelog notes the AddSchema `lastColumnId` deprecation/optional change. When the value is omitted, Nessie computes from the schema max field ID and only increases the stored table-level value: https://github.com/projectnessie/nessie/blob/8f3a35c103ccf98fad9a95c22f2cf50993026a7f/catalog/format/iceberg/src/main/java/org/projectnessie/catalog/formats/iceberg/rest/IcebergMetadataUpdate.java#L348-L371 and https://github.com/projectnessie/nessie/blob/8f3a35c103ccf98fad9a95c22f2cf50993026a7f/catalog/format/iceberg/src/main/java/org/projectnessie/catalog/formats/iceberg/nessie/NessieModelIceberg.java#L1198-L1250 - Polaris uses Iceberg's `UpdateTableRequest`/`MetadataUpdate` model for commits and applies updates via `MetadataUpdate.applyTo(...)`, so it follows the Iceberg parser/model behavior instead of requiring the deprecated request field. Its copied REST spec also marks the field deprecated and optional: https://github.com/apache/polaris/blob/e3f26cf69f6ee189328f1b7adb90ec83cbbb26d9/api/iceberg-service/src/main/java/org/apache/polaris/service/types/CommitTableRequest.java#L19-L23 and https://github.com/apache/polaris/blob/e3f26cf69f6ee189328f1b7adb90ec83cbbb26d9/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/CatalogHandlerUtils.java#L408-L522 and https://github.com/apache/polaris/blob/e3f26cf69f6ee189328f1b7adb90ec83cbbb26d9/spec/iceberg-rest-catalog-open-api.yaml#L2969-L2988 - The `tabulario/iceberg-rest` reference image is a thin wrapper around Apache Iceberg's `RESTCatalogAdapter`, so it does not add a separate `last-column-id` validator. Its exact behavior is tied to the bundled Iceberg version, which is why the fix belongs in Iceberg's REST metadata update serialization/model: https://github.com/databricks/iceberg-rest-image/blob/62a5078032e0675d34b377d2ead2eea814c1da48/src/main/java/org/apache/iceberg/rest/RESTServerCatalogAdapter.java#L28-L43 Catalogs that still send the deprecated field remain supported because parsing is unchanged. The omission path is part of the REST contract; server implementations that need the table-level last assigned field ID should use existing table metadata and/or `AssertLastAssignedFieldId` requirements rather than depending on the deprecated `AddSchemaUpdate` field. ## What changed - Stop serializing deprecated `last-column-id` for `add-schema` metadata updates. - Keep parsing legacy payloads that still include `last-column-id`. - Add request serialization coverage for an add-schema update representing the schema after dropping the highest field ID. ## Testing - `JAVA_HOME=$(/usr/libexec/java_home -v 21) ./gradlew :iceberg-core:test --tests org.apache.iceberg.TestMetadataUpdateParser --tests org.apache.iceberg.rest.requests.TestUpdateTableRequestParser --tests org.apache.iceberg.rest.TestCatalogHandlers --console=plain --warning-mode=none` - `JAVA_HOME=$(/usr/libexec/java_home -v 21) ./gradlew :iceberg-core:spotlessJavaCheck --console=plain --warning-mode=none` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
