yhyyz opened a new pull request, #18722: URL: https://github.com/apache/iceberg/pull/18722
Follow-up to #18610, answering @pvary's question in https://github.com/apache/iceberg/pull/18610 (nested equality columns such as `a.b`). **Depends on #18721.** This branch is stacked on it; until #18721 merges, the diff here also shows its commit. Only the last commit, "Flink: Support nested equality fields in ConvertEqualityDeletes", belongs to this PR. I will rebase once #18721 is in. ### Current behavior `ConvertEqualityDeletes` accepts a nested equality column such as `user.id`, but every cycle fails in the reader: ``` java.lang.UnsupportedOperationException: Cannot serialize type: STRUCT at org.apache.iceberg.types.Conversions.toByteBuffer(Conversions.java:145) at org.apache.iceberg.flink.maintenance.operator.StructLikeSerializer.writeField(...) at ...EqualityConvertReader.processDataFile(...) ``` `TypeUtil.select` keeps the parent struct around a nested key field, so the key serializer received a struct value. The error aborts the cycle; no wrong keys are produced. ### Change - `RecordSerializer` exposes the key to the byte encoder as its primitive leaf fields in depth-first schema order (new package-private `FlattenedEqualityKey`). This matches how core's `DeleteFilter` / `StructProjection` apply equality deletes on nested fields. A leaf under a null parent struct is serialized as null. - A key made of top-level fields only flattens to the original key type, so the serialized format (Flink keyed state) is unchanged; the existing byte-identity tests in `TestRecordSerializer` pass unmodified. - The builder now fails fast if an equality column is not a primitive (e.g. `user`) or is nested in a list or map (e.g. `tags.element`), and the javadoc documents this. ### Testing - `TestConvertEqualityDeletes`: end to end, an equality delete on `user.id` is converted to a DV and removes the right rows; rejection of `user` and `tags.element`. - `TestEqualityConvertReader`: the data-row key equals the delete key for `{id, user.id}`; a non-matching row produces a different key. - `TestRecordSerializer`: exact bytes for `{id, user.id, user.birthday}` (including a DATE leaf) and null leaves under a null parent struct. - `TestFlattenedEqualityKey`: depth-first order, null parent, top-level keys unchanged, list rejected. Flink 2.3 only; backports to 1.20 and 2.2 will follow after merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
