mxm commented on code in PR #18525:
URL: https://github.com/apache/iceberg/pull/18525#discussion_r4335447584
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java:
##########
@@ -94,21 +94,9 @@ public void serialize(DynamicRecordInternal toSerialize,
DataOutputView dataOutp
}
dataOutputView.writeInt(toSerialize.writerKey());
- final RowDataSerializer rowDataSerializer;
- if (writeSchemaAndSpec) {
- rowDataSerializer =
- serializerCache.serializer(
- toSerialize.tableName(), toSerialize.schema(),
toSerialize.spec());
- } else {
- // Check that the schema id can be resolved. Not strictly necessary for
serialization.
- Tuple3<RowDataSerializer, Schema, PartitionSpec> serializer =
- serializerCache.serializerWithSchemaAndSpec(
- toSerialize.tableName(),
- toSerialize.schema().schemaId(),
- toSerialize.spec().specId());
- rowDataSerializer = serializer.f0;
- }
-
+ RowDataSerializer rowDataSerializer =
+ serializerCache.serializer(
+ toSerialize.tableName(), toSerialize.schema(), toSerialize.spec());
Review Comment:
Peter's point is valid. To your point, Swapna, it's true that for the
immediate update path we will always use the table schema instead of the
user-provided one, but on cache refresh in
[TableMetadataCache](https://github.com/apache/iceberg/blob/0bd929a02be35113000932f6af8f288243a45ab8/flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableMetadataCache.java#L119),
we reload the entire
[`Table`](https://github.com/apache/iceberg/blob/0bd929a02be35113000932f6af8f288243a45ab8/flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableMetadataCache.java#L220),
which triggers creating a new `Schema` object and subsequently will trigger
creating a new serializer here. This was not an issue before because we used
the schema id to de-duplicate the schema instances.
For the immediate update path, should we use the schema id instead of the
schema reference to cache the serializer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
