mxm commented on code in PR #18525:
URL: https://github.com/apache/iceberg/pull/18525#discussion_r4413266374
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java:
##########
@@ -96,17 +96,10 @@ public void serialize(DynamicRecordInternal toSerialize,
DataOutputView dataOutp
dataOutputView.writeInt(toSerialize.writerKey());
final RowDataSerializer rowDataSerializer;
if (writeSchemaAndSpec) {
- rowDataSerializer =
- serializerCache.serializer(
- toSerialize.tableName(), toSerialize.schema(),
toSerialize.spec());
+ rowDataSerializer = serializerCache.serializer(toSerialize.tableName(),
toSerialize.schema());
} else {
- // Check that the schema id can be resolved. Not strictly necessary for
serialization.
- Tuple3<RowDataSerializer, Schema, PartitionSpec> serializer =
- serializerCache.serializerWithSchemaAndSpec(
- toSerialize.tableName(),
- toSerialize.schema().schemaId(),
- toSerialize.spec().specId());
- rowDataSerializer = serializer.f0;
+ rowDataSerializer =
+ serializerCache.serializerById(toSerialize.tableName(),
toSerialize.schema());
Review Comment:
Could we add a test to `TestDynamicRecordInternalSerializer` which
serializes a record with `writeSchemaAndSpec=false` for a non-existing table?
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableSerializerCache.java:
##########
@@ -112,17 +123,32 @@ int maximumSize() {
private class SerializerInfo {
private final String tableName;
- private final Map<Schema, RowDataSerializer> serializers;
+ private final Map<Schema, RowDataSerializer> serializersBySchema;
+ private final Map<Integer, RowDataSerializer> serializersBySchemaId;
private Map<Integer, Schema> schemas;
private Map<Integer, PartitionSpec> specs;
SerializerInfo(String tableName) {
this.tableName = tableName;
- this.serializers = Maps.newHashMapWithExpectedSize(2);
+ this.serializersBySchema = new LRUCache<>(maximumSize);
Review Comment:
Going forward, I think we need more cache options. `maximumSize` is used for
the maximum number of tables to cache in other places, and here it is used for
the maximum number of schemas.
How about `inputSchemasPerTableCacheMaxSize`? I think this is closer to what
we want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
