pvary commented on code in PR #13340:
URL: https://github.com/apache/iceberg/pull/13340#discussion_r2156735013
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordProcessor.java:
##########
@@ -31,10 +33,14 @@
import org.apache.iceberg.Schema;
import org.apache.iceberg.catalog.Catalog;
import org.apache.iceberg.flink.CatalogLoader;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.dynamic.convert.RowDataConverter;
@Internal
class DynamicRecordProcessor<T> extends ProcessFunction<T,
DynamicRecordInternal>
implements Collector<DynamicRecord> {
+ private static final int ROW_DATA_CONVERTER_CACHE_MAXIMUM_SIZE = 1000;
Review Comment:
@aiborodin: Tests are needed, but my guess that adding the converter to the
`TableMetadataCache` is a good idea. The cache timeout is only used for missing
items. So I don't think we will have an issue with the refresh.
A bit concerned with the size of the converters if they are in the
`TableMetadataCache`. The cache items are only evicted if they are not used, or
the tables are updated. So old converters are still kept until the table is not
used anymore. This might not be a big issue, but we have to be aware of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
