pvary commented on code in PR #16791:
URL: https://github.com/apache/iceberg/pull/16791#discussion_r4023340382
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/data/FlinkParquetWriters.java:
##########
@@ -574,32 +578,32 @@ private MapDataWriter(
LogicalType keyType,
LogicalType valueType) {
super(definitionLevel, repetitionLevel, keyWriter, valueWriter);
- this.keyType = keyType;
- this.valueType = valueType;
+ this.keyGetter = ArrayData.createElementGetter(keyType);
+ this.valueGetter = ArrayData.createElementGetter(valueType);
+ this.entryIterator = new EntryIterator();
}
@Override
protected Iterator<Map.Entry<K, V>> pairs(MapData map) {
- return new EntryIterator<>(map);
- }
-
- private class EntryIterator<K, V> implements Iterator<Map.Entry<K, V>> {
- private final int size;
- private final ArrayData keys;
- private final ArrayData values;
- private final ParquetValueReaders.ReusableEntry<K, V> entry;
- private final ArrayData.ElementGetter keyGetter;
- private final ArrayData.ElementGetter valueGetter;
+ // The parent writer fully consumes the iterator inside a single write()
call, so a single
+ // reusable instance (and its reusable entry) avoids allocating per row.
Review Comment:
Are we sure that this remains the case always?
Buffering, or lazy consuming in Parquet code could cause issues later.
Could we just reuse the getters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
