RussellSpitzer commented on code in PR #10200:
URL: https://github.com/apache/iceberg/pull/10200#discussion_r1583299839
##########
core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java:
##########
@@ -109,18 +112,34 @@ protected abstract class BaseEqualityDeltaWriter
implements Closeable {
private final StructProjection structProjection;
private RollingFileWriter dataWriter;
private RollingEqDeleteWriter eqDeleteWriter;
- private SortedPosDeleteWriter<T> posDeleteWriter;
+ private FileWriter<PositionDelete<T>, DeleteWriteResult> posDeleteWriter;
private Map<StructLike, PathOffset> insertedRowMap;
protected BaseEqualityDeltaWriter(StructLike partition, Schema schema,
Schema deleteSchema) {
+ this(partition, schema, deleteSchema, DeleteGranularity.PARTITION);
+ }
+
+ protected BaseEqualityDeltaWriter(
+ StructLike partition,
+ Schema schema,
+ Schema deleteSchema,
+ DeleteGranularity deleteGranularity) {
Review Comment:
We discussed a bit on slack. I think no-conf is good, in my mind if there
was a benefit from having the two modes as an option it's something flink
should be choosing at runtime rather than a user defined config. I think this
is also mostly a perf difference so I wouldn't mind changing the behavior from
the previous behavior in those edge cases with enough data to write multiple
files in the same partition in one batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
