aokolnychyi commented on code in PR #10200:
URL: https://github.com/apache/iceberg/pull/10200#discussion_r1581637078
##########
core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java:
##########
@@ -109,18 +112,34 @@ protected abstract class BaseEqualityDeltaWriter
implements Closeable {
private final StructProjection structProjection;
private RollingFileWriter dataWriter;
private RollingEqDeleteWriter eqDeleteWriter;
- private SortedPosDeleteWriter<T> posDeleteWriter;
+ private FileWriter<PositionDelete<T>, DeleteWriteResult> posDeleteWriter;
private Map<StructLike, PathOffset> insertedRowMap;
protected BaseEqualityDeltaWriter(StructLike partition, Schema schema,
Schema deleteSchema) {
+ this(partition, schema, deleteSchema, DeleteGranularity.PARTITION);
+ }
+
+ protected BaseEqualityDeltaWriter(
+ StructLike partition,
+ Schema schema,
+ Schema deleteSchema,
+ DeleteGranularity deleteGranularity) {
Review Comment:
We don't have a lot of input from the Spark side as it was shipped in 1.5
for the first time. There, the situation is also very different as the number
of deletes is proposal to the number of changes in a batch.
@chenjunjiedada, you faced the concurrent compaction issue in the past.
Would you have a chance to try this PR and see how it performs for your use
cases?
@stevenzwu @rdblue @RussellSpitzer, would love to hear your thoughts as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
