singhpk234 commented on code in PR #11825:
URL: https://github.com/apache/iceberg/pull/11825#discussion_r1913830176
##########
core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java:
##########
@@ -84,13 +86,30 @@ private boolean shouldRewrite(List<FileScanTask> group) {
return enoughInputFiles(group)
|| enoughContent(group)
|| tooMuchContent(group)
- || anyTaskHasTooManyDeletes(group);
+ || anyTaskHasTooManyDeletes(group)
+ || anyTaskHasTooHighDeleteRatio(group);
}
private boolean anyTaskHasTooManyDeletes(List<FileScanTask> group) {
return group.stream().anyMatch(this::tooManyDeletes);
}
+ private boolean anyTaskHasTooHighDeleteRatio(List<FileScanTask> group) {
+ return group.stream().anyMatch(this::tooHighDeleteRatio);
+ }
+
+ private boolean tooHighDeleteRatio(FileScanTask task) {
+ if (ContentFileUtil.containsSingleDV(task.deletes())) {
+ DeleteFile file = Iterables.getOnlyElement(task.deletes());
+ double deletedRecords = (double) Math.min(file.recordCount(),
task.file().recordCount());
+ double deleteRatio = deletedRecords / task.file().recordCount();
Review Comment:
> @singhpk234 the min calculation here is really only used for the case
where we have more deletes than actual records
I was referring for the case when we have SplitScan task ? as if one file in
diff splits so each split will have a part of file, and will have a whole DV
attached.
please consider a 1GB file broken into 128 MB splits so ~8 splits for 1 file
and each split file will have the whole DV attached.
So in this case when we are looking in a scan task it will have number of
records in part of file < total records of DV ?
> Do we operate on split tasks here? I thought we were still dealing with
entire files, but I can be wrong.
Need to take a deeper look but IMHO this can be
(https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkBinPackDataRewriter.java#L50),
need to double check though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
