nastra commented on code in PR #11825:
URL: https://github.com/apache/iceberg/pull/11825#discussion_r1924915138
##########
core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java:
##########
@@ -84,13 +87,34 @@ private boolean shouldRewrite(List<FileScanTask> group) {
return enoughInputFiles(group)
|| enoughContent(group)
|| tooMuchContent(group)
- || anyTaskHasTooManyDeletes(group);
+ || anyTaskHasTooManyDeletes(group)
+ || anyTaskHasTooHighDeleteRatio(group);
}
private boolean anyTaskHasTooManyDeletes(List<FileScanTask> group) {
return group.stream().anyMatch(this::tooManyDeletes);
}
+ private boolean anyTaskHasTooHighDeleteRatio(List<FileScanTask> group) {
+ return group.stream().anyMatch(this::tooHighDeleteRatio);
+ }
+
+ private boolean tooHighDeleteRatio(FileScanTask task) {
+ if (null == task.deletes() || task.deletes().isEmpty()) {
+ return false;
+ }
+
+ if (ContentFileUtil.containsSingleDV(task.deletes())
+ || task.deletes().stream().allMatch(ContentFileUtil::isFileScoped)) {
+ long sum =
task.deletes().stream().mapToLong(ContentFile::recordCount).sum();
+ double deletedRecords = (double) Math.min(sum,
task.file().recordCount());
Review Comment:
yes, this addressed an edge case that is being tested by
`testDataFilesRewrittenWithMaxDeleteRatio` where we produce more positions to
be deleted than actual records and the final delete ratio would be > 1.0 (which
technically still produces the same end result, but I'd like to keep the delete
ratio between 0.0 and 1.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
