aokolnychyi commented on code in PR #9611:
URL: https://github.com/apache/iceberg/pull/9611#discussion_r1561911029
##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java:
##########
@@ -909,6 +909,47 @@ public void testParallelPartialProgressWithCommitFailure()
{
shouldHaveACleanCache(table);
}
+ @Test
+ public void testParallelPartialProgressWithMaxFailedCommits() {
+ Table table = createTable(20);
+ int fileSize = averageFileSize(table);
+
+ List<Object[]> originalData = currentData();
+
+ RewriteDataFilesSparkAction realRewrite =
+ basicRewrite(table)
+ .option(
+ RewriteDataFiles.MAX_FILE_GROUP_SIZE_BYTES,
Integer.toString(fileSize * 2 + 1000))
+ .option(RewriteDataFiles.MAX_CONCURRENT_FILE_GROUP_REWRITES, "3")
+ .option(RewriteDataFiles.PARTIAL_PROGRESS_ENABLED, "true")
+ .option(RewriteDataFiles.PARTIAL_PROGRESS_MAX_COMMITS, "3")
+ .option(RewriteDataFiles.PARTIAL_PROGRESS_MAX_FAILED_COMMITS, "0");
+
+ RewriteDataFilesSparkAction spyRewrite = Mockito.spy(realRewrite);
+
+ // Fail groups 1, 3, and 7 during rewrite
+ GroupInfoMatcher failGroup = new GroupInfoMatcher(1, 3, 7);
+ doThrow(new RuntimeException("Rewrite Failed"))
+ .when(spyRewrite)
+ .rewriteFiles(any(), argThat(failGroup));
+
+ assertThatThrownBy(() -> spyRewrite.execute())
+ .isInstanceOf(RuntimeException.class)
+ .hasMessageContaining(
+ "1 rewrite commits failed. This is more than the maximum allowed
failures of 0");
+
+ table.refresh();
+
+ List<Object[]> postRewriteData = currentData();
+ assertEquals("We shouldn't have changed the data", originalData,
postRewriteData);
+
+ // With 10 original groups and Max Commits of 3, we should have commits
with 4, 4, and 2.
+ // removing 3 groups leaves us with only 2 new commits, 4 and 3
Review Comment:
What does `4 and 3` mean in this case at the end of this line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
