amogh-jahagirdar commented on code in PR #9400:
URL: https://github.com/apache/iceberg/pull/9400#discussion_r1442236190
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:
##########
@@ -345,7 +345,7 @@ private Result doExecuteWithPartialProgress(
.noRetry()
.onFailure(
(fileGroup, exception) -> {
- LOG.error("Failure during rewrite group {}", fileGroup.info(),
exception);
+ LOG.warn("Failure during rewrite group {}", fileGroup.info(),
exception);
Review Comment:
Another idea is that we log an error if the commit results size is less than
some threshold rather than completely empty. This would eliminate the task
level noise from errors but at the same time indicate to logging systems that
the compaction isn't super effective.
```
if (commitResults.size() == 0) {
LOG.error(
"{} is true but no rewrite commits succeeded. Check the logs to
determine why the individual "
+ "commits failed. If this is persistent it may help to
increase {} which will break the rewrite operation "
+ "into smaller commits.",
PARTIAL_PROGRESS_ENABLED,
PARTIAL_PROGRESS_MAX_COMMITS);
}
```
to
````
if (commitResults.size()/commitAttempts <= SOME_THRESHOLD) {
LOG.error(
"{} is true but less than SOME_THRESHOLD rewrite commits
succeeded. Check the logs to determine why the individual "
+ "commits failed. If this is persistent it may help to
increase {} which will break the rewrite operation "
+ "into smaller commits.",
PARTIAL_PROGRESS_ENABLED,
PARTIAL_PROGRESS_MAX_COMMITS);
}
```
but maybe determining commitAttempts is overcomplicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
