openinx commented on code in PR #18463:
URL: https://github.com/apache/iceberg/pull/18463#discussion_r4533938834
##########
gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java:
##########
@@ -327,16 +331,56 @@ public void deleteFiles(Iterable<String> pathsToDelete)
throws BulkDeletionFailu
@SuppressWarnings("resource")
private void internalDeleteFiles(Stream<BlobId> blobIdsToDelete) {
- Streams.stream(
- Iterators.partition(
- blobIdsToDelete.iterator(),
-
clientForStoragePath(ROOT_STORAGE_PREFIX).gcpProperties().deleteBatchSize()))
- .forEach(
- batch -> {
- if (!batch.isEmpty()) {
-
clientForStoragePath(batch.get(0).toGsUtilUri()).storage().delete(batch);
- }
- });
+ List<String> failedDeletions =
+ Streams.stream(
+ Iterators.partition(
+ blobIdsToDelete.iterator(),
+
clientForStoragePath(ROOT_STORAGE_PREFIX).gcpProperties().deleteBatchSize()))
+ .filter(batch -> !batch.isEmpty())
+ .flatMap(
+ batch ->
+
deleteBatch(clientForStoragePath(batch.get(0).toGsUtilUri()).storage(), batch)
+ .stream())
+ .toList();
Review Comment:
**nit: Keep the failure count instead of all failed paths**
We save every failed file path in a list, but only use the list’s size. If
many deletes fail, this list can consume a lot of memory. Can we add up each
batch’s failure count instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
