[
https://issues.apache.org/jira/browse/HADOOP-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631263#comment-15631263
]
Kai Zheng commented on HADOOP-13768:
------------------------------------
Looking at the following codes:
{code}
+ List<String> deleteFailed = keysToDelete;
+ while(CollectionUtils.isNotEmpty(deleteFailed)) {
+ List<String> l = new ArrayList<>();
+ List<List<String>> smallerLists = Lists.partition(deleteFailed, 1000);
+ for (List<String> smallerList : smallerLists) {
+ DeleteObjectsRequest deleteRequest =
+ new DeleteObjectsRequest(bucketName);
+ deleteRequest.setKeys(smallerList);
+ deleteRequest.setQuiet(true);
+ DeleteObjectsResult result = ossClient.deleteObjects(deleteRequest);
+ l.addAll(result.getDeletedObjects());
+ }
+ deleteFailed = l;
+ tries++;
+ if (tries == retry) {
+ break;
+ }
+ }
{code}
1. Please give {{l}} a more readable name.
2. Can you give some comments to explain some bit about the procedure? I
(probably others) wouldn't know why it's like that without querying the SDK's
manual. I know it now, there're 2 modes in the {{ossClient.deleteObjects}}
operation, one mode to return successfully deleted objects, and the other
returning the deleting-failed objects. You're using the latter, and use a loop
to try some times to delete and delete the failed-to-delete objects.
> AliyunOSS: handle deleteDirs reliably when too many objects to delete
> ---------------------------------------------------------------------
>
> Key: HADOOP-13768
> URL: https://issues.apache.org/jira/browse/HADOOP-13768
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
> Affects Versions: 3.0.0-alpha2
> Reporter: Genmao Yu
> Assignee: Genmao Yu
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13768.001.patch, HADOOP-13768.002.patch
>
>
> Note in Aliyun OSS SDK, DeleteObjectsRequest has 1000 objects limit. This
> needs to improve {{deleteDirs}} operation to make it pass when more objects
> than the limit to delete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
