[
https://issues.apache.org/jira/browse/HADOOP-18966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18966:
------------------------------------
Summary: s3a file rename does double HEAD or LIST on source file/dir (was:
s3a file rename does double HEAD or LIST on source fiile/dir)
> s3a file rename does double HEAD or LIST on source file/dir
> -----------------------------------------------------------
>
> Key: HADOOP-18966
> URL: https://issues.apache.org/jira/browse/HADOOP-18966
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.6
> Reporter: Steve Loughran
> Priority: Minor
>
> going to add this as a v2 issue, though really it's been latent.
> when you do a rename() we do a getFileStatus call to get the list/file
> status..if this is a dir that's a single LIST; a file then it'll be LIST +
> HEAD.
> when the actual copy is kicked off,
> * if the source is a dir, then a duplicate LIST is initiated straight
> afterwards.
> * FIle: there's a second HEAD to get the metadata for the copy
> Proposed.
> * LIST initiates a full list, and if it returns a list of objects, that list
> iterator is passed in to rename.
> * full result of HEAD preserved and passed down to copyFile()
> will cut out one round trip regardless of source type
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
