[
https://issues.apache.org/jira/browse/HADOOP-12038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566900#comment-14566900
]
Chen He commented on HADOOP-12038:
----------------------------------
Thanks, [~steve_l].
Actually, Openstack community has another version of swift driver for Hadoop.
It supports files that are larger than 5GB, what I did is to add those
functions to hadoop-openstack module. I don't know why Hadoo community does not
have similar solution. The error was reported during my test process.
Openstack driver is call Sahara. It breaks file (larger than 5GB) into a
configurable chunks (default 4.6GB) and create a manifest fold in swift file
system and point to those chunks. However, since swift "rename" process is to
create a new file instead of changing original file's name (Because of Swift
DHT using name to do the hash). It is inefficient for large file copying. I
resolved this issue and will create issue and post patch later.
> SwiftNativeOutputStream should check whether a file exists or not before
> deleting
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-12038
> URL: https://issues.apache.org/jira/browse/HADOOP-12038
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.7.0
> Reporter: Chen He
> Assignee: Chen He
> Priority: Minor
> Attachments: HADOOP-12038.000.patch
>
>
> 15/05/27 15:27:03 WARN snative.SwiftNativeOutputStream: Could not delete
> /tmp/hadoop-root/output-3695386887711395289.tmp
> It should check whether the file exists or not before deleting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
