[
https://issues.apache.org/jira/browse/HADOOP-11750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382068#comment-14382068
]
Chen He commented on HADOOP-11750:
----------------------------------
Hi [[email protected]], thank you for reviewing this issue.
If we use "dfs -cp", it is a single "JVM" serial copy, right? What if users
want to copy 10TB data from swift to HDFS? Serial copy is impractical.
IMHO, DistCp is a tool that can help people copy data across different
filesystems in parallel but not limited to HDFS. Our team is working on
resolving this problem, please at least leave one or two days for further
discussion before close it directly.
> distcp fails if we copy data from swift to secure HDFS
> ------------------------------------------------------
>
> Key: HADOOP-11750
> URL: https://issues.apache.org/jira/browse/HADOOP-11750
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/swift
> Affects Versions: 3.0.0, 2.3.0
> Reporter: Chen He
> Assignee: Chen He
>
> ERROR tools.DistCp: Exception encountered
> java.lang.IllegalArgumentException: java.net.UnknownHostException:
> babynames.main
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:373)
> at
> org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:258)
> at
> org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:301)
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:523)
> at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:507)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:121)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
> at
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:133)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:83)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:84)
> at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:353)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:160)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
> Caused by: java.net.UnknownHostException: babynames.main
> ... 17 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
