[
https://issues.apache.org/jira/browse/HADOOP-10007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393303#comment-16393303
]
Steve Loughran commented on HADOOP-10007:
-----------------------------------------
latest stack
{code}2018-03-09 17:59:08,752 [LocalJobRunner Map Task Executor #0] ERROR
util.RetriableCommand (RetriableCommand.java:execute(89)) - Failure in
Retriable command: Copying
file:/Users/stevel/hadoop-trunk/hadoop-common-project/hadoop-auth/target/classes3/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider$1.class
to
ftp://ftpserver/home/scratch/auth/target/classes3/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider$1.class
java.io.IOException: Cannot rename source:
ftp://ftpserver/home/scratch/auth/.distcp.tmp.attempt_local1253814949_0001_m_000000_0
to
ftp://ftpserver/home/scratch/auth/target/classes3/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider$1.class
-only same directory renames are supported
at org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:674)
at org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:613)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:249)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:140)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:256)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:217)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-03-09 17:59:12,700 [LocalJobRunner Map Task Exec
{code}
> distcp / mv is not working on ftp
> ---------------------------------
>
> Key: HADOOP-10007
> URL: https://issues.apache.org/jira/browse/HADOOP-10007
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Environment: Ubuntu 12.04.2 LTS
> Hadoop 2.0.0-cdh4.2.1
> Subversion
> file:///var/lib/jenkins/workspace/generic-package-ubuntu64-12-04/CDH4.2.1-Packaging-Hadoop-2013-04-22_09-50-19/hadoop-2.0.0+960-1.cdh4.2.1.p0.9~precise/src/hadoop-common-project/hadoop-common
> -r 144bd548d481c2774fab2bec2ac2645d190f705b
> Compiled by jenkins on Mon Apr 22 10:26:30 PDT 2013
> From source with checksum aef88defdddfb22327a107fbd7063395
> Reporter: Fabian Zimmermann
> Priority: Major
>
> i'm just trying to backup some files to our ftp-server.
> hadoop distcp hdfs:///data/ ftp://user:pass@server/data/
> returns after some minutes with:
> Task TASKID="task_201308231529_97700_m_000002" TASK_TYPE="MAP"
> TASK_STATUS="FAILED" FINISH_TIME="1380217916479"
> ERROR="java\.io\.IOException: Cannot rename parent(source):
> ftp://x:x@backup2/data/, parent(destination): ftp://x:x@backup2/data/
> at
> org\.apache\.hadoop\.fs\.ftp\.FTPFileSystem\.rename(FTPFileSystem\.java:557)
> at
> org\.apache\.hadoop\.fs\.ftp\.FTPFileSystem\.rename(FTPFileSystem\.java:522)
> at
> org\.apache\.hadoop\.mapred\.FileOutputCommitter\.moveTaskOutputs(FileOutputCommitter\.java:154)
> at
> org\.apache\.hadoop\.mapred\.FileOutputCommitter\.moveTaskOutputs(FileOutputCommitter\.java:172)
> at
> org\.apache\.hadoop\.mapred\.FileOutputCommitter\.commitTask(FileOutputCommitter\.java:132)
> at
> org\.apache\.hadoop\.mapred\.OutputCommitter\.commitTask(OutputCommitter\.java:221)
> at org\.apache\.hadoop\.mapred\.Task\.commit(Task\.java:1000)
> at org\.apache\.hadoop\.mapred\.Task\.done(Task\.java:870)
> at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:329)
> at org\.apache\.hadoop\.mapred\.Child$4\.run" TASK_ATTEMPT_ID="" .
> I googled a bit and added
> fs.ftp.host = backup2
> fs.ftp.user.backup2 = user
> fs.ftp.password.backup2 = password
> to core-site.xml, then I was able to execute:
> hadoop fs -ls ftp:///data/
> hadoop fs -rm ftp:///data/test.file
> but as soon as I try
> hadoop fs -mv file:///data/test.file ftp:///data/test2.file
> mv: `ftp:///data/test.file': Input/output error
> I enabled debug-logging in our ftp-server and got:
> Sep 27 15:24:33 backup2 ftpd[38241]: command: LIST /data
> Sep 27 15:24:33 backup2 ftpd[38241]: <--- 150
> Sep 27 15:24:33 backup2 ftpd[38241]: Opening BINARY mode data connection for
> '/bin/ls'.
> Sep 27 15:24:33 backup2 ftpd[38241]: <--- 226
> Sep 27 15:24:33 backup2 ftpd[38241]: Transfer complete.
> Sep 27 15:24:33 backup2 ftpd[38241]: command: CWD ftp:/data
> Sep 27 15:24:33 backup2 ftpd[38241]: <--- 550
> Sep 27 15:24:33 backup2 ftpd[38241]: ftp:/data: No such file or directory.
> Sep 27 15:24:33 backup2 ftpd[38241]: command: RNFR test.file
> Sep 27 15:24:33 backup2 ftpd[38241]: <--- 550
> looks like the generation of "CWD" is buggy, hadoop tries to cd into
> "ftp:/data", but should use "/data"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
