[
https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785778#comment-13785778
]
Haohui Mai commented on HADOOP-10016:
-------------------------------------
Indeed this is related to HADOOP-8828.
After discussions with [~sureshms], [~jingzhao], [~sanjay.radia], we believe
that it is a valid use case that distcp copies data from a secure cluster to an
insecure cluster. That is, distcp runs inside the secure cluster and writes to
the insecure cluster. The set up is the same as the one described in
HADOOP-8828.
It is particularly important to support the use case of copying data from a
secure Hadoop 1 cluster to an insecure Hadoop 2 cluster, since this gives users
a path to migrate data from a secure Hadoop 1 cluster to a new installation of
insecure Hadoop 2 cluster.
The problem here is that in this set up, both distcp and map-reduce try to ask
for delegation tokens in order to authenticate with the insecure cluster, in
which case the insecure cluster returns an error. Currently Hadoop 2 can mostly
handle this case (see HADOOP-10017), since it uses RPCv9 which supports
negotiation and fallback during authentication.
For Hadoop 1, however, the above use case is fundamentally _broken_ since RPCv8
does not support negotiation at all. What it means is that you cannot write to
the insecure Hadoop 2 cluster directly via HDFS. You cannot write to the
cluster via WebHDFS either because neither distcp nor map-reduce can
successfully get delegation tokens from the insecure cluster.
> Distcp should support copy from a secure Hadoop 1 cluster to an insecure
> Hadoop 2 cluster
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-10016
> URL: https://issues.apache.org/jira/browse/HADOOP-10016
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Haohui Mai
> Assignee: Haohui Mai
>
> Distcp should be able to copy from a secure cluster to an insecure cluster.
> This functionality is important for operators to migrate data to a new Hadoop
> installation.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
