Hello, List, I'm new here and I hope you are all very fine. I'm trying different combinations of distcp in order to restore data that I receive from standard input. Example:
============================== ============================== === echo data | /etc/hadoop/bin/hadoop distcp file:///dev/stdin hdfs://hadoop2:54310/a ============================== ============================== === I tried different options of distcp but the MapReduce always stalls. E.g.: ============================== ============================== ==== 7-08-15 06:59:26,665 INFO [main] org.apache.hadoop.metrics2. impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2017-08-15 06:59:26,802 INFO [main] org.apache.hadoop.metrics2. impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-08-15 06:59:26,802 INFO [main] org.apache.hadoop.metrics2. impl.MetricsSystemImpl: MapTask metrics system started 2017-08-15 06:59:26,813 INFO [main] org.apache.hadoop.mapred. YarnChild: Executing with tokens: 2017-08-15 06:59:26,814 INFO [main] org.apache.hadoop.mapred. YarnChild: Kind: mapreduce.job, Service: job_1502794712113_0001, Ident: (org.apache.hadoop.mapreduce. security.token. JobTokenIdentifier@467f0da4) 2017-08-15 06:59:26,996 INFO [main] org.apache.hadoop.mapred. YarnChild: Sleeping for 0ms before retrying again. Got null now. 2017-08-15 06:59:27,518 INFO [main] org.apache.hadoop.mapred. YarnChild: mapreduce.cluster.local.dir for child: /root/hdfs/hadoop-tmp-dir/nm- local-dir/usercache/root/ appcache/application_ 1502794712113_0001 2017-08-15 06:59:28,926 INFO [main] org.apache.hadoop.conf. Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-08-15 06:59:29,783 INFO [main] org.apache.hadoop.mapreduce. lib.output. FileOutputCommitter: File Output Committer Algorithm version is 1 2017-08-15 06:59:29,804 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-08-15 06:59:30,139 INFO [main] org.apache.hadoop.mapred. MapTask: Processing split: /tmp/hadoop-yarn/staging/root/ .staging/_distcp-298457134/ fileList.seq:0+176 2017-08-15 06:59:30,145 INFO [main] org.apache.hadoop.mapreduce. lib.output. FileOutputCommitter: File Output Committer Algorithm version is 1 2017-08-15 06:59:30,250 INFO [main] org.apache.hadoop.tools. mapred.CopyMapper: Copying file:/dev/stdin to hdfs://hadoop2:54310/aaa 2017-08-15 06:59:30,259 INFO [main] org.apache.hadoop.tools. mapred. RetriableFileCopyCommand: Creating temp file: hdfs://hadoop2:54310/.distcp. tmp.attempt_1502794712113_ 0001_m_000000_0 ============================== ============================== ==== Regards, -- =========================================================================== Heitor Medrado de Faria | CEO Bacula do Brasil | Visto EB-1 | LPIC-III | EMC 05-001 | ITIL-F • Não seja tarifado pelo tamanho dos seus backups, conheça o Bacula Enterprise: http://www.bacula.com.br/enterprise/ • Ministro treinamento e implementação in-company do Bacula Community: http://www.bacula.com.br/in-company/ +55 61 98268-4220 | www.bacula.com.br ============================================================================ Indicamos também as capacitações complementares: • Shell básico e Programação em Shell com Julio Neves. • Zabbix com Adail Host. ============================================================================
