Hi, I try to setup a hadoop cluser with 1 namenode and 5 datanodes with two networks. The cluster should use a 192.168 address for its own communication, client traffic should happen over a 10 network.
Allthough reading a bunch of websides about this I cannot get this to to work, there are allways connection errors showing up. Maybe someone has done this and could help me with that... The setup is: each node has two nics. The datanodes are 192.168.10.1 to 192.168.10.5, the namenode is 192.168.10.6 The second nic in each node has network 10 address. The clients connect via the 10 network to the cluster, also the dns resolves to the 10 network Listed in /etc/host are short names with the corresponding 192.168.. address aand the fqdn with the 10 address. The cluster nodes should talk to each other via the 192 addresses, with no firewall between them. My config includes: namenode: dfs.namenode.rpc-bind-host: 0.0.0.0 dfs.namenode.servicerpc-bind-host: 0.0.0. datanodes: dfs.namenode.servicerpc-address: master01:9000 dfs.client.use.datanode.hostname: true the slaves file contains the short names which are listed in the /etc/hosts file. With this config the cluster starts up without errors, hdfs dfs -ls / works too, but "hdfs dfs -put ..." doesn't work: DFSClient: Exception in createBlockOutputStream, Connection refused. Abandoning BP....-10.x.x.x-7872394857:blk_23423433_1013 Excluding DatanodeInfoWithStorage(192.168.10.2 ) I thought the datanodes listen on 0.0.0 as shown in https://hadoop.apache.org/docs/r2.7.7/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml What I am missing? I tried various combinations of config values, maybe overlooking the obvious right one... Thanks for helping, Michael --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
