Hi, As you have only a min of 1 (minReplication=1, meaning no replication) you infrastructure won't provide a guarantee of not loosing data in case of failure. If a user do a command like this :
hdfs dfs -Ddfs.replication=1 -put localfile hdfsfile The file will not be replicated and the block will be definitely loosed when the datanode goes down. You should set a minReplication of 2 Regards, Philippe On Sat, Jul 1, 2017 at 2:05 AM, Brahma Reddy Battula < [email protected]> wrote: > > 1.Yes, those will ensure that file will be written to available nodes . > > > 2. > > BlockManager: defaultReplication = 2 > > This is the Default block replication which you configured in server > (Namenode). The actual number of replications can be specified when the > file is created. The default is used if replication is not specified in > create time. > > > > 3. *"dfs.replication"* is client(in your case confluent kafka) side > property.May be,you can cross check this configuration in kafka. > > > > -Brahma Reddy Battula > ------------------------------ > *From:* Nishant Verma <[email protected]> > *Sent:* Friday, June 30, 2017 7:50 PM > *To:* [email protected] > *Subject:* Ensure High Availability of Datanodes in a HDFS cluster > > > Hi > > I have a two master and three datanode HDFS cluster setup. They are AWS > EC2 instances. > > I have to test High Availability of Datanodes i.e., if during load run > where data is written on HDFS, a datanode dies then there is no data loss. > The two remaning datanodes which are alive should take care of the data > writes. > > I have set below properties in hdfs-site.xml. dfs.replication = 2 (because > if any one datanode dies, then there is no issue of not able to meet > replication factor) > > dfs.client.block.write.replace-datanode-on-failure.policy = ALWAYS > dfs.client.block.write.replace-datanode-on-failure.enable = true > dfs.client.block.write.replace-datanode-on-failure.best-effort = true > > My questions are: > > 1 - Does setting up above properties suffice my Datanode High > Availability? Or something else is needed? 2 - On dfs service startup, I do > see below INFO on namenode logs: > > 2017-06-27 10:51:52,546 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > defaultReplication = 2 > 2017-06-27 10:51:52,546 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication > = 512 > 2017-06-27 10:51:52,546 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication > = 1 > 2017-06-27 10:51:52,546 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > maxReplicationStreams = 2 > > But I still see that the files being created on HDFS are with replication > factor 3. Why is that so? This would hurt my High Availability of Datanodes. > > -rw-r--r-- 3 hadoopuser supergroup 247373 2017-06-29 09:36 > /topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+210+0001557358+0001557452 > -rw-r--r-- 3 hadoopuser supergroup 1344 2017-06-29 08:33 > /topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+228+0001432839+0001432850 > -rw-r--r-- 3 hadoopuser supergroup 3472 2017-06-29 09:03 > /topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+228+0001432851+0001432881 > -rw-r--r-- 3 hadoopuser supergroup 2576 2017-06-29 08:33 > /topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+23+0001236477+0001236499 > > P.S. - My records are written on HDFS by Confluent Kafka Connect HDFS Sink > Connector. > > > Thanks > > Nishant > -- Philippe Kernévez Directeur technique (Suisse), [email protected] +41 79 888 33 32 Retrouvez OCTO sur OCTO Talk : http://blog.octo.com OCTO Technology http://www.octo.ch
