Hi Hilmi, thanks for your response. I have already done the following: 1. Defined the rack topology in a script and added that to core-site.xml. I think this ensures that the hadoop on namenode knows the ip addresses of datanodes and corresponding rack numbers. 2. I have gone through the default block placement policy code multiple times and I can see that a parameter called favored nodes has been used. I believe this parameter is of interest to me.
The trouble is that I have a set of nodes with known ip addresses but I cant see any specific ip addresses being used in the placement policy code. I specifically want to know how to tell my hadoop code to place a particular block of data on a particular datanode given it's ip address. I think this problem can be resolved by setting the given datanode as favored node for the time when the data block of interest is being dequeued to be sent to datanodes but I can't how that should be done. Warm regards, Shuubham Ojha On Fri, Apr 19, 2019 at 4:12 AM Hilmi Egemen Ciritoğlu < [email protected]> wrote: > Hi Shuubham, > > You can simply create your own block placement class by extending > BlockPlacementPolicy. > As a starting point, you may want to have a look at Block Placement > Default Policy Source Code > <https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java> > . > > Regards, > H. Egemen Ciritoglu > > On Thu, 18 Apr 2019 at 22:34, Shuubham Ojha <[email protected]> > wrote: > >> Can anyone give me some idea as to how I can write my own block >> placement strategy in hadoop 3. >> >> >> Shuubham Ojha >> >
