Favoured nodes is a per-file property that you can set at the time of creation. Note that future rebalancing may not respect this. You can read more about it here - https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L1171
If there is a fixed relation b/w your files and the datanode they need to be on, then as Hilmi suggested you are better off writing a custom policy that extends the default one. You would need to extend the chooseTarget method to apply your logic there. ~ Hari On Fri, Apr 19, 2019 at 12:44 PM Shuubham Ojha <[email protected]> wrote: > Hi Hilmi, thanks for your response. I have already done the following: > 1. Defined the rack topology in a script and added that to core-site.xml. > I think this ensures that the hadoop on namenode knows the ip addresses of > datanodes and corresponding rack numbers. > 2. I have gone through the default block placement policy code multiple > times and I can see that a parameter called favored nodes has been used. I > believe this parameter is of interest to me. > > The trouble is that I have a set of nodes with known ip addresses but I > cant see any specific ip addresses being used in the placement policy code. > I specifically want to know how to tell my hadoop code to place a > particular block of data on a particular datanode given it's ip address. I > think this problem can be resolved by setting the given datanode as favored > node for the time when the data block of interest is being dequeued to be > sent to datanodes but I can't how that should be done. > > Warm regards, > Shuubham Ojha > > On Fri, Apr 19, 2019 at 4:12 AM Hilmi Egemen Ciritoğlu < > [email protected]> wrote: > >> Hi Shuubham, >> >> You can simply create your own block placement class by extending >> BlockPlacementPolicy. >> As a starting point, you may want to have a look at Block Placement >> Default Policy Source Code >> <https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java> >> . >> >> Regards, >> H. Egemen Ciritoglu >> >> On Thu, 18 Apr 2019 at 22:34, Shuubham Ojha <[email protected]> >> wrote: >> >>> Can anyone give me some idea as to how I can write my own block >>> placement strategy in hadoop 3. >>> >>> >>> Shuubham Ojha >>> >>
