Hi,
Actually we just want to implement our block placement strategy
in hadoop 3 and compare results. I think that we should see some
improvements. Can you shed some more light on the difference between
general block placement policy and balancer / mover thing.
Warm regards,
Shuubham Ojha
On Thu, Dec 20, 2018 at 9:24 PM Wei-Chiu Chuang <[email protected]>
wrote:
> +Hdfs-dev <[email protected]>
> Hi Shuubham,
>
> Just like to clarify a bit. What's the purpose of this work? Is this for
> the general block placement policy in HDFS, or the
> balancer/mover/diskbalancer, or decommissioning/recommissioning? Block
> placement is determined by NameNode. Do you intend to shorten the time to
> decide where a block is placed? Do you want to reduce the time such that
> re-replication takes less time?
>
> I'm asking this because I don't think there's ever a placementmonitor or a
> blockmonitor class.
>
> On Wed, Dec 19, 2018 at 10:36 PM Shuubham Ojha <[email protected]>
> wrote:
>
>> Hello All,
>>
>> I am Shuubham Ojha a graduate researcher with the
>> University Of Melbourne. We have developed a block placement strategy which
>> optimises delay associated with reconstruction. As a result of this
>> optimisation problem, we get a placement matrix for blocks which tells us
>> which block has to be placed at which node. We have been able to implement
>> this strategy in Hadoop 2 by tweaking the file *placementmonitor.java*
>> and *blockmover.java* where *placementmonitor.java* monitors the
>> placement process and calls *blockmover.java* when the placement is not
>> according the strategy. However, I can't find any such file analogous to
>> *placementmonitor.java* in Hadoop 3 although I think that the closest
>> file which performs this function is *balancer.java* located in
>> hadoop-hdfs-project. Can anybody please provide me more information on this
>> front?
>>
>>
>> Warm Regards,
>>
>> Shuubham Ojha
>>
>> University Of Melbourne,
>>
>> Victoria, Australia- 3010
>>
>
