[
https://issues.apache.org/jira/browse/HADOOP-7359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059384#comment-13059384
]
Steve Loughran commented on HADOOP-7359:
----------------------------------------
I played for a while with a dynamic subclass of JobConf that could be changed
from CM tooling in a live system. There are a lot of places where those configs
get serialized, and it's surprisingly hard to change things on the fly: the
nodes generally assume things are static for the life of the process. That's a
hard thing to deal with in the workers
LDAP/JNDI might be a better option for central sites over JDBC. This doesn't
mean that I', a fan of JNDI, only that it's easier to write back ends to than a
generic JDBC driver -and LDAP is designed with better redundancy in from the
outset.
> Pluggable interface for cluster membership
> ------------------------------------------
>
> Key: HADOOP-7359
> URL: https://issues.apache.org/jira/browse/HADOOP-7359
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Travis Crawford
> Attachments: HADOOP-7359.diff
>
>
> Currently Hadoop uses local files to determine cluster membership. With HDFS
> for example, dfs.hosts and dfs.hosts.exclude are used.
> To enable tighter integrations cluster membership should be an interface,
> with the current file-based functionality provided as the default
> implementation. The common case would be no functional change, however, sites
> could plug an alternative implementation in, such as pulling the machine
> lists from a machine database.
> DETAILS:
> Two machine lists, includes and excludes, are used to define cluster
> membership and state. HostsFileReader currently handles reading these lists
> from files, who's names are passed in by FSNamesystem for HDFS and JobTracker
> for MR.
> The proposed change is adding a HostsReader interface to common, and changing
> HostsFileReader to an abstract class that functions the same as today.
> Two new classes, DFSHostsFileReader and MRHostsFileReader, extend
> HostsFileReader and simply pass the appropriate file names in. These new
> classes are needed because config key names live outside common.
> Two new conf keys, defaulting to the file-based readers, would be added to
> choose a different hosts reader: dfs.namenode.hosts.reader.class
> mapreduce.jobtracker.hosts.reader.class
> Comments/suggestions? I have most of this written already but would love some
> feedback on the general idea before posting the diff.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
