[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012839#comment-14012839
]
Colin Patrick McCabe commented on HADOOP-10640:
-----------------------------------------------
With this patch, libhdfs.so continues to implement the same API as before. At
runtime, we select which native filesystem implementation to use based on by
looking at {{<scheme>.native.handler}}. So, if the scheme of the URI we're
handling is HDFS, we'd look at {{hdfs.native.handler}}. If there is no
scheme-specific handler, we fall back to {{default.native.handler}}.
There are two libhdfs native filesystems available at the moment. One is
jniFS, the existing JNI-based filesystem. The other is NDFS, the new native
filesystem that we're developing. As mentioned earlier, jniFS has the
advantage of working with things like s3 or HDFS encryption, whereas NDFS has
the advantage that it doesn't require JNI, is more efficient, etc. The goal
here is to have a {{libhdfs.so}} that we can drop into existing installations
and have everything "just work."
The libhdfs.so library doesn't depend on JNI... it can be installed in cases
where JNI is not present. The reason is because it uses {{dlopen}} (by means
of the libuv shims) to open the {{libjvm.so}} library as-needed. So clients
who don't want to install or use JNI don't have to.
The build changed a little bit to have a {{config.h.cmake}} file which gets
transformed into {{config.h}} based on which features are present at
compile-time. This is how the existing libhdfs build works (we need it to
detect some platform-specific features.) I also hide all library symbols
except for the public symbols, so that our library internals are not visible to
the outside world, using {{fvisibility=hidden}}.
source overview:
* common/hadoop_err: added the {{hadoop_err_prepend}}, {{hadoop_err_copy}}, and
{{terror}} utility functions.
* common/hconf: this is a native implementation of the Java Configuration class
which reads XML files. It uses libexpat.
* common/htable: a hash table used in a few places.
* common/net: added {{net_ipv4_name_and_port}}, {{get_first_ipv4_addr}} utility
functions
* common/string: added the {{strdupto}} utility function plus a unit test.
* common/uri: parses HDFS-style URIs into scheme, authority, port, etc. using
the uriparser library.
* fs/fs.c: the entry points for all libhdfs functions. Contains a vtable used
to call the appropriate implementation function.
* fs/hdfs.h: the public API of libhdfs. This is just a rename.
* jni/: all the stuff in this directory was moved from the previous libhdfs
location.
* net/proxy, net/reactor: fixed a bug in the synchronous RPC code
* test/: I created a test directory paralleling the source directory. Moved
test.h to this directory.
Running {{NAMENODE_URI=hdfs://localhost:6000 ./test_libhdfs_meta_ops}} will
give an idea of how this operates.
> Implement Namenode RPCs in HDFS native client
> ---------------------------------------------
>
> Key: HADOOP-10640
> URL: https://issues.apache.org/jira/browse/HADOOP-10640
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: HADOOP-10388
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
>
> Implement the parts of libhdfs that just involve making RPCs to the Namenode,
> such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
