[
https://issues.apache.org/jira/browse/HADOOP-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949388#comment-14949388
]
Chris Nauroth commented on HADOOP-11127:
----------------------------------------
[~alanburlison], thank you again for picking this up. The proposal essentially
jives with the HADOOP-11064.003.patch attached to this JIRA, which is something
Colin had coded a while ago.
I agree with Colin that option #3 is possible. I might not have made this
clear in my earlier notes, but the proposal is that hadoop-common.jar would
contain a compatible native build for every supported target platform, not just
one. It would contain a liibhadoop.so, hadoop.dll, winutils.exe, and now
Solaris artifacts too thanks to your efforts. At runtime, hadoop-common.jar
can detect the platform and bind to the corresponding native artifact. This
has a lot of benefits, because linking to a correct version would then be
achieved completely by the Maven dependency resolution process. There wouldn't
be any external dependencies. This might even completely eliminate a common
category of problems for end users on Windows: how to track down a good
hadoop.dll and winutils.exe and configure Hadoop to find them.
Steve mentioned snappy-java as an example of a project that already
demonstrates that this is possible. Here is a link.
https://github.com/xerial/snappy-java
Additionally, hadoop-lzo recently added similar support for its native
components.
https://github.com/twitter/hadoop-lzo/pull/81
Option #3 also is admittedly the most difficult to implement. The coding isn't
the difficult part. The difficult part is the build and release process, which
would look much different from the current Apache Hadoop release process.
Unfortunately, I think versioning with a composite name introduces some new
problems. Steve already articulated this very well. Also, in the examples in
your proposal, it uses X.Y versions, i.e. libhadoop-2.7.so, without the .Z
part. Were you thinking of expanding this to include the .Z part, i.e.
libhadoop-2.7.1.so, or were you thinking that the internal interfaces are still
frozen within an X.Y version?
> Improve versioning and compatibility support in native library for downstream
> hadoop-common users.
> --------------------------------------------------------------------------------------------------
>
> Key: HADOOP-11127
> URL: https://issues.apache.org/jira/browse/HADOOP-11127
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Reporter: Chris Nauroth
> Assignee: Alan Burlison
> Attachments: HADOOP-11064.003.patch, proposal.txt
>
>
> There is no compatibility policy enforced on the JNI function signatures
> implemented in the native library. This library typically is deployed to all
> nodes in a cluster, built from a specific source code version. However,
> downstream applications that want to run in that cluster might choose to
> bundle a hadoop-common jar at a different version. Since there is no
> compatibility policy, this can cause link errors at runtime when the native
> function signatures expected by hadoop-common.jar do not exist in
> libhadoop.so/hadoop.dll.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
