[
https://issues.apache.org/jira/browse/HADOOP-13687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13687:
-----------------------------------
Attachment: HADOOP-13687-trunk.001.patch
HADOOP-13687-branch-2.001.patch
I'm attaching revision 001 patches for trunk and branch-2.
* I named the new module hadoop-hcfs-client, though I'm open to suggestions.
* The pom.xml has dependencies on all HCFS implementations within the Hadoop
source tree.
* I've used exclusions to trim down the transitive dependencies, taking
inspiration from hadoop-client. Maybe we could trim this down even more.
* Due to a Maven quirk, I needed to put in explicit dependencies on a few
Jackson jars. Without this, my test application that depended on
hadoop-hcfs-client was picking up version 2.5.3 of jackson-databind
transitively through the AWS SDK. That would conflict with our other Jackson
jars, which are 2.2.3. This is a little baffling to me. I would have expected
Maven to version manage down to 2.2.3 for my application, just like what
happens when you build hadoop-hcfs-client itself.
* I updated a hadoop-openstack dependency on the hadoop-common test jar so that
it has test scope instead of compile scope. This way, hadoop-hcfs-client users
won't pick up the hadoop-common test jar transitively.
I tested a full distro and wrote a small custom application with a dependency
on hadoop-hcfs-client to verify that functionality works correctly against all
HCFS implementations.
> Provide a unified dependency artifact that transitively includes all
> Hadoop-compatible file systems.
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13687
> URL: https://issues.apache.org/jira/browse/HADOOP-13687
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-13687-branch-2.001.patch,
> HADOOP-13687-trunk.001.patch
>
>
> Currently, downstream projects that want to integrate with different
> Hadoop-compatible file systems like WASB and S3A need to list dependencies on
> each one. This creates an ongoing maintenance burden for those projects,
> because they need to update their build whenever a new Hadoop-compatible file
> system is introduced. This issue proposes adding a new artifact that
> transitively includes all Hadoop-compatible file systems. Similar to
> hadoop-client, this new artifact will consist of just a pom.xml listing the
> individual dependencies. Downstream users can depend on this artifact to
> sweep in everything, and picking up a new file system in a future version
> will be just a matter of updating the Hadoop dependency version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
