[
https://issues.apache.org/jira/browse/HADOOP-13687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549905#comment-15549905
]
Chris Nauroth commented on HADOOP-13687:
----------------------------------------
Some applications are choosing not to run with the Hadoop classpath, and
sometimes even supporting a deployment model where they are not co-located with
a full Hadoop distro. For those applications, they can't rely on execution
through {{hadoop jar}} or {{java -cp $(hadoop classpath)}} or similar.
Another use case for this is to support integration testing in downstream
components, where they'll be able to add a test dependency on this new artifact
and run against WASB, S3A, etc.
For applications that do not need to operate under this limitation, I agree
that relying on the Hadoop classpath is the better approach.
I think your last comment is referring to HCFS implementations that reside
outside of the Apache Hadoop source tree. The intent is not to force those to
ship an artifact within Hadoop, and also the intent is not for Hadoop to add an
external Maven dependency on them. The scope of this only applies to the
implementations already shipped in-tree. Specifically, that's hadoop-aws,
hadoop-azure, hadoop-azure-datalake and hadoop-openstack.
> Provide a unified dependency artifact that transitively includes all
> Hadoop-compatible file systems.
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13687
> URL: https://issues.apache.org/jira/browse/HADOOP-13687
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
>
> Currently, downstream projects that want to integrate with different
> Hadoop-compatible file systems like WASB and S3A need to list dependencies on
> each one. This creates an ongoing maintenance burden for those projects,
> because they need to update their build whenever a new Hadoop-compatible file
> system is introduced. This issue proposes adding a new artifact that
> transitively includes all Hadoop-compatible file systems. Similar to
> hadoop-client, this new artifact will consist of just a pom.xml listing the
> individual dependencies. Downstream users can depend on this artifact to
> sweep in everything, and picking up a new file system in a future version
> will be just a matter of updating the Hadoop dependency version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
