[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039655#comment-16039655
]
Allen Wittenauer commented on HADOOP-14498:
-------------------------------------------
HADOOP\_OPTIONAL\_TOOLS basically triggers a read of
"libexec/shellprofile.d/(whatever).sh" that is created at build time by some
maven magic and "dev-support/bin/dist-tools-hooks-maker" .
The inside of this file (after cutting out the boiler plate) says, effectively:
{code}
function _hadoop-azure-datalake_hadoop_classpath
{
if [[ -f
"${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/azure-data-lake-store-sdk-2.1.4.jar"
]]; then
hadoop_add_classpath
"${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/azure-data-lake-store-sdk-2.1.4.jar"
fi
hadoop_add_classpath
"${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/hadoop-azure-datalake-3.0.0-alpha4-SNAPSHOT.jar"
}
{code}
ie, we're going to add azure-data-lake-store-sdk-2.1.4.jar and
hadoop-azure-datalake-3.0.0-alpha4-SNAPSHOT.jar to the classpath.
hadoop-azure.sh, meanwhile, says:
{code}
if [[ -f
"${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/azure-storage-4.2.0.jar" ]];
then
hadoop_add_classpath
"${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/azure-storage-4.2.0.jar"
fi
if [[ -f
"${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/azure-keyvault-core-0.8.0.jar"
]]; then
hadoop_add_classpath
"${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/azure-keyvault-core-0.8.0.jar"
fi
hadoop_add_classpath
"${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/hadoop-azure-3.0.0-alpha4-SNAPSHOT.jar"
{code}
ie., azure-storage-4.2.0.jar, azure-keyvault-core-0.8.0.jar, and
hadoop-azure-3.0.0-alpha4-SNAPSHOT.jar.
Different dependencies are getting generated by the build and are either
incorrect/incomplete in the pom, a bug in the dependency file generator, or
something else going haywire. It is not a bug in how HADOOP\_OPTIONAL\_TOOLS
is getting parsed post-build.
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> ------------------------------------------
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 3.0.0-alpha1
> Reporter: Mingliang Liu
> Priority: Critical
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
