[
https://issues.apache.org/jira/browse/HADOOP-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114453#comment-16114453
]
Sean Mackrory commented on HADOOP-13952:
----------------------------------------
Changing findfileindir in dist-layout-stitching to only omit copying files if
they're under share/hadoop/common (rather than anywhere under share) fixes the
problems, but has some other effects as well. It increases total tarball size
by 4MB and adds the following files:
{code}
./hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/mapreduce/lib: # dir didn't exist at
all before
-rw-rw-r--. 1 sean sean 4467 aopalliance-1.0.jar
-rw-rw-r--. 1 sean sean 668235 guice-4.0.jar
-rw-rw-r--. 1 sean sean 76983 guice-servlet-4.0.jar
-rw-rw-r--. 1 sean sean 2497 javax.inject-1.jar
-rw-rw-r--. 1 sean sean 16151 jersey-guice-1.19.jar
-rw-rw-r--. 1 sean sean 1045744 leveldbjni-all-1.8.jar
./hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/tools/lib:
-rw-rw-r--. 1 sean sean 38302 jetty-util-ajax-9.3.11.v20160721.jar
-rw-rw-r--. 1 sean sean 85449 metrics-core-3.0.1.jar
-rw-rw-r--. 1 sean sean 319099 okhttp-2.4.0.jar
-rw-rw-r--. 1 sean sean 64661 okio-1.4.0.jar
./hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/yarn/lib:
-rw-rw-r--. 1 sean sean 38302 jetty-util-ajax-9.3.11.v20160721.jar
-rw-rw-r--. 1 sean sean 1045744 leveldbjni-all-1.8.jar
-rw-rw-r--. 1 sean sean 1779991 netty-all-4.0.23.Final.jar
{code}
This seems like the correct solution to me, although it perhaps indicates a few
other dependencies could be moved to common since they are in fact shared
between multiple things apparently. The alternative is to not consider
dependencies missing if they're provided by HDFS, but having tools share
classpath with HDFS doesn't feel right. Although I'd expect they actually do
that in practice now?
> tools dependency hooks are throwing errors
> ------------------------------------------
>
> Key: HADOOP-13952
> URL: https://issues.apache.org/jira/browse/HADOOP-13952
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 3.0.0-alpha2
> Reporter: Allen Wittenauer
> Assignee: Sean Mackrory
> Priority: Critical
> Attachments: HADOOP-13952.preview.patch
>
>
> During build, we are throwing these errors:
> {code}
> ERROR: hadoop-aliyun has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> ERROR: hadoop-archive-logs has missing dependencies:
> jasper-compiler-5.5.23.jar
> ERROR: hadoop-archives has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aws has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-azure has missing dependencies:
> jetty-util-ajax-9.3.11.v20160721.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
> ERROR: hadoop-extras has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-gridmix has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-kafka has missing dependencies: lz4-1.2.0.jar
> ERROR: hadoop-kafka has missing dependencies: kafka-clients-0.8.2.1.jar
> ERROR: hadoop-openstack has missing dependencies: commons-httpclient-3.1.jar
> ERROR: hadoop-rumen has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: metrics-core-3.0.1.jar
> ERROR: hadoop-streaming has missing dependencies: jasper-compiler-5.5.23.jar
> {code}
> Likely a variety of reasons for the failures. Kafka is HADOOP-12556, but
> others need to be investigated. Probably just need to look at more than just
> common/lib in dist-tools-hooks-maker now that shading has gone in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
