[
https://issues.apache.org/jira/browse/HADOOP-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114925#comment-16114925
]
Allen Wittenauer commented on HADOOP-13952:
-------------------------------------------
*shrugs* None of the solutions are great, honestly. Most of the gains that we
got by doing the dedupe were quickly ate up by YARN's bloat anyway. :(
Code-wise:
1. using an array will be significantly faster. (Yes, getting rid of awk's and
cut's is my new thing, now that I've just thrown in the towel on arrays.)
{code}
function depline_to_jarname
{
depline="${*}"
IFS=':' read -ra fields <<< "${depline}"
case "${#fields[@]}" in
"5")
# format: groupId:artifactId:type:version:scope
jarname="${fields[1]}-${fields[3]}.jar"
;;
"6")
# format: groupId:artifactId:type:classifier:version:scope
jarname="${fields[1]}-${fields[4]}-${fields[5]}.jar"
;;
*)
echo "Error: unrecognized number of fields in mvn dependency:list output"
exit 1
;;
esac
echo "${jarname}"
}
{code}
2. Probably better to do bash-style math:
{code}
declare -i errors=0
...
((errors=errors+1))
{code}
> tools dependency hooks are throwing errors
> ------------------------------------------
>
> Key: HADOOP-13952
> URL: https://issues.apache.org/jira/browse/HADOOP-13952
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 3.0.0-alpha2
> Reporter: Allen Wittenauer
> Assignee: Sean Mackrory
> Priority: Critical
> Attachments: HADOOP-13952.000.patch, HADOOP-13952.preview.patch
>
>
> During build, we are throwing these errors:
> {code}
> ERROR: hadoop-aliyun has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> ERROR: hadoop-archive-logs has missing dependencies:
> jasper-compiler-5.5.23.jar
> ERROR: hadoop-archives has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aws has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-azure has missing dependencies:
> jetty-util-ajax-9.3.11.v20160721.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
> ERROR: hadoop-extras has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-gridmix has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-kafka has missing dependencies: lz4-1.2.0.jar
> ERROR: hadoop-kafka has missing dependencies: kafka-clients-0.8.2.1.jar
> ERROR: hadoop-openstack has missing dependencies: commons-httpclient-3.1.jar
> ERROR: hadoop-rumen has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: metrics-core-3.0.1.jar
> ERROR: hadoop-streaming has missing dependencies: jasper-compiler-5.5.23.jar
> {code}
> Likely a variety of reasons for the failures. Kafka is HADOOP-12556, but
> others need to be investigated. Probably just need to look at more than just
> common/lib in dist-tools-hooks-maker now that shading has gone in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
