[
https://issues.apache.org/jira/browse/HADOOP-19696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18020920#comment-18020920
]
ASF GitHub Bot commented on HADOOP-19696:
-----------------------------------------
steveloughran commented on PR #7980:
URL: https://github.com/apache/hadoop/pull/7980#issuecomment-3303127263
* This puts the hadoop-azure, hadoop-aws &c binaries into common/lib and so
on the classpath everywhere
* some problem with gcs instantiation during enum (will file later, as while
it surfaces here, I think it's unrelated)
* my local builds end up (today) with some versioned jars as well as the
-SNAPSHOT. I think this is from me tainting my maven repo, would like to see
what others see
```
total 1401704
-rw-r--r--@ 1 stevel staff 106151 Sep 17 12:57
aliyun-java-core-0.2.11-beta.jar
-rw-r--r--@ 1 stevel staff 194215 Sep 17 12:57
aliyun-java-sdk-core-4.5.10.jar
-rw-r--r--@ 1 stevel staff 163698 Sep 17 12:57
aliyun-java-sdk-kms-2.11.0.jar
-rw-r--r--@ 1 stevel staff 220800 Sep 17 12:57
aliyun-java-sdk-ram-3.1.0.jar
-rw-r--r--@ 1 stevel staff 928456 Sep 17 12:57 aliyun-sdk-oss-3.18.1.jar
-rw-r--r--@ 1 stevel staff 2470776 Sep 17 12:57
analyticsaccelerator-s3-1.3.0.jar
-rw-r--r--@ 1 stevel staff 27006 Sep 17 13:11
aopalliance-repackaged-2.6.1.jar
-rw-r--r--@ 1 stevel staff 20891 Sep 17 13:11
audience-annotations-0.12.0.jar
-rw-r--r--@ 1 stevel staff 651391 Sep 17 13:11 avro-1.11.4.jar
-rw-r--r--@ 1 stevel staff 113966 Sep 17 12:57
azure-data-lake-store-sdk-2.3.9.jar
-rw-r--r--@ 1 stevel staff 10288 Sep 17 12:57
azure-keyvault-core-1.0.0.jar
-rw-r--r--@ 1 stevel staff 815331 Sep 17 12:57 azure-storage-7.0.1.jar
-rw-r--r--@ 1 stevel staff 8324412 Sep 17 13:11 bcprov-jdk18on-1.78.1.jar
-rw-r--r--@ 1 stevel staff 641534749 Sep 17 12:57 bundle-2.29.52.jar
-rw-r--r--@ 1 stevel staff 223979 Sep 17 13:11 checker-qual-3.33.0.jar
-rw-r--r--@ 1 stevel staff 75479 Sep 17 13:11 commons-cli-1.9.0.jar
-rw-r--r--@ 1 stevel staff 353793 Sep 17 13:11 commons-codec-1.15.jar
-rw-r--r--@ 1 stevel staff 751914 Sep 17 13:11
commons-collections4-4.4.jar
-rw-r--r--@ 1 stevel staff 1079377 Sep 17 13:11
commons-compress-1.26.1.jar
-rw-r--r--@ 1 stevel staff 657516 Sep 17 13:11
commons-configuration2-2.10.1.jar
-rw-r--r--@ 1 stevel staff 24239 Sep 17 13:11 commons-daemon-1.0.13.jar
-rw-r--r--@ 1 stevel staff 508826 Sep 17 13:11 commons-io-2.16.1.jar
-rw-r--r--@ 1 stevel staff 673587 Sep 17 13:11 commons-lang3-3.17.0.jar
-rw-r--r--@ 1 stevel staff 70816 Sep 17 13:11 commons-logging-1.3.0.jar
-rw-r--r--@ 1 stevel staff 2213560 Sep 17 13:11 commons-math3-3.6.1.jar
-rw-r--r--@ 1 stevel staff 316431 Sep 17 13:11 commons-net-3.9.0.jar
-rw-r--r--@ 1 stevel staff 238400 Sep 17 13:11 commons-text-1.10.0.jar
-rw-r--r--@ 1 stevel staff 8661164 Sep 17 12:57 cos_api-bundle-5.6.19.jar
-rw-r--r--@ 1 stevel staff 2983237 Sep 17 13:11 curator-client-5.2.0.jar
-rw-r--r--@ 1 stevel staff 336384 Sep 17 13:11
curator-framework-5.2.0.jar
-rw-r--r--@ 1 stevel staff 315569 Sep 17 13:11 curator-recipes-5.2.0.jar
-rw-r--r--@ 1 stevel staff 583996 Sep 17 13:11 dnsjava-3.6.1.jar
-rw-r--r--@ 1 stevel staff 324655 Sep 17 12:57 dom4j-2.1.4.jar
-rw-r--r--@ 1 stevel staff 670059 Sep 17 12:57
esdk-obs-java-3.20.4.2.jar
-rw-r--r--@ 1 stevel staff 4617 Sep 17 13:11 failureaccess-1.0.1.jar
-rw-r--r--@ 1 stevel staff 249277 Sep 17 13:11 gson-2.9.0.jar
-rw-r--r--@ 1 stevel staff 3037368 Sep 17 13:11 guava-32.0.1-jre.jar
-rw-r--r--@ 1 stevel staff 94013 Sep 17 12:57
hadoop-aliyun-3.5.0-20250916.124028-685.jar
-rw-r--r--@ 1 stevel staff 14456 Sep 17 13:11
hadoop-annotations-3.5.0-SNAPSHOT.jar
-rw-r--r--@ 1 stevel staff 114335 Sep 17 13:11
hadoop-auth-3.5.0-SNAPSHOT.jar
-rw-r--r--@ 1 stevel staff 930516 Sep 17 12:57
hadoop-aws-3.5.0-20250916.124028-686.jar
-rw-r--r--@ 1 stevel staff 827349 Sep 17 12:57
hadoop-azure-3.5.0-20250916.124028-685.jar
-rw-r--r--@ 1 stevel staff 33363 Sep 17 12:57
hadoop-azure-datalake-3.5.0-20250916.124028-685.jar
-rw-r--r--@ 1 stevel staff 70007 Sep 17 12:57
hadoop-cos-3.5.0-20250916.124028-683.jar
-rw-r--r--@ 1 stevel staff 138447 Sep 17 12:57
hadoop-gcp-3.5.0-SNAPSHOT.jar
-rw-r--r--@ 1 stevel staff 142274 Sep 17 12:57
hadoop-huaweicloud-3.5.0-SNAPSHOT.jar
-rw-r--r--@ 1 stevel staff 3519516 Sep 17 13:11
hadoop-shaded-guava-1.4.0.jar
-rw-r--r--@ 1 stevel staff 1952967 Sep 17 13:11
hadoop-shaded-protobuf_3_25-1.4.0.jar
-rw-r--r--@ 1 stevel staff 4019589 Sep 17 12:57
hadoop-tos-3.5.0-20250916.124028-202.jar
-rw-r--r--@ 1 stevel staff 200223 Sep 17 13:11 hk2-api-2.6.1.jar
-rw-r--r--@ 1 stevel staff 203358 Sep 17 13:11 hk2-locator-2.6.1.jar
-rw-r--r--@ 1 stevel staff 131590 Sep 17 13:11 hk2-utils-2.6.1.jar
-rw-r--r--@ 1 stevel staff 780321 Sep 17 13:11 httpclient-4.5.13.jar
-rw-r--r--@ 1 stevel staff 328593 Sep 17 13:11 httpcore-4.4.13.jar
-rw-r--r--@ 1 stevel staff 102220 Sep 17 12:57 ini4j-0.5.4.jar
-rw-r--r--@ 1 stevel staff 29807 Sep 17 13:11
istack-commons-runtime-3.0.12.jar
-rw-r--r--@ 1 stevel staff 9301 Sep 17 13:11
j2objc-annotations-2.8.jar
-rw-r--r--@ 1 stevel staff 76636 Sep 17 13:11
jackson-annotations-2.14.3.jar
-rw-r--r--@ 1 stevel staff 473081 Sep 17 13:11 jackson-core-2.14.3.jar
-rw-r--r--@ 1 stevel staff 1617187 Sep 17 13:11
jackson-databind-2.14.3.jar
-rw-r--r--@ 1 stevel staff 68453 Sep 17 13:11
jakarta.activation-1.2.2.jar
-rw-r--r--@ 1 stevel staff 44399 Sep 17 13:11
jakarta.activation-api-1.2.1.jar
-rw-r--r--@ 1 stevel staff 25058 Sep 17 13:11
jakarta.annotation-api-1.3.5.jar
-rw-r--r--@ 1 stevel staff 18140 Sep 17 13:11 jakarta.inject-2.6.1.jar
-rw-r--r--@ 1 stevel staff 82973 Sep 17 13:11
jakarta.servlet-api-4.0.4.jar
-rw-r--r--@ 1 stevel staff 53683 Sep 17 13:11
jakarta.servlet.jsp-api-2.3.6.jar
-rw-r--r--@ 1 stevel staff 91930 Sep 17 13:11
jakarta.validation-api-2.0.2.jar
-rw-r--r--@ 1 stevel staff 140376 Sep 17 13:11
jakarta.ws.rs-api-2.1.6.jar
-rw-r--r--@ 1 stevel staff 115638 Sep 17 13:11
jakarta.xml.bind-api-2.3.3.jar
-rw-r--r--@ 1 stevel staff 7771 Sep 17 12:57
java-trace-api-0.2.11-beta.jar
-rw-r--r--@ 1 stevel staff 18432 Sep 17 12:57 java-xmlbuilder-1.2.jar
-rw-r--r--@ 1 stevel staff 794714 Sep 17 13:11 javassist-3.30.2-GA.jar
-rw-r--r--@ 1 stevel staff 95806 Sep 17 13:11
javax.servlet-api-3.1.0.jar
-rw-r--r--@ 1 stevel staff 1019097 Sep 17 13:11 jaxb-runtime-2.3.9.jar
-rw-r--r--@ 1 stevel staff 4722 Sep 17 13:11
jcip-annotations-1.0-1.jar
-rw-r--r--@ 1 stevel staff 327806 Sep 17 12:57 jdom2-2.0.6.1.jar
-rw-r--r--@ 1 stevel staff 311826 Sep 17 13:11 jersey-client-2.46.jar
-rw-r--r--@ 1 stevel staff 1267957 Sep 17 13:11 jersey-common-2.46.jar
-rw-r--r--@ 1 stevel staff 32929 Sep 17 13:11
jersey-container-servlet-2.46.jar
-rw-r--r--@ 1 stevel staff 75742 Sep 17 13:11
jersey-container-servlet-core-2.46.jar
-rw-r--r--@ 1 stevel staff 80272 Sep 17 13:11 jersey-hk2-2.46.jar
-rw-r--r--@ 1 stevel staff 964550 Sep 17 13:11 jersey-server-2.46.jar
-rw-r--r--@ 1 stevel staff 90184 Sep 17 12:57 jettison-1.5.4.jar
-rw-r--r--@ 1 stevel staff 249911 Sep 17 13:11
jetty-http-9.4.57.v20241219.jar
-rw-r--r--@ 1 stevel staff 183011 Sep 17 13:11
jetty-io-9.4.57.v20241219.jar
-rw-r--r--@ 1 stevel staff 118496 Sep 17 13:11
jetty-security-9.4.57.v20241219.jar
-rw-r--r--@ 1 stevel staff 739348 Sep 17 13:11
jetty-server-9.4.57.v20241219.jar
-rw-r--r--@ 1 stevel staff 146064 Sep 17 13:11
jetty-servlet-9.4.57.v20241219.jar
-rw-r--r--@ 1 stevel staff 588962 Sep 17 13:11
jetty-util-9.4.57.v20241219.jar
-rw-r--r--@ 1 stevel staff 66643 Sep 17 13:11
jetty-util-ajax-9.4.57.v20241219.jar
-rw-r--r--@ 1 stevel staff 140308 Sep 17 13:11
jetty-webapp-9.4.57.v20241219.jar
-rw-r--r--@ 1 stevel staff 68894 Sep 17 13:11
jetty-xml-9.4.57.v20241219.jar
-rw-r--r--@ 1 stevel staff 282591 Sep 17 13:11 jsch-0.1.55.jar
-rw-r--r--@ 1 stevel staff 19936 Sep 17 13:11 jsr305-3.0.2.jar
-rw-r--r--@ 1 stevel staff 4519 Sep 17 13:11 jul-to-slf4j-1.7.36.jar
-rw-r--r--@ 1 stevel staff 223129 Sep 17 13:11 kerb-core-2.0.3.jar
-rw-r--r--@ 1 stevel staff 115065 Sep 17 13:11 kerb-crypto-2.0.3.jar
-rw-r--r--@ 1 stevel staff 36361 Sep 17 13:11 kerb-util-2.0.3.jar
-rw-r--r--@ 1 stevel staff 100095 Sep 17 13:11 kerby-asn1-2.0.3.jar
-rw-r--r--@ 1 stevel staff 30190 Sep 17 13:11 kerby-config-2.0.3.jar
-rw-r--r--@ 1 stevel staff 200581 Sep 17 13:11 kerby-pkix-2.0.3.jar
-rw-r--r--@ 1 stevel staff 40787 Sep 17 13:11 kerby-util-2.0.3.jar
-rw-r--r--@ 1 stevel staff 2199 Sep 17 13:11
listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
-rw-r--r--@ 1 stevel staff 136314 Sep 17 13:11 metrics-core-3.2.4.jar
-rw-r--r--@ 1 stevel staff 4554 Sep 17 13:11
netty-all-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 339045 Sep 17 13:11
netty-buffer-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 355199 Sep 17 13:11
netty-codec-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 67192 Sep 17 13:11
netty-codec-dns-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 37789 Sep 17 13:11
netty-codec-haproxy-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 674362 Sep 17 13:11
netty-codec-http-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 490985 Sep 17 13:11
netty-codec-http2-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 44736 Sep 17 13:11
netty-codec-memcache-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 113699 Sep 17 13:11
netty-codec-mqtt-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 46015 Sep 17 13:11
netty-codec-redis-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 21344 Sep 17 13:11
netty-codec-smtp-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 121032 Sep 17 13:11
netty-codec-socks-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 34636 Sep 17 13:11
netty-codec-stomp-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 19823 Sep 17 13:11
netty-codec-xml-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 719225 Sep 17 13:11
netty-common-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 580162 Sep 17 13:11
netty-handler-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 25650 Sep 17 13:11
netty-handler-proxy-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 26833 Sep 17 13:11
netty-handler-ssl-ocsp-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 37842 Sep 17 13:11
netty-resolver-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 188360 Sep 17 13:11
netty-resolver-dns-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 9145 Sep 17 13:11
netty-resolver-dns-classes-macos-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 19825 Sep 17 13:11
netty-resolver-dns-native-macos-4.1.118.Final-osx-aarch_64.jar
-rw-r--r--@ 1 stevel staff 19629 Sep 17 13:11
netty-resolver-dns-native-macos-4.1.118.Final-osx-x86_64.jar
-rw-r--r--@ 1 stevel staff 521428 Sep 17 13:11
netty-transport-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 147621 Sep 17 13:11
netty-transport-classes-epoll-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 108558 Sep 17 13:11
netty-transport-classes-kqueue-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 42321 Sep 17 13:11
netty-transport-native-epoll-4.1.118.Final-linux-aarch_64.jar
-rw-r--r--@ 1 stevel staff 36594 Sep 17 13:11
netty-transport-native-epoll-4.1.118.Final-linux-riscv64.jar
-rw-r--r--@ 1 stevel staff 40644 Sep 17 13:11
netty-transport-native-epoll-4.1.118.Final-linux-x86_64.jar
-rw-r--r--@ 1 stevel staff 6193 Sep 17 13:11
netty-transport-native-epoll-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 25741 Sep 17 13:11
netty-transport-native-kqueue-4.1.118.Final-osx-aarch_64.jar
-rw-r--r--@ 1 stevel staff 25170 Sep 17 13:11
netty-transport-native-kqueue-4.1.118.Final-osx-x86_64.jar
-rw-r--r--@ 1 stevel staff 44157 Sep 17 13:11
netty-transport-native-unix-common-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 18241 Sep 17 13:11
netty-transport-rxtx-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 50814 Sep 17 13:11
netty-transport-sctp-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 32189 Sep 17 13:11
netty-transport-udt-4.1.118.Final.jar
-rw-r--r--@ 1 stevel staff 779369 Sep 17 13:11
nimbus-jose-jwt-9.37.2.jar
-rw-r--r--@ 1 stevel staff 425763 Sep 17 12:57 okhttp-3.14.2.jar
-rw-r--r--@ 1 stevel staff 91980 Sep 17 12:57 okio-1.17.2.jar
-rw-r--r--@ 1 stevel staff 141734 Sep 17 12:57
opentelemetry-api-1.38.0.jar
-rw-r--r--@ 1 stevel staff 47252 Sep 17 12:57
opentelemetry-context-1.38.0.jar
-rw-r--r--@ 1 stevel staff 18189 Sep 17 12:57
opentracing-api-0.33.0.jar
-rw-r--r--@ 1 stevel staff 10542 Sep 17 12:57
opentracing-noop-0.33.0.jar
-rw-r--r--@ 1 stevel staff 7504 Sep 17 12:57
opentracing-util-0.33.0.jar
-rw-r--r--@ 1 stevel staff 281989 Sep 17 12:57
org.jacoco.agent-0.8.5-runtime.jar
-rw-r--r--@ 1 stevel staff 19479 Sep 17 13:11
osgi-resource-locator-1.0.3.jar
-rw-r--r--@ 1 stevel staff 128414 Sep 17 13:11 re2j-1.1.jar
-rw-r--r--@ 1 stevel staff 11369 Sep 17 12:57
reactive-streams-1.0.3.jar
-rw-r--r--@ 1 stevel staff 332398 Sep 17 13:11 reload4j-1.2.22.jar
-rw-r--r--@ 1 stevel staff 41125 Sep 17 13:11 slf4j-api-1.7.36.jar
-rw-r--r--@ 1 stevel staff 9824 Sep 17 13:11 slf4j-reload4j-1.7.36.jar
-rw-r--r--@ 1 stevel staff 2112099 Sep 17 13:11 snappy-java-1.1.10.4.jar
-rw-r--r--@ 1 stevel staff 195909 Sep 17 13:11 stax2-api-4.2.1.jar
-rw-r--r--@ 1 stevel staff 72007 Sep 17 13:11 txw2-2.3.9.jar
-rw-r--r--@ 1 stevel staff 443788 Sep 17 12:57
wildfly-openssl-2.1.4.Final.jar
-rw-r--r--@ 1 stevel staff 522679 Sep 17 13:11 woodstox-core-5.4.0.jar
-rw-r--r--@ 1 stevel staff 1323991 Sep 17 13:11 zookeeper-3.8.4.jar
-rw-r--r--@ 1 stevel staff 254932 Sep 17 13:11 zookeeper-jute-3.8.4.jar
```
> hadoop binary distribution to move cloud connectors to hadoop common/lib
> ------------------------------------------------------------------------
>
> Key: HADOOP-19696
> URL: https://issues.apache.org/jira/browse/HADOOP-19696
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, fs/gcs, fs/huawei, fs/s3
> Affects Versions: 3.4.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Place all the cloud connector hadoop-* artifacts and dependencies into
> hadoop/common/lib so that the stores can be directly accessed.
> * filesystem operations against abfs, s3a, gcs, etc don't need any effort
> setting things up.
> * Releases without the aws bundle.jar can be trivially updated by adding any
> version of the sdk libraries to the common/lib dir.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
