steveloughran commented on PR #8094: URL: https://github.com/apache/hadoop/pull/8094#issuecomment-3580952919
@pan3793 happy to explain, and happy to hear your concerns. the build is set to keep out all the new transitive dependencies as much for security reasons as size: fewer dependencies, fewer dependabot alerts about CVEs, fewer jars to keep updating. but the hadoop-gcs and hadoop-cos (or is it hadoop-tos? it's 3.5+ only I think) both build shaded releases with all their dependencies in their JARs if you do a -Pdist build, which of course ASF releases do. This makes for bigger artifacts but not so big that they create a distribution problem...and that shading makes direct use of the fs through an import of the jar or hadoop-cloud-storage pom easier. Changing that packaging to not do the shading adds many, many more libraries to hadoop common (all enumerated and listed in LICENSE-binary), and would complicate use through maven declarations: classpath hell. Although I don't use Aliyun, tos, cos, volcano connectors myself, I don't want to do anything to stop them being used, and with these changes make it easy for people to build a hadoop-release with out-the-box support for them. FWIW, in cloudera we * keep hadoop-aws and hadoop-azure in tools/lib * and add google gcs (not the one in trunk, the original) * plus some associated auth/permissions * somehow add them to the classpath for FS commands * *and* strip out the hadoop-cos artifact as its sdk caused problems talking to newer s3 regions due to an outdated `mozilla/public-suffix-list.txt` resource confusing aws sdk about what were toplevel domains. Good bug to track down :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
