[
https://issues.apache.org/jira/browse/HADOOP-19733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18056965#comment-18056965
]
ASF GitHub Bot commented on HADOOP-19733:
-----------------------------------------
brandonvin commented on PR #8048:
URL: https://github.com/apache/hadoop/pull/8048#issuecomment-3861934530
@steveloughran, There's a bit more implementation work to ensure the custom
signer loading uses the right classloader. I've started that work, but not
quite finished.
I've also been trying to run the S3A test suite and have had trouble with
that.
It's pretty close. I'll try to find some time to work on this soon.
> S3A: Credentials provider classes not found despite setting
> `fs.s3a.classloader.isolation` to `false`
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-19733
> URL: https://issues.apache.org/jira/browse/HADOOP-19733
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.4.2
> Reporter: Brandon
> Assignee: Brandon
> Priority: Minor
> Labels: pull-request-available
>
> HADOOP-18993 added the option `fs.s3a.classloader.isolation` to support, for
> example, a Spark job using an AWS credentials provider class that is bundled
> into the Spark job JAR. In testing this, the AWS credentials provider classes
> are still not found.
> I think the cause is:
> * `fs.s3a.classloader.isolation` is implemented by setting (or not setting)
> a classloader on the `Configuration`
> * However, code paths to load AWS credential provider call
> `S3AUtils.getInstanceFromReflection`, which uses the classloader that loaded
> the S3AUtils class. That's likely to be the built-in application classloader,
> which won't be able to load classes in a Spark job JAR.
> And the fix seems small:
> * Change `S3AUtils.getInstanceFromReflection` to load classes using the
> `Configuration`'s classloader. Luckily we already have the Configuration in
> this method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
