[
https://issues.apache.org/jira/browse/HADOOP-19384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17952297#comment-17952297
]
Venkatasubrahmanian Narayanan commented on HADOOP-19384:
--------------------------------------------------------
[[email protected]] I ran the Hadoop itests after cherrypicking to branch-3.4,
and the only errors are a timeout (
ITestConnectionTimeouts.testObjectUploadTimeouts
) (similar to what we've seen earlier), and a couple of failures in
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAfterNormalRead
(stacktrace below) . I ran the itests with the current trunk and I see the same
behavior. Is the testVectoredReadAfterNormalRead failure something that was
introduced by a different, more recent, change?
PR: [https://github.com/apache/hadoop/pull/7690]
Stacktrace:
{code:java}
[ERROR]
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAfterNormalRead
Time elapsed: 3.259 s <<< ERROR!
java.io.IOException: Cannot read block file
at
software.amazon.s3.analyticsaccelerator.io.physical.data.Block.getDataWithRetries(Block.java:296)
at
software.amazon.s3.analyticsaccelerator.io.physical.data.Block.read(Block.java:244)
at
software.amazon.s3.analyticsaccelerator.io.physical.data.Blob.read(Blob.java:112)
at
software.amazon.s3.analyticsaccelerator.io.physical.impl.PhysicalIOImpl.lambda$read$3(PhysicalIOImpl.java:162)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measure(DefaultTelemetry.java:102)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.Telemetry.measureVerbose(Telemetry.java:253)
at
software.amazon.s3.analyticsaccelerator.io.physical.impl.PhysicalIOImpl.read(PhysicalIOImpl.java:151)
at
software.amazon.s3.analyticsaccelerator.io.logical.impl.DefaultLogicalIOImpl.lambda$read$1(DefaultLogicalIOImpl.java:92)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measureConditionally(DefaultTelemetry.java:141)
at
software.amazon.s3.analyticsaccelerator.io.logical.impl.DefaultLogicalIOImpl.read(DefaultLogicalIOImpl.java:81)
at
software.amazon.s3.analyticsaccelerator.S3SeekableInputStream.lambda$read$3(S3SeekableInputStream.java:155)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measure(DefaultTelemetry.java:102)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.Telemetry.measureVerbose(Telemetry.java:253)
at
software.amazon.s3.analyticsaccelerator.S3SeekableInputStream.read(S3SeekableInputStream.java:145)
at
org.apache.hadoop.fs.s3a.impl.streams.AnalyticsStream.read(AnalyticsStream.java:123)
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:200)
at
org.apache.hadoop.fs.contract.AbstractContractVectoredReadTest.testVectoredReadAfterNormalRead(AbstractContractVectoredReadTest.java:490)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: At least one
of the pre-conditions you specified did not hold (Service: S3, Status Code:
412, Request ID: KS1PFC35DGJBNDKC, Extended Request ID:
fE+vsi9+3saUIGQQR86oaZqNajrUSYV3WwEMADn2rB3Bl70LgI68Iw7RNMt8JTWAaLT+yEmboCOGshMejg8IvTknw0kSvc/pYTwv/ndiHdw=)
at
software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104)
at
software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58)
at
software.amazon.awssdk.protocols.query.internal.unmarshall.AwsXmlErrorUnmarshaller.unmarshall(AwsXmlErrorUnmarshaller.java:98)
at
software.amazon.awssdk.protocols.query.unmarshall.AwsXmlErrorProtocolUnmarshaller.handle(AwsXmlErrorProtocolUnmarshaller.java:102)
at
software.amazon.awssdk.protocols.query.unmarshall.AwsXmlErrorProtocolUnmarshaller.handle(AwsXmlErrorProtocolUnmarshaller.java:82)
at
software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.lambda$handle$0(MetricCollectingHttpResponseHandler.java:52)
at
software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
at
software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:95)
at
software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.handle(MetricCollectingHttpResponseHandler.java:52)
at
software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2079)
at
software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
at
software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
at
software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$DataCountingPublisher$1.onComplete(ResponseHandler.java:519)
at
software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:254)
at
software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$600(ResponseHandler.java:77)
at
software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:375)
at
software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.publishMessage(HandlerPublisher.java:402)
at
software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.flushBuffer(HandlerPublisher.java:338)
at
software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.receivedDemand(HandlerPublisher.java:291)
at
software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.access$200(HandlerPublisher.java:61)
at
software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495)
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at
software.amazon.awssdk.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at
software.amazon.awssdk.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
... 1 more{code}
> Add support for ProfileCredentialsProvider
> ------------------------------------------
>
> Key: HADOOP-19384
> URL: https://issues.apache.org/jira/browse/HADOOP-19384
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.4.1
> Reporter: Venkatasubrahmanian Narayanan
> Assignee: Venkatasubrahmanian Narayanan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> Hadoop currently doesn't support AWS' ProfileCredentialsProvider. A thin
> wrapper is sufficient to get it to work, since it just needs us to fetch the
> credentials file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
