[
https://issues.apache.org/jira/browse/HDFS-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18069015#comment-18069015
]
ASF GitHub Bot commented on HDFS-17893:
---------------------------------------
dlmarion commented on PR #8348:
URL: https://github.com/apache/hadoop/pull/8348#issuecomment-4143287447
I added a test that causes an NPE in DFSStripedInputStream.useDirectBuffer.
On trunk, the test fails with the following in the log:
```
[main] INFO hdfs.TestDFSStripedInputStreamReadFailures
(TestDFSStripedInputStreamReadFailures.java:testReadWithXceiverExhaustion(164))
- 7 exceptions occurred
[main] ERROR hdfs.TestDFSStripedInputStreamReadFailures
(TestDFSStripedInputStreamReadFailures.java:lambda$testReadWithXceiverExhaustion$3(165))
- Exception details
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.preferDirectBuffer()"
because "this.decoder" is null
at
org.apache.hadoop.hdfs.DFSStripedInputStream.useDirectBuffer(DFSStripedInputStream.java:119)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.resetCurStripeBuffer(DFSStripedInputStream.java:124)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:299)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:421)
at
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:960)
at java.base/java.io.DataInputStream.read(DataInputStream.java:158)
at
org.apache.hadoop.hdfs.StripedFileTestUtil.verifyStatefulRead(StripedFileTestUtil.java:128)
at
org.apache.hadoop.hdfs.TestDFSStripedInputStreamReadFailures.lambda$testReadWithXceiverExhaustion$0(TestDFSStripedInputStreamReadFailures.java:146)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
The other 6 exceptions have the same stack trace. When this test is run on
this branch it still fails, but in a different way which is probably handled
better by a client because an IOException is expected exception. The failure in
this branch looks like:
```
[main] INFO hdfs.TestDFSStripedInputStreamReadFailures
(TestDFSStripedInputStreamReadFailures.java:testReadWithXceiverExhaustion(164))
- 9 exceptions occurred
[main] ERROR hdfs.TestDFSStripedInputStreamReadFailures
(TestDFSStripedInputStreamReadFailures.java:lambda$testReadWithXceiverExhaustion$3(165))
- Exception details
java.io.IOException: 6 missing blocks, the stripe is:
AlignedStripe(Offset=0, length=1048576, fetchedChunksNum=0,
missingChunksNum=6); locatedBlocks is: LocatedBlocks{; fileLength=6291579;
underConstruction=false;
blocks=[LocatedStripedBlock{BP-1063483206-10.113.15.120-1774623121364:blk_-9223372036854775728_1005;
getBlockSize()=6291579; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:38777,DS-5d4bafbc-f78a-4608-b4d7-635c26439ef6,DISK],
DatanodeInfoWithStorage[127.0.0.1:33155,DS-b94250d0-39ca-47b9-ae38-14b0eccc3e2a,DISK],
DatanodeInfoWithStorage[127.0.0.1:33469,DS-0ef39e40-8f52-497f-bcc4-7b74f5359200,DISK],
DatanodeInfoWithStorage[127.0.0.1:40287,DS-346798b6-b6c5-4f85-ba85-84a701ced827,DISK],
DatanodeInfoWithStorage[127.0.0.1:32995,DS-9be75b23-9ac5-4278-a46b-ee4a52f4ca2f,DISK],
DatanodeInfoWithStorage[127.0.0.1:33789,DS-e6ff79de-2b1b-48da-a84d-a8da0fba80ff,DISK],
DatanodeInfoWithStorage[127.0.0.1:46867,DS-dfa3ee1d-0964-4e1d-818d-77bdcae7c4e0,DISK],
DatanodeInfoWithStorage[127.0.0.1:44495,DS-8d4dd57f-c417-4010-9278-253236ea4ff3,DISK],
DatanodeInfoWithStorage[127.0.0.1:34009,DS-1af2db95-5652-4bdf-b6a8-99ef2eea080f,DISK]];
indices=[0, 1, 2, 3, 4, 5, 6, 7, 8]}];
lastLocatedBlock=LocatedStripedBlock{BP-1063483206-10.113.15.120-1774623121364:blk_-9223372036854775728_1005;
getBlockSize()=6291579; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:38777,DS-5d4bafbc-f78a-4608-b4d7-635c26439ef6,DISK],
DatanodeInfoWithStorage[127.0.0.1:33155,DS-b94250d0-39ca-47b9-ae38-14b0eccc3e2a,DISK],
DatanodeInfoWithStorage[127.0.0.1:33469,DS-0ef39e40-8f52-497f-bcc4-7b74f5359200,DISK],
DatanodeInfoWithStorage[127.0.0.1:40287,DS-346798b6-b6c5-4f85-ba85-84a701ced827,DISK],
DatanodeInfoWithStorage[127.0.0.1:32995,DS-9be75b23-9ac5-4278-a46b-ee4a52f4ca2f,DISK],
DatanodeInfoWithStorage[127.0.0.1:33789,DS-e6ff79de-2b1b-48da-a84d-a8da0fba80ff,DISK],
DatanodeInfoWithStorage[127.0.0.1:46867,DS-dfa3ee1d-0964-4e1d-818d-77bdcae7c4e0,DISK],
DatanodeInfoWithStorage[127.0.0.1:44495,DS-8d4dd57f-c417-4010-9278-253236ea4ff3,DISK],
DatanodeInfoWithStorage[127.0.0.1:34009,DS-1af2db95-5652-4bdf-b6a8-99ef2eea080f,DISK]];
indices=[0, 1, 2, 3, 4, 5, 6, 7, 8]}; isLastBlockComplete=true;
ecPolicy=ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs,
numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1]}
at
org.apache.hadoop.hdfs.StripeReader.checkMissingBlocks(StripeReader.java:180)
at
org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:350)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:322)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:421)
at
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:960)
at java.base/java.io.DataInputStream.read(DataInputStream.java:158)
at
org.apache.hadoop.hdfs.StripedFileTestUtil.verifyStatefulRead(StripedFileTestUtil.java:128)
at
org.apache.hadoop.hdfs.TestDFSStripedInputStreamReadFailures.lambda$testReadWithXceiverExhaustion$0(TestDFSStripedInputStreamReadFailures.java:146)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
> NPE in DFSStripedInputStream.readWithStrategy
> ----------------------------------------------
>
> Key: HDFS-17893
> URL: https://issues.apache.org/jira/browse/HDFS-17893
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec
> Affects Versions: 3.5.0
> Reporter: Dave Marion
> Priority: Major
>
> [DFSStripedInputStream.readWithStrategy|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java#L391]
> checks to see if it is closed, then enters a retry loop where it ends up
> calling *readOneStripe* (line 421). *readOneStripe* calls *readStripe* on the
> {*}StatefulStripeReader{*}, which defaults to its parent class
> {*}StripeReader{*}. *StripeReader.readStripe* (line 356) on an exception
> calls close on the *DFSStripedInputStream* which ends up setting *decoder* to
> null.
> *DFSStripedInputStream.readWithStrategy* catches the IOException and retries
> calling *readOneStripe* again, which throws a NullPointerException in the
> call to *resetCurStripeBuffer* because the *decoder* is null when
> *useDirectBuffer* is called.
>
> With the retries that DFSStripedInputStream is doing in readWithStrategy,
> it's likely that either the StripedReader should not be closing the
> DFSStripedInputStream, or the closed check needs to happen again at the top
> of the retry loop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
