[
https://issues.apache.org/jira/browse/HADOOP-19654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18030433#comment-18030433
]
ASF GitHub Bot commented on HADOOP-19654:
-----------------------------------------
steveloughran commented on PR #7882:
URL: https://github.com/apache/hadoop/pull/7882#issuecomment-3412377933
Now that 3rd party is good, I'm getting S3 express happy, mainly by test
tuning. But many, many errors with vectored reads
```
[ERROR] Errors:
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testAllRangesMergedIntoOne
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testAllRangesMergedIntoOne » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testAllRangesMergedIntoOne » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testBufferSlicing
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testBufferSlicing » IO Server error
accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testBufferSlicing » IO Server error
accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testConsecutiveRanges
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testConsecutiveRanges » IO Server
error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testConsecutiveRanges » IO Server
error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testMultipleVectoredReads
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testMultipleVectoredReads » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testMultipleVectoredReads » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testNormalReadAfterVectoredRead
[INFO] Run 1: PASS
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testNormalReadAfterVectoredRead »
IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testReadVectoredWithAALStatsCollection
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testReadVectoredWithAALStatsCollection
» IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testReadVectoredWithAALStatsCollection
» IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testSomeRandomNonOverlappingRanges
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testSomeRandomNonOverlappingRanges
» IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testSomeRandomNonOverlappingRanges
» IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAfterNormalRead
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAfterNormalRead »
IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAfterNormalRead »
IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAndReadFully
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadAndReadFully:220
» IO test/vectored_file.txt: Stream is closed!
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadAndReadFully:220
» IO test/vectored_file.txt: Stream is closed!
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadMultipleRanges
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadMultipleRanges:206
» Execution java.io.IOException: Server error accessing
s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadMultipleRanges:206
» Execution java.io.IOException: Server error accessing
s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadWholeFile
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadWholeFile » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadWholeFile » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[INFO]
```
```
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testReadVectoredWithAALStatsCollection
> Upgrade AWS SDK to 2.33.x
> -------------------------
>
> Key: HADOOP-19654
> URL: https://issues.apache.org/jira/browse/HADOOP-19654
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, fs/s3
> Affects Versions: 3.5.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Upgrade to a recent version of 2.33.x or later while off the critical path of
> things.
> HADOOP-19485 froze the sdk at a version which worked with third party stores.
> Apparently the new version works; early tests show that Bulk Delete calls
> with third party stores complain about lack of md5 headers, so some tuning is
> clearly going to be needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
