[
https://issues.apache.org/jira/browse/HADOOP-19654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18030429#comment-18030429
]
ASF GitHub Bot commented on HADOOP-19654:
-----------------------------------------
steveloughran commented on PR #7882:
URL: https://github.com/apache/hadoop/pull/7882#issuecomment-3412290514
@ahmarsuhail
I'm handling the retries now by requiring the md5 plugin to be explicitly
requested (i.e. third party stores); also making it easier to switch checksum
generation from ALWAYS to WHEN_REQUESTED. So for AWS S3: stricter checksums, no
md5. Other stores: configure it as needed.
Still wondering if we should make this more automated, but not in a way
which causes problems later.
---
I am now seeing failings against s3 express
```
org.opentest4j.AssertionFailedError: [Counter named audit_request_execution
with expected value 4]
Expecting:
<11L>
to be equal to:
<4L>
but was not.
Expected :4
Actual :11
<Click to see difference>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
org.apache.hadoop.fs.statistics.IOStatisticAssertions.verifyStatisticValue(IOStatisticAssertions.java:274)
at
org.apache.hadoop.fs.statistics.IOStatisticAssertions.verifyStatisticCounterValue(IOStatisticAssertions.java:175)
at
org.apache.hadoop.fs.s3a.ITestS3AAnalyticsAcceleratorStreamReading.testMultiRowGroupParquet(ITestS3AAnalyticsAcceleratorStreamReading.java:186)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at java.util.ArrayList.forEach(ArrayList.java:1259)
```
I'm changing this test to measure the # of audited requests before the file
opening begins and then assert on the difference between them.
> Upgrade AWS SDK to 2.33.x
> -------------------------
>
> Key: HADOOP-19654
> URL: https://issues.apache.org/jira/browse/HADOOP-19654
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, fs/s3
> Affects Versions: 3.5.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Upgrade to a recent version of 2.33.x or later while off the critical path of
> things.
> HADOOP-19485 froze the sdk at a version which worked with third party stores.
> Apparently the new version works; early tests show that Bulk Delete calls
> with third party stores complain about lack of md5 headers, so some tuning is
> clearly going to be needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
