[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565100#comment-16565100
]
Ewan Higgs commented on HADOOP-15576:
-------------------------------------
I wonder if I've misconfigured the test because they actually run 3 times for
me:
{code:java}
$ mvn install -DskipShade -Dmaven.javadoc.skip=true -Pdist,parallel-tests
-DtestsThreadCount=8 -Djava.awt.headless=true -DS3Guard -DAuth
-Dit.test=ITestS3AContractMultipartUploader -pl :hadoop-aws -Dtest=wibble
{code}
{code:java}
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test
(default-integration-test) @ hadoop-aws ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.541 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test
(sequential-integration-tests) @ hadoop-aws ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.987 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test (default) @ hadoop-aws
---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.966 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO]
{code}
{quote}tests take a v long time.{quote}
Out of interest, how long do they take for you?
{quote}
I'm going to do a revision of the patch, so I'll see if I can identify the
problem. I'll do what I can about scaling the patch down to be viable, and make
it a -Dscale test only.
{quote}
Thanks
> S3A Multipart Uploader to work with S3Guard and encryption
> -----------------------------------------------------------
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2
> Reporter: Steve Loughran
> Assignee: Ewan Higgs
> Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
