[
https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526478#comment-15526478
]
Steve Loughran commented on HADOOP-13560:
-----------------------------------------
Patch 006; commit 88e443; scale test runs, with one NPE in logging caught.
* fix an NPE in a debug log statement for close-with-no-active block;
* remove unimplemented (and hard to implement meaningfully) bandwidth gauge;
* diff against branch-2 to reduce delta as much as possible (IDE import changes)
Lots of work on improving many-GB test runs, including:
* configurable test timeout in maven
* preflight validation oftest timeout in big files (and a suggestion of a new
timeout size to use); This stops a test being killed after an hour because a
timeout was too low in maven or in the test case. Validation assumes 1MB/s
bandwidth,
* bandwidth stats printed on intermediate writes and on upload callbacks, so
helping differentiate buffer write and upload speeds, and give someone logging
the files something interesting to look at. This becomes quite useful when you
are doing a many-GB test and what a bit of a clue what is going on
{code}
2016-09-27 16:17:25,592 [JUnit] INFO scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:test_010_CreateHugeFile(181)) - [50%] Buffered
10.00 MB out of 20 MB; PUT 0 bytes (8388608 pending) in 1 operations (1
active); elapsedTime=0.86s; write to buffer bandwidth=11.58 MB/s
2016-09-27 16:17:25,592 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
TRANSFER_PART_STARTED_EVENT, bytes: 0
2016-09-27 16:17:25,593 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
CLIENT_REQUEST_STARTED_EVENT, bytes: 0
2016-09-27 16:17:25,593 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_REQUEST_STARTED_EVENT, bytes: 0
2016-09-27 16:17:25,613 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
TRANSFER_PART_STARTED_EVENT, bytes: 0
2016-09-27 16:17:25,613 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
CLIENT_REQUEST_STARTED_EVENT, bytes: 0
2016-09-27 16:17:25,614 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_REQUEST_STARTED_EVENT, bytes: 0
2016-09-27 16:17:25,616 [JUnit] INFO scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:test_010_CreateHugeFile(181)) - [100%] Buffered
20.00 MB out of 20 MB; PUT 0 bytes (16777216 pending) in 2 operations (2
active); elapsedTime=0.89s; write to buffer bandwidth=22.48 MB/s
2016-09-27 16:17:25,617 [JUnit] INFO scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:test_010_CreateHugeFile(196)) - Closing file
and completing write operation
2016-09-27 16:17:25,627 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
TRANSFER_PART_STARTED_EVENT, bytes: 0
2016-09-27 16:17:25,627 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
CLIENT_REQUEST_STARTED_EVENT, bytes: 0
2016-09-27 16:17:25,627 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_REQUEST_STARTED_EVENT, bytes: 0
2016-09-27 16:17:35,552 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
2016-09-27 16:17:35,553 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_RESPONSE_STARTED_EVENT, bytes: 0
2016-09-27 16:17:35,553 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
2016-09-27 16:17:35,553 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
2016-09-27 16:17:35,553 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(260)) - Event
TRANSFER_PART_COMPLETED_EVENT, bytes: 0; total uploaded=12 MB in 10.8s;
effective upload bandwidth = 1.11 MB/s
2016-09-27 16:17:41,427 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
2016-09-27 16:17:41,427 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_RESPONSE_STARTED_EVENT, bytes: 0
2016-09-27 16:17:41,427 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
2016-09-27 16:17:41,427 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
2016-09-27 16:17:41,428 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(260)) - Event
TRANSFER_PART_COMPLETED_EVENT, bytes: 0; total uploaded=20 MB in 16.7s;
effective upload bandwidth = 1.20 MB/s
2016-09-27 16:17:41,438 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
2016-09-27 16:17:41,438 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_RESPONSE_STARTED_EVENT, bytes: 0
2016-09-27 16:17:41,438 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
2016-09-27 16:17:41,438 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(270)) - Event
CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
2016-09-27 16:17:41,439 [java-sdk-progress-listener-callback-thread] INFO
scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:progressChanged(260)) - Event
TRANSFER_PART_COMPLETED_EVENT, bytes: 0; total uploaded=20 MB in 16.7s;
effective upload bandwidth = 1.20 MB/s
2016-09-27 16:17:41,919 [JUnit] INFO contract.ContractTestUtils
(ContractTestUtils.java:end(1365)) - Duration of time to close() output stream:
16,302,440,012 nS
2016-09-27 16:17:41,919 [JUnit] INFO contract.ContractTestUtils
(ContractTestUtils.java:end(1365)) - Duration of time to write 20 MB in blocks
of 65536: 17,193,443,672 nS
2016-09-27 16:17:41,920 [JUnit] INFO scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:test_010_CreateHugeFile(206)) - Time per MB to
write = 859,672,183 nS
2016-09-27 16:17:41,921 [JUnit] INFO contract.ContractTestUtils
(ContractTestUtils.java:bandwidth(917)) - Bandwidth = 1.163234 MB/S
2016-09-27 16:17:41,921 [JUnit] INFO scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:test_010_CreateHugeFile(211)) - PUT 20971520
bytes in 3 operations; 6 MB/operation
2016-09-27 16:17:41,921 [JUnit] INFO scale.AbstractSTestS3AHugeFiles
(AbstractSTestS3AHugeFiles.java:test_010_CreateHugeFile(214)) - Time per PUT
5,731,147,890 nS
2016-09-27 16:17:42,303 [JUnit] INFO scale.S3AScaleTestBase
(S3AScaleTestBase.java:describe(161)) -
{code}
Tests: s3 ireland; using test files of 4GB. Saw a couple of socket timeouts
reading the data back in; S3a had logged and retried. Created HADOOP-13664 to
consider using a Hadoop RetryPolicy here.
> S3ABlockOutputStream to support huge (many GB) file writes
> ----------------------------------------------------------
>
> Key: HADOOP-13560
> URL: https://issues.apache.org/jira/browse/HADOOP-13560
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-13560-branch-2-001.patch,
> HADOOP-13560-branch-2-002.patch, HADOOP-13560-branch-2-003.patch,
> HADOOP-13560-branch-2-004.patch
>
>
> An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights
> that metadata isn't copied on large copies.
> 1. Add a test to do that large copy/rname and verify that the copy really
> works
> 2. Verify that metadata makes it over.
> Verifying large file rename is important on its own, as it is needed for very
> large commit operations for committers using rename
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
