[
https://issues.apache.org/jira/browse/HADOOP-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130472#comment-14130472
]
Steve Loughran commented on HADOOP-10400:
-----------------------------------------
-1: no tests. Sorry -but I don't want that test coverage put off as a "later".
h2. Tests
That's {{FSContractTestBase}} and the new {{AbstractFSContract}} stuff. The
tests are there, its a matter of subclassing, running the tests and then fixing
where it breaks.
WRT to tests that would fail if they were part of the patch
# fix seek() to throw an EOFException on a negative parameter
# I don't think rename() works in all corner cases. Just a guess from time
spent understanding the depths of rename and writing tests for it.
h2. replace IOExceptions with meaningful subclasses
These are things that the Abstract Contract test will show up if the FS
declares that it supports strict exceptions
# {{FSDataOutputStream create()}} should throw {{FileAlreadyExistsException}}
# {{S3AFileSystem.initialize()}} could throw a stricter IOE on bad bucket,
FileNotFoundException?
# {{open()}} should throw FileAlreadyExistsException...
...etc. Trivial to do & the contract tests will show where they are needed.
h2. other recommended pre-commit changes
Things that could be done after commit, but which are easier to do now
# linelength is too long ... reformat to the hadoop guidelines
# move to SLF4j as the log API
# the hadoop-aws J POMs should be set up so that instead of excluding the new
AWS lib, people only get the hadoop-aws JAR and dependencies if they ask for
it ... it shouldn't be another thing hadoop-client drags in.
Things to not get in to the patch
* {{hadoop-common-project/hadoop-common/src/test/resources/core-site.xml}}
shouldn't have those entries
h2. Longer term,
# there's a lot of old->new double config reads. Do we need this? If so, could
that deprecated config feature used elsewhere in Hadoop be used?
the following things should be lifted from the openstack stuff
# implementing forward seeks just by skipping characters ... this is
significantly faster over long-haul and HTTPS
# collection of stats of HTTP verb & operation performance
> Incorporate new S3A FileSystem implementation
> ---------------------------------------------
>
> Key: HADOOP-10400
> URL: https://issues.apache.org/jira/browse/HADOOP-10400
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3
> Affects Versions: 2.4.0
> Reporter: Jordan Mendelson
> Assignee: Jordan Mendelson
> Attachments: HADOOP-10400-1.patch, HADOOP-10400-2.patch,
> HADOOP-10400-3.patch, HADOOP-10400-4.patch, HADOOP-10400-5.patch,
> HADOOP-10400-6.patch, HADOOP-10400-7.patch, HADOOP-10400-branch-2.patch
>
>
> The s3native filesystem has a number of limitations (some of which were
> recently fixed by HADOOP-9454). This patch adds an s3a filesystem which uses
> the aws-sdk instead of the jets3t library. There are a number of improvements
> over s3native including:
> - Parallel copy (rename) support (dramatically speeds up commits on large
> files)
> - AWS S3 explorer compatible empty directories files "xyz/" instead of
> "xyz_$folder$" (reduces littering)
> - Ignores s3native created _$folder$ files created by s3native and other S3
> browsing utilities
> - Supports multiple output buffer dirs to even out IO when uploading files
> - Supports IAM role-based authentication
> - Allows setting a default canned ACL for uploads (public, private, etc.)
> - Better error recovery handling
> - Should handle input seeks without having to download the whole file (used
> for splits a lot)
> This code is a copy of https://github.com/Aloisius/hadoop-s3a with patches to
> various pom files to get it to build against trunk. I've been using 0.0.1 in
> production with CDH 4 for several months and CDH 5 for a few days. The
> version here is 0.0.2 which changes around some keys to hopefully bring the
> key name style more inline with the rest of hadoop 2.x.
> *Tunable parameters:*
> fs.s3a.access.key - Your AWS access key ID (omit for role authentication)
> fs.s3a.secret.key - Your AWS secret key (omit for role authentication)
> fs.s3a.connection.maximum - Controls how many parallel connections
> HttpClient spawns (default: 15)
> fs.s3a.connection.ssl.enabled - Enables or disables SSL connections to S3
> (default: true)
> fs.s3a.attempts.maximum - How many times we should retry commands on
> transient errors (default: 10)
> fs.s3a.connection.timeout - Socket connect timeout (default: 5000)
> fs.s3a.paging.maximum - How many keys to request from S3 when doing
> directory listings at a time (default: 5000)
> fs.s3a.multipart.size - How big (in bytes) to split a upload or copy
> operation up into (default: 104857600)
> fs.s3a.multipart.threshold - Until a file is this large (in bytes), use
> non-parallel upload (default: 2147483647)
> fs.s3a.acl.default - Set a canned ACL on newly created/copied objects
> (private | public-read | public-read-write | authenticated-read |
> log-delivery-write | bucket-owner-read | bucket-owner-full-control)
> fs.s3a.multipart.purge - True if you want to purge existing multipart
> uploads that may not have been completed/aborted correctly (default: false)
> fs.s3a.multipart.purge.age - Minimum age in seconds of multipart uploads
> to purge (default: 86400)
> fs.s3a.buffer.dir - Comma separated list of directories that will be used
> to buffer file writes out of (default: uses ${hadoop.tmp.dir}/s3a )
> *Caveats*:
> Hadoop uses a standard output committer which uploads files as
> filename.COPYING before renaming them. This can cause unnecessary performance
> issues with S3 because it does not have a rename operation and S3 already
> verifies uploads against an md5 that the driver sets on the upload request.
> While this FileSystem should be significantly faster than the built-in
> s3native driver because of parallel copy support, you may want to consider
> setting a null output committer on our jobs to further improve performance.
> Because S3 requires the file length and MD5 to be known before a file is
> uploaded, all output is buffered out to a temporary file first similar to the
> s3native driver.
> Due to the lack of native rename() for S3, renaming extremely large files or
> directories make take a while. Unfortunately, there is no way to notify
> hadoop that progress is still being made for rename operations, so your job
> may time out unless you increase the task timeout.
> This driver will fully ignore _$folder$ files. This was necessary so that it
> could interoperate with repositories that have had the s3native driver used
> on them, but means that it won't recognize empty directories that s3native
> has been used on.
> Statistics for the filesystem may be calculated differently than the s3native
> filesystem. When uploading a file, we do not count writing the temporary file
> on the local filesystem towards the local filesystem's written bytes count.
> When renaming files, we do not count the S3->S3 copy as read or write
> operations. Unlike the s3native driver, we only count bytes written when we
> start the upload (as opposed to the write calls to the temporary local file).
> The driver also counts read & write ops, but they are done mostly to keep
> from timing out on large s3 operations.
> The AWS SDK unfortunately passes the multipart threshold as an int which means
> fs.s3a.multipart.threshold can not be greater than 2^31-1 (2147483647).
> This is currently implemented as a FileSystem and not a AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
