[
https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996548#comment-15996548
]
Steve Loughran commented on HADOOP-13887:
-----------------------------------------
If you look at HDFS-11644 we're discussing how to make the capabilities of a
stream discoverable on demand.
but actually the real issue is more fundamental. I expect to be able to get the
length of a file
{code}
status = fs.getFileStatus(path)
{code}
create a buffer from it
{code}
buffer = new byte[status.getLen()]
{code}
and then read that in
{code}
s = fs.open(path)
s.readFully(0, buffer)
{code}
(or do the same in a for() loop)
That runs through a lot of the code: the length of the file is used to
determine the followon actions, rather than just read() until a -1 is returned.
I don't really know what we can do here to address the mismatch, except in the
special cases in the code where we can look at it and see if we can address the
situation "the file is shorter than we thought". I'd look at distcp here,
because at a quick scan, it may fail on the mismatch, and its the foundational
one you could use to bootstrap: copy encrypted data down locally, work on it,
push things back later.
I don't know enough about the HDFS crypto stuff to see how that would link in.
I'd suggest you subscribe to the hdfs-dev mailing list and start the topic of
conversation there.
> Support for client-side encryption in S3A file system
> -----------------------------------------------------
>
> Key: HADOOP-13887
> URL: https://issues.apache.org/jira/browse/HADOOP-13887
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Jeeyoung Kim
> Assignee: Igor Mazur
> Priority: Minor
> Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch,
> HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch,
> HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch,
> HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch,
> HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch,
> HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch,
> HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch
>
>
> Expose the client-side encryption option documented in Amazon S3
> documentation -
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html
> Currently this is not exposed in Hadoop but it is exposed as an option in AWS
> Java SDK, which Hadoop currently includes. It should be trivial to propagate
> this as a parameter passed to the S3client used in S3AFileSystem.java
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
