[
https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110202#comment-17110202
]
Steve Loughran edited comment on HADOOP-13887 at 5/18/20, 12:17 PM:
--------------------------------------------------------------------
S3 now supports unpadded CSE, so client side encryption will be safe to use.
This is what EMR supports
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-emrfs-encryption-cse.html
if someone wants to add support for that, and that only, things will work
This will have to work with Delegation Tokens. The enum of encryption options
to serialize has always had options for CSE_KMS and CSE_CUSTOM, so the wire
format will not change at all. the patch will need to
* get the current secret/kms key into
org.apache.hadoop.fs.s3a.auth.delegation.EncryptionSecrets
* extend DT tests to verify it will marshall/unmarshall such that an FS
instance will have the same options
and equally critically; file length is always that of the data you can read,
both from HEAD and list requests, seek() to declared EOF -1 and read() works,
especially for an unguarded store. do this with varying file lengths to ensure
that success isn't just because the length of the test file is such that no
padding is needed
Best way to verify the tests are good would be start with padded encryption,
verify failure, switch to unpadded and expect the faiures to go away.
was (Author: [email protected]):
S3 now supports unpadded CSE, so client side encryption will be safe to use.
This is what e
> Encrypt S3A data client-side with AWS SDK (S3-CSE)
> --------------------------------------------------
>
> Key: HADOOP-13887
> URL: https://issues.apache.org/jira/browse/HADOOP-13887
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Jeeyoung Kim
> Assignee: Igor Mazur
> Priority: Minor
> Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch,
> HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch,
> HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch,
> HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch,
> HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch,
> HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch,
> HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf
>
>
> Expose the client-side encryption option documented in Amazon S3
> documentation -
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html
> Currently this is not exposed in Hadoop but it is exposed as an option in AWS
> Java SDK, which Hadoop currently includes. It should be trivial to propagate
> this as a parameter passed to the S3client used in S3AFileSystem.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
