[
https://issues.apache.org/jira/browse/HADOOP-18752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729218#comment-17729218
]
ASF GitHub Bot commented on HADOOP-18752:
-----------------------------------------
dannycjones commented on code in PR #5689:
URL: https://github.com/apache/hadoop/pull/5689#discussion_r1217662352
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/directory_markers.md:
##########
@@ -12,35 +12,40 @@
limitations under the License. See accompanying LICENSE file.
-->
-# Experimental: Controlling the S3A Directory Marker Behavior
+# Controlling the S3A Directory Marker Behavior
-This document discusses an experimental feature of the S3A
-connector since Hadoop 3.3.1: the ability to retain directory
-marker objects above paths containing files or subdirectories.
+This document discusses an performance feature of the S3A
+connector: directory markers are not deleted unless the
+client is explicitly configured to do so.
## <a name="compatibility"></a> Critical: this is not backwards compatible!
This document shows how the performance of S3 I/O, especially applications
creating many files (for example Apache Hive) or working with versioned S3
buckets can
increase performance by changing the S3A directory marker retention policy.
-Changing the policy from the default value, `"delete"` _is not backwards
compatible_.
+The default policy in this release of hadoop is "keep",
+which _is not backwards compatible_ with hadoop versions
+released before 2021.
-Versions of Hadoop which are incompatible with other marker retention policies,
-as of August 2020.
+The compatibility table of older releases is as follows:
-| Branch | Compatible Since | Supported |
-|------------|------------------|---------------------|
-| Hadoop 2.x | n/a | WONTFIX |
-| Hadoop 3.0 | check | Read-only |
-| Hadoop 3.1 | check | Read-only |
-| Hadoop 3.2 | check | Read-only |
-| Hadoop 3.3 | 3.3.1 | Done |
+| Branch | Compatible Since | Supported | Released |
+|------------|------------------|-----------|----------|
+| Hadoop 2.x | 2.10.2 | Read-only | 05/2022 |
+| Hadoop 3.0 | n/a | WONTFIX | |
+| Hadoop 3.1 | n/a | WONTFIX | |
+| Hadoop 3.2 | 3.2.2 | Read-only | 01/2022 |
+| Hadoop 3.3 | 3.3.1 | Done | 01/2021 |
Review Comment:
nice, that's great then
> Change fs.s3a.directory.marker.retention to "keep"
> --------------------------------------------------
>
> Key: HADOOP-18752
> URL: https://issues.apache.org/jira/browse/HADOOP-18752
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.5
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Change the default value of "fs.s3a.directory.marker.retention" to keep;
> update docs to match.
> maybe include with HADOOP-17802 so we don't blow up with fewer markers being
> created.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
