This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a change to branch HDDS-10685
in repository https://gitbox.apache.org/repos/asf/ozone.git
omit 15ee286339 HDDS-11622. Support domain socket creation (#7397)
add 47ec4ddf87 HDDS-11671. Refer to website for supported versions (#7412)
add 889ba80766 HDDS-11653. Bump Ratis to 3.1.2 (#7427)
add fd5c6d8cc2 HDDS-11689. Extract scheduled workflow for populate-cache
(#7429)
add 5275ded92b HDDS-10133. Add a method to check key name in OMKeyRequest
(#6012)
add 12419fae1f HDDS-11695. SCM follower should not log NotLeaderException
during Pipeline Report processing. (#7430)
add 4066c7c2e4 HDDS-117. Add convenience methods for port management in
DatanodeDetails (#7408)
add dd22dbef89 HDDS-11615. Add Upgrade Action for Initial Schema
Constraints for Unhealthy Container Table in Recon. (#7372)
add f60ad6101e HDDS-11705. Snapshot operations on linked buckets should
work on actual underlying bucket (#7434)
add 238f232420 HDDS-11692. Skip spotbugs for modules with only generated
code. (#7428)
add dbda703281 HDDS-11736. Bump maven-javadoc-plugin to 3.11.1 (#7444)
add 036e727327 HDDS-11732. Fix ACL check on bucket resolution while
reading from snapshot (#7446)
add 3e278b7609 HDDS-10655. Support PutObjectTagging, GetObjectTagging, and
DeleteObjectTagging (#6756)
add 3e798e6ce5 HDDS-11698. Use hadoop images from GitHub in CI (#7432)
add 79ca95672a HDDS-8829. Symmetric Keys for Delegation Tokens (#7394)
add b25218172d HDDS-10804. Include only limited set of ports in Pipeline
proto (#6655)
add 20e4969679 HDDS-11737. UnsupportedOperationException in S3
setBucketAcl (#7448)
add ebcbce78e4 HDDS-11644. Close OMLayoutVersionManager (#7445)
add e96e314b28 HDDS-11697. Integrate Ozone Filesystem Implementation with
Ozone ListStatusLight API (#7440)
add 2cef3935e0 HDDS-11696. Limit max number of entries in list keys/status
response (#7431)
add 0e0d5e9bad HDDS-11163. Improve Heatmap page UI (#7420)
add 0f7104eec0 HDDS-11708. Recon ListKeys API should return a proper http
response status code if NSSummary rebuild is in progress. (#7437)
add c7f65e7ebb HDDS-11740. Add debug command to show internal component
versions (#7450)
add 6b40003bb2 HDDS-11739. Extract generic unmarshaller for S3 requests
(#7449)
add 433c7bb186 HDDS-11729. Update skipRecon property to skip only frontend
build (#7454)
add 3ba347484b HDDS-11755. mktemp --suffix does not work on Mac (#7457)
add a8db9cd476 HDDS-11749. Extract moveToTrash implementation to client
code (#7453)
add fc6a2ea144 HDDS-11650. ContainerId list to track all containers
created in a datanode (#7402)
add 9945de6c75 HDDS-11667. Validating DatanodeID on any request to the
datanode (#7418)
add 1a49991372 HDDS-11617. Update hadoop to 3.4.1 (#7376)
add d52615ae54 HDDS-11733. Remove okio versioning and unused okhttp
dependency (#7447)
add d6a54880fd HDDS-11713. Use seek to reach the start transaction. (#7460)
add f4a9ee0c2f HDDS-11702. Merge test_bucket_encryption into robot
compatibility test (#7451)
add 3b8ed58c6a HDDS-11682. Bump maven-resources-plugin to 3.3.0 (#7384)
add 6ca7230b49 HDDS-11769. Add tools folder into ozone src package. (#7466)
add 1d8abd6989 HDDS-11789. Bump zstd-jni to 1.5.6-7 (#7477)
add 6547de7217 HDDS-11790. Bump commons-lang3 to 3.17.0 (#7475)
add d7f8235fde HDDS-11788. Bump log4j2 to 2.24.2 (#7479)
add 6a1ff845fd HDDS-11791. Bump commons-io to 2.18.0 (#7478)
add f8e4db926f HDDS-11793. Bump maven-checkstyle-plugin to 3.6.0 (#7476)
add b090312574 HDDS-11780. Increase client write retry when SCM is in safe
mode (#7470)
add 20c4cfa349 HDDS-11386. Multithreading bug in ContainerBalancerTask
(#7339)
add c4b2056e6f Update documentation to mention that container schemaV3 is
default (#7481)
add cc1a37476b HDDS-11704. Hadoop test leaves running containers in case
of failure (#7435)
add 579a38e6c4 HDDS-11723. Tool to better micro benchmark hbase
performance in Ozone (#7463)
add 9bc9145d48 HDDS-10411. Support incremental ChunkBuffer checksum
calculation (#7189)
add 6871547273 HDDS-11716. Address Incomplete Upgrade Scenario in Recon
Upgrade Framework (#7452)
add a99ab2706e HDDS-11243. SCM SafeModeRule Support EC. (#7008)
add f98eac2099 HDDS-11561. Refactor Open Key Search Endpoint and
Consolidate with OmDBInsightEndpoint Using StartPrefix Parameter. (#7336)
add 1383c18bbe HDDS-11718. Some CI checks passing despite error (#7483)
add f0a2c87f30 HDDS-11773. Prevent frequent DataNode Ratis snapshotting.
(#7473)
add 3f92663aed HDDS-11785. DataNode aborts ContainerStateMachine if it
does not know any follower next index (#7480)
add 9b261560d6 HDDS-11811. rocksdbjni deleted on exit could be used by
other components #7493
add 7f40624ebf HDDS-11821. Mark
TestECKeyOutputStream#testECKeyCreatetWithDatanodeIdChange as unhealthy
add ebcdc6a1ef HDDS-11810. Secure acceptance test on arm64 fails with
LoginException: Checksum failed (#7498)
add 2c6c1169b6 HDDS-11265. Add Ozone 1.4.1 to compatibility acceptance
tests (#7503)
add 34f9d9e870 HDDS-11742. Update metrics with leaderId if known when
starting SCM (#7471)
add 850306def3 HDDS-11823. Bump cyclonedx-maven-plugin to 2.9.1 (#7508)
add ad867bcef6 HDDS-11824. Bump sqlite-jdbc to 3.47.1.0 (#7511)
add 09a4a904d2 HDDS-11827. Bump exec-maven-plugin to 3.5.0 (#7512)
add e6bd3f5c52 HDDS-11829. Bump zstd-jni to 1.5.6-8 (#7510)
new a102c43063 HDDS-11622. Support domain socket creation (#7397)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (15ee286339)
\
N -- N -- N refs/heads/HDDS-10685 (a102c43063)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/ci.yml | 24 +-
.github/workflows/intermittent-test-check.yml | 2 +-
.github/workflows/populate-cache.yml | 3 +-
.../workflows/scheduled-cache-update.yml | 23 +-
SECURITY.md | 14 +-
.../apache/hadoop/hdds/scm/OzoneClientConfig.java | 2 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 2 +-
.../hadoop/hdds/scm/XceiverClientManager.java | 3 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 22 +-
.../hadoop/hdds/client/ECReplicationConfig.java | 5 +
.../hadoop/hdds/client/RatisReplicationConfig.java | 5 +
.../hadoop/hdds/client/ReplicationConfig.java | 2 +
.../hdds/client/StandaloneReplicationConfig.java | 5 +
.../hadoop/hdds/protocol/DatanodeDetails.java | 132 ++++++-
.../common/helpers/ContainerWithPipeline.java | 3 +-
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 13 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 5 -
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 1 +
.../apache/hadoop/ozone/OzoneManagerVersion.java | 3 +
.../org/apache/hadoop/ozone/common/Checksum.java | 91 ++++-
.../apache/hadoop/ozone/common/ChecksumCache.java | 122 +++++++
.../common/src/main/resources/ozone-default.xml | 16 +-
.../hadoop/hdds/protocol/TestDatanodeDetails.java | 23 +-
.../container/common/helpers/TestExcludeList.java | 16 +-
.../apache/hadoop/ozone/common/TestChecksum.java | 38 +-
.../hadoop/ozone/common/TestChecksumCache.java | 75 ++++
.../ozone/container/ContainerTestHelper.java | 22 +-
.../ozone/container/common/impl/ContainerSet.java | 130 ++++++-
.../container/common/impl/HddsDispatcher.java | 6 +-
.../ozone/container/common/interfaces/Handler.java | 4 +-
.../common/transport/server/XceiverServerGrpc.java | 5 +-
.../server/ratis/ContainerStateMachine.java | 35 +-
.../container/common/volume/MutableVolumeSet.java | 5 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 59 +++-
.../container/metadata/AbstractDatanodeStore.java | 202 +++--------
.../ozone/container/metadata/AbstractRDBStore.java | 135 +++++++
.../ozone/container/metadata/DBStoreManager.java | 71 ++++
.../ozone/container/metadata/DatanodeStore.java | 49 +--
.../metadata/WitnessedContainerDBDefinition.java | 71 ++++
.../metadata/WitnessedContainerMetadataStore.java | 28 +-
.../WitnessedContainerMetadataStoreImpl.java | 78 +++++
.../ozone/container/ozoneimpl/ContainerReader.java | 3 +-
.../ozoneimpl/OnDemandContainerDataScanner.java | 3 +
.../ozone/container/ozoneimpl/OzoneContainer.java | 33 +-
.../container/replication/ContainerImporter.java | 2 +-
.../ozone/container/common/ContainerTestUtils.java | 8 +-
.../container/common/TestDatanodeStateMachine.java | 15 +-
.../common/helpers/TestContainerUtils.java | 3 +-
.../common/helpers/TestDatanodeIdYaml.java | 2 +-
.../container/common/impl/TestHddsDispatcher.java | 9 +-
.../TestCloseContainerCommandHandler.java | 9 +-
.../common/volume/TestVolumeSetDiskChecks.java | 1 +
.../container/keyvalue/TestKeyValueHandler.java | 9 +-
.../container/ozoneimpl/TestOzoneContainer.java | 31 +-
.../replication/TestGrpcReplicationService.java | 6 +-
.../upgrade/TestDataNodeStartupSlvLessThanMlv.java | 9 +-
.../docs/content/feature/dn-merge-rocksdb.md | 8 +-
.../docs/content/feature/dn-merge-rocksdb.zh.md | 6 +-
.../hadoop/hdds/security/OzoneSecretManager.java | 5 +-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 9 +-
.../hadoop/hdds/utils/db/InMemoryTestTable.java | 133 +++++++
hadoop-hdds/interface-admin/pom.xml | 9 +-
hadoop-hdds/interface-client/pom.xml | 10 +-
hadoop-hdds/interface-server/pom.xml | 8 +-
.../hadoop/hdds/utils/NativeLibraryLoader.java | 29 +-
.../hadoop/hdds/utils/TestNativeLibraryLoader.java | 50 ++-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 1 +
.../scm/block/DeletedBlockLogStateManagerImpl.java | 4 +-
.../hdds/scm/container/ContainerReportHandler.java | 7 +
.../balancer/AbstractFindTargetGreedy.java | 4 +-
.../container/balancer/ContainerBalancerTask.java | 76 ++--
.../scm/container/balancer/FindSourceGreedy.java | 4 +-
.../apache/hadoop/hdds/scm/events/SCMEvents.java | 7 +
.../hadoop/hdds/scm/ha/SCMHAManagerStub.java | 8 +
.../apache/hadoop/hdds/scm/ha/SCMRatisServer.java | 3 +
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 11 +
.../hdds/scm/node/NodeDecommissionManager.java | 4 +-
.../hdds/scm/pipeline/PipelineReportHandler.java | 47 ++-
...lockLocationProtocolServerSideTranslatorPB.java | 3 +-
.../hdds/scm/safemode/ContainerSafeModeRule.java | 281 +++++++++++----
.../hdds/scm/safemode/SCMSafeModeManager.java | 11 +
.../hadoop/hdds/scm/safemode/SafeModeMetrics.java | 16 +
.../hdds/scm/security/SecretKeyManagerService.java | 3 +-
.../scm/server/SCMDatanodeHeartbeatDispatcher.java | 12 +
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 2 +-
.../hdds/scm/server/StorageContainerManager.java | 6 +-
.../main/resources/webapps/scm/scm-overview.html | 2 +-
.../src/main/resources/webapps/scm/scm.js | 10 +-
.../org/apache/hadoop/hdds/scm/HddsTestUtils.java | 26 ++
.../hdds/scm/ha/TestReplicationAnnotation.java | 7 +
.../hadoop/hdds/scm/ha/TestSCMRatisServerImpl.java | 108 ++++++
.../hdds/scm/node/TestNodeDecommissionManager.java | 38 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 6 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 149 +++++++-
.../java/org/apache/ozone/test/MetricsAsserts.java | 7 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 53 ++-
.../apache/hadoop/ozone/client/OzoneVolume.java | 7 +-
.../ozone/client/protocol/ClientProtocol.java | 34 ++
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 55 +++
.../TestBlockOutputStreamIncrementalPutBlock.java | 4 +
.../client/io/TestECBlockOutputStreamEntry.java | 2 +-
.../apache/hadoop/fs/ozone/OzoneTrashPolicy.java | 208 +++++++++++
.../org/apache/hadoop/fs/ozone/package-info.java} | 33 +-
.../main/java/org/apache/hadoop/ozone/OFSPath.java | 13 +
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 3 +
.../apache/hadoop/ozone/om/IOmMetadataReader.java | 8 +
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 5 +
.../hadoop/ozone/om/helpers/BasicOmKeyInfo.java | 4 +
.../hadoop/ozone/om/helpers/KeyValueUtil.java | 4 +-
.../ozone/om/protocol/OzoneManagerProtocol.java | 25 ++
...OzoneManagerProtocolClientSideTranslatorPB.java | 74 +++-
.../ozone/security/OzoneTokenIdentifier.java | 100 +++---
hadoop-ozone/dev-support/checks/_post_process.sh | 50 +++
hadoop-ozone/dev-support/checks/acceptance.sh | 14 +-
hadoop-ozone/dev-support/checks/author.sh | 14 +-
hadoop-ozone/dev-support/checks/bats.sh | 8 +-
hadoop-ozone/dev-support/checks/checkstyle.sh | 13 +-
hadoop-ozone/dev-support/checks/docs.sh | 9 +-
hadoop-ozone/dev-support/checks/findbugs.sh | 23 +-
hadoop-ozone/dev-support/checks/junit.sh | 10 +-
hadoop-ozone/dev-support/checks/kubernetes.sh | 9 +-
hadoop-ozone/dev-support/checks/license.sh | 10 +-
hadoop-ozone/dev-support/checks/rat.sh | 8 +-
hadoop-ozone/dev-support/checks/repro.sh | 11 +-
hadoop-ozone/dev-support/checks/sonar.sh | 2 +-
.../dist/dev-support/bin/dist-layout-stitching | 3 +-
hadoop-ozone/dist/pom.xml | 19 +-
.../dist/src/main/assemblies/ozone-src.xml | 42 +--
.../src/main/compose/common/hadoop-secure.yaml | 6 +-
.../dist/src/main/compose/common/hadoop-test.sh | 28 +-
.../dist/src/main/compose/common/hadoop.yaml | 4 +-
.../dist/src/main/compose/common/init-kdc.sh | 86 +++++
.../dist/src/main/compose/common/s3a-test.sh | 5 +-
.../src/main/compose/compatibility/docker-config | 2 +-
.../src/main/compose/ozone-balancer/docker-config | 2 +-
.../dist/src/main/compose/ozone-csi/docker-config | 2 +-
.../dist/src/main/compose/ozone-ha/docker-config | 2 +-
.../src/main/compose/ozone-om-ha/docker-config | 2 +-
.../main/compose/ozone-om-prepare/docker-config | 2 +-
.../src/main/compose/ozone-topology/docker-config | 2 +-
.../dist/src/main/compose/ozone/docker-config | 2 +-
.../src/main/compose/ozoneblockade/docker-config | 2 +-
.../src/main/compose/ozonescripts/docker-config | 2 +-
.../dist/src/main/compose/ozonesecure-ha/.env | 1 +
.../compose/ozonesecure-ha/docker-compose.yaml | 4 +-
.../src/main/compose/ozonesecure-ha/docker-config | 2 +-
.../dist/src/main/compose/ozonesecure-mr/.env | 1 +
.../compose/ozonesecure-mr/docker-compose.yaml | 4 +-
.../src/main/compose/ozonesecure-mr/docker-config | 2 +-
.../dist/src/main/compose/ozonesecure/.env | 1 +
.../main/compose/ozonesecure/docker-compose.yaml | 4 +-
.../src/main/compose/ozonesecure/docker-config | 2 +-
.../dist/src/main/compose/restart/docker-config | 2 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 8 +-
.../dist/src/main/compose/upgrade/compose/ha/.env | 1 +
.../compose/upgrade/compose/ha/docker-compose.yaml | 4 +-
.../main/compose/upgrade/compose/ha/docker-config | 2 +-
.../compose/upgrade/compose/non-ha/docker-config | 2 +-
.../compose/upgrade/compose/om-ha/docker-config | 2 +-
hadoop-ozone/dist/src/main/compose/upgrade/test.sh | 3 +-
hadoop-ozone/dist/src/main/compose/xcompat/.env | 1 +
.../dist/src/main/compose/xcompat/clients.yaml | 9 +
.../dist/src/main/compose/xcompat/docker-config | 2 +-
.../dist/src/main/compose/xcompat/new-cluster.yaml | 4 +-
.../dist/src/main/compose/xcompat/old-cluster.yaml | 4 +-
hadoop-ozone/dist/src/main/compose/xcompat/test.sh | 15 +-
.../src/main/k8s/definitions/ozone/config.yaml | 2 +-
.../examples/getting-started/config-configmap.yaml | 2 +-
.../k8s/examples/minikube/config-configmap.yaml | 2 +-
.../k8s/examples/ozone-dev/config-configmap.yaml | 2 +-
.../k8s/examples/ozone-ha/config-configmap.yaml | 2 +-
.../main/k8s/examples/ozone/config-configmap.yaml | 2 +-
hadoop-ozone/dist/src/main/keytabs/HTTP.keytab | Bin 580 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/dn.keytab | Bin 576 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/hadoop.keytab | Bin 146 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/httpfs.keytab | Bin 616 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/jhs.keytab | Bin 142 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/nm.keytab | Bin 138 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/om.keytab | Bin 576 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/recon.keytab | Bin 606 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/rm.keytab | Bin 138 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/s3g.keytab | Bin 586 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/scm.keytab | Bin 586 -> 0 bytes
hadoop-ozone/dist/src/main/keytabs/testuser.keytab | Bin 908 -> 0 bytes
.../dist/src/main/keytabs/testuser2.keytab | Bin 920 -> 0 bytes
.../dist/src/main/keytabs/update-keytabs.sh | 98 ------
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 3 +
hadoop-ozone/dist/src/main/license/jar-report.txt | 4 +-
.../src/main/license/src/licenses/IMPORTANT.md | 6 +-
.../src/main/smoketest/compatibility/read.robot | 3 +
.../src/main/smoketest/compatibility/write.robot | 7 +
.../main/smoketest/debug/ozone-debug-tests.robot | 5 +
.../dist/src/main/smoketest/s3/commonawslib.robot | 4 +-
.../dist/src/main/smoketest/s3/objectputget.robot | 1 +
.../dist/src/main/smoketest/s3/objecttagging.robot | 73 ++++
.../src/main/smoketest/s3/s3_compatbility_check.sh | 1 +
.../smoketest/security/bucket-encryption.robot | 45 ---
hadoop-ozone/dist/src/shell/conf/log4j.properties | 9 +-
hadoop-ozone/integration-test/pom.xml | 5 +
.../fs/ozone/AbstractOzoneFileSystemTest.java | 46 ++-
.../ozone/AbstractRootedOzoneFileSystemTest.java | 2 +-
.../java/org/apache/hadoop/fs/ozone/TestHSync.java | 3 +-
.../hadoop/fs/ozone/TestOzoneFsSnapshot.java | 18 +
.../apache/hadoop/hdds/scm/TestSecretKeysApi.java | 20 +-
.../hdds/scm/TestStorageContainerManagerHA.java | 83 ++++-
.../java/org/apache/hadoop/ozone/TestDataUtil.java | 44 ++-
.../hadoop/ozone/TestMultipartObjectGet.java | 4 +-
.../hadoop/ozone/TestSecureOzoneCluster.java | 32 +-
.../ozone/client/rpc/OzoneRpcClientTests.java | 132 +++++++
.../rpc/TestContainerStateMachineFailures.java | 55 +++
.../ozone/client/rpc/TestECKeyOutputStream.java | 153 ++++++--
.../TestOzoneRpcClientWithKeyLatestVersion.java | 7 +
.../transport/server/ratis/TestCSMMetrics.java | 16 +-
.../container/metrics/TestContainerMetrics.java | 4 +-
.../container/ozoneimpl/TestOzoneContainer.java | 169 ++++++++-
.../ozoneimpl/TestSecureOzoneContainer.java | 3 +-
.../replication/TestContainerReplication.java | 2 +-
.../container/server/TestContainerServer.java | 7 +-
.../server/TestSecureContainerServer.java | 4 +-
.../ozone/freon/TestHadoopDirTreeGenerator.java | 13 +-
.../hadoop/ozone/freon/TestHsyncGenerator.java | 9 +-
.../org/apache/hadoop/ozone/om/TestListKeys.java | 2 +
.../hadoop/ozone/om/TestListKeysWithFSO.java | 2 +
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 16 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 32 +-
.../om/TestOzoneManagerListVolumesSecure.java | 2 +
.../hadoop/ozone/om/snapshot/TestOmSnapshot.java | 152 ++++----
.../om/snapshot/TestOmSnapshotFileSystem.java | 70 ++--
.../om/snapshot/TestOmSnapshotFileSystemFso.java | 4 +-
...tOmSnapshotFileSystemFsoWithLinkedBuckets.java} | 6 +-
.../snapshot/TestOmSnapshotFileSystemLegacy.java | 4 +-
...SnapshotFileSystemLegacyWithLinkedBuckets.java} | 6 +-
.../snapshot/TestOmSnapshotFsoWithNativeLib.java | 2 +-
...SnapshotFsoWithNativeLibWithLinkedBuckets.java} | 6 +-
.../TestOmSnapshotFsoWithoutNativeLib.java | 2 +-
...pshotFsoWithoutNativeLibWithLinkedBuckets.java} | 6 +-
.../om/snapshot/TestOmSnapshotObjectStore.java | 2 +-
...estOmSnapshotObjectStoreWithLinkedBuckets.java} | 6 +-
... => TestOmSnapshotWithBucketLinkingLegacy.java} | 6 +-
... TestOmSnapshotWithoutBucketLinkingLegacy.java} | 6 +-
.../ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java | 13 +-
.../ozone/shell/TestDeletedBlocksTxnShell.java | 22 ++
.../tools/contract/AbstractContractDistCpTest.java | 47 ++-
hadoop-ozone/interface-client/pom.xml | 10 +-
.../src/main/proto/OmClientProtocol.proto | 37 +-
.../ozone/om/codec/TokenIdentifierCodec.java | 10 +-
.../org/apache/hadoop/ozone/audit/OMAction.java | 6 +-
.../org/apache/hadoop/ozone/om/KeyManager.java | 12 +
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 10 +
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 37 ++
.../hadoop/ozone/om/OMPerformanceMetrics.java | 20 +-
.../apache/hadoop/ozone/om/OmMetadataReader.java | 40 +++
.../hadoop/ozone/om/OmMetadataReaderMetrics.java | 4 +
.../org/apache/hadoop/ozone/om/OmSnapshot.java | 6 +
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 9 +-
.../apache/hadoop/ozone/om/OmSnapshotMetrics.java | 17 +
.../org/apache/hadoop/ozone/om/OzoneManager.java | 114 ++++--
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 168 +--------
.../hadoop/ozone/om/helpers/OMAuditLogger.java | 3 +
.../om/ratis/utils/OzoneManagerRatisUtils.java | 10 +
.../BucketLayoutAwareOMKeyRequestFactory.java | 21 ++
.../ozone/om/request/OMClientRequestUtils.java | 13 +
.../om/request/file/OMDirectoryCreateRequest.java | 7 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 17 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 15 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 14 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 13 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 75 ++++
.../s3/tagging/S3DeleteObjectTaggingRequest.java | 190 ++++++++++
.../S3DeleteObjectTaggingRequestWithFSO.java | 169 +++++++++
.../s3/tagging/S3PutObjectTaggingRequest.java | 192 ++++++++++
.../tagging/S3PutObjectTaggingRequestWithFSO.java | 171 +++++++++
.../s3/tagging/package-info.java} | 36 +-
.../request/snapshot/OMSnapshotCreateRequest.java | 24 +-
.../request/snapshot/OMSnapshotDeleteRequest.java | 7 +
.../request/snapshot/OMSnapshotRenameRequest.java | 8 +
.../s3/tagging/S3DeleteObjectTaggingResponse.java | 73 ++++
.../S3DeleteObjectTaggingResponseWithFSO.java | 72 ++++
.../s3/tagging/S3PutObjectTaggingResponse.java | 72 ++++
.../tagging/S3PutObjectTaggingResponseWithFSO.java | 72 ++++
.../s3/tagging/package-info.java} | 36 +-
.../hadoop/ozone/om/upgrade/OMLayoutFeature.java | 3 +-
.../protocolPB/OzoneManagerRequestHandler.java | 46 ++-
.../OzoneDelegationTokenSecretManager.java | 77 +++-
.../hadoop/ozone/om/TestOMMultiTenantManager.java | 3 +-
.../request/TestBucketLayoutAwareOMKeyFactory.java | 6 +-
.../request/file/TestOMDirectoryCreateRequest.java | 1 +
.../file/TestOMDirectoryCreateRequestWithFSO.java | 1 +
.../ozone/om/request/key/TestOMKeyRequest.java | 22 ++
.../tagging/TestS3DeleteObjectTaggingRequest.java | 202 +++++++++++
.../TestS3DeleteObjectTaggingRequestWithFSO.java | 74 ++++
.../s3/tagging/TestS3PutObjectTaggingRequest.java | 254 ++++++++++++++
.../TestS3PutObjectTaggingRequestWithFSO.java | 95 +++++
.../snapshot/TestOMSnapshotCreateRequest.java | 26 ++
.../snapshot/TestOMSnapshotDeleteRequest.java | 27 ++
.../snapshot/TestOMSnapshotRenameRequest.java | 28 ++
.../tagging/TestS3DeleteObjectTaggingResponse.java | 89 +++++
.../TestS3DeleteObjectTaggingResponseWithFSO.java | 74 ++++
.../s3/tagging/TestS3PutObjectTaggingResponse.java | 88 +++++
.../TestS3PutObjectTaggingResponseWithFSO.java | 70 ++++
.../snapshot/TestSnapshotRequestAndResponse.java | 6 +
.../protocolPB/TestOzoneManagerRequestHandler.java | 175 ++++++++++
.../hadoop/ozone/security/SecretKeyTestClient.java | 73 ++++
.../TestOzoneDelegationTokenSecretManager.java | 43 ++-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 50 ++-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 40 ++-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 83 +++--
.../fs/ozone/BasicRootedOzoneFileSystem.java | 47 ++-
.../apache/hadoop/fs/ozone/OzoneClientAdapter.java | 3 +-
.../apache/hadoop/fs/ozone/OzoneClientUtils.java | 6 +
hadoop-ozone/ozonefs-shaded/pom.xml | 6 +
hadoop-ozone/pom.xml | 14 +-
.../recon/schema/ContainerSchemaDefinition.java | 35 +-
hadoop-ozone/recon/pom.xml | 4 +
.../recon/ReconSchemaVersionTableManager.java | 45 ++-
.../org/apache/hadoop/ozone/recon/ReconServer.java | 4 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 106 +++++-
.../ozone/recon/api/OMDBInsightEndpoint.java | 236 ++++++++-----
.../ozone/recon/api/OMDBInsightSearchEndpoint.java | 350 -------------------
.../ozone/recon/api/ServiceNotReadyException.java} | 20 +-
.../scm/ReconStorageContainerManagerFacade.java | 12 +-
.../upgrade/InitialConstraintUpgradeAction.java | 114 ++++++
.../recon/upgrade/ReconLayoutVersionManager.java | 58 ++-
.../ozone/recon/upgrade/ReconUpgradeAction.java | 4 +-
.../src/v2/components/navBar/navBar.tsx | 2 +-
.../src/v2/components/plots/heatmapPlot.tsx | 151 ++++++++
.../src/v2/constants/heatmap.constants.tsx} | 45 ++-
.../src/v2/pages/heatmap/heatmap.less | 86 +++++
.../src/v2/pages/heatmap/heatmap.tsx | 388 +++++++++++++++++++++
.../recon/ozone-recon-web/src/v2/routes-v2.tsx | 5 +-
.../ozone-recon-web/src/v2/types/heatmap.types.ts} | 41 ++-
.../ozone/recon/api/TestContainerEndpoint.java | 13 +
.../recon/api/TestNSSummaryEndpointWithFSO.java | 13 +-
.../ozone/recon/api/TestOmDBInsightEndPoint.java | 72 ++--
...dpoint.java => TestOpenKeysSearchEndpoint.java} | 170 +++++----
.../TestInitialConstraintUpgradeAction.java | 192 ++++++++++
.../upgrade/TestReconLayoutVersionManager.java | 145 ++++++--
.../org/apache/hadoop/ozone/audit/S3GAction.java | 5 +-
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 3 +-
...CompleteMultipartUploadRequestUnmarshaller.java | 45 +--
.../hadoop/ozone/s3/endpoint/EndpointBase.java | 68 ++--
...tUnmarshaller.java => MessageUnmarshaller.java} | 42 ++-
.../endpoint/MultiDeleteRequestUnmarshaller.java | 54 +--
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 115 +++++-
.../endpoint/PutBucketAclRequestUnmarshaller.java | 59 +---
.../ozone/s3/endpoint/PutTaggingUnmarshaller.java} | 36 +-
.../apache/hadoop/ozone/s3/endpoint/S3Tagging.java | 155 ++++++++
.../hadoop/ozone/s3/exception/S3ErrorTable.java | 4 +
.../hadoop/ozone/s3/metrics/S3GatewayMetrics.java | 90 +++++
.../org/apache/hadoop/ozone/s3/util/S3Consts.java | 1 +
.../hadoop/ozone/client/ClientProtocolStub.java | 16 +
.../hadoop/ozone/client/OzoneBucketStub.java | 31 ++
.../s3/endpoint/TestAbortMultipartUpload.java | 4 +-
.../hadoop/ozone/s3/endpoint/TestListParts.java | 14 +-
.../s3/endpoint/TestMultipartUploadComplete.java | 2 +-
.../s3/endpoint/TestMultipartUploadWithCopy.java | 6 +-
.../hadoop/ozone/s3/endpoint/TestObjectDelete.java | 2 +-
.../hadoop/ozone/s3/endpoint/TestObjectGet.java | 22 +-
.../hadoop/ozone/s3/endpoint/TestObjectPut.java | 66 ++--
.../ozone/s3/endpoint/TestObjectTaggingDelete.java | 152 ++++++++
.../ozone/s3/endpoint/TestObjectTaggingGet.java | 125 +++++++
.../ozone/s3/endpoint/TestObjectTaggingPut.java | 263 ++++++++++++++
.../hadoop/ozone/s3/endpoint/TestPartUpload.java | 14 +-
.../s3/endpoint/TestPartUploadWithStream.java | 8 +-
.../ozone/s3/endpoint/TestPermissionCheck.java | 48 ++-
.../ozone/s3/endpoint/TestUploadWithStream.java | 4 +-
.../ozone/s3/metrics/TestS3GatewayMetrics.java | 146 +++++++-
.../hadoop/ozone/debug/DBDefinitionFactory.java | 4 +-
.../apache/hadoop/ozone/debug/VersionDebug.java | 77 ++++
.../ozone/debug/container/ContainerCommands.java | 2 +-
.../ozone/freon/ClosedContainerReplicator.java | 19 +-
.../hadoop/ozone/freon/DatanodeSimulator.java | 6 +-
.../apache/hadoop/ozone/freon/HsyncGenerator.java | 128 ++++---
.../hadoop/ozone/freon/SCMThroughputBenchmark.java | 9 +-
pom.xml | 60 ++--
375 files changed, 10321 insertions(+), 2882 deletions(-)
copy hadoop-ozone/dev-support/checks/author.sh =>
.github/workflows/scheduled-cache-update.yml (64%)
mode change 100755 => 100644
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumCache.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/common/TestChecksumCache.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/AbstractRDBStore.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/DBStoreManager.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/WitnessedContainerDBDefinition.java
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotLegacy.java
=>
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/WitnessedContainerMetadataStore.java
(58%)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/WitnessedContainerMetadataStoreImpl.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/InMemoryTestTable.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/TestSCMRatisServerImpl.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/fs/ozone/OzoneTrashPolicy.java
copy
hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataReaderMetrics.java
=> common/src/main/java/org/apache/hadoop/fs/ozone/package-info.java} (60%)
create mode 100644 hadoop-ozone/dev-support/checks/_post_process.sh
create mode 100755 hadoop-ozone/dist/src/main/compose/common/init-kdc.sh
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/HTTP.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/dn.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/hadoop.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/httpfs.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/jhs.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/nm.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/om.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/recon.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/rm.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/s3g.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/scm.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/testuser.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/testuser2.keytab
delete mode 100755 hadoop-ozone/dist/src/main/keytabs/update-keytabs.sh
create mode 100644 hadoop-ozone/dist/src/main/smoketest/s3/objecttagging.robot
delete mode 100644
hadoop-ozone/dist/src/main/smoketest/security/bucket-encryption.robot
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/{TestOmSnapshotFileSystemFso.java
=> TestOmSnapshotFileSystemFsoWithLinkedBuckets.java} (83%)
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/{TestOmSnapshotFileSystemLegacy.java
=> TestOmSnapshotFileSystemLegacyWithLinkedBuckets.java} (82%)
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/{TestOmSnapshotFsoWithNativeLib.java
=> TestOmSnapshotFsoWithNativeLibWithLinkedBuckets.java} (85%)
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/{TestOmSnapshotFsoWithoutNativeLib.java
=> TestOmSnapshotFsoWithoutNativeLibWithLinkedBuckets.java} (82%)
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/{TestOmSnapshotObjectStore.java
=> TestOmSnapshotObjectStoreWithLinkedBuckets.java} (83%)
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/{TestOmSnapshotLegacy.java
=> TestOmSnapshotWithBucketLinkingLegacy.java} (84%)
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/{TestOmSnapshotLegacy.java
=> TestOmSnapshotWithoutBucketLinkingLegacy.java} (83%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/tagging/S3DeleteObjectTaggingRequest.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/tagging/S3DeleteObjectTaggingRequestWithFSO.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/tagging/S3PutObjectTaggingRequest.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/tagging/S3PutObjectTaggingRequestWithFSO.java
copy
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/{OmMetadataReaderMetrics.java
=> request/s3/tagging/package-info.java} (59%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/tagging/S3DeleteObjectTaggingResponse.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/tagging/S3DeleteObjectTaggingResponseWithFSO.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/tagging/S3PutObjectTaggingResponse.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/tagging/S3PutObjectTaggingResponseWithFSO.java
copy
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/{OmMetadataReaderMetrics.java
=> response/s3/tagging/package-info.java} (59%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/tagging/TestS3DeleteObjectTaggingRequest.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/tagging/TestS3DeleteObjectTaggingRequestWithFSO.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/tagging/TestS3PutObjectTaggingRequest.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/tagging/TestS3PutObjectTaggingRequestWithFSO.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/tagging/TestS3DeleteObjectTaggingResponse.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/tagging/TestS3DeleteObjectTaggingResponseWithFSO.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/tagging/TestS3PutObjectTaggingResponse.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/tagging/TestS3PutObjectTaggingResponseWithFSO.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/protocolPB/TestOzoneManagerRequestHandler.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/security/SecretKeyTestClient.java
delete mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightSearchEndpoint.java
copy
hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotFileSystemFso.java
=>
recon/src/main/java/org/apache/hadoop/ozone/recon/api/ServiceNotReadyException.java}
(73%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/InitialConstraintUpgradeAction.java
create mode 100644
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/v2/components/plots/heatmapPlot.tsx
copy
hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotLegacy.java
=>
recon/src/main/resources/webapps/recon/ozone-recon-web/src/v2/constants/heatmap.constants.tsx}
(57%)
create mode 100644
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/v2/pages/heatmap/heatmap.less
create mode 100644
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/v2/pages/heatmap/heatmap.tsx
rename
hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotLegacy.java
=>
recon/src/main/resources/webapps/recon/ozone-recon-web/src/v2/types/heatmap.types.ts}
(51%)
rename
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/{TestOMDBInsightSearchEndpoint.java
=> TestOpenKeysSearchEndpoint.java} (83%)
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/upgrade/TestInitialConstraintUpgradeAction.java
copy
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/{PutBucketAclRequestUnmarshaller.java
=> MessageUnmarshaller.java} (74%)
copy
hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataReaderMetrics.java
=>
s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/PutTaggingUnmarshaller.java}
(60%)
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/S3Tagging.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingDelete.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingGet.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingPut.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/VersionDebug.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
