This is an automated email from the ASF dual-hosted git repository.
tanvipenumudy pushed a change to branch HDDS-13177
in repository https://gitbox.apache.org/repos/asf/ozone.git
from bc237f2c8c HDDS-13187. Extend Recon events handling to
MultipartInfoTable (#8651)
add 9b713d0b65 HDDS-12395. Add Mismatched Replica tab to the Containers
Page (#8407)
add 14748e831d HDDS-13401. Mark TestContainerCommandReconciliation as flaky
add 1315df6461 HDDS-13337. Reduce config required to enable Byteman
listener (#8701)
add b610fd0a42 HDDS-13327. Improve log and error handling while starting
Recon (#8688)
add 5255c8cbc7 HDDS-13122. Add acceptance test for `ozone repair om
compact` (#8703)
add bca868865e HDDS-13098. Create PicoCLI mixin to standardize datanode
selection options (#8674)
add bf9cdd0df9 HDDS-13162. Improve Container Replica Debugger Tool
documentation (#8597)
add 05c7e62f45 HDDS-13404. Improve persistent test data dir
creation/cleanup (#8762)
add 03fade02d0 HDDS-13383. Improve Logs in Ozone documentation. (#8744)
add c3fe4586e4 HDDS-12726. Update Ozone documentation with the latest S3
API compatibility (#8735)
add 13af30cf43 HDDS-13380. Docs: Enhance Ozone Manager documentation with
cross-references (#8738)
add 06944809ce HDDS-13354. Run multitenancy acceptance test with Ranger
(#8722)
add 2ae8d6da28 HDDS-13379. Document SCM Safe Mode and its configuration
properties. (#8737)
add e08b001044 HDDS-13006. Use yaml files to host Ozone snapshot local
properties (#8555)
add a9755a0491 HDDS-13373. [Docs] Add leader transfer command and service
role listing command in the High Availability pages (#8730)
add c706c7ac0d HDDS-13228. Take snapshot cache lock during the last
iteration of tarball transfer. (#8678)
add 7477431c93 HDDS-13372. [Docs] Compare Ozone to other storage
technologies (#8736)
add 1ad1b2d58a HDDS-13409. Add supported operations for Ofs. (#8763)
add 1a2ab3dfab HDDS-13397. Fix liveness probe failures for httpfs and s3g
pods (#8775)
add 1159d9b7b4 HDDS-13369. [Docs] Maintenance mode. (#8728)
add f7eb0104d3 HDDS-13421. [Docs] Ofs migration guide (#8784)
add e0b804ef99 HDDS-13095. Support sorting by most/least used nodes in
ozone admin datanode list (#8520)
add 490e6bbd90 HDDS-13402. Speed up TestOmSnapshotFileSystem (#8756)
add c696942607 HDDS-10931. Schedule on demand scan of containers after
import (#8550)
add 8d1a0adf7c HDDS-13335. Implement and adopt lightweight proto for Recon
listKeys API (#8699)
add 59c2ec3ad9 HDDS-13386. Replace leftover HADOOP_ variables (#8759)
add 8651aa4a76 HDDS-13343. Consider delegation token lifetime for secret
key expiry (#8742)
add eebc7b5e66 HDDS-13364. Use properties for Java/Maven version
enforcement (#8724)
add 583730dc6d HDDS-12843. Intermittent NPE in
TestDecommissionAndMaintenance (#8643)
add a1e7513188 HDDS-13361. Attempt to delete non-empty tenant fails after
revoke (#8776)
add f856adae60 HDDS-8781. Allow on demand metadata scanning of open
containers (#8442)
add 5ea69402e9 HDDS-13427. Bump awssdk to 2.31.78 (#8792)
add 9aad5bc9d2 HDDS-13424. Use lsof instead of fuser to find if file is
used in AbstractTestChunkManager (#8790)
add 96cf8bed03 HDDS-11679. Support multiple S3Gs in MiniOzoneCluster
(#8733)
add fb75be7a89 HDDS-13376. Add server-side limit note to ozone sh snapshot
diff --page-size option (#8791)
add 3171688d81 HDDS-11514. Set optimal default values for delete
configurations based on live cluster testing. (#8766)
add f0dd23676f HDDS-13405. ozone admin container create runs forever
without kinit (#8765)
add 5acd3d8bc5 HDDS-13387. OMSnapshotCreateRequest logs invalid warning
about DefaultReplicationConfig (#8760)
add 918b2848ee HDDS-13213. KeyDeletingService should limit task size by
both key count and serialized size. (#8757)
add d76f4ee287 HDDS-11797. Remove cyclic dependency between
SCMSafeModeManager and SafeModeRules (#8782)
add b220c99e4b HDDS-13403. [Docs] Make feature proposal process more
visible. (#8758)
add d15e8a6347 HDDS-13378. [Docs] Add a Production page under Getting
Started (#8734)
add d4617d0f12 HDDS-13428. Recon - Retrigger of build whole NSSummary tree
task submission inconsistency. (#8793)
add 18c928d23b Merge remote-tracking branch 'origin/master' into HDDS-13177
No new revisions were added by this update.
Summary of changes:
CONTRIBUTING.md | 4 +
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 2 +-
.../java/org/apache/hadoop/hdds/scm/ScmConfig.java | 2 +-
.../hadoop/hdds/utils/db/DelegatedCodec.java | 11 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 15 ++
.../common/src/main/resources/ozone-default.xml | 24 +-
.../ozone/container/common/impl/ContainerSet.java | 6 +-
.../common/statemachine/DatanodeConfiguration.java | 6 +-
.../SetNodeOperationalStateCommandHandler.java | 16 +-
.../BackgroundContainerMetadataScanner.java | 19 +-
.../container/ozoneimpl/ContainerScanHelper.java | 34 ++-
...aScanner.java => OnDemandContainerScanner.java} | 18 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 4 +-
.../container/replication/ContainerImporter.java | 5 +-
.../container/common/impl/TestContainerSet.java | 6 +-
...stContainerReconciliationWithMockDatanodes.java | 6 +-
.../container/keyvalue/TestKeyValueHandler.java | 4 +-
.../keyvalue/impl/AbstractTestChunkManager.java | 4 +-
...nner.java => TestOnDemandContainerScanner.java} | 34 ++-
.../replication/TestContainerImporter.java | 117 ++++++---
hadoop-hdds/docs/content/concept/Comparison.md | 77 ++++++
hadoop-hdds/docs/content/concept/OzoneManager.md | 16 +-
.../content/concept/StorageContainerManager.md | 21 ++
hadoop-hdds/docs/content/feature/Maintenance.md | 78 ++++++
hadoop-hdds/docs/content/feature/OM-HA.md | 78 ++++++
hadoop-hdds/docs/content/feature/SCM-HA.md | 93 ++++++-
hadoop-hdds/docs/content/interface/Ofs.md | 158 +++++++++++
hadoop-hdds/docs/content/interface/S3.md | 75 ++++--
.../docs/content/start/ProductionDeployment.md | 89 +++++++
.../docs/content/start/ProductionDeployment.zh.md | 69 +++++
.../docs/content/tools/ContainerLogParser.md | 26 +-
.../docs/content/tools/ContainerLogParser.zh.md | 239 +++++++++++++++++
hadoop-hdds/docs/content/tools/LogsInOzone.md | 81 +++++-
.../java/org/apache/hadoop/hdds/utils/HAUtils.java | 37 ++-
.../hadoop/hdds/utils/MapBackedTableIterator.java | 12 +-
.../hdds/scm/safemode/ECContainerSafeModeRule.java | 1 -
.../hdds/scm/safemode/SCMSafeModeManager.java | 4 +-
.../hdds/scm/safemode/SafeModeRuleFactory.java | 14 +-
.../hdds/scm/safemode/TestSafeModeRuleFactory.java | 11 +-
.../cli/datanode/DecommissionStatusSubCommand.java | 33 +--
.../hdds/scm/cli/datanode/ListInfoSubcommand.java | 115 ++++++--
.../hdds/scm/cli/datanode/NodeSelectionMixin.java | 66 +++++
.../hdds/scm/cli/datanode/UsageInfoSubcommand.java | 25 +-
.../ozone/admin/om/UpdateRangerSubcommand.java | 2 +-
.../datanode/TestDecommissionStatusSubCommand.java | 35 ++-
.../scm/cli/datanode/TestListInfoSubcommand.java | 108 ++++++++
.../ozone/shell/snapshot/SnapshotDiffHandler.java | 5 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 2 +-
.../hadoop/ozone/om/helpers/SnapshotInfo.java | 4 +-
hadoop-ozone/dist/pom.xml | 2 +-
.../dist/src/main/compose/common/security.conf | 3 +
.../dist/src/main/compose/ozone-om-ha/startOM.sh | 6 +-
.../dist/src/main/compose/ozone-om-prepare/test.sh | 14 +-
.../dist/src/main/compose/ozonesecure-ha/.env | 2 -
.../src/main/compose/ozonesecure-ha/byteman.yaml | 47 ----
.../src/main/compose/ozonesecure-ha/docker-config | 8 +-
.../compose/ozonesecure-ha/test-debug-tools.sh | 17 +-
.../src/main/compose/ozonesecure-ha/test-ranger.sh | 6 +-
.../{test-byteman.sh => test-repair-tools.sh} | 19 +-
.../dist/src/main/compose/ozonesecure-ha/test.sh | 6 +
hadoop-ozone/dist/src/main/compose/restart/test.sh | 3 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 17 +-
.../src/main/compose/upgrade/compose/ha/load.sh | 3 +-
.../main/compose/upgrade/compose/non-ha/load.sh | 3 +-
.../dist/src/main/compose/upgrade/testlib.sh | 15 --
.../dist/src/main/dockerlibexec/entrypoint.sh | 4 +
.../src/main/k8s/definitions/ozone/config.yaml | 2 +
.../src/main/k8s/definitions/ozone/s3g-ss.yaml | 2 +-
.../examples/getting-started/config-configmap.yaml | 2 +
.../examples/getting-started/s3g-statefulset.yaml | 2 +-
.../k8s/examples/minikube/config-configmap.yaml | 2 +
.../k8s/examples/minikube/s3g-statefulset.yaml | 2 +-
.../k8s/examples/ozone-dev/config-configmap.yaml | 2 +
.../k8s/examples/ozone-dev/s3g-statefulset.yaml | 2 +-
.../k8s/examples/ozone-ha/config-configmap.yaml | 2 +
.../k8s/examples/ozone-ha/s3g-statefulset.yaml | 2 +-
.../main/k8s/examples/ozone/config-configmap.yaml | 2 +
.../main/k8s/examples/ozone/s3g-statefulset.yaml | 2 +-
.../src/main/smoketest/admincli/datanode.robot | 10 +-
.../{kinit.robot => admincli/lib.resource} | 20 +-
hadoop-ozone/dist/src/main/smoketest/kinit.robot | 4 +-
.../src/main/smoketest/repair/om-compact.robot | 60 +++++
.../test1.robot => scmha/container-create.robot} | 11 +-
.../smoketest/security/ozone-secure-tenant.robot | 74 +++---
.../dist/src/main/smoketest/upgrade/lib.robot | 16 +-
.../httpfsgateway/src/main/conf/httpfs-env.sh | 6 +-
.../main/libexec/shellprofile.d/hadoop-httpfs.sh | 54 ----
hadoop-ozone/integration-test-s3/pom.xml | 40 +++
.../hadoop/ozone/s3/LoadBalanceStrategy.java} | 17 +-
.../hadoop/ozone/s3/MultiS3GatewayService.java | 99 +++++++
.../org/apache/hadoop/ozone/s3/ProxyServer.java | 215 +++++++++++++++
.../ozone/s3/ProxyServerIntegrationTest.java | 163 ++++++++++++
.../apache/hadoop/ozone/s3/RoundRobinStrategy.java | 32 +--
.../ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java | 4 +-
.../ozone/s3/awssdk/v2/AbstractS3SDKV2Tests.java | 4 +-
.../hadoop/hdds/scm/TestSecretKeySnapshot.java | 4 +
.../apache/hadoop/hdds/scm/TestSecretKeysApi.java | 5 +
.../org/apache/hadoop/ozone/TestBlockTokens.java | 4 +
.../hadoop/ozone/TestSecureOzoneCluster.java | 29 ++-
.../TestContainerCommandReconciliation.java | 11 +-
...> TestOnDemandContainerScannerIntegration.java} | 47 +++-
.../org/apache/hadoop/ozone/om/TestKeyPurging.java | 9 +-
.../TestOMDbCheckpointServletInodeBasedXfer.java | 183 +++++++++++--
.../hadoop/ozone/om/snapshot/SnapshotTests.java | 82 ++++++
.../om/snapshot/TestOmSnapshotFileSystem.java | 121 ++++-----
.../om/snapshot/TestOmSnapshotFileSystemFso.java | 27 --
...stOmSnapshotFileSystemFsoWithLinkedBuckets.java | 27 --
.../snapshot/TestOmSnapshotFileSystemLegacy.java | 27 --
...mSnapshotFileSystemLegacyWithLinkedBuckets.java | 27 --
.../om/snapshot/TestOzoneSnapshotsNonHA.java} | 8 +-
.../org/apache/ozone/test/ClusterForTests.java | 6 +
.../src/main/proto/OmClientProtocol.proto | 19 ++
hadoop-ozone/ozone-manager/pom.xml | 4 +
.../org/apache/hadoop/ozone/om/KeyManager.java | 8 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 44 +++-
.../om/OMDBCheckpointServletInodeBasedXfer.java | 35 +++
.../hadoop/ozone/om/OmSnapshotLocalData.java | 288 +++++++++++++++++++++
.../hadoop/ozone/om/OmSnapshotLocalDataYaml.java | 259 ++++++++++++++++++
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 15 ++
.../org/apache/hadoop/ozone/om/OzoneManager.java | 53 +++-
.../hadoop/ozone/om/codec/OMDBDefinition.java | 10 +
.../multitenant/MultiTenantAccessController.java | 36 +++
.../request/s3/tenant/OMTenantDeleteRequest.java | 21 +-
.../request/snapshot/OMSnapshotCreateRequest.java | 8 +-
.../ozone/om/service/DirectoryDeletingService.java | 5 +-
.../ozone/om/service/KeyDeletingService.java | 35 ++-
.../ozone/om/service/SnapshotDeletingService.java | 7 +-
.../hadoop/ozone/om/snapshot/SnapshotCache.java | 1 +
.../OzoneDelegationTokenSecretManager.java | 4 +
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 23 +-
.../ozone/om/TestOmSnapshotLocalDataYaml.java | 209 +++++++++++++++
.../ozone/om/service/TestKeyDeletingService.java | 20 +-
.../org/apache/hadoop/ozone/recon/ReconServer.java | 46 +++-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 19 +-
.../ozone/recon/api/OMDBInsightEndpoint.java | 31 +--
.../recon/api/types/KeyEntityInfoProtoWrapper.java | 144 -----------
.../ozone/recon/api/types/ListKeysResponse.java | 6 +-
.../recon/api/types/ReconBasicOmKeyInfo.java} | 213 +++++++--------
.../ozone/recon/fsck/ReconSafeModeMgrTask.java | 6 +-
.../recon/recovery/ReconOMMetadataManager.java | 6 +-
.../recon/recovery/ReconOmMetadataManagerImpl.java | 6 +-
.../scm/ReconStorageContainerManagerFacade.java | 3 +
.../spi/impl/OzoneManagerServiceProviderImpl.java | 3 +
.../webapps/recon/ozone-recon-web/api/db.json | 40 +++
.../src/v2/components/tables/containersTable.tsx | 48 +++-
.../src/v2/pages/containers/containers.less | 13 +-
.../src/v2/pages/containers/containers.tsx | 27 +-
.../src/v2/types/container.types.ts | 2 +
.../ozone/recon/api/TestOmDBInsightEndPoint.java | 18 +-
pom.xml | 18 +-
150 files changed, 4173 insertions(+), 1136 deletions(-)
rename
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/{OnDemandContainerDataScanner.java
=> OnDemandContainerScanner.java} (89%)
rename
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/{TestOnDemandContainerDataScanner.java
=> TestOnDemandContainerScanner.java} (91%)
create mode 100644 hadoop-hdds/docs/content/concept/Comparison.md
create mode 100644 hadoop-hdds/docs/content/feature/Maintenance.md
create mode 100644 hadoop-hdds/docs/content/start/ProductionDeployment.md
create mode 100644 hadoop-hdds/docs/content/start/ProductionDeployment.zh.md
create mode 100644 hadoop-hdds/docs/content/tools/ContainerLogParser.zh.md
create mode 100644
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/NodeSelectionMixin.java
delete mode 100644
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/byteman.yaml
rename hadoop-ozone/dist/src/main/compose/ozonesecure-ha/{test-byteman.sh =>
test-repair-tools.sh} (74%)
copy hadoop-ozone/dist/src/main/smoketest/{kinit.robot =>
admincli/lib.resource} (71%)
create mode 100644 hadoop-ozone/dist/src/main/smoketest/repair/om-compact.robot
copy hadoop-ozone/dist/src/main/smoketest/{failing/test1.robot =>
scmha/container-create.robot} (68%)
delete mode 100755
hadoop-ozone/httpfsgateway/src/main/libexec/shellprofile.d/hadoop-httpfs.sh
copy
hadoop-ozone/{s3gateway/src/main/java/org/apache/hadoop/ozone/s3/GatewayApplication.java
=>
integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/LoadBalanceStrategy.java}
(71%)
create mode 100644
hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/MultiS3GatewayService.java
create mode 100644
hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/ProxyServer.java
create mode 100644
hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/ProxyServerIntegrationTest.java
copy
hadoop-hdds/test-utils/src/test/java/org/apache/ozone/test/SpyInputStream.java
=>
hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/RoundRobinStrategy.java
(59%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/scanner/{TestOnDemandContainerDataScannerIntegration.java
=> TestOnDemandContainerScannerIntegration.java} (79%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/SnapshotTests.java
delete mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotFileSystemFso.java
delete mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotFileSystemFsoWithLinkedBuckets.java
delete mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotFileSystemLegacy.java
delete mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotFileSystemLegacyWithLinkedBuckets.java
copy
hadoop-ozone/integration-test/src/test/java/org/apache/{ozone/test/TestFreon.java
=> hadoop/ozone/om/snapshot/TestOzoneSnapshotsNonHA.java} (87%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotLocalDataYaml.java
delete mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/KeyEntityInfoProtoWrapper.java
copy
hadoop-ozone/{common/src/main/java/org/apache/hadoop/ozone/om/helpers/BasicOmKeyInfo.java
=>
recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ReconBasicOmKeyInfo.java}
(51%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
