pan3793 commented on code in PR #8347: URL: https://github.com/apache/hadoop/pull/8347#discussion_r3333161811
########## hadoop-cloud-storage-project/hadoop-bos/src/test/resources/README-TEST.md: ########## @@ -0,0 +1,326 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# BOS FileSystem Testing Guide + +This document provides guidance on running and understanding the BOS FileSystem test suite. + +## Table of Contents + +1. [Prerequisites](#prerequisites) +2. [Test Configuration](#test-configuration) +3. [Running Tests](#running-tests) +4. [Test Structure](#test-structure) +5. [Troubleshooting](#troubleshooting) + +## Prerequisites + +### Required + +- Java 8 or higher +- Maven 3.3 or higher +- Access to a Baidu BOS bucket for testing +- BOS credentials (Access Key ID and Secret Access Key) + +### BOS Bucket Setup + +1. Create a BOS bucket for testing (or use an existing one) +2. Ensure the bucket is accessible with your credentials +3. Note: Tests will create and delete files under `/test` directory in the bucket + +## Test Configuration + +### Step 1: Create Test Configuration File + +Copy the template configuration file: + +```bash +cd src/test/resources +cp contract-test-options.xml.template contract-test-options.xml +``` + +### Step 2: Configure BOS Credentials + +Edit `contract-test-options.xml` and set your BOS configuration: + +```xml +<property> + <name>fs.contract.test.fs.bos</name> + <value>bos://your-bucket-name/test</value> +</property> + +<property> + <name>fs.bos.endpoint</name> + <value>http://bd.bcebos.com</value> +</property> +``` + +### Step 3: Set Authentication + +**Recommended: Use Environment Variables** + +```bash +export FS_BOS_ACCESS_KEY=your_access_key_id +export FS_BOS_SECRET_ACCESS_KEY=your_secret_access_key +# Optional: for STS/temporary credentials +# export FS_BOS_SESSION_TOKEN_KEY=your_session_token +``` + +Ensure the credentials provider is set to `EnvironmentVariableCredentialsProvider`: + +```xml +<property> + <name>fs.bos.credentials.provider</name> + <value>org.apache.hadoop.fs.bos.credentials.EnvironmentVariableCredentialsProvider</value> +</property> +``` + +**Alternative: Configuration-based (Not Recommended)** + +You can also configure credentials directly in `contract-test-options.xml`, but this is **NOT recommended** for security reasons: + +```xml +<property> + <name>fs.bos.access.key</name> + <value>YOUR_ACCESS_KEY</value> +</property> + +<property> + <name>fs.bos.secret.key</name> + <value>YOUR_SECRET_KEY</value> +</property> +``` + +**⚠️ SECURITY WARNING**: Never commit `contract-test-options.xml` with credentials to version control! + +## Running Tests + +### Run All Tests + +```bash +mvn test +``` + +### Run Contract Tests Only + +```bash +mvn test -Dtest="**/contract/*Test.java" +``` + +### Run Specific Contract Test Suite + +```bash +# Test file creation +mvn test -Dtest=TestBosContractCreate + +# Test rename operations +mvn test -Dtest=TestBosContractRename + +# Test delete operations +mvn test -Dtest=TestBosContractDelete +``` + +### Run Integration Tests + +```bash +mvn test -Dtest="**/integration/*Test.java" +``` + +### Run with Specific Bucket Type + +For namespace-enabled bucket: +```bash +mvn test -Dfs.contract.test.fs.bos=bos://namespace-bucket/test +``` + +For flat (non-namespace) bucket: +```bash +mvn test -Dfs.contract.test.fs.bos=bos://flat-bucket/test +``` + +### Skip Tests + +```bash +mvn package -DskipTests +``` + +## Test Structure + +### Contract Tests (`org.apache.hadoop.fs.bos.contract`) + +These tests verify that BOS FileSystem conforms to the Hadoop FileSystem contract: + +- **TestBosContractCreate**: File creation operations +- **TestBosContractOpen**: File open and read operations +- **TestBosContractDelete**: File and directory deletion +- **TestBosContractMkdir**: Directory creation +- **TestBosContractRename**: File and directory rename +- **TestBosContractSeek**: Random access (seek) operations +- **TestBosContractGetFileStatus**: Metadata operations +- **TestBosContractRootDir**: Root directory operations (disabled by default) +- **TestBosContractContentSummary**: Content summary operations + +### Integration Tests (`org.apache.hadoop.fs.bos.integration`) + +These tests verify BOS-specific functionality: + +- **TestBosIntegrationIO**: Input/Output stream operations +- **TestBosIntegrationChecksum**: Checksum calculation and verification +- **TestBosIntegrationMultipart**: Multi-part upload/download (if applicable) + +### Credential Tests (`org.apache.hadoop.fs.bos.credentials`) + +- **TestConfigurationCredentialsProvider**: Configuration-based authentication + +## Test Features and Limitations + +### Supported Features + +- ✅ File create, read, delete +- ✅ Directory operations (mkdir, delete) +- ✅ Rename (via copy + delete) +- ✅ Random access (seek) +- ✅ Content summary +- ✅ Checksum verification + +### Unsupported Features (Tests Will Skip) + +- ❌ Append to files +- ❌ File concatenation +- ❌ File truncation +- ❌ Symbolic links +- ❌ Extended attributes (xattr) +- ❌ Full Unix permissions (limited support in namespace mode) + Review Comment: you seem to miss `hflush/hsync operations` item which you mentioned in another place, and what's the behavior if the user calls `hflush/hsync`? noop or throw an exception? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
