hello-stephen opened a new pull request, #70412: URL: https://github.com/apache/doris/pull/70412
### Background The regression framework supplies S3 credentials to an AWS SDK client and to SQL assembled by individual suites. Some suites also generate objects in the same bucket used for immutable input fixtures. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Temporary access keys require a session token, but the SDK helper constructs two-part credentials and many SQL call sites omit the configured token. This prevents affected tests from reaching their data assertions. Introduce one credential snapshot for SDK requests and explicit SQL construction, and let generated objects use a separate write bucket, prefix, endpoint and region. Keep immutable fixture references and deliberate invalid-credential cases distinct. ### Details - **Change map:** Regression infrastructure: 8 files (+251/-5); framework unit tests: 4 files (+331); suite migrations: 134 files (+904/-1473); configuration documentation and logging: 2 files (+14/-2). No FE/BE production code or generated result files change. Most suite changes replace repeated AK/SK pairs with an explicit helper call. - **Frequency:** Whenever these SDK/SQL paths run with temporary credentials that require a session token. - **Root cause:** `getS3Client()` uses `BasicAWSCredentials`, while individual statements copy only AK/SK. Generated-object destinations also assume write access to the fixture bucket. - **Why existing tests missed it:** Long-lived keys do not require a token, and environments with write access to their fixture bucket do not expose the destination assumption. - **Fix:** Use session credentials when configured, render the supported S3/AWS/TVF credential namespaces explicitly, and provide opt-in write-destination getters. Redact configured credentials in log encoders and TeamCity failure messages without modifying executed SQL or suppressing errors. - **Other paths and branches:** Migrate observed TVF, LOAD, OUTFILE, resource, repository and storage-vault call sites. Preserve 56 storage-vault negative blocks byte-for-byte, including uppercase property names, missing/empty roots and invalid credentials; preserve the leading-slash positive case. Other branches and remaining call sites need independent review. - **Behavior change:** Missing write settings fall back to the existing bucket/endpoint/region and unprefixed keys. Configured session tokens now reach SDK/SQL consumers. Write prefixes affect only explicit generated-object call sites. Resource connectivity probes use an explicit scoped root. This change does not acquire credentials, renew a lease, change cloud permissions, or rewrite SQL at runtime. Credentials must remain valid for the complete run; SDK requests and newly constructed statements read the current configuration, while previously persisted SQL objects retain their statement-time credentials. ### Testing - Standard `bash run-regression-test.sh --clean --compile` on Linux with JDK 17: all 63 framework tests passed; Java UDF packaging succeeded using the script's existing build behavior. - New tests exercise actual SDK HTTP requests against a local server: PUT/GET session-token headers, an updated credential snapshot, no-token compatibility and propagated HTTP 403 failures. Logback and TeamCity failure-output tests verify redaction without changing the original event or turning failure into success. - Groovy parsing/conversion passed for all 134 migrated suite files. `git diff --check` passed. - Real object-storage validation and complete Doris SQL/regression execution are pending. These compile/unit/syntax results do not establish that every migrated case passes. ### Notes for Reviewers Start with `S3TestCredentials`, the `Suite` getters, and the new tests. The suite diff makes credential and write-destination choices explicit. No new exclusions or changes to expected result files are included. Read and write endpoints can differ, while both use the configured provider and credential identity. ### Release note None ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No; regression configuration comments document the new test-only settings. - Pre-submission status: Not ready: Draft pending real storage and Doris regression validation and human review. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Implementation, suite migrations, tests and this description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
