Jetiaime opened a new pull request, #4519: URL: https://github.com/apache/streampark/pull/4519
## What changes were proposed in this pull request Closes #4518 Fix checkpoint/savepoint deletion for YARN Application records whose stored HDFS path has no authority, for example `hdfs:/tmp/checkpoints/chk-1`. Hadoop accepts this URI and resolves its authority using `fs.defaultFS`. The current `HdfsOperator.toHdfsPath` only recognizes `hdfs://`, so it prepends the default filesystem to a single-slash HDFS URI. With `fs.defaultFS=hdfs://namenode:8020`, that produces `hdfs://namenode:8020hdfs:/tmp/checkpoints/chk-1`, and the target directory is not deleted. Preserve paths starting with `hdfs:/` and let Hadoop resolve them. This includes the existing triple-slash and fully qualified forms. Paths without a scheme continue to use the existing default-filesystem prefix behavior. Based on the current upstream `dev` at `829466b5470d749773793193f1fc1d46e8613d61`. ## Brief change log - Recognize HDFS URIs with or without an authority in `HdfsOperator.toHdfsPath`. - Add a 12-case parameterized regression test that calls `HdfsOperator.delete` and checks the resolved deletion target. It covers single-slash, triple-slash, fully qualified and absolute paths across a host/port default filesystem and a logical nameservice with/without a trailing slash. - No schema, public API or configuration changes. ## Verifying this change This change adds regression coverage and was verified as follows. ### Automated regression tests and project checks ```shell ./mvnw -B -ntp -pl streampark-common test checkstyle:check spotless:check apache-rat:check ``` Passed: **121 tests, 0 failures, 0 errors, 0 skipped**; Checkstyle, Spotless and Apache RAT passed. Also ran these goals against a fresh source archive in a Linux container using Maven 3.9.9 / Temurin 11.0.27 with the same result. The host run used OpenJDK 17 with the project's Java 11 target. The newly added regression test fails against the original implementation (3 affected single-slash cases) and passes with this change (12/12). ### Real HDFS deletion validation Ran an isolated Hadoop 3.3.5 `MiniDFSCluster` with a real NameNode and one DataNode under Java 11. The test creates a target directory containing `_metadata`, confirms the file exists, invokes the production `HdfsOperator.delete`, then checks the directory is absent and an adjacent file still contains its original data. | Input path | Upstream implementation | This change | | --- | --- | --- | | `hdfs:/...` | Fails; directory remains (2/2 configurations) | Pass (2/2) | | `hdfs:///...` | Pass (2/2) | Pass (2/2) | | `hdfs://host:port/...` | Pass (2/2) | Pass (2/2) | | `/...` | Pass (2/2) | Pass (2/2) | The two configurations use `fs.defaultFS` with and without a trailing slash. The adjacent file remained intact in all cases. The baseline loads `HdfsOperator` compiled from upstream `dev`; the fixed run loads this branch's production class. MiniDFSCluster is temporary test tooling and is not added as a project dependency. ### Console API end-to-end validation Started the freshly built 3.0.0 Console package with an isolated file-backed H2 database and the real MiniDFSCluster. The container had no external network access. Used the bundled demo application with `deploy_mode=4` (YARN Application) and the standard authentication/permission checks. For each of the four path forms above: 1. Created a checkpoint-directory fixture with a `_metadata` file and an adjacent file to retain; inserted a matching savepoint record into the isolated H2 database before Console startup. 2. Logged in through `POST /passport/signin` and confirmed the record through `POST /flink/savepoint/history`. 3. Called authenticated `POST /flink/savepoint/delete` with `id`, `appId` and `teamId`. 4. Asserted `data=true`, absence of the record from history, absence of the directory in real HDFS, and unchanged content of the adjacent file. 5. Stopped the Console and directly queried H2 to confirm that all four fixture rows were deleted. **Result: 4/4 API cases passed; final H2 row count: 0.** This covers HTTP authentication → controller → savepoint service → HDFS deletion → persisted database removal. It uses seeded checkpoint fixtures, not a live Flink/YARN job, and does not cover browser interaction. During fixture setup, the bundled H2 seed script had an unrelated demo-project column-count mismatch; initialization continued as configured by the application, and the required account/application/savepoint fixtures were verified before testing. ### Backend package build ```shell ./mvnw -B -ntp -pl streampark-console/streampark-console-service -am -Pshaded -DskipTests package ``` Passed: all 30 reactor modules, including the Console runtime package. Tests are skipped by this packaging command; test execution is reported separately above. The frontend was not built. The repository's current `E2E - Result` workflow is a retired-suite compatibility check, so it is not used as evidence of the validation above. ## Does this pull request potentially affect one of the following parts - Dependencies (does it add or upgrade a dependency): **no** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
