88fantasy opened a new pull request, #4500: URL: https://github.com/apache/streampark/pull/4500
## What is the purpose of the change Makes Flink 2.x job submission work on REMOTE mode. Today no Flink 2.x SQL job can be submitted at all, while Flink 1.x is unaffected — which is why this went unnoticed. Closes #4499 > **Stacked on #4496.** This branch contains that PR's commit as its base, so the diff to review here is the second commit only. If #4496 merges first this rebases away cleanly; if it is closed, this needs its `setUserClassPaths` change kept. ## Brief change log Five independent causes on one path — each only becomes visible once the previous is fixed: 1. **`FlinkShimsProxy` was not loading the per-version shims jar.** It matches a name shaped `streampark-flink-shims_flink-<major>_<scala>`, which those artifacts carried until `e770d2e8e` renamed them without the Scala suffix. Both spellings are accepted now. 2. **The same rename silenced the rule that pulls in the rest of StreamPark's Flink jars** (`^streampark-.*_<scala>.*$`), so the whole client stack was loaded by the console's own classloader and resolved `org.apache.flink.*` from the console's fixed baseline Flink rather than the registered target version. That, I believe, is the actual mechanism behind #4483 — the `ServiceLoader` mismatch it reports is what a half-populated shims classloader looks like from the outside. 3. **`shims-base` and `shims-base-v2` share twelve class names** (v2 redeclares them for Flink 2.x, inheriting the rest) yet both were added to every shims classloader in `listFiles()` order — so which Flink version a class was compiled for was decided by the filesystem. A 1.x target no longer sees v2 at all; a 2.x target gets v2 ahead of the base. 4. **API removed in Flink 2.x.** `SavepointConfigOptions` is gone (2.x declares the same keys in `StateRecoveryOptions`), and `Configuration`'s typed accessors over a `ConfigOption` went with it. Since this module is compiled once against a single baseline but submits to whichever version the user registered, both are addressed portably: the savepoint options are declared from their keys — byte-identical across 1.17–2.3, verified against the distributions — and the generic `get`/`set` replace the typed accessors. 5. **`ClusterClient#submitJob` widened its parameter** from `JobGraph` to `ExecutionPlan` in 2.x. The instance satisfies either signature, so the call is made reflectively. Additionally, a `FLINK_SQL` program's classloader is told to resolve `org.apache.streampark.*` parent-first: the fat jar bundles whichever shims it was built against, while the parent is now the shims classloader for the version actually registered, and loading both ends in a `LinkageError` as soon as one references the other. ## Verifying this change Verified against real standalone clusters by driving StreamPark's own submission path out-of-process against the deployed console's `lib/`, so each fix could be confirmed against the actual artifacts: - A Flink SQL job **submits and reaches `FINISHED` on Flink 2.2.1** — previously impossible. - The same job on **Flink 1.20.4, which worked before this change, still does**. This is the important one: cause (3) surfaced as a regression on 1.20 while fixing (1) and (2), and was only caught because 1.x was re-tested at every step. - The classloader behaviour underlying (1)/(2) was checked directly: with the target version's jars reachable, `org.apache.flink.util.ParameterTool` resolves from `flink-dist-2.2.1.jar`; without them it fails exactly as reported. - The savepoint option keys were read out of both distributions' bytecode and are identical. ## Does this pull request potentially affect one of the following parts - Dependencies (does it add or upgrade a dependency): **no** - The public API: **no** - The runtime per-record code paths (performance sensitive): **no** — client-side submission only - Anything that affects deployment: **yes, in effect** — the shims classloader is now populated as it was designed to be, so the submission stack binds to the registered Flink version rather than the console's baseline. That is the intended behaviour, but it does change which classes a submission runs against, which is why 1.x was re-verified end to end. **Touches `FlinkShimsProxy`**, which `AGENTS.md` marks high-sensitivity. No static state is introduced; the changes are to which jars go into the classloader and in what order. **Not addressed:** `LocalClient` and `KubernetesNativeSessionClient` use the same removed `Configuration` accessors and will fail the same way on Flink 2.x. Neither is reachable in the environment this was verified in, so I have left them for a change that can be tested rather than guessed at. ## Documentation - Does this pull request introduce a new feature? **no** — it makes an advertised one (Flink 2.x support, added in `e770d2e8e`) actually work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
