shangeyao opened a new issue, #4410:
URL: https://github.com/apache/streampark/issues/4410
## Summary
StreamPark **3.0** will ship two breaking infrastructure changes in a
**single release**:
1. **Remove Scala** from the codebase (#4408)
2. **Raise Console baseline to JDK 11** (#4409)
This issue is the **umbrella execution plan**. It defines PR slicing rules
so each change stays small and reviewable.
Related design issues:
- #4408 — Remove Scala design
- #4409 — JDK 8 → 11 design
---
## PR Sizing Rules (mandatory)
Each PR must satisfy **all** of the following:
| Rule | Guideline |
|------|-----------|
| **Single concern** | One module layer, one package, one Flink shims
version, or one connector — not multiple |
| **Diff size** | Target **≤ 500 LOC** changed; hard cap **~800 LOC**
(excluding generated/moved files) |
| **No drive-by** | No unrelated formatting, import reorder, or refactors
outside the stated scope |
| **Tests co-located** | ScalaTest → JUnit / parity tests land in the **same
PR** as the migration |
| **No mixed breaking axes** | Do **not** combine "Scala → Java" and "JDK
compile bump" in one PR unless the module is trivial (<10 files) |
| **CI green** | Each PR must pass existing CI; JDK 8 matrix may still pass
until Phase F |
| **Title format** | `[Module] Imperative description` per project
convention |
If a PR exceeds the cap, split by sub-package (e.g. `common/util` part 1 /
part 2).
---
## Release Strategy: Two Tracks, One Finish Line
```
Phase A–C ──► JDK 11 runtime ready (Console runs on 11, still compiles to 8)
Phase D–H ──► Scala → Java (module by module)
Phase F ──► Compile baseline 11 + remove Scala toolchain + Docker/CI
Phase G ──► 3.0 docs + release notes
```
**Key idea:** get Console **running** on JDK 11 early (low risk), migrate
code to Java in small PRs, flip compile/toolchain **once at the end** when no
Scala remains.
---
## PR Checklist (ordered by merge dependency)
> Check off in order. Items marked `(parallel)` can proceed concurrently
after their dependency is merged.
### Phase A — JDK 11 runtime readiness (no compile bump, no Scala removal)
| # | PR title | Scope | ~LOC | Depends on |
|---|----------|-------|------|------------|
| A1 | `[Console] Enable JDK 11 JVM opts and fix startup classpath` |
`jvm_opts.sh`, `streampark.sh` — enable `--add-opens`, remove `jre/lib`, update
GC logging | 80 | — |
| A2 | `[Console] Add explicit javax.annotation-api for JDK 11` |
`console-service/pom.xml` only | 20 | — |
| A3 | `[Common] Fix ClassLoaderUtils ucp lookup for JDK 9+ module system` |
`ClassLoaderUtils` only (Java rewrite or ucp walk fix) | 150 | — |
| A4 | `[Docs] Add StreamPark 3.0 upgrade guide skeleton` | docs only:
Console JDK 11 vs job JDK matrix | 100 | — |
**Exit gate:** Console starts on JDK 11, existing CI still green.
---
### Phase B — `streampark-common` → Java (foundation, blocks everything)
| # | PR title | Scope | ~LOC | Depends on |
|---|----------|-------|------|------------|
| B1 | `[Common] Migrate conf classes to Java` | `FlinkVersion`,
`SparkVersion`, `CommonConfig`, `ConfigKeys`, `Workspace`, … | 400 | A3 |
| B2 | `[Common] Migrate util helpers to Java (part 1)` | `Utils`,
`FileUtils`, `PropertiesUtils`, `JsonUtils`, `DateUtils`, `AssertUtils` | 500 |
B1 |
| B3 | `[Common] Migrate util helpers to Java (part 2)` | `HadoopUtils`,
`YarnUtils`, `HttpClientUtils`, `CommandUtils` | 500 | B2 |
| B4 | `[Common] Migrate util helpers to Java (part 3)` |
`ClassLoaderUtils`, `FlinkConfigurationUtils`, `SparkConfigurationUtils`,
`DeflaterUtils` | 400 | A3, B3 |
| B5 | `[Common] Migrate fs package to Java` | `FsOperator`, `HdfsOperator`,
`LfsOperator` | 300 | B1 |
| B6 | `[Common] Remove Scala sources and tests from streampark-common` |
Delete `.scala`, migrate remaining ScalaTest → JUnit, drop scala deps from
module POM | 200 | B4, B5 |
**Exit gate:** `streampark-common` has zero `.scala`; Console still compiles.
---
### Phase C — Flink proxy & client facades
| # | PR title | Scope | ~LOC | Depends on |
|---|----------|-------|------|------------|
| C1 | `[Flink] Rewrite FlinkShimsProxy in Java` | `streampark-flink-proxy`
| 250 | B6 |
| C2 | `[Flink] Migrate Flink client beans to Java` | `SubmitRequest`,
`CancelRequest`, responses, … | 400 | B6 |
| C3 | `[Flink] Rewrite FlinkClient facade in Java` | `FlinkClient`,
entrypoint wiring | 200 | C1, C2 |
| C4 | `[Flink] Migrate shims-base core traits to Java interfaces` |
`FlinkClientTrait`, `FlinkTableTrait`, initializers, `SqlCommandParser` | 500 |
C1 |
---
### Phase D — Flink shims (one version per PR)
| # | PR title | Scope | ~LOC | Depends on |
|---|----------|-------|------|------------|
| D1 | `[Flink] Migrate shims_flink-1.20 to Java` | 5 shims classes + test |
300 | C4 |
| D2 | `[Flink] Migrate shims_flink-1.19 to Java` | (parallel after D1
template) | 300 | D1 |
| D3 | `[Flink] Migrate shims_flink-1.18 to Java` | (parallel) | 300 | D1 |
| D4 | `[Flink] Migrate shims_flink-1.17 to Java` | (parallel) | 300 | D1 |
| D5 | `[Flink] Migrate shims_flink-1.16 to Java` | (parallel) | 300 | D1 |
| D6 | `[Flink] Migrate shims_flink-1.15 to Java` | (parallel) | 300 | D1 |
| D7 | `[Flink] Migrate shims_flink-1.14 to Java` | (parallel) | 300 | D1 |
| D8 | `[Flink] Migrate shims_flink-1.13 to Java` | (parallel) | 300 | D1 |
| D9 | `[Flink] Migrate shims_flink-1.12 to Java` | (parallel) | 300 | D1 |
| D10 | `[Flink] Migrate shims_flink-2.x to Java` | 2.0 / 2.1 / 2.2 (one PR
if identical structure) | 400 | D1 |
**Rule:** do not batch multiple Flink versions in one PR unless diff is
literally identical copy-paste.
---
### Phase E — Flink user API & runtime modules
| # | PR title | Scope | ~LOC | Depends on |
|---|----------|-------|------|------------|
| E1 | `[Flink] Add Java user API for FlinkStreaming and FlinkTable` | New
Java base classes + deprecate Scala traits | 400 | C4 |
| E2 | `[Flink] Remove deprecated Scala user API from flink-core` | Delete
scala traits after E1 merged + doc | 100 | E1, **3.0 only** |
| E3 | `[Flink] Migrate flink-client-core to Java` | Yarn/Remote/K8s clients
| 500 | C3 |
| E4 | `[Flink] Migrate flink-kubernetes to Java (part 1)` | json4s →
Jackson, model classes | 400 | B6 |
| E5 | `[Flink] Migrate flink-kubernetes to Java (part 2)` | watchers,
ingress, controller | 500 | E4 |
| E6 | `[Flink] Migrate flink-packer to Java` | pipeline + docker templates
| 500 | B6 |
| E7 | `[Flink] Migrate flink-sqlclient to Java` | CLI entry | 100 | C4 |
---
### Phase F — Flink connectors (one connector per PR)
| # | PR title | Scope | ~LOC | Depends on |
|---|----------|-------|------|------------|
| F1 | `[Flink] Migrate connector-kafka to Java` | kafka module only | 400 |
B6 |
| F2 | `[Flink] Migrate connector-jdbc to Java` | jdbc module only | 400 |
B6 |
| F3 | `[Flink] Migrate connector-redis to Java` | | 300 | B6 |
| F4 | `[Flink] Migrate connector-mongo to Java` | | 300 | B6 |
| F5 | `[Flink] Migrate connector-clickhouse to Java` | | 300 | B6 |
| F6 | `[Flink] Migrate connector-doris to Java` | | 300 | B6 |
| F7 | `[Flink] Migrate connector-hbase to Java` | | 400 | B6 |
| F8 | `[Flink] Migrate connector-http to Java` | | 300 | B6 |
| F9 | `[Flink] Migrate connector-influx to Java` | | 200 | B6 |
| F10 | `[Flink] Migrate connector-elasticsearch to Java` | all ES
sub-modules in one PR if shared code | 500 | B6 |
---
### Phase G — Spark modules (`-Pspark`, parallel after B6)
| # | PR title | Scope | ~LOC | Depends on |
|---|----------|-------|------|------------|
| G1 | `[Spark] Migrate spark-client to Java` | proxy + client + beans | 400
| B6 |
| G2 | `[Spark] Add Java user API and remove SparkStreaming/Batch Scala
traits` | spark-core | 300 | G1 |
| G3 | `[Spark] Migrate spark-connector-kafka to Java` | | 400 | B6 |
| G4 | `[Spark] Migrate spark-connector-base and remaining connectors to
Java` | split if >800 LOC | 500 | B6 |
| G5 | `[Spark] Migrate spark-sqlclient to Java` | | 100 | G1 |
---
### Phase H — Toolchain & distribution (3.0 finalization)
| # | PR title | Scope | ~LOC | Depends on |
|---|----------|-------|------|------------|
| H1 | `[Build] Bump project.build.jdk to 11` | root `pom.xml`, Spotless
formatter, module compiler overrides | 150 | B6, all Scala removed |
| H2 | `[Build] Remove Scala Maven toolchain` | drop scala-maven-plugin,
scalatest, scalafmt, silencer | 100 | H1 |
| H3 | `[Docker] Migrate official image to JDK 11` | `docker/Dockerfile`,
compose | 50 | A1 |
| H4 | `[CI] Make JDK 11 primary and remove JDK 8 matrix` | `backend.yml`,
`unit-test.yml`, `e2e.yml`, `docker-push.yml` | 80 | H1, H3 |
| H5 | `[Docs] StreamPark 3.0 release notes and migration guide` | breaking
changes, Scala API removal, JDK 11 | 200 | E2, H4 |
| H6 | `[Build] Update AGENTS.md for Java-only conventions` | remove Scala
section | 50 | H2 |
**Exit gate:** zero `.scala` files, `./mvnw -Pshaded,webapp,dist clean
install` on JDK 11 only, Docker on JDK 11.
---
## Optional / Follow-up (not blocking 3.0)
| PR | Scope |
|----|-------|
| `[Console] Upgrade Pac4j for JDK 11` | SSO regression test required |
| `[Console] Frontend refactor` | independent of 3.0 backend work |
---
## Parallelization Map
After **B6** merges, these tracks proceed in parallel:
```
┌── Phase D (shims 1.12–2.x, one PR each)
├── Phase E (flink k8s / packer / client-core)
├── Phase F (connectors, one each)
└── Phase G (Spark)
↓
Phase H (toolchain — serial, last)
```
Within Phase D, up to **4 shims PRs** can be in flight simultaneously
(different modules, no conflict).
---
## What NOT to put in one PR
- ❌ `streampark-common` entire module (split B1–B6)
- ❌ Multiple Flink shims versions
- ❌ Multiple connectors
- ❌ Scala removal + JDK 11 compile bump + Docker change
- ❌ User API breaking change + unrelated Console RBAC fix
- ❌ Flink kubernetes + packer together
---
## 3.0 Breaking Changes (release note summary)
1. **JDK 11+ required** to run StreamPark Console.
2. **Scala development API removed** — use Java `FlinkStreamingJob` /
`FlinkTableJob` (names TBD in E1).
3. Flink/Spark **job JDK** still configured per environment (not forced to
11).
4. Docker `latest` image based on JDK 11.
---
## Estimated PR count
| Phase | PRs |
|-------|-----|
| A | 4 |
| B | 6 |
| C | 4 |
| D | 10 |
| E | 7 |
| F | 10 |
| G | 5 |
| H | 6 |
| **Total** | **~52 PRs** |
At ~500 LOC each, reviewers can typically review one PR in 15–30 minutes.
---
## Suggested GitHub Milestone
Create milestone **`3.0`** and link all sub-issues/PRs. Use labels:
- `3.0-jdk11`
- `3.0-no-scala`
- `good-first-3.0` (for isolated shims/connector PRs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
