shangeyao opened a new issue, #4421:
URL: https://github.com/apache/streampark/issues/4421
## Background
StreamPark currently ships **nine Flink shims modules**
(`streampark-flink-shims_flink-1.12` through `1.20`), each isolating
version-specific Flink API differences behind `FlinkShimsProxy`. This matrix
creates ongoing maintenance cost:
1. **EOL versions**: Flink 1.12–1.16 are end-of-life upstream; keeping shims
and E2E coverage for them slows 3.0 delivery.
2. **Flink 2.x gap**: Apache Flink 2.x is Java-only (Scala DataStream API
removed). StreamPark 3.0 removes Scala (#4408) and must add dedicated 2.x shims.
3. **Artifact naming**: Flink 2.x distributions use `flink-dist-2.x.jar` (no
`_2.12` suffix), requiring updated version detection and shims loading logic.
4. **Shims architecture split**: Flink 1.x and 2.x API surfaces diverge
enough that a shared `shims-base` is insufficient — a separate `shims-base-v2`
is needed.
This issue tracks the **design and execution plan** for Flink version
baseline changes in StreamPark 3.0.
**Implementation PR:** #4419 — `[Flink] Drop 1.12–1.16 shims, baseline
1.17+, add 2.0/2.1/2.2 support`
**Depends on:** #4418 (Remove Scala) → #4417 (JDK 11 baseline)
**Parent roadmap:** #4410
---
## Goals
- [ ] Drop Flink **1.12–1.16** shims modules and all related references
(assembly, UI, E2E).
- [ ] Retain Flink **1.17, 1.18, 1.19, 1.20** shims (Java, post–Remove
Scala).
- [ ] Add Flink **2.0, 2.1, 2.2** support via new
`streampark-flink-shims-base-v2` and thin version modules.
- [ ] Update `FlinkVersion`, `FlinkShimsProxy`, `EnvInitializer`, console
packaging, and frontend env registration.
- [ ] Document supported Flink versions and upgrade path for 3.0.
## Non-Goals
- Supporting Flink 1.12–1.16 in 3.0 (users must upgrade clusters to 1.17+).
- Migrating backend code to Java — handled by #4408 / #4418.
- Splitting version work into per-version micro-PRs — **one consolidated
PR** (#4419).
---
## Current Baseline (2.x)
| Area | Today |
|------|-------|
| Shims modules | 1.12, 1.13, 1.14, 1.15, 1.16, 1.17, 1.18, 1.19, 1.20 |
| Shims language | Scala (pre–#4418) → Java (post–#4418) |
| Flink 2.x | Not supported |
| Default `streampark.flink.shims.version` | 1.14 |
| Version detection | `flink-dist_*_2.12*.jar`, `bin/flink --version` |
---
## 3.0 Target State
| Area | Target |
|------|--------|
| **Supported 1.x** | 1.17, 1.18, 1.19, 1.20 |
| **Supported 2.x** | 2.0.2, 2.1.2, 2.2.1 (patch versions follow upstream) |
| **Dropped** | 1.12, 1.13, 1.14, 1.15, 1.16 |
| Shims layout | `shims-base` (1.x) + `shims-base-v2` (2.x) + thin
per-version modules |
| Artifact suffix | No `_2.12` suffix for 2.x shims jars |
| Default shims version | 1.17 |
---
## Proposed Architecture
### Shims module layout
```
streampark-flink-shims/
├── streampark-flink-shims-base/ # 1.x shared interfaces (existing,
Java)
├── streampark-flink-shims-base-v2/ # 2.x shared interfaces (new)
├── streampark-flink-shims_flink-1.17/
├── streampark-flink-shims_flink-1.18/
├── streampark-flink-shims_flink-1.19/
├── streampark-flink-shims_flink-1.20/
├── streampark-flink-shims_flink-2.0/
├── streampark-flink-shims_flink-2.1/
└── streampark-flink-shims_flink-2.2/
```
### Key code changes (#4419)
| Component | Change |
|-----------|--------|
| `FlinkVersion` | Support `flink-dist-2.x.jar` naming; validate 1.17+ and
2.x only |
| `FlinkShimsProxy` | Route 2.x major versions to `shims-base-v2`
classloader path |
| `EnvInitializer` | Copy 1.17–1.20 and 2.0–2.2 shims jars into dist `lib/` |
| `console-service/pom.xml` | Assembly deps for supported shims only |
| Frontend `flinkHome.ts` | Remove 1.12–1.16 from env registration hints |
| E2E | Remove legacy Flink version test cases |
### FlinkShimsProxy routing (conceptual)
```
majorVersion = flinkVersion.majorVersion()
if majorVersion >= 2.0:
load shims-base-v2 + shims_flink-{major}.{minor}
else:
load shims-base + shims_flink-{major}.{minor}
```
**High sensitivity:** preserve `ChildFirstClassLoader` isolation and
per-version cache semantics (see AGENTS.md).
---
## Execution Plan
Delivered as **one PR** (#4419) on branch `feature/flink-1.17-plus`:
1. Delete shims modules 1.12–1.16 and update `shims/pom.xml`
2. Add `shims-base-v2` and 2.0/2.1/2.2 modules
3. Update version detection, proxy routing, console packaging, frontend, E2E
4. CI green on JDK 11 + Java codebase (#4418 merged first)
**Can run in parallel with:** #4420 (Spark version track) after #4418 merges.
---
## Risks & Mitigations
| Risk | Impact | Mitigation |
|------|--------|------------|
| Users on Flink 1.12–1.16 | High | 3.0 release notes; upgrade guide to
1.17+ |
| Flink 2.x API differences | Medium | Separate `shims-base-v2`; integration
tests on 2.0/2.1/2.2 |
| Shims classloader regressions | High | Submit/cancel/SQL verify per
supported version |
| Incomplete E2E coverage for 2.x | Medium | Add 2.x E2E case in follow-up
if not in #4419 |
---
## Acceptance Criteria
1. `./mvnw clean compile -Pfast -pl
streampark-flink/streampark-flink-shims,streampark-console/streampark-console-service
-am` succeeds on JDK 11.
2. Console registers Flink 1.17+ and 2.x homes; version detection correct.
3. Submit/cancel/SQL validation works on representative 1.20 and 2.x
clusters.
4. No shims modules or references for Flink 1.12–1.16 remain.
5. Supported version matrix documented in upgrade guide.
---
## Open Questions
1. ~~Include Flink 2.x in 3.0 or 3.1?~~ → **Included in 3.0 (#4419)**
2. Exact patch pins for 2.0/2.1/2.2 shims — follow upstream release tags?
3. Flink 1.17 job JDK: document minimum job-side JDK per version?
---
## References
- Parent roadmap: #4410
- Remove Scala: #4408 / #4418
- JDK 11 baseline: #4409 / #4417
- Implementation PR: #4419
- `FlinkShimsProxy` / `ChildFirstClassLoader` — high sensitivity (AGENTS.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
