shangeyao opened a new issue, #4408:
URL: https://github.com/apache/streampark/issues/4408
## Background
StreamPark was originally built with a Scala-first development model: user
applications extend `FlinkStreaming` / `FlinkTable` / `SparkStreaming` /
`SparkBatch` traits, Flink multi-version support is implemented via Scala shims
behind `FlinkShimsProxy`, and most runtime modules (`streampark-common`, Flink
client/K8s/packer/connectors, Spark client/connectors) are written in Scala.
This creates several long-term costs:
1. **Build complexity**: `scala-maven-plugin`, Scalafmt, Scalastyle,
silencer, ScalaTest, and mixed Java/Scala compilation increase CI time and
contributor onboarding friction.
2. **Ecosystem alignment**: Apache Flink 2.x removed the Scala DataStream
API; continuing Scala-centric APIs fights upstream direction.
3. **Console alignment**: `streampark-console-service` is Java/Spring Boot
but depends heavily on Scala-compiled libraries (`HadoopUtils`, `FlinkClient`,
`FlinkShimsProxy`, packer pipelines, etc.).
4. **Maintenance multiplier**: Every shims change must be replicated across
**12 Flink version modules** (1.12–1.20, plus future 2.x), most of which are
Scala.
This issue tracks the **design and execution plan** to remove Scala from the
StreamPark codebase.
---
## Goals
- [ ] Eliminate Scala source code from all StreamPark modules (including
tests).
- [ ] Remove Scala from the Maven build toolchain (`scala-maven-plugin`,
Scalafmt/Scalastyle, ScalaTest, silencer, `mockito-scala`).
- [ ] Provide **Java-first public APIs** for user application development
and SQL/streaming lifecycle.
- [ ] Preserve functional parity for Console operations
(submit/cancel/deploy, SQL validation, K8s tracking, packaging).
- [ ] Document migration path and deprecation timeline for existing
Scala-based user jobs.
## Non-Goals (initial phase)
- Rewriting third-party Flink/Spark dependencies that still ship `_2.12`
artifacts (we stop **authoring** Scala, but may still consume Flink's binary
artifacts).
- Changing frontend stack (Vue/TS) — out of scope.
- Big-bang rewrite in a single PR — migration must be phased and CI-green at
each step.
---
## Current Inventory (baseline)
| Area | Scala `.scala` files (approx.) | Notes |
|------|-------------------------------|-------|
| `streampark-common` | 61 (51 main + 10 test) | Utils, config, FS; Java
enums already exist alongside Scala |
| `streampark-flink-connector` | 76 | Sink/source helpers, often Flink Scala
API |
| `streampark-flink-shims` (+ base) | 62 | 12 version modules × ~5 files +
shared base traits |
| `streampark-flink-kubernetes` | 38 | K8s watcher, ingress; uses json4s |
| `streampark-flink-packer` | 34 | Build pipelines |
| `streampark-flink-client` | 27 | `FlinkClient`, deploy/submit clients |
| `streampark-flink-core` | 4 | **Public user API**: `FlinkStreaming`,
`FlinkTable`, `DataStreamExt` |
| `streampark-flink-proxy` | 1 | `FlinkShimsProxy` (classloader isolation) |
| `streampark-flink-sqlclient` | 1 | SQL CLI entry |
| `streampark-spark` (all) | 39 | Core traits, client, connectors, sqlclient
|
| `streampark-console` | **0** | Pure Java; consumes Scala JARs |
| **Total** | **~343** | 22 Scala test files |
### Key Scala-specific patterns in use
| Pattern | Example | Java replacement strategy |
|---------|---------|---------------------------|
| `object` singleton facades | `FlinkClient`, `FlinkShimsProxy`, `Utils` |
`final class` + static methods, or Spring `@Component` where appropriate |
| `trait` lifecycle API | `FlinkStreaming`, `FlinkTable`, `SparkStreaming` |
Abstract Java class or interface + template method |
| `implicit` conversions | `Implicits`, `EnhancerImplicit`, `DataStreamExt`
| Explicit static helper classes; no implicit magic |
| `enumeratum` enums | `SqlCommandParser` command types | Java `enum`
(already used in `common/enums`) |
| `ClassTag` / reflection | `FlinkClient` shims dispatch | Plain Java
classes + `ClassLoader` reflection (already partially used) |
| json4s | K8s REST JSON parsing | Jackson (already in dependency tree) |
| scalikejdbc | Kafka offset store | JDBC via HikariCP / MyBatis-style thin
wrapper |
| ScalaTest | common/flink tests | JUnit 5 + AssertJ (Console standard) |
### Critical dependency graph
```
streampark-console (Java)
└── streampark-common (Scala) ─────────────────┐
└── streampark-flink-client (Scala) │
└── streampark-flink-proxy / FlinkShimsProxy │
└── streampark-flink-packer (Scala) ├── all Scala today
└── streampark-flink-kubernetes (Scala) │
└── streampark-spark-client (Scala) ───────────┘
User Flink apps
└── streampark-flink-core (Scala traits)
└── Flink Scala DataStream API (removed in Flink 2.x)
```
---
## Proposed Target Architecture
### 1. User development API (breaking change, needs migration guide)
Replace Scala traits with Java lifecycle base classes:
```java
public abstract class FlinkStreamingJob {
protected StreamExecutionEnvironment env;
protected ParameterTool parameter;
public final void main(String[] args) {
init(args);
ready();
handle();
execute();
destroy();
}
protected void ready() {}
protected abstract void handle() throws Exception;
protected void destroy() {}
protected void configure(StreamExecutionEnvironment env, ParameterTool
parameter) {}
}
```
- Package: `org.apache.streampark.flink.core.java` (mirror current `scala`
package layout during transition).
- **Deprecation**: keep Scala traits as thin deprecated wrappers delegating
to Java base for **one major release**, then remove.
- For Flink 1.12–1.20: use **Java DataStream API**
(`org.apache.flink.streaming.api.environment.StreamExecutionEnvironment`) even
if Flink also ships Scala API.
- For Flink 2.x: already Java-only upstream — StreamPark Java API becomes
the canonical path.
Same pattern for `FlinkTable`, `FlinkStreamTable`, `SparkStreaming`,
`SparkBatch`.
### 2. Shims layer refactor
**Problem**: 12 near-identical Scala shims modules; `FlinkShimsProxy` is
Scala `object` with mutable cache.
**Target**:
- Define **Java interfaces** in `streampark-flink-shims-base` (e.g.
`FlinkShimsService`, `FlinkSqlValidator`, `FlinkClusterClient`).
- Each version module implements interfaces in **Java** (5 classes/module →
~60 small Java files, but mechanically similar).
- Rewrite `FlinkShimsProxy` → `FlinkShimsClassLoader` (Java) with unchanged
classloader isolation semantics (ChildFirstClassLoader, per-version cache).
**Do not introduce static state leaking across classloaders.**
- Consider consolidating duplicate shims code via a shared Java abstract
base parameterized by Flink version differences (where API is stable).
### 3. `streampark-common` → Java
Priority order (most Console callers first):
1. `util/`: `Utils`, `HadoopUtils`, `YarnUtils`, `HttpClientUtils`,
`ClassLoaderUtils`, `FileUtils`, …
2. `conf/`: `FlinkVersion`, `SparkVersion`, `CommonConfig`, `ConfigKeys`
wrappers
3. `fs/`: `FsOperator`, `HdfsOperator`, `LfsOperator`
4. Remove `Implicits.scala` — replace call sites with plain Java collections
/ `AutoCloseable` try-with-resources.
`FlinkVersion` already exposes Java-friendly methods via `@BeanProperty` /
explicit getters — migrate to pure Java POJO.
### 4. Flink runtime modules
| Module | Migration notes |
|--------|-----------------|
| `streampark-flink-client` | Convert `SubmitRequest`/`SubmitResponse` beans
to Java records or Lombok `@Data`; `FlinkClient` → Java facade |
| `streampark-flink-kubernetes` | Replace json4s with Jackson; convert
watchers/controllers to Java |
| `streampark-flink-packer` | Pipeline pattern maps cleanly to Java
interfaces |
| `streampark-flink-connector-*` | Largest bulk; many sinks use Flink Scala
`DataStream` — rewrite against Java `DataStream` API |
| `streampark-flink-sqlclient` | Java `main` entry |
### 5. Spark modules (`-Pspark`)
Same strategy as Flink client/core/connectors. Spark still supports Scala
API upstream, but StreamPark should expose Java lifecycle only.
### 6. Build & CI cleanup
Remove from root `pom.xml` and module POMs:
- `scala.version`, `scala-maven-plugin`, `maven-scalastyle-plugin`
- Spotless Scalafmt config (`tools/checkstyle/.scalafmt.conf`)
- `scalatest`, `mockito-scala`, `silencer-plugin`
- `scala-library` / `scala-compiler` dependency management entries
Update `AGENTS.md` coding conventions (Scala section → Java-only).
---
## Phased Execution Plan
### Phase 0 — Design sign-off & compatibility policy (this issue)
- [ ] PMC/committer agreement on breaking API change policy
- [ ] Define deprecation window (proposal: **2 minor releases** for Scala
user API)
- [ ] Publish user migration guide skeleton in docs
### Phase 1 — Foundation (`streampark-common` → Java)
- [ ] Convert `util/*` and `conf/*` to Java with parity tests (JUnit 5)
- [ ] Keep Scala modules delegating to Java via thin wrappers (temporary)
**or** use dual compilation bridge
- [ ] CI green; Console integration tests pass
**Estimated scope**: ~60 files, **1–2 PRs**
### Phase 2 — Proxy & client facades
- [ ] `FlinkShimsProxy` → Java
- [ ] `FlinkClient` / request-response beans → Java
- [ ] `SparkShimsProxy`, `SparkClient` → Java
- [ ] Console services unchanged at source level (same public Java API)
**Estimated scope**: ~30 files, **2–3 PRs**
### Phase 3 — Shims base + one pilot version (e.g. 1.20)
- [ ] Java interfaces in shims-base
- [ ] Migrate `streampark-flink-shims_flink-1.20` to Java
- [ ] Validate submit/cancel/SQL verify against 1.20 cluster
- [ ] Document mechanical migration template for remaining versions
**Estimated scope**: **1 PR per Flink version** (11 remaining) — can
parallelize
### Phase 4 — User-facing `streampark-flink-core` / `streampark-spark-core`
- [ ] Introduce Java lifecycle base classes
- [ ] Deprecate Scala traits (annotation + docs)
- [ ] Update archetypes/examples
**Breaking change release note required**
### Phase 5 — K8s, packer, connectors
- [ ] `streampark-flink-kubernetes` (json4s → Jackson)
- [ ] `streampark-flink-packer`
- [ ] `streampark-flink-connector-*` (largest effort; batch by connector)
- [ ] Spark connectors + sqlclient
**Estimated scope**: ~150 files, **multiple PRs**
### Phase 6 — Remove Scala completely
- [ ] Delete remaining `.scala` files and Scala test sources
- [ ] Remove Scala build plugins/deps
- [ ] Remove Scalafmt/Scalastyle from Spotless/CI
- [ ] Final grep: zero `.scala` in repo
---
## Risks & Mitigations
| Risk | Impact | Mitigation |
|------|--------|------------|
| Breaking user Scala jobs | High | Deprecation period; migration guide;
compat module optional |
| Shims classloader regressions | High | Dedicated integration tests per
Flink version; no change to cache key semantics |
| Connector rewrite bugs | Medium | Port tests first; one connector per PR |
| Large PR fatigue | Medium | Strict phase gates; module-by-module |
| Flink 1.12–1.19 Java API gaps | Medium | Use Flink Java DataStream API;
shims hide version diffs |
| json4s / scalikejdbc removal | Low | Jackson + JDBC already available |
---
## Acceptance Criteria
1. `./mvnw clean install -DskipTests` succeeds **without**
`scala-maven-plugin`.
2. `./mvnw clean install` full test suite passes (Java tests only).
3. Console can submit/cancel Flink & Spark apps on representative deploy
modes (YARN, K8s session/application).
4. SQL validation works for supported Flink versions.
5. No `.scala` files under `src/main` or `src/test`.
6. User migration guide published on streampark.apache.org.
---
## Suggested Sub-Issues (to be created after sign-off)
1. `[Common] Migrate streampark-common utilities to Java`
2. `[Flink] Rewrite FlinkShimsProxy and FlinkClient in Java`
3. `[Flink] Java shims template + Flink 1.20 pilot`
4. `[Flink] Migrate remaining Flink shims versions (1.12–1.19, 2.x)`
5. `[Flink] Java user API for FlinkStreaming/FlinkTable + deprecation`
6. `[Flink] Migrate kubernetes module (json4s → Jackson)`
7. `[Flink] Migrate packer module to Java`
8. `[Flink] Migrate connector modules to Java (track per connector)`
9. `[Spark] Migrate Spark modules to Java`
10. `[Build] Remove Scala toolchain and update AGENTS.md`
---
## Open Questions
1. **Deprecation timeline**: one or two minor releases for Scala user API?
2. **Flink version support window**: migrate all 1.12–1.20 shims, or drop
EOL versions first to reduce work?
3. **Optional compat artifact**: ship `streampark-flink-core-scala`
deprecated module for one release?
4. **Spark profile**: migrate Spark in same epic or separate track?
5. **JDK baseline**: stay on JDK 8 for Console, or raise baseline as part of
this effort?
---
## References
- Flink 2.x removes Scala DataStream API (StreamPark Flink 2.x work should
be Java-first)
- `FlinkShimsProxy` / `ChildFirstClassLoader` — high sensitivity (see
AGENTS.md)
- `FlinkStreaming` lifecycle contract — must preserve execution order
(`main` → `init` → `ready` → `handle` → `destroy`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
