hi: This is a great proposal, I've reviewed the PR, overall it looks good, but there are some details that need further discussion. Could you provide the relevant architectural design? How do you plan to complete this major feature in several steps ? Are there corresponding test cases? I suggest you can create an umbrella to tracking these issues, e.g: https://github.com/apache/incubator-streampark/issues/2879
BTW, I will soon initiate a vote about StreamPark support Apache Spark job development and management functionality. Looking forward to hearing everyone's opinions. Let's discuss and work together to complete this feature. Best, Huajie Wang 程杰 <[email protected]> 于2024年2月18日周日 17:09写道： > Hi everyone, I'd like to start a discussion on streampark support submit > spark task to yarn > > Reference: > GitHub PR: <https://github.com/apache/incubator-streampark/pull/3537> > https://github.com/apache/incubator-streampark/pull/3537 > > Currently completed functions: > 1.streampark-console module adds spark related interfaces to support spark > task submit, spark environment creation, and Release application building > pipeline.These interfaces are new and will not affect previous > Flink-related interfaces > > 2.Added streampark-spark-client module to support spark jar task submit > > 3.Modified the streampark-flink-packer module to support the Building > pipeline for spark yarn application mode.The subsequent plan is to extract > this module from flink > > 4.`t_spark_env` and `t_spark_app` are added to the database > > > Features under development： > 1.StreamPark support spark sql task submit > 2.spark task status tracking > > Sample task submission > > > > > > > > > > > >
