1. Thanks for your proposal. This is gread feature, Many users are using pyflink, which is also a must-have capability in strampark. However, I have a few small questions:
1. Does this venv(python environment) need to be bundled with each job? If so, the dependency packages for the job will be large. 2. 2. How does pyflink support dependencies like flink connectors? Best, Huajie Wang 程杰 <[email protected]> 于2023年8月23日周三 13:59写道： > Hi everyone, I'd like to start a discussion on streampark support for > pyflink > > Reference: > GitHub issue: https://github.com/apache/incubator-streampark/issues/2437 > GitHub PR: https://github.com/apache/incubator-streampark/pull/2956 > > streampark pyflink currently supports yarn, k8s will be supported in the > future > 1. Before using pyflink, users need to compile venv.zip and upload it to > hdfs and $WORKSPACE/python(whether this can be optimized, directly compile > venv.zip and upload it to streampark directory or other solutions), please > refer to the document: > https://github.com/apache/incubator-streampark/pull/2956/files#diff-abc9ac8d84878215b2c86971e3320fbe0a7333a9a6dde9eb82b6 > 583393792b40 > > 2.yarn application mode can be extended by adding PythonOptions > parameters, such as' python.archives' and 'python.client.executable'. > > 3.yarn perjob mode can be extended by modifying getJobGraph to generate > different Jobgraphs based on whether python or jar is uploaded > > >
