Thank you for participating in the pyflink discussion 1.venv.zip is related to the flink version, and PyFlink dependencies need to be installed when venv.zip is packaged, such as: pip install "apache-flink==1.16.2 ", then specify the venv.zip environment through "python.archives" and "python.client.executable" and other parameters. A flink version needs to be packaged with a venv.zip, and each time you submit a python task you only need to upload the.py file, there is no need to repackage
2. How does pyflink support flink connector, which I am still developing and researching ---- Replied Message ---- | From | Huajie Wang<[email protected]> | | Date | 08/23/2023 21:16 | | To | [email protected] | | Cc | | | Subject | Re: StreamPark supports pyflink | 1. Thanks for your proposal. This is gread feature, Many users are using pyflink, which is also a must-have capability in strampark. However, I have a few small questions: 1. Does this venv(python environment) need to be bundled with each job? If so, the dependency packages for the job will be large. 2. 2. How does pyflink support dependencies like flink connectors? Best, Huajie Wang 程杰 <[email protected]> 于2023年8月23日周三 13:59写道： > Hi everyone, I'd like to start a discussion on streampark support for > pyflink > > Reference: > GitHub issue: https://github.com/apache/incubator-streampark/issues/2437 > GitHub PR: https://github.com/apache/incubator-streampark/pull/2956 > > streampark pyflink currently supports yarn, k8s will be supported in the > future > 1. Before using pyflink, users need to compile venv.zip and upload it to > hdfs and $WORKSPACE/python(whether this can be optimized, directly compile > venv.zip and upload it to streampark directory or other solutions), please > refer to the document: > https://github.com/apache/incubator-streampark/pull/2956/files#diff-abc9ac8d84878215b2c86971e3320fbe0a7333a9a6dde9eb82b6 > 583393792b40 > > 2.yarn application mode can be extended by adding PythonOptions > parameters, such as' python.archives' and 'python.client.executable'. > > 3.yarn perjob mode can be extended by modifying getJobGraph to generate > different Jobgraphs based on whether python or jar is uploaded > > >
