GitHub user pykenny added a comment to the discussion: Separate Python virtual environments for each DAG(not Task)
I read the [slides about remote worker](https://airflowsummit.org/sessions/2024/running-airflow-tasks-anywhere-in-any-language/) (or edge worker, to be more specifically) introduced on last year's Airflow Summit and feel that it could possibly be the solution when the feature becomes fully available. Edge executor still need to run on an environment with the same Python+Airflow version as the core service does, but at runner level it won't matter as long as it supports the task execution protocol (using compatible Task SDK to communicate with the ). Unfortunately at the time of writing, Airflow officials haven't release any SDK that supports language outside of Python (Golang, JavaScript, TypeScript, etc.) yet, and current Python Task SDK has to take care of [backward compatibility of existing codebase](https://airflow.apache.org/blog/airflow-three-point-oh-is-here/#:~:text=The%20Airflow%203%20release%20includes%20the%20Python%20TaskSDK%20which%20enables%20backward%20compatibility%20for%20existing%20DAGs.%20TaskSDKs%20for%20additional%20languages%2C%20starting%20with%20Golang%20will%20be%20released%20over%20the%20next%20few%20months.), meaning it's [heavily tied to the core system itself](https://github.com/apache/airflow/blob/35186bccf3da10d5d01b744966be8156721be5c7/task-sdk/pyproject.toml#L49) (runner using Python SDK should run in an environment with Airflow installed), and doesn't offer the same flexibility as demonstrated in the summit. GitHub link: https://github.com/apache/airflow/discussions/40651#discussioncomment-13072576 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
