This is an automated email from the ASF dual-hosted git repository. pierrejeambrun pushed a commit to branch v2-5-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a18a73e6e294f02990a4e383a5a0507792e07534 Author: Rafael Carrasco <[email protected]> AuthorDate: Wed Mar 15 19:06:35 2023 -0500 Adding more information regarding top level code (#30040) --------- Co-authored-by: Tzu-ping Chung <[email protected]> (cherry picked from commit f1e40cf799c5ae73ec6f7991efe604f2088d8622) --- docs/apache-airflow/best-practices.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/apache-airflow/best-practices.rst b/docs/apache-airflow/best-practices.rst index b29cab34b4..01bf6d6ef5 100644 --- a/docs/apache-airflow/best-practices.rst +++ b/docs/apache-airflow/best-practices.rst @@ -166,7 +166,7 @@ Good example: @task() def print_array(): """Print Numpy array.""" - import numpy as np # <- THIS IS HOW NUMPY SHOULD BE IMPORTED IN THIS CASE + import numpy as np # <- THIS IS HOW NUMPY SHOULD BE IMPORTED IN THIS CASE! a = np.arange(15).reshape(3, 5) print(a) @@ -174,6 +174,8 @@ Good example: print_array() +In the Bad example, NumPy is imported each time the DAG file is parsed, which will result in suboptimal performance in the DAG file processing. In the Good example, NumPy is only imported when the task is running. + Dynamic DAG Generation ---------------------- Sometimes writing DAGs manually isn't practical. @@ -213,7 +215,7 @@ or if you need to deserialize a json object from the variable : {{ var.json.<variable_name> }} -Make sure to use variable with template in operator, not in the top level code. +In top-level code, variables using jinja templates do not produce a request until a task is running, whereas, ``Variable.get()`` produces a request every time the dag file is parsed by the scheduler. Using ``Variable.get()`` will lead to suboptimal performance in the dag file processing. In some cases this can cause the dag file to timeout before it is fully parsed. Bad example:
