This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 1b592527429 Improve dag bundle documentation (#49342)
1b592527429 is described below
commit 1b592527429a42e89aa9111ffb1e85459e0bbdf5
Author: Tzu-ping Chung <[email protected]>
AuthorDate: Wed Apr 16 17:50:32 2025 +0800
Improve dag bundle documentation (#49342)
I was reading the doc and noticed the introduction part has some
duplicated descriptions on the feature, so I cleaned up and rearranged
it a bit.
Also tweaked the intentation a bit so it reads better.
---
.../administration-and-deployment/dag-bundles.rst | 52 +++++++++++-----------
.../src/airflow/config_templates/config.yml | 21 ++++-----
2 files changed, 38 insertions(+), 35 deletions(-)
diff --git a/airflow-core/docs/administration-and-deployment/dag-bundles.rst
b/airflow-core/docs/administration-and-deployment/dag-bundles.rst
index 27b21daa5ed..7dc03ebab14 100644
--- a/airflow-core/docs/administration-and-deployment/dag-bundles.rst
+++ b/airflow-core/docs/administration-and-deployment/dag-bundles.rst
@@ -18,20 +18,22 @@
Dag Bundles
===========
-Dag bundle are a collection of dags and other files (think the Airflow 2 dags
folder). Unlike Airflow 2, where dags were required to be on local disk and
getting
-the dags there was the sole responsibility of the deployment manager, Airflow
3 is now able to pull dags from external systems
-as well. And since dag bundles support versioning, it also allows Airflow to
run a task using a specific version
-of the dag bundle, allowing for a dag run to use the same code for the whole
run, even if the dag is updated mid way through the run.
-
-What's in a dag bundle? One or more dag files along with their associated
files, such as
-other Python scripts, configuration files, or other resources. By keeping the
bundle at a higher level, it allows for versioning
-everything the dag needs to run.
-
-Dag bundles can source the dags from various locations, such as local
directories, Git repositories, or other external systems.
-Deployment administrators can also write their own dag bundle classes to
support custom sources.
-You can also define more than 1 dag bundle in an Airflow deployments, allowing
for better organization of your dags.
-
-Why Are dag bundles important?
+A dag bundle is a collection of one or more dags, files along with their
associated files, such as other
+Python scripts, configuration files, or other resources. Dag bundles can
source the dags from various
+locations, such as local directories, Git repositories, or other external
systems. Deployment administrators
+can also write their own dag bundle classes to support custom sources. You can
also define more than one dag
+bundle in an Airflow deployments, allowing for better organization of your
dags. By keeping the bundle at a
+higher level, it allows for versioning everything the dag needs to run.
+
+This is similar, but more powerful than the *dags folder* in Airflow 2 or
earlier, where dags were required to
+be in one place on the local disk, and getting the dags there was solely the
responsibility of the deployment
+manager.
+
+Since dag bundles support versioning, they also allow Airflow to run a task
using a specific version of the
+dag bundle, allowing for a dag run to use the same code for the whole run,
even if the dag is updated mid-way
+through the run.
+
+Why are dag bundles important?
------------------------------
- **Version Control**: By supporting versioning, dag bundles allow dag runs to
use the same code for the whole run, even if the dag is updated mid way through
the run.
@@ -61,17 +63,17 @@ For example, adding multiple dag bundles to your
``airflow.cfg`` file:
[dag_processor]
dag_bundle_config_list = [
- {
- "name": "my_git_repo",
- "classpath": "airflow.dag_processing.bundles.git.GitDagBundle",
- "kwargs": {"tracking_ref": "main", "git_conn_id":
"my_git_conn"}
- }
- {
- "name": "dags-folder",
- "classpath":
"airflow.dag_processing.bundles.local.LocalDagBundle",
- "kwargs": {}
- }
- ]
+ {
+ "name": "my_git_repo",
+ "classpath": "airflow.dag_processing.bundles.git.GitDagBundle",
+ "kwargs": {"tracking_ref": "main", "git_conn_id": "my_git_conn"}
+ },
+ {
+ "name": "dags-folder",
+ "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle",
+ "kwargs": {}
+ }
+ ]
.. note::
diff --git a/airflow-core/src/airflow/config_templates/config.yml
b/airflow-core/src/airflow/config_templates/config.yml
index 604b6bcdcfb..1b71e5ae019 100644
--- a/airflow-core/src/airflow/config_templates/config.yml
+++ b/airflow-core/src/airflow/config_templates/config.yml
@@ -2456,22 +2456,23 @@ dag_processor:
type: string
example: >
[
- {
+ {
"name": "my-git-repo",
"classpath": "airflow.providers.git.bundles.git.GitDagBundle",
"kwargs": {
- "subdir": "dags",
- "tracking_ref": "main",
- "refresh_interval": 0
- }
+ "subdir": "dags",
+ "tracking_ref": "main",
+ "refresh_interval": 0
+ }
+ }
]
default: >
[
- {{
- "name": "dags-folder",
- "classpath":
"airflow.dag_processing.bundles.local.LocalDagBundle",
- "kwargs": {{}}
- }}
+ {{
+ "name": "dags-folder",
+ "classpath":
"airflow.dag_processing.bundles.local.LocalDagBundle",
+ "kwargs": {{}}
+ }}
]
refresh_interval:
description: |
