This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v3-0-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 63ec6f4cc7e1668cc0fdbb570f8a448fa6e620e9 Author: Amogh Desai <[email protected]> AuthorDate: Wed Apr 23 09:50:19 2025 +0530 Updating Airflow executor docs for AF3 (#49389) (cherry picked from commit eca6a53110e7365bc013a181ea5a4746444ddc9e) --- airflow-core/docs/core-concepts/executor/index.rst | 40 +++++++++++++++++++--- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/airflow-core/docs/core-concepts/executor/index.rst b/airflow-core/docs/core-concepts/executor/index.rst index c83f10cb050..33427ff2bcf 100644 --- a/airflow-core/docs/core-concepts/executor/index.rst +++ b/airflow-core/docs/core-concepts/executor/index.rst @@ -48,8 +48,6 @@ If you want to check which executor is currently set, you can use the ``airflow $ airflow config get-value core executor LocalExecutor - - Executor Types -------------- @@ -232,6 +230,40 @@ Some reasons you may want to write a custom executor include: * You'd like to use an executor that leverages a compute service from your preferred cloud provider. * You have a private tool/service for task execution that is only available to you or your organization. +Workloads +^^^^^^^^^ + +A workload in context of an Executor is the fundamental unit of execution for an executor. It represents a discrete +operation or job that the executor runs on a worker. For example, it can run user code encapsulated in an Airflow task +on a worker. + +Example: + +.. code-block:: python + + ExecuteTask( + token="mock", + ti=TaskInstance( + id=UUID("4d828a62-a417-4936-a7a6-2b3fabacecab"), + task_id="mock", + dag_id="mock", + run_id="mock", + try_number=1, + map_index=-1, + pool_slots=1, + queue="default", + priority_weight=1, + executor_config=None, + parent_context_carrier=None, + context_carrier=None, + queued_dttm=None, + ), + dag_rel_path=PurePosixPath("mock.py"), + bundle_info=BundleInfo(name="n/a", version="no matter"), + log_path="mock.log", + type="ExecuteTask", + ) + Important BaseExecutor Methods ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -239,7 +271,7 @@ Important BaseExecutor Methods These methods don't require overriding to implement your own executor, but are useful to be aware of: * ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks. -* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor. +* ``queue_workload``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the *workloads* (check section above to understand) to an internal list of queued workloads to run within the executor. All executors present in the repository use this method. * ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing. * ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running. * ``send_callback``: Sends any callbacks to the sink configured on the executor. @@ -251,7 +283,7 @@ Mandatory Methods to Implement The following methods must be overridden at minimum to have your executor supported by Airflow: * ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler. -* ``execute_async``: Executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor. +* ``execute_async``: Executes a *workload* asynchronously. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor. Optional Interface Methods to Implement
