GitHub user michaelosthege added a comment to the discussion: High CPU and database load caused by Airflow 3 dag-processor and scheduler
The scheduler spent 75 % in [the `_do_scheduling(session)` call](https://github.com/apache/airflow/blob/3.0.2/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L1296) and only 3.7 % in [the `time.sleep` idling](https://github.com/apache/airflow/blob/3.0.2/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L1342). So it took 97 % of the default `AIRFLOW__SCHEDULER__SCHEDULER_IDLE_SLEEP_TIME=1` seconds to find out that there's nothing to do, because my 2 DAGs are only scheduled every 10 minutes... Airflow 2 was way more efficient in that regard, but I guess I can `SCHEDULER_IDLE_SLEEP_TIME` here. GitHub link: https://github.com/apache/airflow/discussions/53177#discussioncomment-13734451 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
