GitHub user michaelosthege added a comment to the discussion: High CPU and database load caused by Airflow 3 dag-processor and scheduler
Turns out the dag-processor loop is fine, but the healthcheck was the issue! My `dag-processor` healthcheck (copied from the docker-compose example) was: ``` healthcheck: test: ["CMD-SHELL", 'airflow jobs check --job-type DagProcessorJob'] interval: 30s timeout: 20s retries: 5 start_period: 30s ``` I disabled it yesterday and the container's CPU load went down to ~1 %. Now I just ran ``` watch -n 5 docker exec -it airflow-dag-processor airflow jobs check --job-type DagProcessorJob ``` and the CPU of the dag-processor, as well as the database reads go 🚀 🔥 ------------- The CPU load by the Postgres container remained high regardless. Here it's definitely the scheduler, but I don't know why. GitHub link: https://github.com/apache/airflow/discussions/53177#discussioncomment-13783682 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
