GitHub user potiuk added a comment to the discussion: Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler
> 'l @kyungjunleeme right, i want to get airflow slowness solution in airflow > 2.10.3 Now we are planning to implement pgbouner on prod airflow. I am not sure pgbouner will help in airflow slowness. PGBouncer is HIGHLY recommended for Airflow 2 (look at the database documentation in Airflow docs for 2.10.3 if you care about performance and have high load, I am actually quite surprised you do not have it already, this is almost mandatory requirement with high load, because Airflow 2 task instances open - each - new connection and Postgress creates a new process for every connection, which likely causes it use a LOT of memory and that's the reason for slowness (not locking). We also recommend you have monitoring of your database and alerting to be able to diagnose such situations. In general - there will be no changes nor optimizations any more for Airflow 2 - just critical bugfixes. So you should also seriously consider migrating to Airflow 3 asap. Architecture is changed there and task instance no longer open DB connections directly to database. Converting to discussion if more discussion needed. GitHub link: https://github.com/apache/airflow/discussions/53538#discussioncomment-13817131 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
