potiuk commented on issue #53491: URL: https://github.com/apache/airflow/issues/53491#issuecomment-3092380152
> 'l @kyungjunleeme right, i want to get airflow slowness solution in airflow 2.10.3 Now we are planning to implement pgbouner on prod airflow. I am not sure pgbouner will help in airflow slowness. PGBouncer is HIGHLY recommended for Airflow 2 (look at the database documentation in Airflow docs for 2.10.3 if you care about performance and have high load, I am actually quite surprised you do not have it already, this is almost mandatory requirement with high load, because Airflow 2 task instances open - each - new connection and Postgress creates a new process for every connection, which likely causes it use a LOT of memory and that's the reason for slowness (not locking). We also recommend you have monitoring of your database and alerting to be able to diagnose such situations. In general - there will be no changes nor optimizations any more for Airflow 2 - just critical bugfixes. So you should also seriously consider migrating to Airflow 3 asap. Architecture is changed there and task instance no longer open DB connections directly to database. Converting to discussion if more discussion needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
