GitHub user leninkumar-sv-office closed a discussion: Max tasks across dags stuck at 487
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.8.1 ### What happened? I have the airflow running on AKS cluster with just the kubernetes executor. The following configs are set for attaining the max concurrency. Parallelism: 128 (per scheduler) No of Schedulers: 10 Default pool size: 1200 DB: Postgres Airflow Version: 2.8.1 Platform: Ubuntu We are running big data workloads and the number of Max task runs can reach 1000. However, with the above configurations, the Max task runs across all the dags is stuck at the magic number 487 and the remaining ones are queued. I have checked the docs about any other parameters that need to be set, however only these are mentioned. Appreciate the suggestions from the community to identify the missing configuration. Thanks ### What you think should happen instead? _No response_ ### How to reproduce this setup is running on Azure K8s. The number of concurrent tasks with kubernetes executor is stuck at 487 mac. ### Operating System Ubuntu ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) GitHub link: https://github.com/apache/airflow/discussions/44293 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
