The AWS Airflow team is happy to announce that the dashboard we have been building which summarizes the health status of system tests within the Amazon provider package is now available here: https://aws-mwaa.github.io/open-source/system-tests/dashboard.html. The purpose is to share with anyone who is interested in the results of the latest executions of our system tests. These system tests are executed frequently internally (every hour if new commits have arrived during that hour) using the last main version and this dashboard shares the results of these executions in a transparent way.
In this dashboard you can find the list of all system tests within the Amazon provider package and some statistics about the last executions, such as: number of invocations, number of successes, number of failures and average duration. You can also see the status (succeed/failed) of the last 10 executions for each system test. What is the use? This dashboard can be used in multiple different scenarios. Here are some examples: * When releasing new Airflow provider packages, this dashboard can be used to check the Amazon provider package health status * Being able to detect early some failures or bugs in Amazon provider package because of new code changes Why some system tests are not listed in this dashboard? If a system test in the Amazon provider package system test directory (https://github.com/apache/airflow/tree/main/tests/system/providers/amazon/aws) is not listed in this dashboard, it can be due to two reasons: * It is a relatively new system test and we need internally to configure it to run in our testing pipeline. Once this configuration is done, it will show up in the dashboard * We cannot support this system test for various reasons (e.g. it needs a third party account/resource). This is decided case by case so if you ever wonder why a given system test is not listed in the dashboard, feel free to ask in #airflow-aws, we’ll happily answer How to write a system test myself? First of all, any contribution is welcome so please add system tests where applicable. When doing so, please follow the guidelines defined here https://github.com/apache/airflow/tree/main/tests/system/providers/amazon. Once the new system test is merged, as mentioned above, the AWS Airflow team will write the configuration needed to run it. Once this is done, the system test you wrote will appear in the dashboard. A system test is failing, does it mean something is broken? Not necessarily, it can also be due to an issue in our internal infrastructure which runs the system tests. In such a situation, out team will investigate the system test failure and act upon it. If it is a legitimate regression in Airflow/Amazon Provider Package we may work on the fix ourselves or create a GitHub issue in the Airflow repository to track the fix. This allows the community to get involved. This is just a first step and we will try to improve it over time. Any suggestions/recommendations/improvements are very welcome so please do reach out (through the different medias) if you have any.
