Hi Vikram, Great thoughts on this.
I agree with this: > I am concerned that we are overwhelming our audience segment (1) with the > work and configurations around running Airflow at scale. I would like to see a better separation on 1) how to set up an airflow cluster in production, how to monitor the health of an airflow cluster 2) how to write better airflow DAGs. I also like the idea of segmenting the release notes for different groups. Thanks, Ping On Thu, Jun 30, 2022 at 8:50 PM Vikram Koka <[email protected]> wrote: > Hi everyone, > > As I have been looking through the recent AIPs, development features, and > mailing list discussions, it struck me that we have effectively three > different audiences here for Airflow. > > 1. Individuals and small teams using Airflow for their purpose, > 2. Enterprises managing Airflow for large teams of data engineers and data > scientists, and > 3. Service providers making "Airflow as a service" available for many > customers, either external or internal. > > Why does this even matter? Let me elaborate below: > > - Clearly, a lot of "data practitioners", people who are primarily > focused on creating pipelines and working with data are spread across all > three audiences above. > - However, "Airflow administrators" i.e. people who are focused on > running Airflow for data practitioners, especially at scale are primarily > in the audiences (2) and (3) above. > - It is my observation that a lot of work being done right now in > Airflow such as multi-tenancy (but not limited to it), is focused on > Airflow administration. > - I am concerned that we are overwhelming our audience segment (1) > with the work and configurations around running Airflow at scale. > > > If this is true, I would like to propose that we segment our Airflow > configurations, our packaging including our docs, and even our release > notes to make it easier for our audience (1), who is almost certainly the > largest block of our Airflow user community. > > I would like the opinions of the community on this topic. > > Best regards, > Vikram > >
