You won't believe, I was thinking about something like this just last week. So happy to see the community also moving in the same direction
On Sat, Apr 4, 2026 at 6:57 PM Aritra Basu <[email protected]> wrote: > Overall a big +1 to this, I've added a few comments on it. But good read, > would love to see this implemented and always open to helping get it to the > finish line! Thanks for the proposal! > Thanks and Regards, > Aritra Basu > > > On Fri, Apr 3, 2026 at 6:33 PM Shahar Epstein <[email protected]> wrote: > > > Hello everyone, > > > > Since I first opened the thread discussing the MCP server, I've been > > thinking for a long while about how we can practically bring AI-assisted > > debugging and operational insights directly into Airflow. As > orchestration > > environments grow more complex, the cost of troubleshooting, such as > > navigating across Dag code, task instances, scheduler logs, and > > configurations, translates directly to lost on-call time and delayed > > pipelines. > > > > Today, organizations that want AI-assisted debugging are forced to build > > custom, ad-hoc integrations or rely on external paid solutions. This > leads > > to fragmented user experiences, duplicated effort, and most critically, > > inconsistent security controls that risk exposing sensitive metadata or > > bypassing Airflow's native Role-Based Access Control (RBAC). > > > > I think we can do better, by proposing AIP-101: Airflow AI Assistant - > > Phase 1 (Read-only assistance). This AIP introduces an official, opt-in > > plugin that provides a conversational UI directly within Airflow to > answer > > user questions about their instances, explain errors, and help > troubleshoot > > failures. > > > > To ensure this is done safely and securely, Phase 1 is strictly > read-only. > > The assistant does not modify Airflow state, nor does it operate > > autonomously. Instead, it relies on the newly proposed Airflow MCP Server > > (AIP-91) as its data-retrieval engine. By leveraging the MCP standard, > the > > assistant guarantees that its answers are grounded in live system state > > while strictly enforcing the authenticated user's RBAC permissions so the > > AI never accesses data the user cannot see. > > > > tl;dr of the proposed implementation: > > > > Packaging: Delivered as an opt-in, standalone plugin package within the > > apache/airflow monorepo (with an independent release cycle). > > Frontend: A conversational UI embedded directly in the Airflow web > > interface. > > Backend: A FastAPI-based plugin backend utilizing pydantic-ai to safely > > orchestrate external LLM calls. > > Data Access: Relies entirely on the Airflow MCP Server (AIP-91) to fetch > > read-only state. > > > > --- > > > > Because the assistant is heavily coupled with the secure tool-calling > > execution provided by the MCP server, which is covered in a separate AIP > > (AIP-91) - please note the ongoing discussion here, as well as AIP-91 > > itself: https://lists.apache.org/thread/xgd66v6s7zf0xkvy3c7ysqvn4csgmw06 > > https://cwiki.apache.org/confluence/x/G4q3FQ > > > > --- > > > > AIP-101 is available here: > > https://cwiki.apache.org/confluence/x/8Ic8G > > > > A quick warning before you read: the AIP is quite long! (sorry Jarek) > > Because integrating AI into an orchestrator opens up a lot of potential > > pitfalls, [ChatGPT and] I tried to be extremely thorough in covering all > > the possible stuff that could go wrong :) > > If you find a specific section to be overly detailed or repetitive, > please > > comment in the AIP and I'll try to handle it. > > > > I've managed to build a very inital POC, screenshots are available in > this > > section: > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620144#AIP101AirflowAIAssistantPhase1(Readonlyassistance)-BehavioralModel > > > > I would love to hear your thoughts. Please comment on the AIP and/or > reply > > to this thread. > > > > Thank you, > > > > Shahar > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
