Hi Shahar , Thank you for putting together this comprehensive proposal. The decision to architect the MCP server as a stateless proxy that delegates all RBAC evaluation to Airflow's native Auth Manager is a fantastic design choice. It cleanly avoids the Confused Deputy problem and ensures per-user permission enforcement without forcing us to maintain a parallel authorization layer.
I've been reviewing the AIP-91 design specifications against the current codebase (specifically the v3-2-test branch), and I wanted to raise a few concrete findings regarding the implementation. Because AI agents consume data fundamentally differently than human operators do, there are a few architectural gaps we might need to address: *1. Task Logs Are Not Redacted on the Read Path*AIP-91 states: The MCP Server MUST pipe all retrieved free-text artifacts (specifically Task Logs and Rendered Templates) through Airflow's native SecretsMasker before serializing the tool response. However, looking at the current API execution path (get_log -> TaskLogReader.read_log_chunks -> FileTaskHandler.read), there is no read-time invocation of SecretsMasker.redact(). Because SecretsMasker is a logging.Filter, it only masks secrets at write time. If a secret was leaked via a standard out print() before being registered, or if the masker simply wasn't active, the REST API will serve the leaked credential as-is. Question: Should we introduce a read-time redaction layer directly within TaskLogReader, or should we explicitly make this the responsibility of the MCP proxy? *2. Lack of Heuristic Detection for Unregistered Secrets*The native SecretsMasker is highly optimized for deterministic masking (key-name matching against fields like password or api_key, and exact-string matching). It does not use regex heuristics to detect high-entropy strings, which makes sense given historical performance concerns with regex in the main logging pipeline. However, for an AI assistant blindly consuming arbitrary tracebacks, unregistered secrets (like an AWS AKIA... key or a JWT eyJ... token) represent a severe exfiltration risk. Question: Should the MCP server implement a secondary, regex-based heuristic scrubbing layer exclusively for LLM payloads before they leave the network boundary? *3. The get_log_tail Tool Relies on Non-Existent API Functionality*AIP-91 specifies that truncated log payloads will instruct the LLM to use a get_log_tail tool. Currently, the log API handles pagination via an opaque continuation token (URLSafeSerializer). Because this is a forward-only cryptographic cursor, it cannot be used for a bounded page backwards from the end of the file navigation, which is exactly what an AI needs to quickly read a stack trace. Question: Will get_log_tail require us to build a new byte-offset/tailing endpoint into the Core API, or will the MCP server cache the full log stream internally to virtualize this capability? *4. JWT Token Refresh for Bearer-Authenticated Clients*The Core API's JWTRefreshMiddleware is currently optimized for browser UI sessions (cookies). When the MCP server authenticates via a Bearer token in the Authorization header,the standard for machine-to-machine integration expired tokens trigger an exception rather than a refresh. The Execution API handles this beautifully with a JWTReissueMiddleware that auto-refreshes tokens with <20% validity remaining. Question: For long-running AI debugging sessions, should we harmonize the Core API to adopt the Execution API's reissue middleware, or is the MCP server expected to handle token rotation out-of-band? *5. Resource Saturation and Rate Limiting*AIP-91 rightly proposes token-bucket rate limiting. It’s worth noting that the current Core API does not enforce rate limits on resource-intensive endpoints (like NDJSON log streaming). An LLM caught in a hallucination loop could easily overwhelm the webserver's connection pool. Question: This seems to reinforce the idea that the MCP server must implement its own distributed rate limiting (e.g., backed by Redis) rather than relying on the Core API to throttle rogue traffic. Do we agree this is a hard requirement for Phase 1? I am highly supportive of this AIP and the proxy-based approach. I would be more than happy to help contribute to the implementation or assist with testing these specific security boundaries. Best regards, Viquar Khan On Fri, Apr 3, 2026 at 6:38 AM Shahar Epstein <[email protected]> wrote: > Hey everyone, > > A lot of water has passed under the AI bridge since we first discussed > this, and I think it's a good time to get things moving. > > I have drafted AIP-91 for the official Airflow MCP Server: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=364349979 > > tl;dr - In its 1st phase, the main use case of the MCP server will be to > serve the planned Airflow AI Assistant (which will be discussed in a > separate thread for AIP-101). To ensure strict security and per-user RBAC > enforcement, both the MCP server and the assistant will be strictly in > read-only mode for Phase 1. > > A quick note: While I'm proposing this as a separate AIP, I highly > recommend reading it in the context of AIP-101 to see exactly how the two > pieces fit together. > > I'd love to get your thoughts and feedback on the design. > > > Shahar > > On 2025/05/28 17:25:06 Shahar Epstein wrote: > > Dear community, > > > > Following the thread on Slack [1], initiated by Jason Sebastian Kusuma, > I'd > > like to start an effort to officially support MCP in Airflow's codebase. > > > > *Some background * > > Model Context Protocol (MCP) is an open standard, open-source framework > > that standardizes the way AI models like LLM integrate and share data > with > > external tools, systems and data sources. Think of it as a "USB-C for > AI" - > > a universal connector that simplifies and standardizes AI integrations. A > > notable example of an MCP server is GitHub's official implementation > [3], which > > allows LLMs such as Claude, Copilot, and OpenAI (or: "MCP clients") to > > fetch pull request details, analyze code changes, and generate review > > summaries. > > > > *How could an MCP server be useful in Airflow?* > > Imagine the possibilities when LLMs can seamlessly interact with > Airflow’s > > API: triggering DAGs using natural language, retrieving DAG run history, > > enabling smart debugging, and more. This kind of integration opens the > door > > to a more intuitive, conversational interface for workflow orchestration. > > > > *Why do we need to support it officially?* > > Quid pro quo - LLMs become an integral part of the modern development > > experience, while Airflow evolves into the go-to for orchestrating AI > > workflows. By officially supporting it, we’ll enable multiple users to > > interact with Airflow through their LLMs, streamlining automation and > > improving accessibility across diverse workflows. All of that is viable > > with relatively small development effort (see next paragraph). > > > > *How should it be implemented?* > > As of today, there have been several implementations of MCP servers for > > Airflow API, the most visible one [4] made by Abhishek Bhakat from > > Astronomer. > > The efforts of implementing it and maintaining it in our codebase > shouldn't > > be too cumbersome (at least in theory), as we could utilize packages like > > fastmcp to auto-generate the server using the existing OpenAPI specs. I'd > > be very happy if Abhishek could share his experience in this thread. > > > > *Where else could we utilize MCP?* > > Beyond the scope of the public API, I could also imagine using it to > > communicate with Breeze. > > > > *How do we proceed from here?* > > Feel free to share your thoughts here in this discussion. > > If there are no objections, I'll be happy to start working on an AIP. > > > > > > Sincerely, > > Shahar Epstein > > > > > > *References:* > > [1] Slack discussion, > > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746121916951569 > > [2] Introducing the model context protocol, > > https://www.anthropic.com/news/model-context-protocol > > [3] GitHub Official MCP server, > https://github.com/github/github-mcp-server > > [4] Unofficial MCP Server made by Abhishek Hakat, > > https://github.com/abhishekbhakat/airflow-mcp-server > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
