This is an automated email from the ASF dual-hosted git repository. wusheng pushed a commit to branch feature/mcp-observability in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit d6b2705b0222d172952a1cb5ca123ae262f9115c Author: Wu Sheng <[email protected]> AuthorDate: Sun Apr 5 00:13:21 2026 +0800 Support MCP observability for Envoy AI Gateway --- docs/en/changes/changes.md | 1 + .../backend/backend-envoy-ai-gateway-monitoring.md | 114 ++++++---- .../src/main/resources/application.yml | 2 +- .../src/main/resources/lal/envoy-ai-gateway.yaml | 51 ++++- .../envoy-ai-gateway/gateway-mcp-instance.yaml | 80 +++++++ .../envoy-ai-gateway/gateway-mcp-service.yaml | 85 ++++++++ .../envoy-ai-gateway-instance.json | 156 ++++++++++++++ .../envoy_ai_gateway/envoy-ai-gateway-service.json | 234 +++++++++++++++++++++ .../cases/envoy-ai-gateway/docker-compose.yml | 32 ++- test/e2e-v2/cases/envoy-ai-gateway/e2e.yaml | 35 ++- .../envoy-ai-gateway/envoy-ai-gateway-cases.yaml | 24 ++- .../cases/envoy-ai-gateway/expected/logs.yml | 2 + 12 files changed, 754 insertions(+), 62 deletions(-) diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 0918ae388d..281be6c3b1 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -10,6 +10,7 @@ * Push `taskId` filter down to the storage layer in `IAsyncProfilerTaskLogQueryDAO`, removing in-memory filtering from `AsyncProfilerQueryService`. * Fix missing parentheses around OR conditions in `JDBCZipkinQueryDAO.getTraces()`, which caused the table filter to be bypassed for all but the first trace ID. Replaced with a proper `IN` clause. * Fix missing `and` keyword in `JDBCEBPFProfilingTaskDAO.getTaskRecord()` SQL query, which caused a syntax error on every invocation. +* Support MCP (Model Context Protocol) observability for Envoy AI Gateway: MCP metrics (request CPM/latency, method breakdown, backend breakdown, initialization latency, capabilities), MCP access log sampling (errors only), `ai_route_type` searchable log tag, and MCP dashboard tabs. #### UI diff --git a/docs/en/setup/backend/backend-envoy-ai-gateway-monitoring.md b/docs/en/setup/backend/backend-envoy-ai-gateway-monitoring.md index e8620edb9d..7a5c6577ef 100644 --- a/docs/en/setup/backend/backend-envoy-ai-gateway-monitoring.md +++ b/docs/en/setup/backend/backend-envoy-ai-gateway-monitoring.md @@ -4,8 +4,8 @@ [Envoy AI Gateway](https://aigateway.envoyproxy.io/) is a gateway/proxy for AI/LLM API traffic (OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Gemini, etc.) built on top of Envoy Proxy. -It natively emits GenAI metrics and access logs via OTLP, following -[OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). +It natively emits GenAI metrics, MCP (Model Context Protocol) metrics, and access logs via OTLP, +following [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). SkyWalking receives OTLP metrics and logs directly on its gRPC port (11800) — no OpenTelemetry Collector is needed between the AI Gateway and SkyWalking OAP. @@ -15,7 +15,7 @@ Collector is needed between the AI Gateway and SkyWalking OAP. [Envoy AI Gateway getting started](https://aigateway.envoyproxy.io/docs/getting-started/) for installation. ### Data flow -1. Envoy AI Gateway processes LLM API requests and records GenAI metrics (token usage, latency, TTFT, TPOT). +1. Envoy AI Gateway processes LLM API requests and MCP requests, recording GenAI metrics and MCP metrics. 2. The AI Gateway pushes metrics and access logs via OTLP gRPC to SkyWalking OAP. 3. SkyWalking OAP parses metrics with [MAL](../../concepts-and-designs/mal.md) rules and access logs with [LAL](../../concepts-and-designs/lal.md) rules. @@ -27,14 +27,14 @@ in SkyWalking OAP. No OAP-side configuration is needed. Configure the AI Gateway to push OTLP to SkyWalking by setting these environment variables: -| Env Var | Value | Purpose | -|---------|-------|---------| -| `OTEL_SERVICE_NAME` | Per-deployment gateway name (e.g., `my-ai-gateway`) | SkyWalking service name | -| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://skywalking-oap:11800` | SkyWalking OAP gRPC receiver | -| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` | OTLP transport | -| `OTEL_METRICS_EXPORTER` | `otlp` | Enable OTLP metrics push | -| `OTEL_LOGS_EXPORTER` | `otlp` | Enable OTLP access log push | -| `OTEL_RESOURCE_ATTRIBUTES` | See below | Routing + instance + layer | +| Env Var | Value | Purpose | +|-------------------------------|-----------------------------------------------------|------------------------------| +| `OTEL_SERVICE_NAME` | Per-deployment gateway name (e.g., `my-ai-gateway`) | SkyWalking service name | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://skywalking-oap:11800` | SkyWalking OAP gRPC receiver | +| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` | OTLP transport | +| `OTEL_METRICS_EXPORTER` | `otlp` | Enable OTLP metrics push | +| `OTEL_LOGS_EXPORTER` | `otlp` | Enable OTLP access log push | +| `OTEL_RESOURCE_ATTRIBUTES` | See below | Routing + instance + layer | **Required resource attributes** (in `OTEL_RESOURCE_ATTRIBUTES`): - `job_name=envoy-ai-gateway` — Fixed routing tag for MAL/LAL rules. Same for all AI Gateway deployments. @@ -58,47 +58,85 @@ is a service, each pod is an instance. Metrics include per-provider and per-mode #### Service Metrics -| Monitoring Panel | Unit | Metric Name | Description | -|---|---|---|---| -| Request CPM | calls/min | meter_envoy_ai_gw_request_cpm | Requests per minute | -| Request Latency Avg | ms | meter_envoy_ai_gw_request_latency_avg | Average request duration | -| Request Latency Percentile | ms | meter_envoy_ai_gw_request_latency_percentile | P50/P75/P90/P95/P99 | -| Input Token Rate | tokens/min | meter_envoy_ai_gw_input_token_rate | Input (prompt) tokens per minute | -| Output Token Rate | tokens/min | meter_envoy_ai_gw_output_token_rate | Output (completion) tokens per minute | -| TTFT Avg | ms | meter_envoy_ai_gw_ttft_avg | Time to First Token (streaming only) | -| TTFT Percentile | ms | meter_envoy_ai_gw_ttft_percentile | P50/P75/P90/P95/P99 TTFT | -| TPOT Avg | ms | meter_envoy_ai_gw_tpot_avg | Time Per Output Token (streaming only) | -| TPOT Percentile | ms | meter_envoy_ai_gw_tpot_percentile | P50/P75/P90/P95/P99 TPOT | +| Monitoring Panel | Unit | Metric Name | Description | +|----------------------------|------------|----------------------------------------------|----------------------------------------| +| Request CPM | calls/min | meter_envoy_ai_gw_request_cpm | Requests per minute | +| Request Latency Avg | ms | meter_envoy_ai_gw_request_latency_avg | Average request duration | +| Request Latency Percentile | ms | meter_envoy_ai_gw_request_latency_percentile | P50/P75/P90/P95/P99 | +| Input Token Rate | tokens/min | meter_envoy_ai_gw_input_token_rate | Input (prompt) tokens per minute | +| Output Token Rate | tokens/min | meter_envoy_ai_gw_output_token_rate | Output (completion) tokens per minute | +| TTFT Avg | ms | meter_envoy_ai_gw_ttft_avg | Time to First Token (streaming only) | +| TTFT Percentile | ms | meter_envoy_ai_gw_ttft_percentile | P50/P75/P90/P95/P99 TTFT | +| TPOT Avg | ms | meter_envoy_ai_gw_tpot_avg | Time Per Output Token (streaming only) | +| TPOT Percentile | ms | meter_envoy_ai_gw_tpot_percentile | P50/P75/P90/P95/P99 TPOT | #### Provider Breakdown Metrics -| Monitoring Panel | Unit | Metric Name | Description | -|---|---|---|---| -| Provider Request CPM | calls/min | meter_envoy_ai_gw_provider_request_cpm | Requests by provider | -| Provider Token Rate | tokens/min | meter_envoy_ai_gw_provider_token_rate | Token rate by provider | -| Provider Latency Avg | ms | meter_envoy_ai_gw_provider_latency_avg | Latency by provider | +| Monitoring Panel | Unit | Metric Name | Description | +|----------------------|------------|----------------------------------------|------------------------| +| Provider Request CPM | calls/min | meter_envoy_ai_gw_provider_request_cpm | Requests by provider | +| Provider Token Rate | tokens/min | meter_envoy_ai_gw_provider_token_rate | Token rate by provider | +| Provider Latency Avg | ms | meter_envoy_ai_gw_provider_latency_avg | Latency by provider | #### Model Breakdown Metrics -| Monitoring Panel | Unit | Metric Name | Description | -|---|---|---|---| -| Model Request CPM | calls/min | meter_envoy_ai_gw_model_request_cpm | Requests by model | -| Model Token Rate | tokens/min | meter_envoy_ai_gw_model_token_rate | Token rate by model | -| Model Latency Avg | ms | meter_envoy_ai_gw_model_latency_avg | Latency by model | -| Model TTFT Avg | ms | meter_envoy_ai_gw_model_ttft_avg | TTFT by model | -| Model TPOT Avg | ms | meter_envoy_ai_gw_model_tpot_avg | TPOT by model | +| Monitoring Panel | Unit | Metric Name | Description | +|-------------------|------------|-------------------------------------|---------------------| +| Model Request CPM | calls/min | meter_envoy_ai_gw_model_request_cpm | Requests by model | +| Model Token Rate | tokens/min | meter_envoy_ai_gw_model_token_rate | Token rate by model | +| Model Latency Avg | ms | meter_envoy_ai_gw_model_latency_avg | Latency by model | +| Model TTFT Avg | ms | meter_envoy_ai_gw_model_ttft_avg | TTFT by model | +| Model TPOT Avg | ms | meter_envoy_ai_gw_model_tpot_avg | TPOT by model | #### Instance Metrics All service-level metrics are also available per instance (pod) with `meter_envoy_ai_gw_instance_` prefix, including per-provider and per-model breakdowns. +### MCP Metrics + +When the AI Gateway is configured with MCP (Model Context Protocol) routes, SkyWalking collects +MCP-specific metrics. These appear in the **MCP** tab on the service and instance dashboards. + +#### MCP Service Metrics + +| Monitoring Panel | Unit | Metric Name | Description | +|---------------------------------------|-----------|---------------------------------------------------------|-------------------------------------------------------------------| +| MCP Request CPM | calls/min | meter_envoy_ai_gw_mcp_request_cpm | MCP requests per minute | +| MCP Request Latency Avg | ms | meter_envoy_ai_gw_mcp_request_latency_avg | Average MCP request duration | +| MCP Request Latency Percentile | ms | meter_envoy_ai_gw_mcp_request_latency_percentile | P50/P75/P90/P95/P99 | +| MCP Method CPM | calls/min | meter_envoy_ai_gw_mcp_method_cpm | Requests by MCP method (initialize, tools/list, tools/call, etc.) | +| MCP Error CPM | calls/min | meter_envoy_ai_gw_mcp_error_cpm | MCP error requests per minute | +| MCP Initialization Latency Avg | ms | meter_envoy_ai_gw_mcp_initialization_latency_avg | Average MCP session initialization time | +| MCP Initialization Latency Percentile | ms | meter_envoy_ai_gw_mcp_initialization_latency_percentile | P50/P75/P90/P95/P99 | +| MCP Capabilities CPM | calls/min | meter_envoy_ai_gw_mcp_capabilities_cpm | Capabilities negotiated by type | + +#### MCP Backend Breakdown Metrics + +| Monitoring Panel | Unit | Metric Name | Description | +|--------------------------|-----------|----------------------------------------------------------|--------------------------------| +| Backend Request CPM | calls/min | meter_envoy_ai_gw_mcp_backend_request_cpm | Requests by MCP backend | +| Backend Latency Avg | ms | meter_envoy_ai_gw_mcp_backend_request_latency_avg | Latency by MCP backend | +| Backend Method CPM | calls/min | meter_envoy_ai_gw_mcp_backend_method_cpm | Requests by backend and method | +| Backend Error CPM | calls/min | meter_envoy_ai_gw_mcp_backend_error_cpm | Errors by MCP backend | +| Backend Init Latency Avg | ms | meter_envoy_ai_gw_mcp_backend_initialization_latency_avg | Init latency by backend | + +#### MCP Instance Metrics + +All MCP service-level metrics are also available per instance with `meter_envoy_ai_gw_mcp_instance_` prefix. + ### Access Log Sampling -The LAL rules apply a sampling policy to reduce storage: +Access logs are tagged with `ai_route_type` (`llm` or `mcp`) for filtering in the log query UI. +The `ai_route_type` tag is searchable by default. + +**LLM route logs:** - **Error responses** (HTTP status >= 400) — always persisted. - **Upstream failures** — always persisted. -- **High token cost** (>= 10,000 total tokens) — persisted for cost anomaly detection. -- Normal successful responses with low token counts are dropped. +- Normal successful responses are dropped. + +**MCP route logs:** +- **Error responses** (HTTP status >= 400) — always persisted. +- Normal MCP requests are dropped (MCP observability is covered by metrics). -The token threshold can be adjusted in `lal/envoy-ai-gateway.yaml`. +The sampling policy can be adjusted in `lal/envoy-ai-gateway.yaml`. diff --git a/oap-server/server-starter/src/main/resources/application.yml b/oap-server/server-starter/src/main/resources/application.yml index b79ff36a95..a0112f34d3 100644 --- a/oap-server/server-starter/src/main/resources/application.yml +++ b/oap-server/server-starter/src/main/resources/application.yml @@ -118,7 +118,7 @@ core: searchableTracesTags: ${SW_SEARCHABLE_TAG_KEYS:http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker} # Define the set of log tag keys, which should be searchable through the GraphQL. # The max length of key=value should be less than 256 or will be dropped. - searchableLogsTags: ${SW_SEARCHABLE_LOGS_TAG_KEYS:level,http.status_code} + searchableLogsTags: ${SW_SEARCHABLE_LOGS_TAG_KEYS:level,http.status_code,ai_route_type} # Define the set of alarm tag keys, which should be searchable through the GraphQL. # The max length of key=value should be less than 256 or will be dropped. searchableAlarmTags: ${SW_SEARCHABLE_ALARM_TAG_KEYS:level} diff --git a/oap-server/server-starter/src/main/resources/lal/envoy-ai-gateway.yaml b/oap-server/server-starter/src/main/resources/lal/envoy-ai-gateway.yaml index 0af60377b5..bb6b31fdc5 100644 --- a/oap-server/server-starter/src/main/resources/lal/envoy-ai-gateway.yaml +++ b/oap-server/server-starter/src/main/resources/lal/envoy-ai-gateway.yaml @@ -15,29 +15,33 @@ # Envoy AI Gateway access log processing via OTLP. # -# Sampling policy: only persist abnormal or expensive requests. -# Normal 200 responses with low token count and no upstream failure are dropped. +# Two rules: one for LLM route logs, one for MCP route logs. +# LLM sampling: only persist abnormal or expensive requests. +# MCP sampling: only persist error responses. +# Both tag ai_route_type for searchable filtering in the UI. rules: - - name: envoy-ai-gateway-access-log + - name: envoy-ai-gateway-llm-access-log layer: ENVOY_AI_GATEWAY dsl: | filter { + // Only process LLM route logs (gen_ai.request.model is always set for LLM routes, even on errors) + if (tag("gen_ai.request.model") == "" || tag("gen_ai.request.model") == "-") { + abort {} + } + // Drop normal logs: response < 400, no upstream failure, low token count + // Keep only: error responses (>= 400), upstream failures, or high-token requests (>= 10000) if (tag("response_code") != "" && tag("response_code") != "-") { if (tag("response_code") as Integer < 400) { if (tag("upstream_transport_failure_reason") == "" || tag("upstream_transport_failure_reason") == "-") { - if (tag("gen_ai.usage.input_tokens") != "" && tag("gen_ai.usage.input_tokens") != "-" - && tag("gen_ai.usage.output_tokens") != "" && tag("gen_ai.usage.output_tokens") != "-") { - if ((tag("gen_ai.usage.input_tokens") as Integer) + (tag("gen_ai.usage.output_tokens") as Integer) < 10000) { - abort {} - } - } + abort {} } } } extractor { + tag 'ai_route_type': "llm" tag 'gen_ai.request.model': tag("gen_ai.request.model") tag 'gen_ai.response.model': tag("gen_ai.response.model") tag 'gen_ai.provider.name': tag("gen_ai.provider.name") @@ -50,3 +54,32 @@ rules: sink { } } + + - name: envoy-ai-gateway-mcp-access-log + layer: ENVOY_AI_GATEWAY + dsl: | + filter { + // Only process MCP route logs + if (tag("mcp.method.name") == "" || tag("mcp.method.name") == "-") { + abort {} + } + + // Only persist error responses + if (tag("response_code") != "" && tag("response_code") != "-") { + if (tag("response_code") as Integer < 400) { + abort {} + } + } + + extractor { + tag 'ai_route_type': "mcp" + tag 'mcp.method.name': tag("mcp.method.name") + tag 'mcp.provider.name': tag("mcp.provider.name") + tag 'mcp.session.id': tag("mcp.session.id") + tag 'response_code': tag("response_code") + tag 'duration': tag("duration") + } + + sink { + } + } diff --git a/oap-server/server-starter/src/main/resources/otel-rules/envoy-ai-gateway/gateway-mcp-instance.yaml b/oap-server/server-starter/src/main/resources/otel-rules/envoy-ai-gateway/gateway-mcp-instance.yaml new file mode 100644 index 0000000000..344515d3eb --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-rules/envoy-ai-gateway/gateway-mcp-instance.yaml @@ -0,0 +1,80 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Envoy AI Gateway — MCP Instance-level (per-pod) metrics +# +# Same metrics as gateway-mcp-service.yaml but scoped to individual pods. +# All durations are in seconds from the AI Gateway; multiply by 1000 for ms display. + +filter: "{ tags -> tags.job_name == 'envoy-ai-gateway' }" +expSuffix: instance(['service_name'], ['service_instance_id'], Layer.ENVOY_AI_GATEWAY) +metricPrefix: meter_envoy_ai_gw_mcp_instance + +metricsRules: + # ===================== Aggregate MCP metrics ===================== + + # MCP request CPM + - name: request_cpm + exp: mcp_request_duration_count.sum(['service_name', 'service_instance_id']).increase('PT1M') + + # MCP request latency average (ms) + - name: request_latency_avg + exp: mcp_request_duration_sum.sum(['service_name', 'service_instance_id']).increase('PT1M') / mcp_request_duration_count.sum(['service_name', 'service_instance_id']).increase('PT1M') * 1000 + + # MCP request latency percentile (ms) + - name: request_latency_percentile + exp: mcp_request_duration.sum(['le', 'service_name', 'service_instance_id']).increase('PT1M').histogram().histogram_percentile([50,75,90,95,99]) * 1000 + + # MCP method invocation CPM — labeled by mcp_method_name + - name: method_cpm + exp: mcp_method_count.sum(['mcp_method_name', 'service_name', 'service_instance_id']).increase('PT1M') + + # MCP error CPM + - name: error_cpm + exp: mcp_method_count.tagEqual('status', 'error').sum(['service_name', 'service_instance_id']).increase('PT1M') + + # MCP initialization latency average (ms) + - name: initialization_latency_avg + exp: mcp_initialization_duration_sum.sum(['service_name', 'service_instance_id']).increase('PT1M') / mcp_initialization_duration_count.sum(['service_name', 'service_instance_id']).increase('PT1M') * 1000 + + # MCP initialization latency percentile (ms) + - name: initialization_latency_percentile + exp: mcp_initialization_duration.sum(['le', 'service_name', 'service_instance_id']).increase('PT1M').histogram().histogram_percentile([50,75,90,95,99]) * 1000 + + # MCP capabilities negotiated CPM — labeled by capability_type + - name: capabilities_cpm + exp: mcp_capabilities_negotiated.sum(['capability_type', 'service_name', 'service_instance_id']).increase('PT1M') + + # ===================== Per-backend breakdown ===================== + + # Backend request CPM + - name: backend_request_cpm + exp: mcp_request_duration_count.sum(['mcp_backend', 'service_name', 'service_instance_id']).increase('PT1M') + + # Backend request latency average (ms) + - name: backend_request_latency_avg + exp: mcp_request_duration_sum.sum(['mcp_backend', 'service_name', 'service_instance_id']).increase('PT1M') / mcp_request_duration_count.sum(['mcp_backend', 'service_name', 'service_instance_id']).increase('PT1M') * 1000 + + # Backend method CPM + - name: backend_method_cpm + exp: mcp_method_count.sum(['mcp_backend', 'mcp_method_name', 'service_name', 'service_instance_id']).increase('PT1M') + + # Backend error CPM + - name: backend_error_cpm + exp: mcp_method_count.tagEqual('status', 'error').sum(['service_name', 'service_instance_id']).increase('PT1M') + + # Backend initialization latency average (ms) + - name: backend_initialization_latency_avg + exp: mcp_initialization_duration_sum.sum(['mcp_backend', 'service_name', 'service_instance_id']).increase('PT1M') / mcp_initialization_duration_count.sum(['mcp_backend', 'service_name', 'service_instance_id']).increase('PT1M') * 1000 diff --git a/oap-server/server-starter/src/main/resources/otel-rules/envoy-ai-gateway/gateway-mcp-service.yaml b/oap-server/server-starter/src/main/resources/otel-rules/envoy-ai-gateway/gateway-mcp-service.yaml new file mode 100644 index 0000000000..482b8edaa3 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-rules/envoy-ai-gateway/gateway-mcp-service.yaml @@ -0,0 +1,85 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Envoy AI Gateway — MCP Service-level metrics +# +# Source OTLP metrics (dots → underscores by OTel receiver): +# mcp_request_duration — Histogram (Cumulative), unit: seconds, labels: mcp_backend, error_type (on error) +# mcp_method_count — Counter (Cumulative), labels: mcp_backend, mcp_method_name, status +# mcp_initialization_duration — Histogram (Cumulative), unit: seconds, labels: mcp_backend +# mcp_capabilities_negotiated — Counter (Cumulative), labels: mcp_backend, capability_type, capability_side +# +# All durations are in seconds from the AI Gateway; multiply by 1000 for ms display. + +filter: "{ tags -> tags.job_name == 'envoy-ai-gateway' }" +expSuffix: service(['service_name'], Layer.ENVOY_AI_GATEWAY) +metricPrefix: meter_envoy_ai_gw_mcp + +metricsRules: + # ===================== Aggregate MCP metrics ===================== + + # MCP request CPM — count of all MCP requests per minute + - name: request_cpm + exp: mcp_request_duration_count.sum(['service_name']).increase('PT1M') + + # MCP request latency average (ms) + - name: request_latency_avg + exp: mcp_request_duration_sum.sum(['service_name']).increase('PT1M') / mcp_request_duration_count.sum(['service_name']).increase('PT1M') * 1000 + + # MCP request latency percentile (ms) + - name: request_latency_percentile + exp: mcp_request_duration.sum(['le', 'service_name']).increase('PT1M').histogram().histogram_percentile([50,75,90,95,99]) * 1000 + + # MCP method invocation CPM — labeled by mcp_method_name + - name: method_cpm + exp: mcp_method_count.sum(['mcp_method_name', 'service_name']).increase('PT1M') + + # MCP error CPM — only error status + - name: error_cpm + exp: mcp_method_count.tagEqual('status', 'error').sum(['service_name']).increase('PT1M') + + # MCP initialization latency average (ms) + - name: initialization_latency_avg + exp: mcp_initialization_duration_sum.sum(['service_name']).increase('PT1M') / mcp_initialization_duration_count.sum(['service_name']).increase('PT1M') * 1000 + + # MCP initialization latency percentile (ms) + - name: initialization_latency_percentile + exp: mcp_initialization_duration.sum(['le', 'service_name']).increase('PT1M').histogram().histogram_percentile([50,75,90,95,99]) * 1000 + + # MCP capabilities negotiated CPM — labeled by capability_type + - name: capabilities_cpm + exp: mcp_capabilities_negotiated.sum(['capability_type', 'service_name']).increase('PT1M') + + # ===================== Per-backend breakdown ===================== + + # Backend request CPM — labeled by mcp_backend + - name: backend_request_cpm + exp: mcp_request_duration_count.sum(['mcp_backend', 'service_name']).increase('PT1M') + + # Backend request latency average (ms) — labeled by mcp_backend + - name: backend_request_latency_avg + exp: mcp_request_duration_sum.sum(['mcp_backend', 'service_name']).increase('PT1M') / mcp_request_duration_count.sum(['mcp_backend', 'service_name']).increase('PT1M') * 1000 + + # Backend method CPM — labeled by mcp_backend and mcp_method_name + - name: backend_method_cpm + exp: mcp_method_count.sum(['mcp_backend', 'mcp_method_name', 'service_name']).increase('PT1M') + + # Backend error CPM — labeled by mcp_backend + - name: backend_error_cpm + exp: mcp_method_count.tagEqual('status', 'error').sum(['mcp_backend', 'service_name']).increase('PT1M') + + # Backend initialization latency average (ms) — labeled by mcp_backend + - name: backend_initialization_latency_avg + exp: mcp_initialization_duration_sum.sum(['mcp_backend', 'service_name']).increase('PT1M') / mcp_initialization_duration_count.sum(['mcp_backend', 'service_name']).increase('PT1M') * 1000 diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/envoy_ai_gateway/envoy-ai-gateway-instance.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/envoy_ai_gateway/envoy-ai-gateway-instance.json index fe314a11c0..adad4dcca8 100644 --- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/envoy_ai_gateway/envoy-ai-gateway-instance.json +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/envoy_ai_gateway/envoy-ai-gateway-instance.json @@ -459,6 +459,162 @@ } ] }, + { + "name": "MCP", + "children": [ + { + "x": 0, + "y": 0, + "w": 8, + "h": 13, + "i": "0", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_instance_request_cpm" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "MCP Request CPM", + "unit": "calls/min" + } + ], + "widget": { + "title": "MCP Request CPM" + } + }, + { + "x": 8, + "y": 0, + "w": 8, + "h": 13, + "i": "1", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_instance_request_latency_avg" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Avg Latency", + "unit": "ms" + } + ], + "widget": { + "title": "MCP Request Latency Avg" + } + }, + { + "x": 16, + "y": 0, + "w": 8, + "h": 13, + "i": "2", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_instance_request_latency_percentile" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Latency Percentile", + "unit": "ms" + } + ], + "widget": { + "title": "MCP Request Latency Percentile", + "tips": "P50 / P75 / P90 / P95 / P99" + } + }, + { + "x": 0, + "y": 13, + "w": 8, + "h": 13, + "i": "3", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_instance_error_cpm" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Error CPM", + "unit": "calls/min" + } + ], + "widget": { + "title": "MCP Error CPM" + } + }, + { + "x": 8, + "y": 13, + "w": 8, + "h": 13, + "i": "4", + "type": "Widget", + "expressions": [ + "aggregate_labels(meter_envoy_ai_gw_mcp_instance_method_cpm,sum(mcp_method_name))" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Method CPM", + "unit": "calls/min" + } + ], + "widget": { + "title": "MCP Method CPM" + } + }, + { + "x": 16, + "y": 13, + "w": 8, + "h": 13, + "i": "5", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_instance_initialization_latency_avg" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Init Latency", + "unit": "ms" + } + ], + "widget": { + "title": "MCP Initialization Latency Avg" + } + } + ] + }, { "name": "Log", "children": [ diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/envoy_ai_gateway/envoy-ai-gateway-service.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/envoy_ai_gateway/envoy-ai-gateway-service.json index e2599eee1b..dc94841c0d 100644 --- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/envoy_ai_gateway/envoy-ai-gateway-service.json +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/envoy_ai_gateway/envoy-ai-gateway-service.json @@ -459,6 +459,240 @@ } ] }, + { + "name": "MCP", + "children": [ + { + "x": 0, + "y": 0, + "w": 8, + "h": 13, + "i": "0", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_request_cpm" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "MCP Request CPM", + "unit": "calls/min" + } + ], + "widget": { + "title": "MCP Request CPM", + "tips": "Total MCP requests per minute" + } + }, + { + "x": 8, + "y": 0, + "w": 8, + "h": 13, + "i": "1", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_request_latency_avg" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Avg Latency", + "unit": "ms" + } + ], + "widget": { + "title": "MCP Request Latency Avg" + } + }, + { + "x": 16, + "y": 0, + "w": 8, + "h": 13, + "i": "2", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_request_latency_percentile" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Latency Percentile", + "unit": "ms" + } + ], + "widget": { + "title": "MCP Request Latency Percentile", + "tips": "P50 / P75 / P90 / P95 / P99" + } + }, + { + "x": 0, + "y": 13, + "w": 8, + "h": 13, + "i": "3", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_error_cpm" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Error CPM", + "unit": "calls/min" + } + ], + "widget": { + "title": "MCP Error CPM" + } + }, + { + "x": 8, + "y": 13, + "w": 8, + "h": 13, + "i": "4", + "type": "Widget", + "expressions": [ + "aggregate_labels(meter_envoy_ai_gw_mcp_method_cpm,sum(mcp_method_name))" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Method CPM", + "unit": "calls/min" + } + ], + "widget": { + "title": "MCP Method CPM", + "tips": "Requests per minute by MCP method (initialize, tools/list, tools/call, etc.)" + } + }, + { + "x": 16, + "y": 13, + "w": 8, + "h": 13, + "i": "5", + "type": "Widget", + "expressions": [ + "meter_envoy_ai_gw_mcp_initialization_latency_avg" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Init Latency", + "unit": "ms" + } + ], + "widget": { + "title": "MCP Initialization Latency Avg", + "tips": "Average time to initialize MCP session with backend" + } + }, + { + "x": 0, + "y": 26, + "w": 8, + "h": 13, + "i": "6", + "type": "Widget", + "expressions": [ + "aggregate_labels(meter_envoy_ai_gw_mcp_backend_request_cpm,sum(mcp_backend))" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Backend CPM", + "unit": "calls/min" + } + ], + "widget": { + "title": "MCP Request CPM by Backend" + } + }, + { + "x": 8, + "y": 26, + "w": 8, + "h": 13, + "i": "7", + "type": "Widget", + "expressions": [ + "aggregate_labels(meter_envoy_ai_gw_mcp_backend_request_latency_avg,avg(mcp_backend))" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Backend Latency", + "unit": "ms" + } + ], + "widget": { + "title": "MCP Latency Avg by Backend" + } + }, + { + "x": 16, + "y": 26, + "w": 8, + "h": 13, + "i": "8", + "type": "Widget", + "expressions": [ + "aggregate_labels(meter_envoy_ai_gw_mcp_backend_error_cpm,sum(mcp_backend))" + ], + "graph": { + "type": "Line", + "showXAxis": true, + "showYAxis": true + }, + "metricConfig": [ + { + "label": "Backend Error CPM", + "unit": "calls/min" + } + ], + "widget": { + "title": "MCP Error CPM by Backend" + } + } + ] + }, { "name": "Log", "children": [ diff --git a/test/e2e-v2/cases/envoy-ai-gateway/docker-compose.yml b/test/e2e-v2/cases/envoy-ai-gateway/docker-compose.yml index 89669cf6fd..c2a64dc2dd 100644 --- a/test/e2e-v2/cases/envoy-ai-gateway/docker-compose.yml +++ b/test/e2e-v2/cases/envoy-ai-gateway/docker-compose.yml @@ -13,12 +13,13 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Envoy AI Gateway e2e — ai-gateway-cli + Ollama + SkyWalking OAP +# Envoy AI Gateway e2e — ai-gateway-cli + Ollama + MCP Everything Server + SkyWalking OAP # # Architecture: -# trigger → ai-gateway-cli (port 1975) → ollama (port 11434) -# ↓ OTLP gRPC -# oap (port 11800) → banyandb +# LLM trigger → ai-gateway-cli (port 1975) → ollama (port 11434) +# MCP trigger → ai-gateway-cli (port 1975) → mcp-server (port 3001) +# ↓ OTLP gRPC +# oap (port 11800) → banyandb services: banyandb: @@ -52,10 +53,27 @@ services: timeout: 60s retries: 120 + mcp-server: + image: tzolov/mcp-everything-server:v3 + command: ["node", "dist/index.js", "streamableHttp"] + networks: + - e2e + expose: + - 3001 + healthcheck: + test: ["CMD-SHELL", "wget -qO- --header='Content-Type: application/json' --header='Accept: application/json, text/event-stream' --post-data='{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"initialize\",\"params\":{\"protocolVersion\":\"2025-03-26\",\"capabilities\":{},\"clientInfo\":{\"name\":\"healthcheck\",\"version\":\"1.0\"}}}' http://localhost:3001/mcp || exit 1"] + interval: 5s + timeout: 10s + retries: 30 + aigw: # TODO: pin to a release version once ai-gateway-cli HTTP listener is available in a release image: envoyproxy/ai-gateway-cli:latest - command: run --run-id=0 + command: + - "run" + - "--run-id=0" + - "--mcp-json" + - '{"mcpServers":{"everything":{"type":"http","url":"http://mcp-server:3001/mcp"}}}' environment: OPENAI_API_KEY: "dummy-key-not-used" OPENAI_BASE_URL: "http://ollama:11434/v1"; @@ -71,7 +89,7 @@ services: networks: - e2e healthcheck: - test: ["CMD", "aigw", "healthcheck"] + test: ["CMD", "/app", "healthcheck"] interval: 5s timeout: 60s retries: 120 @@ -80,6 +98,8 @@ services: condition: service_healthy ollama: condition: service_healthy + mcp-server: + condition: service_healthy networks: e2e: diff --git a/test/e2e-v2/cases/envoy-ai-gateway/e2e.yaml b/test/e2e-v2/cases/envoy-ai-gateway/e2e.yaml index 18aaa417b8..4d8363403a 100644 --- a/test/e2e-v2/cases/envoy-ai-gateway/e2e.yaml +++ b/test/e2e-v2/cases/envoy-ai-gateway/e2e.yaml @@ -15,14 +15,15 @@ # Envoy AI Gateway e2e test (docker-compose) # -# Validates ENVOY_AI_GATEWAY layer metrics and logs via OTLP from ai-gateway-cli. +# Validates ENVOY_AI_GATEWAY layer LLM + MCP metrics and logs via OTLP from ai-gateway-cli. # # Architecture: -# trigger (curl) → ai-gateway-cli (port 1975) → Ollama (port 11434) -# ↓ OTLP gRPC -# SkyWalking OAP (port 11800) -# ↓ -# BanyanDB +# LLM trigger (curl) → ai-gateway-cli (port 1975) → Ollama (port 11434) +# MCP trigger (curl) → ai-gateway-cli (port 1975) → MCP Everything Server (port 3001) +# ↓ OTLP gRPC +# SkyWalking OAP (port 11800) +# ↓ +# BanyanDB setup: env: compose @@ -47,6 +48,28 @@ setup: -d '{"model":"nonexistent-model","messages":[{"role":"user","content":"Hi"}]}' sleep 1 done + - name: Send MCP requests for metric verification + command: | + # Initialize MCP session and capture session ID + SESSION_ID=$(curl -s -D- --max-time 15 \ + http://${aigw_host}:${aigw_1975}/mcp \ + -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"e2e-test","version":"1.0"}}}' \ + 2>&1 | grep -i 'Mcp-Session-Id' | tr -d '\r' | sed 's/Mcp-Session-Id: //') + sleep 2 + # tools/list + curl -s --max-time 15 -o /dev/null \ + http://${aigw_host}:${aigw_1975}/mcp \ + -H 'Content-Type: application/json' \ + -H "Mcp-Session-Id: $SESSION_ID" \ + -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' + sleep 1 + # tools/call — call a nonexistent tool to generate an error + curl -s --max-time 15 -o /dev/null \ + http://${aigw_host}:${aigw_1975}/mcp \ + -H 'Content-Type: application/json' \ + -H "Mcp-Session-Id: $SESSION_ID" \ + -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"nonexistent_tool","arguments":{}}}' trigger: action: http diff --git a/test/e2e-v2/cases/envoy-ai-gateway/envoy-ai-gateway-cases.yaml b/test/e2e-v2/cases/envoy-ai-gateway/envoy-ai-gateway-cases.yaml index 19f11d7dc7..c9fd498d43 100644 --- a/test/e2e-v2/cases/envoy-ai-gateway/envoy-ai-gateway-cases.yaml +++ b/test/e2e-v2/cases/envoy-ai-gateway/envoy-ai-gateway-cases.yaml @@ -45,6 +45,26 @@ cases: - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_envoy_ai_gw_model_latency_avg --service-name=e2e-ai-gateway expected: expected/metrics-has-value.yml - # Access logs — error requests (404) should be persisted by LAL sampling - - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql logs ls --service-name=e2e-ai-gateway + # ===================== MCP metrics ===================== + + # MCP aggregate metrics + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_envoy_ai_gw_mcp_request_cpm --service-name=e2e-ai-gateway + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_envoy_ai_gw_mcp_request_latency_avg --service-name=e2e-ai-gateway + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_envoy_ai_gw_mcp_request_latency_percentile --service-name=e2e-ai-gateway + expected: expected/metrics-has-value-label.yml + + # MCP method breakdown + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_envoy_ai_gw_mcp_method_cpm --service-name=e2e-ai-gateway + expected: expected/metrics-has-value.yml + + # MCP backend breakdown + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_envoy_ai_gw_mcp_backend_request_cpm --service-name=e2e-ai-gateway + expected: expected/metrics-has-value.yml + + # ===================== Access logs ===================== + + # LLM error requests (404) should be persisted with ai_route_type=llm + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql logs ls --service-name=e2e-ai-gateway --tags=ai_route_type=llm expected: expected/logs.yml diff --git a/test/e2e-v2/cases/envoy-ai-gateway/expected/logs.yml b/test/e2e-v2/cases/envoy-ai-gateway/expected/logs.yml index 37a7f56d41..6efc226799 100644 --- a/test/e2e-v2/cases/envoy-ai-gateway/expected/logs.yml +++ b/test/e2e-v2/cases/envoy-ai-gateway/expected/logs.yml @@ -28,6 +28,8 @@ logs: content: {{ notEmpty .content }} tags: {{- contains .tags }} + - key: ai_route_type + value: llm - key: response_code value: "404" - key: gen_ai.request.model
