Repository navigation
feat(telemetry): trace each MCP request and tool call under the caller's trace - #103
braghettos wants to merge 2 commits into
Conversation
…r's trace Wrap the MCP HTTP handler in otelhttp, so every request gets a server span that is a child of the incoming W3C traceparent. The standalone SSE stream (GET) is filtered out: it stays open for the whole session. The tools/call span is now named "tools/call <tool>" and carries mcp.method.name and gen_ai.tool.name next to the existing mcp.tool.* attributes. It is parented on the server span of the request that made the call. The SDK runs handlers on the session's context, which holds the trace context of the initialize request, so the middleware writes the server span's context into a cloned request's traceparent header and the tool middleware extracts it from CallToolRequest.Extra.Header. No header, argument or result values are added to spans. Part of kagent-dev#102 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016RAxvkp7wRXwXDx4Lx8STi Signed-off-by: Diego Braga <diego.braga86@gmail.com>
|
@braghettos thanks for both PRs! #103 and #107 both change Could you combine them into a single PR? You could stack #103 on top of #107 and close one of them, or squash both into one branch. Please keep the tests from both ( |
|
@braghettos update: #107 is now merged, so please disregard the request to combine. Please rebase #103 on |
Brings in kagent-dev#107 (per-call token kubeconfig, redaction). Conflict in internal/telemetry/middleware.go was the import block only: both redact (kagent-dev#107) and otelhttp (this PR) are kept. pkg/k8s/token_passthrough_test.go (kagent-dev#107) now looks for the tool span under its new name, "tools/call k8s_get_resources". Signed-off-by: Diego Braga <diego.braga86@gmail.com>
Part of #102
What
With tracing enabled, a tool call made over MCP HTTP now shows up in the caller's trace:
telemetry.HTTPMiddlewarenow wraps the handler inotelhttp.NewHandler(go.opentelemetry.io/contrib/instrumentation/net/http/otelhttpv0.72.0, the release that matches the otel v1.47.0 SDK already ingo.mod). The span is a child of the incoming W3Ctraceparent. The standalone SSE stream (GET) is filtered out, because it stays open for the whole session and its span would last for hours and parent nothing. POST responses that stream SSE work as before: otelhttp wraps the writer with httpsnoop, soFlushis kept.ToolMiddleware(the single receiving middleware every tool already goes through) names its spantools/call <tool name>(wasmcp.tool.<tool name>) and adds the MCP semconv attributesmcp.method.nameandgen_ai.tool.name. The existingmcp.tool.*attributes are still there. Error status on anIsErrorresult or a Go error is unchanged.Why the tool span needed more than a server span
The SDK does not run handlers on the HTTP request's context. A stateful session's handlers run on the context of the request that opened the session (
initialize). So even before this change,mcp.tool.*spans joined the trace of the session'sinitializerequest, not the trace of thetools/callthat triggered them. What does reach the handler for each request is that request's header (CallToolRequest.Extra.Header). So:HTTPMiddlewareclones the request and writes the server span's context into the clone'straceparentheader. The caller's request is not modified.ToolMiddlewaredrops whatever span is in the session context and extracts the parent fromExtra.Header. With no header (stdio, in-memory) it keeps the old behaviour.The middleware also no longer re-extracts trace context on top of an active span. That would have replaced the otelhttp server span with its remote parent.
What is not recorded
No request header, tool argument or tool result is added as a span attribute. Headers can carry credentials, and #66 removed header attributes for that reason. otelhttp's defaults record method, path, status, sizes, peer address and
user_agent.original, and no other header.Tests
cmd/tracing_test.gobuilds the HTTP handler the wayrun()does (new helpernewMCPHTTPHandler) and records spans withtracetest.InMemoryExporter. It connects a real go-sdk client whose transport sets a differenttraceparentoninitializeand on eachtools/call, plus anAuthorizationheader. It calls a succeeding tool, a tool that returns an error result, and a tool that returns a Go error. It asserts:tools/callrequest, with the incoming trace ID and a remote parent with the incoming span ID;tools/call <name>span, a child of that server span and in thetools/callrequest's trace (not theinitializetrace);Okfor the succeeding tool andErrorfor both failing tools;mcp.method.nameandgen_ai.tool.nameare set;header,authorizationortoken, and no attribute value contains the bearer credential or the tool argument.When
toolSpanParentis disabled, the test fails on the trace-ID and parent assertions, which confirms the session-context problem described above.make lint(golangci-lint v2.15.0): 0 issues.go vet ./...is clean.go test -tags=test ./pkg/... ./internal/... ./cmd/...passes.Follow-up (not in this PR)
This PR covers only the first half of #102. The second half is the k8s tools: they run
kubectlas a subprocess, which cannot carry trace context. That is left for a follow-up, which would use client-go in-process for the read verbs, with anotelhttptransport, sotraceparentreaches the API server.Note
Dashboards or queries that match the old span name
mcp.tool.<tool>need to switch totools/call <tool>or filter on themcp.tool.name/gen_ai.tool.nameattribute.🤖 Generated with Claude Code
https://claude.ai/code/session_016RAxvkp7wRXwXDx4Lx8STi