LetterMCP

Agent Execution Tracing With OTEL in MCP Pipelines

Unifying fragmented traces across agents and MCP servers reveals what actually happened.

· 10 min read · Updated
Cover illustration for “Agent Execution Tracing With OTEL in MCP Pipelines”
Agentic AI Foundations · September 25, 2026 · 10 min read · 2,173 words

An agent doesn't fail the way a normal service fails. A web server either returns a 200 or throws an error, and that binary tells you something useful. An agent can wander through three tool calls it didn't need, take one action that's syntactically fine but wrong for the task, and still hand back an answer that reads as confident and correct. No alarm fires. No stack trace appears. The dashboard stays green while the actual output is garbage, and that gap between "looks fine" and "is fine" is where agent observability earns its keep.

Part of this comes from how agents behave under the hood. The same prompt, run twice, can produce two different tool-call sequences depending on temperature settings, what retrieval turns up, or which tools happen to be available at that moment. A single clean test run tells you almost nothing about what the agent will do in production tomorrow.

MCP servers make this worse because of where they sit. Most production MCP deployments have no visibility into what happens once a tool call crosses that boundary. Often nobody is watching in real time either. The agent decides to call a tool on its own, and if a downstream API slows down and triggers retries, or the agent hallucinates some fallback behavior to compensate, none of that shows up anywhere unless someone instrumented it in advance.

Underneath all of this sits a structural problem, not just a missing dashboard. The agent produces one trace. The MCP server produces a separate trace. What's logically a single workflow appears in telemetry as two disconnected islands because nothing connects the two traces. When something goes wrong, the question that actually matters is simple: why did the agent do that? Without a shared trace context, that question has no answer, because the record of the agent's reasoning and the record of what the MCP server actually executed live in two unrelated places.

The same fragmentation breaks cost tracking. Every tool call can trigger its own chain of API calls, database hits, or compute spend, and without one unified trace tying it all back to a single agent invocation, there's no way to see which decision caused which cost. That's not a hypothetical risk. In one documented case, a fintech agent ran an undetected loop and racked up $47,000 in costs before anyone caught it.

What the MCP spec requires for trace context propagation

The fragmentation described above now has a direct answer at the protocol level. The MCP specification released on July 28, 2026 documents W3C Trace Context propagation natively, which is the change that finally makes a unified trace between an agent and its MCP server possible.

The mechanism is specific. When an AI host or orchestrator kicks off a tool call, it's now required to inject tracing metadata directly into the JSON-RPC request body, inside the _meta object. That small addition is what links the two halves of the workflow. Instead of an agent's reasoning trace ending the moment a tool call crosses into the MCP server, the trace context travels with the request, and the result is one continuous chain: agent reasoning, tool selection, MCP server execution, and whatever downstream calls the server makes.

The same July 28, 2026 release also retired protocol-level logging. MCP servers used to invent their own log formats, and each one was a little different. The spec now tells servers to log to stderr for stdio transports or to use OpenTelemetry for structured observability instead. Before this change, an MCP log message had no natural connection to any span. Making sense of it meant building custom MCP-specific tooling rather than pointing an existing OpenTelemetry pipeline at the data and letting it work. Every server's custom logging also meant every server defined its own spans differently, so correlating what happened across servers was close to guesswork.

These two changes work as a pair. Deprecating protocol-level logging removes the fragmented, inconsistent alternative that teams had been building around. Mandating W3C propagation gives them a single, consistent path to replace it. Before this spec update, trace propagation across the agent-to-MCP-server boundary was optional, which meant it was inconsistent from one implementation to the next, and that inconsistency made unified tracing unreliable to build on.

GenAI semantic conventions and MCP span identity

W3C propagation lets spans from different processes point to the same trace. It doesn't solve the identity problem. Two correlated spans are still useless if one system calls an attribute tool_name and another calls it function_id. That's the gap the OpenTelemetry GenAI semantic conventions close, by giving every span a consistent, predictable shape so that a correlated trace is also a coherent one.

The OpenTelemetry GenAI SIG formed in April 2024, and the conventions it maintains have grown a long way since then, from simple LLM client call tracing to covering agent orchestration, MCP tool calling, content capture, and quality evaluation across six layers of the stack. Version-by-version evolution that matters for MCP pipelines specifically: v1.39 added MCP semantic conventions, introducing attributes including mcp.method.name, mcp.session.id, and mcp.protocol.version. v1.41 added execute_tool span naming requiring the tool name, reasoning tokens, invoke_workflow, streaming metrics, and split invoke_agent into separate CLIENT and INTERNAL variants.

What this buys teams is a shared vocabulary. Operations like create_agent, invoke_agent, plan, invoke_workflow, and execute_tool now carry the same meaning regardless of which framework produced them, so a trace from one agent system reads the same way as a trace from another.

Two metrics act as the floor for any production setup: gen_ai.client.operation.duration, a histogram tracking latency, and gen_ai.client.token.usage, a histogram breaking consumption down by input and output tokens. Without those two numbers, there's no way to reason about cost or speed at all. A couple of smaller attribute decisions matter too. gen_ai.provider.name and gen_ai.request.model are kept separate because the same model can be reached through different providers, Azure OpenAI versus OpenAI direct being the clearest example. And gen_ai.request.model is tracked separately from gen_ai.response.model because fine-tuned models and version-pinned deployments can diverge between what was asked for and what actually answered.

None of this is frozen yet, so if you're instrumenting today, you need to plan around that. As of v1.41, all gen_ai.* attributes carry Development status badges rather than Stable, so attribute names and structures can still change without a major version bump, and there's no public timeline for when they'll stabilize. The spec offers OTEL_SEMCONV_STABILITY_OPT_IN as a way to manage that transition. Building on the conventions now is still the right call, but it comes with an obligation to expect attribute names to shift and to build instrumentation that can absorb that churn rather than break on it. The core concepts, the span types, the general shape of an agent trace, have settled even while the exact attribute names are still moving.

A unified MCP trace across the full stack

Putting W3C propagation and the GenAI conventions together produces a continuous causal chain running from the agent's reasoning, through the MCP server's internals, down into the infrastructure underneath it, all inside one trace.

Compare that to what an AI-observability-only trace gives you. It stops right at the tool-call span. Everything past that point, the MCP server's database queries, its downstream calls, the hardware it's running on, stays completely opaque.

An agent calls a tool named get_order_status, and the call takes several seconds to return. An AI-only trace tells you the tool was slow. It doesn't tell you why. A full-stack trace, by contrast, can show that the slowness came from a sequential scan across millions of rows, caused by a missing index on order_id, running on a pod that was already under memory pressure, turning "the tool call took 3.8 seconds" into an actual fix: add the index, or move the pod.

Instrumenting the MCP server itself is what produces that depth. Each tool call gets its own span carrying mcp.tool.name, latency, and error status. Nested underneath that are spans for the downstream work, database queries, API calls, cache lookups, and infrastructure context like pod, CPU, and memory, stitched into the same trace ID.

Getting that picture right also depends on how errors get recorded. A tool that returns its error as plain text instead of an HTTP 500 looks like a success to any monitoring system that only watches status codes. The span itself has to check the actual content of the response and explicitly set SpanStatusCode.ERROR when it finds one. Skipping that step lets failures hide in plain sight.

Put together, the span hierarchy runs in a predictable order: an agent reasoning span, an invoke_agent CLIENT span, an execute_tool span carrying the tool name, the MCP server's internal spans, and finally the downstream infrastructure spans, all sharing one trace ID carried across the boundary through the _meta field.

Instrumenting an MCP server with OpenTelemetry in practice

Getting that span hierarchy into production comes down to three things: initializing the SDK before anything else loads, wrapping every tool handler in its own span explicitly, and handling error semantics correctly so text-encoded errors don't slip through as successes.

On the install side, the package list is specific and the pieces stay separate rather than bundled: @opentelemetry/sdk-node, @opentelemetry/api, the OTLP trace exporter package (@opentelemetry/exporter-trace-otlp-http), the OTLP metric exporter package (@opentelemetry/exporter-metrics-otlp-http), plus separate resources and semantic conventions packages.

Where that SDK gets initialized matters as much as what gets installed. It belongs in its own file, something like tracing.js, imported at the very top of the entrypoint before any other import runs. Order matters here because the SDK has to register itself before any instrumentation hooks can fire. Import it late and spans quietly stop getting created, with no error to tell you that happened.

The resource configuration needs ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION, and mcp.server.name set so that every span can be traced back to the exact server instance that produced it. Tool handlers then get wrapped in tracer.startActiveSpan(), tagged with mcp.method.name and gen_ai.tool.name, ideally through a shared utility wrapper so the handler code itself stays readable instead of getting buried in instrumentation boilerplate.

Error handling needs the same explicit treatment described earlier: check the response content for text-encoded errors and set SpanStatusCode.ERROR directly rather than trusting that a thrown exception will catch every failure case. Metrics should travel alongside the traces: a counter for tool call volume, a histogram for call duration, and an error counter keyed by mcp.tool.name, so volume and failure rate are both visible without digging through individual traces.

Where those traces end up depends on what the team already runs. A local Jaeger instance, spun up as an all-in-one Docker container, works well for development. An OTLP endpoint pointed at a hosted backend like Honeycomb works for production. Multiple sinks can run at once, and in high-throughput pipelines, probabilistic sampling, something like a 0.1 sample rate, keeps export volume manageable without losing the signal that matters. Plugging into whatever collection pipeline already exists, rather than standing up something new just for MCP traces, is what matters.

Where real-world OTel deployments in MCP pipelines break down

None of this is automatic, even once the spec requires it. Real-world OTel deployments in MCP pipelines routinely contain incomplete causal trees: missing parentage between spans, broken delegation chains, decision context that simply isn't there. A broader Gartner figure shows that around 15% of GenAI deployments had instrumented observability as of early 2026. The unified trace the spec makes possible is not something teams get for free just by upgrading.

A September 2026 audit of the mcp-hangar core found exactly this pattern. Instrumentation was only partial, with parentage defects reproduced locally (one of which got corrected through same-day re-verification), upstream-status defects, and gaps in decision context, even though 97 real-SDK tests passed. Passing tests and coherent traces turned out to be two different things. The tests checked that components behaved correctly in isolation, not that the trace connecting them told a complete story.

Multi-hop delegation is where propagation breaks most often. When one agent spawns or hands off to another agent, the traceparent chain has to be re-injected at every delegation boundary. If an implementation skips this step, the sub-traces it produces never connect, even though each individual hop is instrumented correctly on its own.

The Development-status instability in the GenAI conventions adds another failure mode on top of this. Instrumentation written against v1.39 attribute names can produce spans that v1.41-based dashboards don't recognize, and that mismatch fails silently: the spans still get created, the trace still exists, but the dashboard query built around the newer attribute names comes back empty. Nobody gets an error. They just stop seeing data and have to work backward to figure out why.

These gaps mean something important: a fully propagated, fully conventioned trace is still only a forensic record. It tells a team what an agent did after the fact, in detail, down to the pod and the query. It doesn't, on its own, stop an agent from doing the wrong thing in the first place. Visibility is the precondition for control, not a substitute for it.

Sources

  1. Observing Agentic AI in Production: MCP Server Tracing with OpenTelemetry and Animal Crossing :: PyData London 2026 :: pretalx
  2. How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools
  3. MCP Observability: From Tool Call to Full-Stack Trace
  4. AI Agent Observability: The OpenTelemetry Standard in 2026
  5. epic: make core tracing explain execution and governance decisions · Issue #1264 · mcp-hangar/mcp-hangar
  6. Key Changes - Model Context Protocol
  7. SEP-414: Document OpenTelemetry Trace Context Propagation Conventions - Model Context Protocol
  8. semantic-conventions-genai/docs/gen-ai/mcp.md at main · open-telemetry/semantic-conventions-genai

More in Agentic AI Foundations