Testing MCP Tool Calls With Automated Agent Harnesses

Testing MCP tool calls with automated agent harnesses means testing both what the agent does and what the server returns. A harness must catch schema drift, broken authentication, poisoned descriptions, and intent drift over the course of a run. That takes several testing layers built around the full sequence of tool calls, with runtime monitoring covering changes that happen after the tests pass.

Why MCP tool calls are structurally harder to test than ordinary API calls

In a conventional API test, the contract sits between a caller and an endpoint. The test sends a payload and checks the response. In MCP, the agent also reads tool names, descriptions, and schemas as context for its reasoning. Those definitions help it decide what to call and how to use the result. A tool can thus take valid input and return valid JSON but still lead the agent to take the wrong action.

Tool descriptions sit beside the instructions in the agent’s working context, creating a trust boundary where a description that has been tampered with can steer what the agent does as well as describe a service. No HTTP response code exposes that failure. Checkmarx’s MCP risk taxonomy separates the host, client, and server into distinct architectural layers, each with its own attack surface. Testing only the server leaves failures at the other layers outside the test.

Authentication adds another structural difference. The July 2026 spec revision made MCP fully stateless, retiring the session handshake and the Mcp-Session-Id header. Each request now has metadata with its protocol version, client identity, and client capabilities. Authentication state cannot simply be assumed to persist from an earlier exchange, because each call stands alone. These differences change what counts as a failure. The harness must be designed around more than endpoint responses.

The four categories of failure a harness must be designed to catch

Schema drift happens when a tool’s input schema changes between the version that was tested and the one the agent sees at runtime. The agent may construct a call the server rejects, or encounter a schema containing malicious fields. Checkmarx’s MCP risk taxonomy identifies schema tampering as a downstream compromise vector. The detection requirement is a comparison of the live tools-list against a pinned baseline.

Auth breaks occur when a stateless call lacks valid authentication for the server receiving it. A setup-time token reused throughout a test does not exercise the required per-call authentication checks. The 2026-07-28 spec requires servers to implement OAuth 2.0 Protected Resource Metadata (RFC 9728) and clients to send the resource parameter (RFC 8707) on every authorization and token request. Because a server must reject tokens that were not issued for it, tests must check replay and cross-server token reuse.

Poisoned descriptions hide instructions in tool metadata that the agent trusts as part of its context. The tool need not be called, because loading its description can be enough to redirect behavior. Tool shadowing takes that attack to a trusted server’s tool, while a rug pull waits until after approval to make the hostile change. The harness needs cases with malicious metadata to test what the agent does when the tool or the timing of the change differs.

Intent drift happens when calls that each seem sound add up to an outcome the user never authorized. OWASP calls this Intent Flow Subversion, and a confused deputy is a compound form in which Server A induces misuse of Server B’s credentials or capabilities. Poor descriptions can also impair task success without an attacker. The February 2026 study “MCP Tool Descriptions Are Smelly!” found widespread description defects, including unstated limitations and opaque parameters. These failures need checks of the whole run as well as checks of each call on its own.

What trajectory evaluation means in practice and why it replaces response assertion

Trajectory evaluation asks which tools the agent called, in what order, with what arguments, and how it interpreted their results. It replaces the final response as the main test of success because the answer can hide a dangerous path. An agent might call a restricted tool, pass sensitive data through an intermediate response, or follow poisoned instructions before getting back on track by chance. An agent that reaches its destination through a confused deputy has failed the test.

The trace becomes the test artifact because it preserves the sequence that needs judgment. A documented Braintrust experiment recorded 14 MCP tool calls across many turns and provided a permalink to the full internal trace. The important distinction is not the length of that run but what remained available for evaluation. Final response text shows the destination, while the trace exposes the route. Assertions can then check whether the agent stayed within what the task allowed it to do along that route.

A mock MCP server lets tests control the conditions for that judgment. Recording and replaying tool responses allows tests to run without external network dependencies or mutations to production data. Deterministic replay gives the harness a stable basis for evaluating known cases, including hostile responses. OpenTelemetry supplies the instrumentation layer, and mcp-eval demonstrates the pattern of deriving assertions directly from OTEL traces. That is an architectural principle rather than a product recommendation, and it applies whether a test uses controlled replay or a live agent-to-server exchange.

How current harness tools approach trajectory testing, schema diffing, and security-specific probes

MCPJam began as a fork of the official MCP Inspector and was later rebuilt from the ground up. It combines protocol inspection, OAuth conformance testing, and agent-driven testing for MCP servers, MCP apps, and ChatGPT apps. Its OAuth coverage directly addresses authentication failures. Protocol inspection can catch malformed schemas and broken flows, but that layer alone does not establish how an agent behaves when processing tool outputs.

For trajectory testing, mcp-eval sends a real agent to a running MCP server and derives its metrics and assertions from OpenTelemetry traces. It supports @task decorators, pytest fixtures, and dataset-driven cases against servers written in any language. Its real-environment approach is distinct from mock-based replay, even though both can support the same emphasis on the call path. Braintrust records full tool-call traces and supports CI gates through GitHub Actions. That makes it suitable for regression testing when server definitions change, with the trace available to judge more than the final answer.

Security probes cover a different part of the problem. Snyk Agent Scan, formerly mcp-scan, reports scored risks in v0.6 and later across prompt injection, untrusted content, private data, and destructive capabilities. In v0.5.x, it named its categories tool poisoning, tool shadowing, and toxic flows. Promptfoo’s redteam-mcp-agent configuration tests real agent behavior when handling tool outputs, including unauthorized cascades and sensitive data leaks through return values. The open-source red-team-blue-team-agent-fabric harness provides executable scenarios spanning MCP, A2A, x402/L402, decision governance, benchmark integrity, human-in-the-loop, and skill supply chain attacks.

These capabilities are not interchangeable. A protocol pass shows only that tools can be listed and calls can return clean JSON. Trace collection makes behavioral assertions possible, but collecting a trace is not itself an assertion. Security probes test hostile cases, while checks for schema drift still need to compare the schema with a baseline. Covering all four failure categories therefore requires deliberate composition of multiple tools.

Composing a harness that covers all four failure categories

The first layer owns schema pinning and diffing before any agent runs. At setup, the harness snapshots the live tools-list and compares it with a pinned baseline. Any added, removed, or modified tool definition is a test failure. Because hash-based pinning is no longer available in Snyk Agent Scan, the harness itself must own the snapshot-and-diff step.

The second layer owns stateless authentication. Each tool call should be tested with its own independently issued token rather than a session-scoped token carried from setup. Cross-server replay cases must test the resource parameter (RFC 8707) and check that the server receiving a token rejects it if it was not issued for that server. For unattended agents, including CI/CD pipelines and background daemons, the tests should also exercise the OAuth Client Credentials extension. That covers direct authentication with the authorization server instead of treating interactive authorization as the only path.

The third layer owns description poisoning probes. The mock server provides descriptions known to be malicious, with hidden instructions, shadowing variants, and rug-pull timing patterns. The assertion checks whether the agent follows those instructions. Promptfoo’s redteam configuration and Snyk Agent Scan provide explicit support for this probe category in CI. Their role is to expose hostile content and unsafe behavior alongside the definition baseline and authentication checks.

The fourth layer checks for intent drift with multi-step trajectory assertions. Each run is judged against a declared expected sequence of tool calls and arguments, with deviation treated as failure even when the answer looks correct. A confused deputy case must explicitly present the agent with credentials for Server B while it processes a response from Server A, then assert that Server B’s credentials are not used. OpenTelemetry is the instrumentation backbone for this layer. mcp-eval’s trace-derived assertion model provides the reference pattern, while its live-server execution remains distinct from the mock infrastructure used for controlled cases.

All four layers belong in CI on every commit that touches a tool definition, server config, or agent prompt. They run against the mock MCP server so known cases can be exercised without changing production data. Each layer has a separate responsibility, which prevents a successful protocol check from standing in for a behavioral test.

What the harness alone cannot catch: runtime threats that require continuous monitoring

A passing harness result applies only to the definitions and conditions tested. The Deadbugz campaign demonstrated that limit by behaving normally for its first three tool calls before changing its metadata to instruct the agent to collect credentials. A harness can test that timing pattern as a known case. It cannot stop a server that was clean from changing after deployment. Rug-pull probes and runtime change detection serve different purposes.

Supply-chain compromise can also happen outside what the tests cover. The August 2026 Shai-Hulud wave used the official MCP Registry as a distribution channel and lingered for an extended period in some cases, with the entry’s removal timeline unconfirmed. The September 2026 incident involved a hijacked maintainer account shipping a malicious release with valid npm provenance. Both bypassed a pre-deployment scan of the original package. An earlier pass says nothing about a replacement that was not part of that test.

Runtime controls need to catch changes to tool definitions after approval. Every server upgrade and tools-list change must be flagged as a configuration change that affects security and checked again before the agent uses the new definitions. Live calls also need per-invocation identity through SSO, SCIM, and just-in-time credentials, so each action is tied to a specific human-authorized identity as well as an agent token. Monitoring must detect tool poisoning, prompt injection in responses, and intent drift as live runs unfold. Test logs alone cannot supply that control unless something consumes and acts on the production evidence.

Tamper-proof audit logs and full OTEL tracing are required for every production tool invocation. Multi-agent accountability depends on pre-action governance, because attribution chains that were not captured in real time cannot be reliably reconstructed later. The harness is the quality gate for behavior against known good configurations. Runtime monitoring governs what happens when those configurations change without warning.