LetterMCP

AI Orchestration Platform Evaluation Criteria

Security gaps in MCP deployments outpace the protocol's ability to prevent them.

· 9 min read · Updated
Cover illustration for “AI Orchestration Platform Evaluation Criteria”
Agentic AI Foundations · August 11, 2026 · 9 min read · 2,019 words

Evaluating an AI orchestration platform starts with understanding what makes the Model Context Protocol different from every integration pattern that came before it: that difference is architectural. In a conventional API, a client asks a server for data and the server responds. MCP inverts that relationship: the server queries and executes actions on behalf of the connected client, opening attack paths that simply have no equivalent in the conventional model. That inversion is the reason a security review built for traditional APIs will miss what matters most in an MCP deployment.

The protocol compounds the problem by blending instructions with data. A tool's natural-language description sits in the same channel as the data it returns, so rewriting that description can redirect an agent's behavior with the same force as rewriting its system prompt, without touching a line of code. The NSA's May 2026 cybersecurity information sheet calls MCP's design "flexible and underspecified," a choice that gives implementers room to build but leaves the question of safe usage open to interpretation. The same sheet notes that some implementations let actor-controlled inputs reach execution environments without proper constraints, and that produces vulnerabilities the agency generally rates as high severity.

Production deployments make the exposure worse by concentrating credentials. MCP servers typically sit in front of several enterprise systems at once, holding the keys to all of them behind a single interface. The convenience that makes MCP attractive to adopt is the same property that turns one compromised server into a single point of failure across every system it touches.

None of this is a theoretical concern confined to early pilots. The Agentics' Enterprise MCP Guide 2026 puts MCP at 97 million monthly downloads, with a substantial share of technical leaders already reporting production use. Only a small fraction of pilots are making it to governed production, and the primary blockers are identity management, auditability, and access-control gaps. Microsoft's security research cites IDC projections showing that the number of active AI agents in enterprises will grow from tens of millions today to billions by 2030. A protocol built with an underspecified security model is becoming load-bearing enterprise infrastructure faster than that model is maturing, and that gap is what every evaluation criterion that follows is meant to close.

The threat taxonomy an evaluation checklist must be built around

Each attack class that has emerged against MCP-based orchestration platforms in 2026 exploits the distance between what a human reviewer approved and what the agent actually carries out at runtime.

Tool poisoning is the clearest example. An attacker alters a tool's natural-language description metadata, not its name and not its user-facing summary, after the tool has already passed approval. Because MCP reflects metadata updates dynamically, those new instructions go live the moment the change is made, in any configuration where description edits don't trigger a fresh review. Microsoft's Incident Response team traced one such attack through a finance workflow: a third-party invoice enrichment server had been approved for production, then had its description silently altered to instruct the agent to pull unpaid invoices and attach them to the next outbound call to an attacker-controlled server. No re-approval fired, because none was built to fire on a description change. The technique was first disclosed by Invariant Labs in April 2025 and has since been observed against a widening range of enterprise agents through 2026.

A second class is exposure through missing authentication. Trend Micro identified 492 MCP servers sitting on the open internet, and you could reach every one of them without authentication.

A third class is credential architecture that was weak from the start. Practical DevSecOps's report, drawing on an Astrix Security audit, found that 53% of scanned MCP servers rely on static API keys or personal access tokens rather than any modern auth standard, and most of those pass the keys through environment variables, with only a small fraction using OAuth at all.

A fourth class covers confused deputy and token passthrough patterns. In the confused deputy version, the MCP server acts using its own elevated privileges instead of checking the requesting user's: a user with no database admin rights asks the agent to run a query, and the server, which does hold admin rights, complies without verifying the user's actual access. Token passthrough compounds the same failure: it forwards tokens that were never issued to the receiving server and skips audience validation entirely, a pattern the MCP specification explicitly forbids.

A fifth class is lateral movement between agents. Cisco's 2026 report documents a case where an attacker poisoned the external data source of a low-privilege data query agent, then used that agent's cross-agent communication channel to send disguised requests to a higher-privilege financial approval agent.

Why authentication and authorization architecture must be evaluated first

Diagram: The Authentication Gap: Where MCP Deployments Actually Stand. Visualizes: Visualize the stark contrast between three authentication metrics that reveal how exposed most MCP deployments are in practice.

Authentication and authorization architecture has to come first in any evaluation because every other control, logging, policy enforcement, lifecycle governance, only functions if the platform can first establish, reliably, who or what is taking an action. A platform with excellent audit logs attached to credentials nobody can trust is still a platform that cannot tell a legitimate request from a forged one.

The industry baseline makes this urgent rather than academic. Practical DevSecOps reports that only 8.5% of MCP servers use OAuth; most still rely on static API keys or personal access tokens passed through environment variables, so it functions as the architectural equivalent of a shared password handed to every system on the network.

Against that baseline, a platform under evaluation needs to demonstrate that unmanaged tool lifecycles are governed through a registry with inventory of all approved servers and skills, version history, approval workflow per version, and automated detection when a tool's metadata diverges from its approved state. It needs audience-bound tokens issued per MCP server, because without that binding, a token stolen from one server is valid against another, and a platform that cannot enforce this fails the evaluation outright. It needs to run on OAuth 2.1 for authentication and SCIM for identity lifecycle management rather than a custom-built mechanism, since custom auth is where implementation flaws concentrate. It needs a governed gateway that enforces tool allowlisting and requires human-in-the-loop approval for high-risk actions, so it can't just function as a bare API passthrough. And it needs single sign-on integration so agent identity ties back into the enterprise's existing identity systems instead of living in a separate, unmanaged credential store.

The confused deputy and token passthrough failures described in the threat taxonomy occur frequently, including under ordinary conditions. They are the direct, predictable consequence of skipping this architecture: a platform that does not bind credentials to the requesting identity by design will produce exactly these failures once it reaches production. That is why auth functions as a gate rather than one line item among many.

Agent identity and credential management as a governance prerequisite

Authorization starts with identity. An enterprise cannot apply differentiated permissions, investigate a misuse case, or shut off one misbehaving agent if several agents are sharing the same account or the same credential.

A platform should assign every production agent a distinct machine identity: a unique identifier, a named business or technical owner, an approved purpose, a defined trust level, a documented lifecycle state, and clear credential and expiration details. This is not yet common practice. A CSA/Strata study, cited in Practical DevSecOps's MCP Security Statistics report, finds that only 23% of organizations have a formal, enterprise-wide agent identity strategy. That makes identity architecture a genuine differentiator between platforms rather than a feature every vendor can be assumed to have already solved.

Just-in-time credential provisioning limits what a compromised agent can do. Credentials that exist only for the duration of a single task can't be replayed once that task finishes, which shrinks the blast radius of any single breach. You should test whether a platform supports this natively, or whether it depends on bolting on a separate secrets management tool. Automated token rotation and cryptographic agent verification address the static API key problem documented earlier, so vendors should be asked to show how often rotation happens and what triggers it.

The lateral movement case Cisco documented in 2026 turns on exactly this gap. The attacker moved from a low-privilege agent to a high-privilege one because the platform had no signed delegation token and no scoped trust boundary separating them. To evaluate a platform on this point, you check whether delegation between agents is cryptographically scoped and whether depth is verified on receipt, not assumed. Identity answers who is acting. Tool governance takes over from there, determining what that identity is allowed to invoke.

Tool lifecycle governance and the re-approval gap that lets poisoning succeed

Diagram: How Tool Poisoning Bypasses a One-Time Security Review. Visualizes: Show the staged sequence of the Microsoft finance workflow attack as a stepped flow: (1) Third-party invoice enrichment server submitted and approved for production; (2)…

Tool poisoning doesn't succeed because platforms skip security review. It succeeds because that review happens once, at initial approval, and most platforms have no mechanism to detect or re-evaluate changes made to a tool afterward.

The Microsoft finance workflow case lays this out clearly. The enrichment server passed its initial review, got approved for production, and was then modified in a way that changed only its description, not its name and not its user-facing summary, so no re-approval ever triggered, because the platform's governance process wasn't watching description fields for changes. Any evaluation checklist has to ask whether the platform treats a tool description change as an event that triggers re-approval. If the answer is no, the initial security review is providing a false sense of coverage.

Lyzr's September 2026 guide names unmanaged tool lifecycles, servers deprecated without anyone formally retiring them, old versions still running because nothing forces a re-review, as one of the four dominant MCP enterprise risks. A governed MCP registry is the operational fix: an inventory of every approved server and skill, a version history, an approval workflow tied to each version, and automated detection when a registered tool's live metadata diverges from what was approved.

The supply chain dimension of this problem appears at scale in marketplace sourcing. Snyk's ToxicSkills audit of the ClawHub marketplace, as covered in CyberDesserts' analysis, found that 13.4% of published skills contained critical issues. Sourcing tools from a marketplace without version pinning and ongoing integrity verification is where this risk compounds across an entire catalog rather than a single server.

The practical test for any vendor is specific: update a tool description on a connected MCP server and watch what the platform does. Does it detect the change, flag it, pause execution, and make the agent wait for re-approval before it can use the updated version? If a platform can't answer that cleanly, it has a lifecycle gap wide enough for the Microsoft finance case to repeat itself, and even a platform that closes this gap still has to show whether anyone can reconstruct after the fact what happened.

Audit trails and observability as discriminating criteria, not table stakes

Vendor marketing sells audit and observability as a solved problem, but put most platforms under real scrutiny and they fail this. Logging that an action occurred is not the same as logging enough context to reconstruct why it happened, who authorized it, and what data it touched. That distinction is what separates a platform that merely claims observability from one that can actually produce evidence an auditor or an investigator can use.

An audit record covering what data was accessed, what decision was made, what context drove that decision, which tool was invoked, which identity authorized it, and whether human approval was in the chain can support SOX, HIPAA, and GDPR review.

An audit trail that can be modified by a compromised agent or a rogue administrator does not satisfy compliance obligations in regulated industries. Full OpenTelemetry tracing, with end-to-end trace IDs connecting every hop, is what lets you correlate across a multi-agent workflow at all. Without that kind of tracing, you can't reconstruct something like the cross-agent lateral movement case Cisco documented, not just with more difficulty, but not at all.

That is the standard an evaluation has to hold platforms to: not whether they log, but whether what they log would actually let a security team rebuild the full chain of a compromise after it's already happened.

Sources

  1. Securing AI agents: When AI tools move from reading to acting
  2. Model Context Protocol (MCP): Security Design ...
  3. A First Measurement Study on Authentication Security in Real-World Remote MCP Servers

More in Agentic AI Foundations