AI Orchestration Platform Evaluation Criteria

Cisco's 2026 survey found that 83% of enterprises have deployed or are actively planning AI agents, while only 29% believe they have adequate security capabilities to support them. That 54-point gap is the actual operating environment most teams are building inside right now, whether they've reckoned with it or not.

The reason that gap exists is structural. Traditional API security was designed around predictable, human-initiated requests. An MCP-based agent operates differently at a fundamental level: it makes autonomous decisions, chains tools dynamically, and carries evolving context across multiple interactions. The attack surface isn't just larger. It behaves in ways the old models weren't built to anticipate.

That surface spans five distinct layers: transport and communication security, authentication and identity, context integrity, authorization and privilege management, and supply chain. A platform securing two or three of these while leaving others exposed has a gap that stays invisible until someone finds it first.

The exposure isn't theoretical. A July 2025 internet scan identified at least 1,862 publicly accessible MCP server instances responding to unauthenticated requests. Not test environments. Deployed infrastructure with no access controls. Between December 2025 and January 2026, a single attacker used an AI agent to breach multiple Mexican government agencies, including the federal tax authority and four state governments, by exploiting exactly this kind of uncontrolled agent access.

Platform selection that ignores these conditions produces something operationally capable and institutionally indefensible, and the second surfaces at the worst possible moment.

Agent-Building Capability: What Actually Matters Beyond the Demo

Every platform leads with its builder UX. The demo is almost always impressive. That's the easy part.

The harder question is whether the builder works for the full population of employees who actually need it, not just the engineers who constructed the demo. Can a finance analyst or HR coordinator configure and deploy an agent without filing a ticket? If every deployment requires engineering support, what you have is a developer tool with enterprise pricing. That distinction matters more than most buyers realize until they're six months in and the backlog is forty requests long.

Tool connectivity is the dimension most buyers underestimate. Pre-built MCP server integrations matter because every gap in the catalog becomes a custom connector that someone has to build and maintain. Custom connectors aren't just engineering overhead; they're security surface area that doesn't benefit from the platform's built-in controls. That cost compounds quietly, then suddenly.

Orchestration model is where capability claims diverge most sharply from reality. Multi-agent coordination, conditional branching, and sequential workflow support are qualitatively different from single-agent, single-task execution. The documentation often implies the former while the platform delivers the latter. Evaluate what it actually does under production conditions.

Framework compatibility matters to teams that have already invested in AutoGen, CrewAI, LangGraph, or similar tools. Platforms requiring full migration to a proprietary model impose switching costs and lock-in risk that grows over time.

One dimension buyers consistently underweight: whether the platform makes the secure path also the easiest path. If security requires a separate, friction-heavy step after the agent is built, most users skip it. Runlayer provides access to thousands of MCP servers from a single platform and supports agent setup from a Slack command, which narrows the gap between the technical user path and the one available to everyone else.

MCP-Specific Threats That a Platform's Security Architecture Must Address

MCP has become a widely adopted integration layer for agentic AI, standardizing how agents connect to external tools, data sources, and APIs through JSON-RPC 2.0. Originally released by Anthropic in November 2024, it's now a Linux Foundation standard. That standardization is genuinely useful. It also means the threat vectors against MCP are now well-documented and reproducible at scale.

Prompt injection is the most pervasive. Malicious instructions embedded in external content — a document, a web page, a tool output — can be read by an agent as authoritative commands. In April 2025, Simon Willison documented how tool descriptions visible to the AI model but hidden from the user interface carry adversarial instructions. This is a structural design risk in MCP, not an implementation bug someone forgot to patch. When agents hold access tokens in context, prompt injection becomes a direct credential exfiltration path, and the distance between "someone embedded bad instructions in a PDF" and "your credentials just left the building" gets uncomfortably short.

Tool poisoning is related but distinct. A malicious or compromised MCP server returns data designed to redirect the agent's next action, not just corrupt its current one.

Cross-server tool shadowing is more sophisticated still. Invariant Labs demonstrated how a single malicious MCP server weaponizes adjacent, trusted servers, because the agent treats all tool outputs as inputs to a shared reasoning context. One bad tool, one corrupted workflow.

Intent drift isn't always adversarial, but it's consistently dangerous. As context accumulates across a multi-step chain, agent behavior diverges from its original task in ways that are genuinely hard to detect and harder to explain after the fact, especially in a regulatory inquiry.

Supply chain risk compounds everything above. A significant share of MCP servers are community-built and third-party operated; 492 have been identified as vulnerable to abuse with no authentication or encryption. Connecting to the broader MCP ecosystem without vetting the supply chain is an exposure decision made by default, whether anyone intended it or not.

OWASP's Top 10 for Agentic Applications, published in 2026, classifies goal hijacking as the primary agentic threat category. A platform that lacks real-time detection for these specific threat patterns cannot address the actual attack surface, regardless of what its general API security monitoring captures.

Authentication and Authorization Architecture: What to Look For at the Protocol Level

TLS or mutual TLS is the minimum for transport security. It's not a differentiator; it's a baseline. Any platform that doesn't enforce encrypted transport by default has already failed before the conversation gets interesting.

Beyond transport, the critical criterion is identity-aware, least-privilege tool access enforced at the point of tool invocation, not just at session initiation. Per-task least-privilege means the agent receives only the permissions required for its current step, not a standing grant covering the entire session. The platform has to understand task context, not just user identity. That's a materially harder problem, and most platforms haven't fully solved it.

Standard role-based access control is insufficient for multi-agent workflows. The same user has different permissions depending on which agent is acting on their behalf, which tool is being invoked, and which data context is involved. Hierarchical, resource-scoped access control handles this; flat RBAC does not, and the gap shows up in exactly the workflows that touch the most sensitive data.

Identity provider federation is non-negotiable at enterprise scale. Okta, Azure AD, and Google Workspace group structures must map directly to agent permissions. Any platform requiring manual user provisioning outside the IdP creates an ongoing administrative and compliance liability that accrues quietly until an audit makes it suddenly visible.

For multi-agent architectures specifically, evaluate whether the platform supports the A2A protocol v1.0, which provides OAuth-scoped skill authorization for agent-to-agent communication. Governed under the Linux Foundation, it's the emerging standard for inter-agent auth, and its absence from a platform's roadmap deserves a direct conversation.

SSO ties every agent action to a verified enterprise identity. SCIM enables automated provisioning and deprovisioning as employees join, change roles, or leave. Runlayer supports OAuth-based credential handling, SSO and SCIM integration, and just-in-time credential issuance, which illustrates what identity-integrated authentication looks like in production rather than in a proof of concept.

One scalability point buyers consistently overlook: authorization check volume grows faster than agent count as orchestration complexity increases. A platform that handles auth checks adequately at ten agents will introduce real latency at a hundred. Evaluate horizontal scalability of the authorization layer explicitly, before you need it.

Credential Management: How Platforms Should Handle the Secrets Agents Need to Act

Here's the structural tension that doesn't resolve cleanly: agents are useful precisely because they hold access to systems and data. That same access makes them a high-value target. The utility is the vulnerability. There's no architectural move that eliminates this; there are only approaches that manage it with more or less discipline.

The exposure vectors specific to agent deployments are worth naming precisely. Environment variable exposure: agents that store credentials in context or configuration make those credentials readable by prompt injection. Long-lived tokens — standing API keys or OAuth tokens that persist across sessions — give an attacker extended access if they're ever captured. Shared credentials across agents mean that compromise of one agent's context compromises every workflow using that credential. These aren't edge cases. They're the default behavior of most platforms that haven't specifically engineered against them.

Good credential management addresses each of these directly. Just-in-time credential issuance means credentials are generated per task and expire at task completion, never persisting in agent context. Credential isolation per agent and per workflow means no shared secrets cross agent boundaries. Credentials should never surface to the LLM's context window; the platform handles the authentication handoff without the model ever seeing the token.

Automated rotation policies and immediate revocation capability are the operational complement to issuance controls. When an agent behaves anomalously, the credential it holds needs to be revocable in seconds, not after a ticket gets triaged.

The vendor question worth asking directly: where exactly do credentials live during a tool call, and what prevents the model from reading them? A clear, specific answer is meaningful. A vague one tells you something too.

Runtime Enforcement and Real-Time Threat Detection During Agent Execution

Most platforms leave a gap their architecture diagrams don't show. Authentication happens at session start. Logging happens after execution completes. Neither one stops a prompt injection or tool poisoning attack occurring in the middle of a workflow. That interval is precisely where the interesting attacks live, and it's the interval most platforms leave effectively unmonitored.

Runtime enforcement means policy is checked at tool-call depth, during the execution path, before each invocation completes. The enforcement happens before the action, not after, which is the only temporal position from which it can actually prevent anything.

The detection capabilities that matter at this layer are specific. Prompt injection detection requires scanning tool inputs and outputs for adversarial instruction patterns before they reach the model. Tool poisoning detection requires flagging anomalous tool return values attempting to redirect agent behavior. Intent drift monitoring requires comparing the agent's current action against its original task definition and surfacing divergence in real time. Exfiltration detection requires recognizing patterns consistent with data being staged or moved to unexpected endpoints.

Shadow AI detection belongs here as well. Enterprise governance platforms should identify unsanctioned MCP usage within the organization: agents or tools operating outside the managed environment, often introduced by individual teams solving immediate problems without any coordination with security. This represents an unmonitored attack surface that grows in direct proportion to how useful the technology actually is.

Prominent orchestration frameworks, including AutoGen, CrewAI, LangGraph, and Google ADK, do not themselves specify autonomy boundaries or intervention triggers. Runtime enforcement must come from the platform layer. Deploying agents through those frameworks without a platform that adds this layer leaves the execution environment effectively unmonitored.

Runlayer's architecture includes real-time detection for tool poisoning, prompt injection, intent drift, and exfiltration, which illustrates what a platform looks like when it actually addresses this layer rather than leaving it as a customer implementation problem.

Audit Logging and Traceability: The Evidence Layer That Makes Governance Real

Audit logging is where governance becomes either real or aspirational. The logs either support an actual investigation under pressure, with a compressed timeline and real stakes, or they don't. There's no middle position when a regulator or an incident response team is asking specific questions.

The minimum bar: immutable logs capturing every model invocation, every tool call, every policy enforcement decision, and every agent handoff, with timestamps and agent attribution on each event. Most platforms claim to meet it. Fewer actually do when you stress-test the specifics.

Above that floor, tamper-proof storage is the first elevated criterion. A mutable log isn't an audit log; it's a record that can be altered after the fact, which makes it legally and operationally unreliable in exactly the moments when reliability matters most.

Full OpenTelemetry-compatible tracing allows teams to reconstruct the complete execution path of any agent workflow, including cross-agent handoffs. This matters specifically for multi-agent architectures where a single user action triggers a chain of sub-agents across different systems, and where "the agent did it" is not an audit-grade answer for GDPR, HIPAA, SOC 2, PCI DSS, or the EU AI Act.

Forensic depth is harder to evaluate from a vendor sheet but critical in practice. Logs should capture what the agent read, what it was instructed to do, what it actually did, and what policy was applied at each step. Identity attribution at the action level is a regulatory requirement, not a preference.

Cross-framework coverage exposes gaps in platforms orchestrating agents built on multiple frameworks. If the audit record is inconsistent depending on whether the agent was built in LangGraph versus AutoGen, the governance layer is effectively framework-specific, which means it isn't a governance layer at all.

The evaluation question is direct: can the platform produce a complete, identity-attributed record of any agent action on demand, or only a partial log of what was explicitly instrumented before the fact?

Governance Architecture: How the Platform Handles the Full Enterprise Agent Estate, Not Just Individual Agents

Individual agent controls, however robust, don't scale. Enterprises deploying dozens or hundreds of agents need a control plane spanning the entire estate, not a collection of per-agent configurations requiring manual coordination to stay consistent.

Continuous discovery is the first requirement at the fleet level. The agent inventory must be automatic: every active agent, every MCP server connection, every skill or tool in use. A manual registry goes stale immediately, because agents are being added faster than anyone is updating the documentation.

Policy enforcement at scale means the same controls that apply to the first agent apply to the hundredth without per-agent configuration. If scaling the agent estate requires proportional growth in security operations effort, you don't have a governance model. You have a staffing plan that will eventually break.

Delegation and sub-delegation controls are where fleet governance gets genuinely complex. When an agent spawns a sub-agent or hands off a task, the original permission scope must attenuate through the chain, not persist or expand. An agent that can delegate its full permissions to a sub-agent, which can delegate its full permissions further down, is a privilege escalation mechanism, not a controlled system.

Autonomy boundaries matter most to the business stakeholders who ultimately own the risk. The platform must allow operators to define, per workflow type, where human-in-the-loop review is required and where autonomous execution is permitted. That boundary should be adjustable as confidence in a workflow's behavior accumulates over time, and the platform needs to support that adjustment without requiring a re-architecture.

Research from Tomašev et al., published in 2026, proposes frameworks covering sub-delegation, permission attenuation, and reputation systems for AI delegation. The fact that this remains an active research area is itself informative: most commercial platforms haven't fully solved the delegation problem, which means buyers should evaluate what a platform has actually built today, not what the category is theoretically capable of.

Gartner projects 40% of enterprise applications will incorporate AI agents by the end of 2026. At that penetration, the platform a company selects now doesn't just determine which agents it can build. It determines the shape of its entire AI operating model, including which risks it has quietly normalized, which compliance gaps it has inherited, and how much remediation it will owe when the regulatory frameworks finish solidifying.