Open Source AI Agents for Enterprise Evaluation

Enterprise teams are rolling out MCP-connected agents fast, but the controls around them haven't caught up, and that's where most of these projects get stuck. The Agentics Enterprise MCP Guide 2026 puts a number on it: only 11 to 14 percent of enterprise agentic AI pilots make it to production. The rest don't stall because the agent can't do the job. They stall on identity, audit, and access-control gaps.

The appetite for MCP is not in question. SDK downloads have reached tens of millions per month by early 2026, and every major AI platform vendor, OpenAI, Google DeepMind, Microsoft Copilot Studio, and AWS, now ships MCP support out of the box. Downloaded and deployed are not the same thing, and governed production is a third category entirely, separated from the first two by a structural problem rather than a staffing shortage⟫.

Evaluating an open-source agent for enterprise use means checking it against five separate dimensions before it ever touches production data: capability fit, identity and authentication, auditability, MCP security posture, and governance readiness. Each one catches a different failure mode.

What "capability fit" means for enterprise open-source agent evaluation

Capability evaluation for enterprise use isn't a question of what an agent can pull off in a demo. You need to know whether the agent's underlying architecture can carry enterprise workload constraints, because if it can't, the security team ends up bolting on controls the framework was never built to support.

Among the open-source frameworks enterprises are evaluating in 2026, multi-agent orchestration tools like CrewAI come up often. CrewAI defines agents with roles, goals, and backstories that work together as a "crew," and it has first-class support for both MCP and the Agent-to-Agent protocol.

When you run a capability evaluation, you need a short list of concrete questions. Does the framework natively support MCP tool integration, or does connecting it to enterprise systems require custom connectors that introduce new, unreviewed code? If it supports multi-agent orchestration, your security team needs to see the communication paths between agents, and limit them. Is there a sandbox or staging environment built into how the framework deploys, so an agent can run against realistic but non-production data before it goes live?

The common failure is evaluating capability on its own: teams ask "can the agent finish this workflow?" and push the governance questions into a future sprint. That sprint rarely comes, because by the time anyone circles back, the agent is already running against production data. Governance controls belong inside the capability evaluation from the start: role-based access control scoped to the individual agent, audit trails for every action and tool call, multi-level approval gates for high-stakes decisions, and sandbox environments built in rather than added on.

A pilot should not move forward until the team can answer, for every tool the agent is allowed to invoke: who approved it, what data it can reach, and what the rollback path looks like if the agent does something unexpected. If that list has gaps, the agent isn't ready, no matter how clean the demo looked.

The identity and authentication gap that open-source agents expose

Most enterprise environments have no real strategy for agent identity. So open-source agents often run under borrowed or shared credentials, and that means access reviews tell you nothing, and nobody can read the audit logs.

The failure is most visible at offboarding. When an employee leaves, any agent deployed under that employee's OAuth credentials should get reviewed, but agents rarely appear on standard offboarding checklists. So you end up with orphaned agents that keep operating, still carrying credentials tied to someone who no longer works there.

NIST SP 800-207A lays out what sound agent identity actually requires. Every production agent needs a distinct machine identity: a unique identifier, a named business owner, an approved purpose, a defined trust level, a documented lifecycle state, and clear credential and expiration details. Credentials need to be short-lived and cryptographically verifiable, checked at each connection and reauthenticated on a regular schedule, not static API keys sitting in a config file somewhere. The control stack follows directly: a distinct identity for each agent, token lifetimes measured in minutes with just-in-time issuance, OAuth 2.1 with PKCE, delegation handled through Token Exchange under RFC 8693, and attribute-based access control enforced at runtime.

The protocol itself has started moving in this direction. The MCP specification dated 2026-07-28 aligns MCP authorization with OAuth 2.1 and OpenID Connect. MCP clients now have to implement Resource Indicators under RFC 8707, so a server can't take a token issued for another and reuse it.

The Enterprise-Managed Authorization extension, stable since June 18, 2026 and already being adopted by Anthropic, Visual Studio Code, and Okta, solves a real problem: it lets organizations manage authorization centrally so employees reach every connected MCP server through one login instead of granting consent server by server. EMA was built for flows where a human is in the loop. But it does not cover an autonomous agent acting with no user present.

A separate extension covering OAuth client credentials addresses unattended client authentication, but you still have to work out standardized agent identity and delegation. So open-source frameworks you evaluate right now may not implement the emerging standard consistently, or at all.

So the test before any pilot advances: for every agent in the proposed deployment, can the team produce, immediately, its unique identifier, its business owner, its approved purpose, its trust level, its lifecycle state, and its credential expiration date? If any of those answers is missing, the identity posture isn't ready for production, regardless of how well the agent performs its tasks.

What auditability requires in open-source agent deployments

If you don't have tool-call-level audit trails tied to a verified agent identity, you can't reconstruct what an agent actually did. You can't demonstrate compliance to a regulator, and you can't catch the specific class of attack where an agent behaves normally through most of a session but quietly pulls data out on one call buried among the rest.

The Supabase MCP incident documented by General Analysis shows how this plays out. An agent holding service-role database privileges was processing a routine support ticket. That ticket contained instructions, embedded in the text, directing the agent to query the integration_tokens table and post the contents back as a new message in the same ticket. No control anywhere in that environment was positioned to evaluate the intent behind the tool call before it ran. The agent had legitimate access and simply used it, following an instruction it had no reason to distrust.

So that incident points at a gap that runs through most enterprise security stacks. Web application firewalls, API gateways, data loss prevention tools, and SIEMs all log what was called, but none of them were built to evaluate why an agent made a given call. They weren't designed with intent in mind, because until recently nothing needed them to be.

If you deploy an open-source agent, production-grade auditability requires a few specific things. Every tool call needs a log entry carrying the agent's identity, the tool invoked, the inputs passed in, the output returned, and a timestamp, covering the full sequence of actions the agent takes rather than just the final action it surfaces to a human user. Those logs need to be tamper-proof: an agent that can edit its own audit trail provides no real assurance to anyone reviewing it later. The logs also need to come out in a format regulators and auditors can actually use: under the EU AI Act, human-oversight records for high-risk deployments are a requirement.

The test here is simple to run and hard to pass if the framework wasn't built for it: ask the vendor or the open-source maintainers for a sample audit log from a multi-step agent execution. If that log doesn't show tool-call-level detail tied to a verified identity, the system isn't ready to be trusted with production data.

The MCP security threats that open-source agent evaluations most often miss

The MCP attack surface is not a hypothetical risk sitting in a research paper somewhere. The threats that matter most for enterprise evaluation are the ones that walk straight past perimeter defenses because they use the agent's own legitimate access and the trust it places in tool definitions.

Wiz Research's framing of the "lethal trifecta" captures the underlying mechanism: untrusted input, access to sensitive data, and the ability to act, combined, turn anything the agent can read into a potential attack vector. Tool poisoning is the clearest expression of that risk and the dominant MCP threat of 2026: an agent reads a tool's description with the same trust it extends to its own system prompt, so instructions buried in that metadata execute without a human ever seeing them. The MCPTox benchmark tested adversarial tool variants against a range of language models and found high average attack success rates, with an average attack success rate of more than a third across all models in the study. MCP also picks up changes to tool descriptions automatically, so a tool approved today and poisoned tomorrow goes live in the agent's context with no re-approval step unless the evaluation process builds one in.

Several named incidents from 2026 give evaluators a concrete threat model to work from instead of an abstract one. The Agentjacking research against Sentry MCP, published in June 2026, found injectable DSNs across a large number of organizations and recorded an 85 percent agent execution rate in testing. Sentry declined full remediation and shipped only a payload-string filter, leaving the underlying architectural risk in place. Antiy CERT confirmed malicious skills across ClawHub, the marketplace serving the OpenClaw agent framework, reaching one in five ecosystem packages at peak. OX Security's research, titled "Mother of All AI Supply Chains," found a command execution vulnerability across Anthropic's official MCP SDKs for Python, TypeScript, Java, and Rust: the STDIO transport passes parameters straight to the host operating system's shell without sanitizing them, so a crafted command can execute on the host whether or not a valid MCP server ever initializes.

OWASP's MCP threat taxonomy has organized these into categories security teams now test against directly: tool poisoning, including schema poisoning and tool shadowing as sub-techniques, command injection and execution, shadow MCP servers running outside any formal governance process, and context injection or over-sharing across sessions.

That last category deserves its own attention, because it's the easiest one to miss. A developer can stand up an MCP server against a production database in an afternoon. Once an agent discovers that server, its tools become part of what the agent can act on, and no security review has happened.

Before approving any MCP server an open-source agent will connect to, the evaluation should require: a known, pinned version; evidence that authentication is actually enforced; a named owner responsible for it; and a re-approval trigger that fires whenever a tool's description changes. None of this replaces the capability, identity, and audit checks you ran earlier. It sits alongside them, because an agent can clear every other test and still be running against a server nobody reviewed. These safeguards align with the practices outlined in Lyzr's MCP Security guide.