Same model, different agent: 17 points apart on 21 MCP servers

Software evaluation group Sineframe released benchmark findings from M3, an open-source evaluation suite built to measure Model Context Protocol server usability across coding assistants. Testing Anthropic Claude Haiku 5.5 through OpenCode 1.19.5 and Claude Code 2.1.312, researchers uncovered a 17-point performance divergence on the exact same 21 MCP servers. According to m3.sineframe.com's report, OpenCode answered 82.0 percent of tasks correctly from the server, whereas Claude Code succeeded on only 65.4 percent.

This report examines how coding agent harnesses route tasks between internal tools and external protocol servers. It details findings across 3,717 evaluation trials for software engineers implementing MCP servers and platform teams configuring agent workflows.

Same model, different agent: what actually changed

The performance gap between the agents stemmed from client tool conflicts rather than underlying model reasoning capabilities. Sineframe separated the 21 test environments into 11 servers with no native agent tool overlap and 10 servers overlapping built-in tools. On the 11 non-overlapping servers covering Postgres, MongoDB, Redis, Kafka, Linear, Jenkins, Grafana, Kubernetes, Gitea, and two browser engines, performance remained virtually identical. Claude Code registered 93.5 percent, OpenCode achieved 96.2 percent, and OpenAI Codex running GPT-6 Luna reached 92.1 percent, placing all three within 4.1 points of each other.

A severe divergence emerged across the 10 overlapping servers, which spanned web search and documentation lookups through Exa, Fetch, Microsoft Learn, DeepWiki, and Terraform, alongside filesystem tools including Git, Filesystem, Desktop Commander, Serena, and memory stores. On these overlapping suites, OpenCode recorded a 66.3 percent server utilization score, whereas Claude Code dropped to 34.5 percent and Codex fell to 17.2 percent. Across all 21 evaluated servers, OpenCode finished at 82.0 percent, Claude Code achieved 65.4 percent, and Codex reached 56.5 percent overall.

Same model, different agent: how it works

Scoring under Sineframe specification 1.6.0 required an agent to return the correct answer while sourcing the required key value directly from an MCP server response. In a test querying the release name and logo theme of Kubernetes v1.37 from August 2026, OpenCode called Exa server web_search_exa and returned Garhwal. Claude Code invoked ToolSearch to load its own native WebSearch tool, located the same official post, and answered Garhwal. While both agents solved the problem, the benchmark counted Claude Code as a miss because it bypassed the server.

Runtime traces confirmed that internal client defaults consistently preempted external servers during execution. Claude Code completed 378 trials on overlapping servers using native utilities, with 342 containing expected values, defaulting to WebSearch, WebFetch, Bash, Read, or inspecting MEMORY.md instead of invoking the Memory server. Codex skipped servers on Microsoft Learn, Terraform, DeepWiki, and Exa entirely, preferring built-in web_search or shell execution. Conversely, OpenCode lacked a built-in web search utility, executing only webfetch across 1,239 trials, leaving the external MCP server as its sole search option.

Same model, different agent: why it matters now

The empirical data demonstrates that MCP servers must actively compete with utilities pre-bundled inside each coding assistant. Across all 21 servers, Codex bypassed external servers on 44.5 percent of should-call trials, Claude Code skipped them on 36.5 percent, and OpenCode skipped on 16.1 percent. Because OpenCode lacked native web search, it achieved a 96 percent success rate on Exa, whereas Claude Code and Codex managed only 11 percent each because their built-in web search utilities preempted external server calls during runtime execution.

Where all three agents shared identical built-in capabilities, external server adoption dropped universally across every harness tested. Because each agent included an integrated shell, all three largely ignored Git, Filesystem, Desktop Commander, and Serena in favor of direct command-line commands. As a result, OpenCode performance advantage narrowed down to a margin between 8 and 37 points on shell and filesystem tasks. Evaluating an MCP integration solely on user success without tracking server calls obscures whether a backend server provides functional utility.

Same model, different agent: who is affected

Developers building commercial or open-source MCP servers face distorted quality metrics when evaluating tools through a single agent environment. A search or documentation server can operate without errors yet experience zero real-world utilization if an agent workspace provides native search primitives. Sineframe noted that an identical server, model, and task scored 96 percent under OpenCode while registering only 11 percent under Claude Code, proving server usability scores depend heavily on client routing choices. Teams must benchmark their servers across multiple distinct coding agent harnesses.

Platform engineers and enterprise architects configuring agent environments must distinguish between user task resolution and backend protocol adoption. To surface this operational distinction, the M3 leaderboard displays an explicit ignore rate alongside its success percentages. A low score accompanied by a high ignore rate indicates that the client environment routed queries around the server entirely, while a low score with a low ignore rate confirms that the agent contacted the server but encountered functional execution failure, providing actionable diagnostic clarity for maintainers.

Same model, different agent: what to watch

The benchmark dataset establishes clear baseline boundaries for evaluating agent architecture. Sineframe executed testing between 29 October and 1 November 2026 across 3,717 scored trials, evaluating approximately 20 tasks per server across seven distinct structural shapes. These shapes tested lookups, searches, multi-step chains, writes, error recovery routines, tool collisions, and negative scenarios where the server should not be called. Local test fixtures were seeded with random values per trial to prevent models from recalling static data, while 25 percent of tasks remained held out.

Engineering teams must account for two structural constraints when interpreting the leaderboard. Because Codex ran GPT-6 Luna rather than Claude Haiku 5.5, its results reflect a blended model-and-agent combination, leaving Claude Code and OpenCode as the only comparison strictly isolating agent software. Furthermore, classifying servers as overlapping a built-in reflects functional categorization rather than statistical scoring patterns. As agent development advances, reconciling internal tool priorities with external protocols will determine whether standardized extensible interfaces like MCP achieve consistent adoption across coding assistants.