Home / Blog / Stripe evals: Measuring real-world coding agent performance
Tech News

Stripe evals: Measuring real-world coding agent performance

Stripe launched Stripe evals across 69 payment scenarios, revealing strict agent pass rates drop to 29.4% despite high headline integration scores.

By Dillip Chowdary • Nov 21, 2026 • Source: stripe.dev

Stripe evals: Measuring real-world coding agent performance
Image: stripe.dev

At a glance

Product
Stripe evals
Release date
2026.11.19
Evaluated scenarios
69 public test cases
Harnesses tested
Claude Code, Codex, OpenCode
Average pass rate
78.1% general, 29.4% strict
Web search change
96% reduction with plugin

Stripe published an evaluation benchmark designed to measure how accurately autonomous coding agents integrate its financial APIs, alongside an agent plugin built on the Model Context Protocol. According to stripe.dev's report, existing software benchmarks evaluate whether generated code runs rather than whether payment workflows adhere to correct API patterns. Stripe created sixty-nine realistic scenarios spanning checkout flows, recurring billing, and marketplace payouts to evaluate fourteen combinations of language models and agent harnesses.

This study covers performance benchmarks, tool-call metrics, category breakdowns, and evaluation tooling across Claude Code, Codex, and OpenCode environments. Stripe built the evaluation harness and the Stripe agent plugin to give developers and enterprise engineering teams empirical data on where agents fail in production workflows. The findings show that while models generate seemingly plausible code, subtle API errors remain frequent without specialized tool support.

01Outdated API calls and hallucination gaps

Coding agents frequently default to legacy API parameters stored in their training data rather than modern payment integration patterns. When generating Stripe integrations without guidance, models miss critical business logic, fail to handle webhook events properly, and omit required error handling for disputes or failed charges. Stripe researchers Kiran Prasad, Yiing Chau Mak, and Manish Kurse found that outdated parameter references accounted for 7% of all parameters in unassisted agent API calls.

When equipped with the Stripe agent plugin, outdated parameter references dropped from 7% to 1.3% across all evaluated scenarios. In one checkout flow, unassisted models wrote deprecated card-specific payment types instead of current dynamic payment method configurations. Providing official tooling prevented these regressions by supplying validated parameters directly during generation.

Note: Outdated API configurations in agent-written code can prevent access to modern payment capabilities and create silent failure paths in production checkouts.

02Sixty-nine test cases across fourteen agents

The Stripe evals harness tests agents against sixty-nine distinct integration scenarios using roleplaying agents to drive real-world developer specifications. The benchmark consists of twenty-four payments tasks, twenty-six subscriptions tasks, and nineteen Connect tasks covering hosted checkouts, metered billing, per-seat tiers, and marketplace payment splitting. Stripe evaluated fourteen model and harness pairings over two complete runs with and without the Stripe agent plugin.

ModelHarnessAverage score: no plugin → pluginScore changeStrict pass rate: no plugin → pluginCost per run ($): no plugin → pluginMinutes per eval: no plugin → plugin
Haiku 5.5Claude Code73.5% → 76.2%+2.7%23.5% → 30.9%0.08 → 0.098.3 → 7.8
Sonnet 5.5Claude Code78.0% → 82.6%+4.6%29.4% → 33.1%1.07 → 1.168.1 → 7.3
Opus 5.5Claude Code83.5% → 86.2%+2.7%35.3% → 36.0%2.77 → 3.1512.4 → 12.4
Fable 5.1Claude Code79.5% → 84.6%+5.1%25.4% → 36.4%5.02 → 6.1315.7 → 19.6
Fable 5.2Claude Code81.1% → 85.2%+4.1%28.4% → 43.2%5.76 → 5.3515.2 → 15.1
GPT-6 LunaCodex72.0% → 79.1%+7.1%17.3% → 29.1%0.05 → 0.077.2 → 8.8
GPT-6.1 SolCodex85.6% → 89.6%+4.0%39.0% → 50.7%0.51 → 0.616.1 → 6.5
GPT-6 AstraCodex85.5% → 90.0%+4.5%41.9% → 52.9%2.93 → 3.636.2 → 7.8
Gemini 4 ArgonOpenCode74.3% → 77.6%+3.3%20.6% → 31.6%1.64 → 3.868.1 → 9.3
Gemini 3.8 FlashOpenCode64.8% → 72.7%+7.9%11.2% → 21.6%2.21 → 2.6711.6 → 9.9
Grok 4.7OpenCode72.0% → 75.8%+3.8%21.3% → 26.5%0.70 → 1.375.1 → 5.3
Kimi K3OpenCode69.2% → 75.0%+5.8%21.1% → 25.0%1.83 → 1.9914.8 → 13.3
GLM 5.3OpenCode68.1% → 72.6%+4.5%18.5% → 21.8%2.26 → 2.4215.7 → 14.7
Minimax M3.1 FlashOpenCode73.4% → 80.8%+7.4%21.5% → 28.6%0.49 → 0.6528.2 → 27.9

Across all tested setups, adding the Stripe agent plugin raised baseline scores between 2.7 and 7.9 percentage points. GPT-6 Astra on Codex achieved the highest overall score with the plugin at 90.0%, followed closely by GPT-6.1 Sol at 89.6%. GPT-6 Luna posted the lowest cost per evaluation at $0.05 unassisted and $0.07 with the plugin.

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

03Strict pass rate drops to twenty-nine percent

Evaluating agents on whether they satisfy every necessary requirement reveals a significant gap between partial functionality and production-ready code. While average scores across all models sat at 78.1%, applying a strict pass rate requiring all criteria to pass caused the average score to plummet to 29.4%. This drop shows that models routinely write almost functional payment logic while failing critical edge cases.

In the radar review fulfillment task, unassisted models verified basic payment intent completion without validating whether orders were eligible for shipping after disputes or refunds. Strict grading exposed how agents skip essential defensive checks despite generating clean, syntactically correct code. Even the highest-performing models failed more than 47% of scenarios under strict grading rules.

04Web searches fall ninety-six percent with plugins

Providing direct Model Context Protocol access through the Stripe plugin replaced open web browsing with targeted documentation lookups. Across all coding agents pooled together, web searches dropped 96% from 1.26 to 0.05 queries per evaluation. Documentation page fetches also fell 63% from 4.84 to 1.78 per run as agents received structured integration skills.

ModelHarnessWeb searches: no plugin → pluginSearch changePage fetches: no plugin → pluginFetch changeTool calls: no plugin → pluginTool call change
Pooled agentsAll1.26 → 0.05−96%4.84 → 1.78−63%59.4 → 61.4+3%
Opus 5.5Claude Code0.36 → 0.07−82%4.14 → 1.92−54%59.9 → 61.3+2%
Sonnet 5.5Claude Code0.82 → 0.02−97%4.67 → 1.28−73%39.0 → 42.8+10%
Haiku 5.5Claude Code1.35 → 0.11−92%4.22 → 1.37−68%45.4 → 45.0−1%
Fable 5.1Claude Code0.40 → 0.03−92%7.26 → 2.74−62%35.8 → 54.8+53%
Fable 5.2Claude Code0.66 → 0.03−95%8.99 → 4.13−54%55.6 → 60.7+9%
GPT-6.1 SolCodex1.31 → 0.01−99%3.99 → 0.17−96%28.8 → 33.6+17%
GPT-6 AstraCodex1.28 → 0.05−96%4.44 → 0.33−92%28.5 → 33.5+17%
GPT-6 LunaCodex4.06 → 0.11−97%3.49 → 0.00−100%52.4 → 52.8+1%
Gemini 4 ArgonOpenCodenot measurednot measured0.17 → 1.9912× more18.7 → 33.7+80%
Gemini 3.8 FlashOpenCodenot measurednot measured0.17 → 0.12−31%47.1 → 34.6−26%
Grok 4.7OpenCodenot measurednot measured3.01 → 3.13+4%40.7 → 57.7+42%
Kimi K3OpenCodenot measurednot measured1.62 → 0.71−56%79.6 → 75.9−5%
GLM 5.3OpenCodenot measurednot measured4.01 → 1.29−68%117.0 → 106.4−9%
Minimax M3.1 FlashOpenCodenot measurednot measured22.92 → 8.77−62%236.1 → 239.5+1%

Individual models adjusted their execution strategies when connected to the plugin. Fable 5.1 increased total tool calls by 53% to gather implementation details, while Gemini 3.8 Flash reduced tool calls by 26%. Furthermore, agents wrote 40% fewer test lines and executed test suites in only 42% of runs compared to 53% without specialized tooling.

05Connect and subscriptions integration differences

Performance varied widely across different Stripe integration surfaces, showing that high competency in basic checkouts does not guarantee success in complex architectures. Coding agents consistently scored lowest in subscription workflows, with baseline accuracy ranging between 51% and 82%. Connect marketplace scenarios saw the largest improvements from plugin access, enabling models to properly navigate multi-party account payouts.

ModelHarnessPayments: no plugin → pluginSubscriptions: no plugin → pluginConnect: no plugin → plugin
Fable 5.2Claude Code88% → 90%69% → 73%89% → 97%
Fable 5.1Claude Code84% → 87%71% → 73%85% → 97%
Opus 5.5Claude Code87% → 89%76% → 78%89% → 94%
Sonnet 5.5Claude Code84% → 90%68% → 70%85% → 91%
Haiku 5.5Claude Code82% → 86%62% → 63%78% → 82%
GPT-6 AstraCodex90% → 96%82% → 83%85% → 92%
GPT-6.1 SolCodex90% → 94%80% → 84%87% → 91%
GPT-6 LunaCodex80% → 89%62% → 67%75% → 83%
Minimax M3.1 FlashOpenCode79% → 88%60% → 66%88% → 92%
Gemini 4 ArgonOpenCode77% → 79%63% → 64%86% → 95%
Gemini 3.8 FlashOpenCode70% → 78%51% → 57%77% → 88%
Grok 4.7OpenCode76% → 81%60% → 60%83% → 90%
Kimi K3OpenCode76% → 80%55% → 59%80% → 91%
GLM 5.3OpenCode75% → 77%55% → 56%77% → 89%

In Connect implementations, Fable 5.1 and Fable 5.2 achieved 97% accuracy when using the plugin, rising from 85% and 89% respectively. GPT-6 Astra reached 96% accuracy on standard payments tasks with plugin support. Stripe committed to maintaining the public benchmark harness and publishing continuous updates as new foundation models and harness versions enter the market.

Frequently asked questions

What is Stripe evals?

Stripe evals is an evaluation framework that benchmarks how accurately autonomous coding agents build payments, subscriptions, and marketplace integrations.

How does the Stripe agent plugin affect model accuracy?

The plugin improves average scores by 3 to 8 percentage points across all 14 evaluated models and lowers outdated API parameters from 7% to 1.3%.

Why is there a large difference between average score and strict pass rate?

The strict pass rate requires all essential criteria to pass, dropping average scores from 78.1% to 29.4% because models miss critical fulfillment checks and edge cases.

Which model achieved the highest integration score?

GPT-6 Astra running on Codex scored the highest overall at 90.0% with the Stripe plugin, including 96% on payments scenarios.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →