Stripe evals: Measuring real-world coding agent performance
Stripe launched Stripe evals across 69 payment scenarios, revealing strict agent pass rates drop to 29.4% despite high headline integration scores.
By Dillip Chowdary • Nov 21, 2026 • Source: stripe.dev

At a glance
- Product
- Stripe evals
- Release date
- 2026.11.19
- Evaluated scenarios
- 69 public test cases
- Harnesses tested
- Claude Code, Codex, OpenCode
- Average pass rate
- 78.1% general, 29.4% strict
- Web search change
- 96% reduction with plugin
In this post
Stripe published an evaluation benchmark designed to measure how accurately autonomous coding agents integrate its financial APIs, alongside an agent plugin built on the Model Context Protocol. According to stripe.dev's report, existing software benchmarks evaluate whether generated code runs rather than whether payment workflows adhere to correct API patterns. Stripe created sixty-nine realistic scenarios spanning checkout flows, recurring billing, and marketplace payouts to evaluate fourteen combinations of language models and agent harnesses.
This study covers performance benchmarks, tool-call metrics, category breakdowns, and evaluation tooling across Claude Code, Codex, and OpenCode environments. Stripe built the evaluation harness and the Stripe agent plugin to give developers and enterprise engineering teams empirical data on where agents fail in production workflows. The findings show that while models generate seemingly plausible code, subtle API errors remain frequent without specialized tool support.
01Outdated API calls and hallucination gaps
Coding agents frequently default to legacy API parameters stored in their training data rather than modern payment integration patterns. When generating Stripe integrations without guidance, models miss critical business logic, fail to handle webhook events properly, and omit required error handling for disputes or failed charges. Stripe researchers Kiran Prasad, Yiing Chau Mak, and Manish Kurse found that outdated parameter references accounted for 7% of all parameters in unassisted agent API calls.
When equipped with the Stripe agent plugin, outdated parameter references dropped from 7% to 1.3% across all evaluated scenarios. In one checkout flow, unassisted models wrote deprecated card-specific payment types instead of current dynamic payment method configurations. Providing official tooling prevented these regressions by supplying validated parameters directly during generation.
Note: Outdated API configurations in agent-written code can prevent access to modern payment capabilities and create silent failure paths in production checkouts.
02Sixty-nine test cases across fourteen agents
The Stripe evals harness tests agents against sixty-nine distinct integration scenarios using roleplaying agents to drive real-world developer specifications. The benchmark consists of twenty-four payments tasks, twenty-six subscriptions tasks, and nineteen Connect tasks covering hosted checkouts, metered billing, per-seat tiers, and marketplace payment splitting. Stripe evaluated fourteen model and harness pairings over two complete runs with and without the Stripe agent plugin.
| Model | Harness | Average score: no plugin → plugin | Score change | Strict pass rate: no plugin → plugin | Cost per run ($): no plugin → plugin | Minutes per eval: no plugin → plugin |
|---|---|---|---|---|---|---|
| Haiku 5.5 | Claude Code | 73.5% → 76.2% | +2.7% | 23.5% → 30.9% | 0.08 → 0.09 | 8.3 → 7.8 |
| Sonnet 5.5 | Claude Code | 78.0% → 82.6% | +4.6% | 29.4% → 33.1% | 1.07 → 1.16 | 8.1 → 7.3 |
| Opus 5.5 | Claude Code | 83.5% → 86.2% | +2.7% | 35.3% → 36.0% | 2.77 → 3.15 | 12.4 → 12.4 |
| Fable 5.1 | Claude Code | 79.5% → 84.6% | +5.1% | 25.4% → 36.4% | 5.02 → 6.13 | 15.7 → 19.6 |
| Fable 5.2 | Claude Code | 81.1% → 85.2% | +4.1% | 28.4% → 43.2% | 5.76 → 5.35 | 15.2 → 15.1 |
| GPT-6 Luna | Codex | 72.0% → 79.1% | +7.1% | 17.3% → 29.1% | 0.05 → 0.07 | 7.2 → 8.8 |
| GPT-6.1 Sol | Codex | 85.6% → 89.6% | +4.0% | 39.0% → 50.7% | 0.51 → 0.61 | 6.1 → 6.5 |
| GPT-6 Astra | Codex | 85.5% → 90.0% | +4.5% | 41.9% → 52.9% | 2.93 → 3.63 | 6.2 → 7.8 |
| Gemini 4 Argon | OpenCode | 74.3% → 77.6% | +3.3% | 20.6% → 31.6% | 1.64 → 3.86 | 8.1 → 9.3 |
| Gemini 3.8 Flash | OpenCode | 64.8% → 72.7% | +7.9% | 11.2% → 21.6% | 2.21 → 2.67 | 11.6 → 9.9 |
| Grok 4.7 | OpenCode | 72.0% → 75.8% | +3.8% | 21.3% → 26.5% | 0.70 → 1.37 | 5.1 → 5.3 |
| Kimi K3 | OpenCode | 69.2% → 75.0% | +5.8% | 21.1% → 25.0% | 1.83 → 1.99 | 14.8 → 13.3 |
| GLM 5.3 | OpenCode | 68.1% → 72.6% | +4.5% | 18.5% → 21.8% | 2.26 → 2.42 | 15.7 → 14.7 |
| Minimax M3.1 Flash | OpenCode | 73.4% → 80.8% | +7.4% | 21.5% → 28.6% | 0.49 → 0.65 | 28.2 → 27.9 |
Across all tested setups, adding the Stripe agent plugin raised baseline scores between 2.7 and 7.9 percentage points. GPT-6 Astra on Codex achieved the highest overall score with the plugin at 90.0%, followed closely by GPT-6.1 Sol at 89.6%. GPT-6 Luna posted the lowest cost per evaluation at $0.05 unassisted and $0.07 with the plugin.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
03Strict pass rate drops to twenty-nine percent
Evaluating agents on whether they satisfy every necessary requirement reveals a significant gap between partial functionality and production-ready code. While average scores across all models sat at 78.1%, applying a strict pass rate requiring all criteria to pass caused the average score to plummet to 29.4%. This drop shows that models routinely write almost functional payment logic while failing critical edge cases.
In the radar review fulfillment task, unassisted models verified basic payment intent completion without validating whether orders were eligible for shipping after disputes or refunds. Strict grading exposed how agents skip essential defensive checks despite generating clean, syntactically correct code. Even the highest-performing models failed more than 47% of scenarios under strict grading rules.
04Web searches fall ninety-six percent with plugins
Providing direct Model Context Protocol access through the Stripe plugin replaced open web browsing with targeted documentation lookups. Across all coding agents pooled together, web searches dropped 96% from 1.26 to 0.05 queries per evaluation. Documentation page fetches also fell 63% from 4.84 to 1.78 per run as agents received structured integration skills.
| Model | Harness | Web searches: no plugin → plugin | Search change | Page fetches: no plugin → plugin | Fetch change | Tool calls: no plugin → plugin | Tool call change |
|---|---|---|---|---|---|---|---|
| Pooled agents | All | 1.26 → 0.05 | −96% | 4.84 → 1.78 | −63% | 59.4 → 61.4 | +3% |
| Opus 5.5 | Claude Code | 0.36 → 0.07 | −82% | 4.14 → 1.92 | −54% | 59.9 → 61.3 | +2% |
| Sonnet 5.5 | Claude Code | 0.82 → 0.02 | −97% | 4.67 → 1.28 | −73% | 39.0 → 42.8 | +10% |
| Haiku 5.5 | Claude Code | 1.35 → 0.11 | −92% | 4.22 → 1.37 | −68% | 45.4 → 45.0 | −1% |
| Fable 5.1 | Claude Code | 0.40 → 0.03 | −92% | 7.26 → 2.74 | −62% | 35.8 → 54.8 | +53% |
| Fable 5.2 | Claude Code | 0.66 → 0.03 | −95% | 8.99 → 4.13 | −54% | 55.6 → 60.7 | +9% |
| GPT-6.1 Sol | Codex | 1.31 → 0.01 | −99% | 3.99 → 0.17 | −96% | 28.8 → 33.6 | +17% |
| GPT-6 Astra | Codex | 1.28 → 0.05 | −96% | 4.44 → 0.33 | −92% | 28.5 → 33.5 | +17% |
| GPT-6 Luna | Codex | 4.06 → 0.11 | −97% | 3.49 → 0.00 | −100% | 52.4 → 52.8 | +1% |
| Gemini 4 Argon | OpenCode | not measured | not measured | 0.17 → 1.99 | 12× more | 18.7 → 33.7 | +80% |
| Gemini 3.8 Flash | OpenCode | not measured | not measured | 0.17 → 0.12 | −31% | 47.1 → 34.6 | −26% |
| Grok 4.7 | OpenCode | not measured | not measured | 3.01 → 3.13 | +4% | 40.7 → 57.7 | +42% |
| Kimi K3 | OpenCode | not measured | not measured | 1.62 → 0.71 | −56% | 79.6 → 75.9 | −5% |
| GLM 5.3 | OpenCode | not measured | not measured | 4.01 → 1.29 | −68% | 117.0 → 106.4 | −9% |
| Minimax M3.1 Flash | OpenCode | not measured | not measured | 22.92 → 8.77 | −62% | 236.1 → 239.5 | +1% |
Individual models adjusted their execution strategies when connected to the plugin. Fable 5.1 increased total tool calls by 53% to gather implementation details, while Gemini 3.8 Flash reduced tool calls by 26%. Furthermore, agents wrote 40% fewer test lines and executed test suites in only 42% of runs compared to 53% without specialized tooling.
05Connect and subscriptions integration differences
Performance varied widely across different Stripe integration surfaces, showing that high competency in basic checkouts does not guarantee success in complex architectures. Coding agents consistently scored lowest in subscription workflows, with baseline accuracy ranging between 51% and 82%. Connect marketplace scenarios saw the largest improvements from plugin access, enabling models to properly navigate multi-party account payouts.
| Model | Harness | Payments: no plugin → plugin | Subscriptions: no plugin → plugin | Connect: no plugin → plugin |
|---|---|---|---|---|
| Fable 5.2 | Claude Code | 88% → 90% | 69% → 73% | 89% → 97% |
| Fable 5.1 | Claude Code | 84% → 87% | 71% → 73% | 85% → 97% |
| Opus 5.5 | Claude Code | 87% → 89% | 76% → 78% | 89% → 94% |
| Sonnet 5.5 | Claude Code | 84% → 90% | 68% → 70% | 85% → 91% |
| Haiku 5.5 | Claude Code | 82% → 86% | 62% → 63% | 78% → 82% |
| GPT-6 Astra | Codex | 90% → 96% | 82% → 83% | 85% → 92% |
| GPT-6.1 Sol | Codex | 90% → 94% | 80% → 84% | 87% → 91% |
| GPT-6 Luna | Codex | 80% → 89% | 62% → 67% | 75% → 83% |
| Minimax M3.1 Flash | OpenCode | 79% → 88% | 60% → 66% | 88% → 92% |
| Gemini 4 Argon | OpenCode | 77% → 79% | 63% → 64% | 86% → 95% |
| Gemini 3.8 Flash | OpenCode | 70% → 78% | 51% → 57% | 77% → 88% |
| Grok 4.7 | OpenCode | 76% → 81% | 60% → 60% | 83% → 90% |
| Kimi K3 | OpenCode | 76% → 80% | 55% → 59% | 80% → 91% |
| GLM 5.3 | OpenCode | 75% → 77% | 55% → 56% | 77% → 89% |
In Connect implementations, Fable 5.1 and Fable 5.2 achieved 97% accuracy when using the plugin, rising from 85% and 89% respectively. GPT-6 Astra reached 96% accuracy on standard payments tasks with plugin support. Stripe committed to maintaining the public benchmark harness and publishing continuous updates as new foundation models and harness versions enter the market.
Frequently asked questions
What is Stripe evals?
Stripe evals is an evaluation framework that benchmarks how accurately autonomous coding agents build payments, subscriptions, and marketplace integrations.
How does the Stripe agent plugin affect model accuracy?
The plugin improves average scores by 3 to 8 percentage points across all 14 evaluated models and lowers outdated API parameters from 7% to 1.3%.
Why is there a large difference between average score and strict pass rate?
The strict pass rate requires all essential criteria to pass, dropping average scores from 78.1% to 29.4% because models miss critical fulfillment checks and edge cases.
Which model achieved the highest integration score?
GPT-6 Astra running on Codex scored the highest overall at 90.0% with the Stripe plugin, including 96% on payments scenarios.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement