Ora benchmarks every major AI agent on Vercel – Customers

What happened

Ora Benchmarks Every Major AI Agent on Vercel

Ora, a startup that tests how ready the web is for autonomous AI agents, has built its benchmarking platform on Vercel and is using it to run Claude Code, ChatGPT, Gemini, Hermes, OpenClaw, and Vercel's own agent framework, eve, against live customer websites. The company was co-founded by Assaf Elovic, whose previous company built a web search engine for AI agents before being acquired earlier this year, and by co-founder Liad Yosef. Ora's core argument is a striking one: by its own estimate, 99% of the web is not agent-ready, and the platform exists to show companies exactly where agents fail and what to change.

This article explains how Ora's benchmarking infrastructure works, what it found when it ran eve head-to-head against Claude Code, what that comparison meant for a 16-person engineering team shipping hundreds of commits a day, and what builders running agent workflows on modern deployment platforms should pay close attention to.

How it works

Vercel published a customer story explaining how Ora runs continuous benchmarks of every major AI agent on its platform. The benchmarking sends agents onto live websites with instructions to sign up for a product, integrate with it, and pay for it, then records cost, latency, and the number of steps each agent needs to finish the task. Ora works with Vercel Engineering as a design partner, giving Vercel's team direct access to the benchmark results.

When Vercel launched eve, its own agent framework, Ora gave it no special treatment. Eve went through the same benchmark under the same conditions as every other harness in the lineup. The initial test put eve against Claude Code across hundreds of real journeys on multiple domains, with both harnesses running the same models — Claude Fable 5 and Haiku 4.5 — and the same instructions on every run.

Agents decompose into two parts: a model that does the reasoning and a harness, the software that gives the model its tools and drives it from step to step. Because no two harnesses want the same infrastructure, each expects its own environment and exposes its steps differently. Ora solves this by running a separate runtime for every harness and tracing every step across all of them. When an agent stalls in a signup flow, the customer sees which step failed and what the agent attempted; without that trace, a benchmark result is just a score with nothing behind it.

Why it matters

The platform itself runs on Vercel, including the agent runtime. Front end, back end, and agent runtime share the same deployment path, logs, and authentication. That consolidation is deliberate: Ido Finder, who leads engineering at Ora, describes it as removing the need to operate separate infrastructure for the agent runtime. Everything lives where the rest of the product lives, which becomes especially significant once coding agents are responsible for operating the stack.

Ora published three numbers from the eve versus Claude Code comparison. Eve completed tasks in 7% fewer steps, achieved 2x native success — meaning twice as many tasks finished on the customer's own site rather than falling back to web search — and found 9% more valid endpoints: more of the API endpoints the agent located were ones it could actually call. Benchmarking also fed back into eve's own development: one run surfaced a prompt-caching issue, the eve team shipped a fix, and Ora's next round of results measured roughly 15% lower total cost after that change.

Who is affected

The feedback loop between a live benchmark and the framework under test is the notable mechanism here. Ora is not just scoring agents in isolation; it is giving framework teams early, reproducible signals from real production sites. For a company that benchmarks every major harness professionally, the decision to then build its own internal agents on eve is itself a data point. By Ora's own measure, 99% of the web still cannot handle an agent that shows up to sign up, integrate, and pay, which means most of the results flowing through the platform reveal real failure modes rather than edge cases.

Ora's direct customers are companies that want to know whether an AI agent can navigate their websites and complete transactions. Those companies get a per-step trace showing where agents fail across Claude Code, ChatGPT, Gemini, Hermes, OpenClaw, and eve. Teams building agent-facing product flows — authentication, checkout, API registration — can use those traces to identify exactly which interaction patterns agents cannot yet complete.

The secondary audience is framework teams themselves. Eve's 15% cost reduction after Ora found the prompt-caching issue shows that continuous third-party benchmarking on real sites catches problems internal tests miss. Engineering teams at smaller companies relying on any of the tested frameworks have a stake in whether that feedback loop continues to run, since the fixes it surfaces land in the frameworks they depend on.

What to watch next

Ora's engineering team is 16 people shipping hundreds of commits a day, with day-to-day infrastructure work handled by coding agents. Finder puts the time saved at a few hours a week at minimum, and co-founder Elovic credits a similar amount to how well coding agents build with Vercel's libraries. The team is now splitting its platform into microservices, all of them on Vercel, with new services deploying to the same infrastructure and talking to each other without extra configuration. The internal agents built on eve will run as one more service in that architecture.

The practical signal for builders is this: verify that whatever agent harness you run exposes per-step traces and that those traces live inside your existing observability infrastructure rather than in a separate silo. Ora's sandbox override — which lets the team swap in its own instrumented environment so eve agents get recorded the same way as every other harness — is the mechanism worth examining. A harness that does not support that kind of override will produce opaque results when something goes wrong, which, by Ora's numbers, happens on the vast majority of the web.