OpenRouter Ori Eval: Prove the Best Model for What You're Building

I'll pull the OpenRouter announcement so the paragraphs stay grounded in real product details rather than invented figures.OpenRouter launched Ori Eval, a product that picks and proves which model fits a specific app instead of relying on leaderboard rankings or social recommendations. The company frames the problem against its catalog of over 500 models, where teams often stick with a default because hand re-evaluating every weekly release is expensive. Ori Eval is meant to answer one question with evidence: which model is best for what you are building, on your harness, data, and prompts.

Technically, Ori Eval acts as a coding agent that scans a codebase for every model call, interviews the developer on priorities such as cost, latency, speed, or tool-call accuracy, then proposes the five latest models that match those constraints. It writes a review.eval.ts file, runs the agent against candidates in parallel through OpenRouter, and returns a comparison table with metrics like catch rate, p50 latency, dollars per PR, and pass or fail. During a run it pins the harness, model, and effort so score changes can be attributed to the model alone. Eval files run under bun test and check three layers: tools that must be called, tools that must not be called, and open-ended answer quality graded by an LLM judge with developer-set criteria and a minimum score.

For engineers shipping AI features, the practical shift is that model choice becomes a regression-tested property of the product rather than a one-off judgment call. A production bug stated in plain language, such as a support agent refunding without looking up the order, becomes an assertion that lookup_order is invoked; the failing eval proves the bug, the fix makes it pass, and the assertion stays in the suite. That turns fragile agent behavior into something CI can fail on before it reaches production.

In market terms, OpenRouter is productizing what most teams still do ad hoc against public benchmarks that measure fixed task sets, not a given app’s tools and prompts. By routing every candidate through OpenRouter, Ori Eval turns multi-lab comparison into a single workflow rather than separate vendor evals, which is the gap between “this model leads a public board” and “this model is cheaper and accurate enough on our PR review agent.” Early beta use includes monthly scheduled comparisons that open a PR when a new model beats the one currently in the codebase.

The immediate takeaway is to install via the spawn-ori-eval skill (curl the skill URL into a coding agent, or run /spawn-ori-eval if you already use the OpenRouter MCP server), complete ori login, and generate an eval from the real call sites in your repo. Wire ori eval into GitHub Actions so failed assertions fail the build, and schedule re-runs when new models ship so cost and quality tradeoffs stay current without a manual bake-off each time.