The most misunderstood graph in AI just got a major update
This breakdown covers METR's updated time horizon benchmark methodology, task scores, and the bottlenecks facing fully autonomous research and development.
By Dillip Chowdary • Nov 21, 2026 • Source: MIT Technology Review

At a glance
- Benchmark
- METR time horizon plot
- Frontier model
- Opus 5.5
- Growth rate
- Task length doubled every 4 months
- Evaluation limit
- 0 AI evaluations deemed usable
- Research bottleneck
- Human research taste and judgment
In this post
Nonprofit evaluation research group METR updated its benchmark tracking how long frontier models can work on complex human tasks, adding open-ended assignments that require judgment alongside standard automated software challenges. According to MIT Technology Review's report, the updated time horizon plot indicates that while frontier artificial intelligence models continue an exponential trajectory on verifiable software tasks, systems still lag behind humans on open-ended problems that resist automated checking.
This breakdown covers METR's updated time horizon benchmark methodology, task scores, and the bottlenecks facing fully autonomous research and development. It is written for engineering leads, machine learning researchers, and system architects assessing model agent limits.
01Task horizons and verifiable automated checks
METR evaluated frontier models by having them attempt a suite of tasks that had previously been completed by human testers. Each task difficulty was scored according to human completion duration, rating models on the maximum length of tasks they could successfully finish. The original plot showed frontier systems doubled the task length they could successfully complete every four months.
The updated plot expands beyond verifiable coding and mathematics into open-ended, judgment-heavy work. Verifiable domains allow researchers to apply reinforcement learning by having systems attempt tasks repeatedly, strengthening pathways that produce success and weakening pathways that fail. It remains difficult to apply this technique to tasks where success is less well defined or takes longer to determine, such as choosing a new research direction.
OpenAI confirmed in a Wednesday blog post that its models do not perform as well on real-world tasks assigned by employees as on the easier, verifiable tasks in METR's original suite. Computer scientists note that training models on easily verifiable tasks may not immediately translate into open-ended problem solving.
02Human evaluations and philosophical arguments
METR researchers introduced open-ended tasks directly relevant to autonomous engineering, including designing new evaluation benchmarks that METR could deploy in its research. Human METR employees scored the resulting benchmarks against research utility standards. Even though frontier models now surpass almost every human at software engineering, not one AI-designed evaluation was considered useful for research by METR staff.
Another evaluation task required systems to score philosophical arguments to measure judgment capability. METR staff disagreed on the relevance of philosophical scoring to automating technical research, and models again fell short of human performance. The philosophical scoring benchmark relied on a single person rating most arguments, judging agents solely by agreement with that individual.
Note: Human completion times in the updated suite are mostly estimates rather than measured trials, because paying qualified experts for tasks requiring hundreds of hours proved prohibitively expensive.
03Evaluation design scores across frontier models
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
Frontier models exhibited measurable shifts on evaluation design tasks between release cycles. While older systems scored in the lowest tier, recent models reached borderline tiers.
| Model Tier | Evaluation Design Score | Research Usability |
|---|---|---|
| Previous Generation | Clearly below the bar | Unusable for METR research |
| Opus 5.5 | Borderline above the bar | Not yet usable for METR research |
Sayash Kapoor, an incoming computer science professor at UC Berkeley, reported in an August preprint that current artificial intelligence models cannot yet reproduce human computer science research independently. Kapoor noted that research taste, specifically the ability to select promising questions, hypotheses, and experiments, remains a primary bottleneck in automating research and development.
04Recursive progress claims at frontier labs
The benchmark update follows a heavily redacted METR report released earlier in the month examining developer velocity at frontier laboratories. That earlier paper claimed artificial intelligence tooling might be accelerating the pace of internal artificial intelligence progress at Anthropic by a factor of two.
The initial findings triggered debate within the technical community following Anthropic's initial public offering. Analysts raised concerns regarding recursive self-improvement, a scenario where software models autonomously design, test, and release their own architectural successors without human intervention.
Despite progress on verifiable software tasks, models remain unable to reliably complete the open-ended design problems that form the foundation of internal laboratory development.
05Task quantification and benchmark extrapolation limits
METR technical staff member Matt Putz stated that predicting progress on hard-to-verify or judgment-laden tasks will remain difficult because they cannot be cleanly quantified. Putz noted researchers will likely not establish trendlines as clean and robust as those observed on easy-to-specify tasks.
Boston University computer scientist and linguist Najoung Kim stated that observed performance gains indicate current capabilities, but caution is warranted when attempting to extrapolate those scores into long-term trends. Experts noted that current benchmarks remain far from answering whether or how quickly automated engineering will arrive.
Kapoor emphasized that moving benchmark design toward open-ended problems is a welcome step, though substantial evaluation work remains before researchers can fully measure autonomous model judgment.
Frequently asked questions
What did the updated METR time horizon plot show?
The updated benchmark showed that while frontier models double the length of verifiable tasks they can finish every four months, they continue to lag behind humans on open-ended tasks requiring judgment.
How did models perform when designing new AI evaluations?
Not a single AI-designed evaluation was considered useful for METR research by employee reviewers, though models like Opus 5.5 reached borderline above the bar performance compared to clearly below the bar for earlier models.
Why are open-ended tasks harder to train with reinforcement learning?
Reinforcement learning requires automated verification of success or failure to strengthen or weaken neural pathways, which cannot be easily applied to tasks with ambiguous or long-term outcomes.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement