Home / Blog / The most misunderstood graph in AI just got a major update
Tech News

The most misunderstood graph in AI just got a major update

This breakdown covers METR's updated time horizon benchmark methodology, task scores, and the bottlenecks facing fully autonomous research and development.

By Dillip Chowdary • Nov 21, 2026 • Source: MIT Technology Review

The most misunderstood graph in AI just got a major update
Image: MIT Technology Review

At a glance

Benchmark
METR time horizon plot
Frontier model
Opus 5.5
Growth rate
Task length doubled every 4 months
Evaluation limit
0 AI evaluations deemed usable
Research bottleneck
Human research taste and judgment

Nonprofit evaluation research group METR updated its benchmark tracking how long frontier models can work on complex human tasks, adding open-ended assignments that require judgment alongside standard automated software challenges. According to MIT Technology Review's report, the updated time horizon plot indicates that while frontier artificial intelligence models continue an exponential trajectory on verifiable software tasks, systems still lag behind humans on open-ended problems that resist automated checking.

This breakdown covers METR's updated time horizon benchmark methodology, task scores, and the bottlenecks facing fully autonomous research and development. It is written for engineering leads, machine learning researchers, and system architects assessing model agent limits.

01Task horizons and verifiable automated checks

METR evaluated frontier models by having them attempt a suite of tasks that had previously been completed by human testers. Each task difficulty was scored according to human completion duration, rating models on the maximum length of tasks they could successfully finish. The original plot showed frontier systems doubled the task length they could successfully complete every four months.

The updated plot expands beyond verifiable coding and mathematics into open-ended, judgment-heavy work. Verifiable domains allow researchers to apply reinforcement learning by having systems attempt tasks repeatedly, strengthening pathways that produce success and weakening pathways that fail. It remains difficult to apply this technique to tasks where success is less well defined or takes longer to determine, such as choosing a new research direction.

OpenAI confirmed in a Wednesday blog post that its models do not perform as well on real-world tasks assigned by employees as on the easier, verifiable tasks in METR's original suite. Computer scientists note that training models on easily verifiable tasks may not immediately translate into open-ended problem solving.

02Human evaluations and philosophical arguments

METR researchers introduced open-ended tasks directly relevant to autonomous engineering, including designing new evaluation benchmarks that METR could deploy in its research. Human METR employees scored the resulting benchmarks against research utility standards. Even though frontier models now surpass almost every human at software engineering, not one AI-designed evaluation was considered useful for research by METR staff.

Another evaluation task required systems to score philosophical arguments to measure judgment capability. METR staff disagreed on the relevance of philosophical scoring to automating technical research, and models again fell short of human performance. The philosophical scoring benchmark relied on a single person rating most arguments, judging agents solely by agreement with that individual.

Note: Human completion times in the updated suite are mostly estimates rather than measured trials, because paying qualified experts for tasks requiring hundreds of hours proved prohibitively expensive.

03Evaluation design scores across frontier models

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

Frontier models exhibited measurable shifts on evaluation design tasks between release cycles. While older systems scored in the lowest tier, recent models reached borderline tiers.

Model TierEvaluation Design ScoreResearch Usability
Previous GenerationClearly below the barUnusable for METR research
Opus 5.5Borderline above the barNot yet usable for METR research

Sayash Kapoor, an incoming computer science professor at UC Berkeley, reported in an August preprint that current artificial intelligence models cannot yet reproduce human computer science research independently. Kapoor noted that research taste, specifically the ability to select promising questions, hypotheses, and experiments, remains a primary bottleneck in automating research and development.

04Recursive progress claims at frontier labs

The benchmark update follows a heavily redacted METR report released earlier in the month examining developer velocity at frontier laboratories. That earlier paper claimed artificial intelligence tooling might be accelerating the pace of internal artificial intelligence progress at Anthropic by a factor of two.

The initial findings triggered debate within the technical community following Anthropic's initial public offering. Analysts raised concerns regarding recursive self-improvement, a scenario where software models autonomously design, test, and release their own architectural successors without human intervention.

Despite progress on verifiable software tasks, models remain unable to reliably complete the open-ended design problems that form the foundation of internal laboratory development.

05Task quantification and benchmark extrapolation limits

METR technical staff member Matt Putz stated that predicting progress on hard-to-verify or judgment-laden tasks will remain difficult because they cannot be cleanly quantified. Putz noted researchers will likely not establish trendlines as clean and robust as those observed on easy-to-specify tasks.

Boston University computer scientist and linguist Najoung Kim stated that observed performance gains indicate current capabilities, but caution is warranted when attempting to extrapolate those scores into long-term trends. Experts noted that current benchmarks remain far from answering whether or how quickly automated engineering will arrive.

Kapoor emphasized that moving benchmark design toward open-ended problems is a welcome step, though substantial evaluation work remains before researchers can fully measure autonomous model judgment.

Frequently asked questions

What did the updated METR time horizon plot show?

The updated benchmark showed that while frontier models double the length of verifiable tasks they can finish every four months, they continue to lag behind humans on open-ended tasks requiring judgment.

How did models perform when designing new AI evaluations?

Not a single AI-designed evaluation was considered useful for METR research by employee reviewers, though models like Opus 5.5 reached borderline above the bar performance compared to clearly below the bar for earlier models.

Why are open-ended tasks harder to train with reinforcement learning?

Reinforcement learning requires automated verification of success or failure to strengthen or weaken neural pathways, which cannot be easily applied to tasks with ambiguous or long-term outcomes.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →