METR’s Updated Time‑Horizon Plot Shows Exponential Gains in Verifiable AI Tasks, Yet Judgment‑Heavy Skills Lag Behind
METR’s updated plot confirms exponential gains in verifiable AI tasks, yet models still fall short on judgment‑heavy challenges like designing evaluations or scoring philosophical arguments.
S
··2 min readSciTech Editorial Team
Newsroom

A fresh report from the nonprofit AI‑evaluation lab METR (Machine Evaluation and Testing Research) revamps its famed time‑horizon plot, confirming that frontier models continue to double the length of verifiable tasks they can solve every four months. The update expands the test suite with longer, more open‑ended challenges, revealing that while exponential gains persist in coding and mathematics, performance on judgment‑laden tasks remains markedly below human levels.
METR’s methodology scores each task by the time a human tester needs to complete it; models are then ranked by the longest‑duration tasks they can finish. The new suite adds tasks that would take humans hours or days, such as designing novel AI evaluation protocols and scoring philosophical arguments. In the evaluation‑design task, even the most advanced model, Opus 5.5, achieved only a “borderline‑above‑the‑bar” rating—insufficient for METR researchers. Likewise, on the philosophical‑argument scoring, models fell short of human judges, underscoring a persistent gap in nuanced judgment.
Experts interpret these findings as a pivotal signal for AI‑driven research and development (R&D). “We’re at the point where AI evaluations need to get more creative,” says Sayash Kapoor, a computer scientist joining UC Berkeley. The inability of current models to autonomously generate useful evaluation criteria or to exhibit reliable research taste suggests that automating the full R&D pipeline remains a distant goal.
Nevertheless, the data hint at incremental progress. Opus 5.5’s borderline performance marks a step up from a year ago, when all models scored “clearly below the bar.” Researchers such as Najoung Kim of Boston University caution that extrapolating from these sparse, hard‑to‑measure tasks is fraught with uncertainty, and METR staff acknowledge that forecasting future gains on such tasks will stay challenging.
The broader implication is clear: while AI continues to accelerate on tasks with automatic verification, the journey toward models that can exercise sound judgment—essential for autonomous scientific discovery—requires new training paradigms and evaluation frameworks. Stakeholders in industry and academia must therefore prioritize the development of metrics that capture the subtleties of open‑ended problem solving, lest the current exponential curve stall when confronted with truly human‑like reasoning.



