WolfBench (2026-03-10)

The impact of time(outs) on agentic benchmarks

by Wolfram Ravenwolf – who evaluates models for breakfast, builds agents at night, and preaches AI usefulness all day long.

In this post, we’ll look at why timeouts are necessary and why I locked WolfBench at 1 hour.

When we launched WolfBench, a four-metric evaluation framework built on Terminal-Bench 2.0, we started with a fixed 2-hour timeout per task. After analyzing nearly 10,000 task results across 8 models, I’m changing that to 1 hour. Here’s why—and why it matters for anyone who cares about meaningful AI benchmarks.

Why agentic benchmarks need timeouts

Terminal-Bench 2.0 is a real-world agentic benchmark: 89 tasks spanning system administration, DevOps, security, data/ML operations, and problem-solving. Models must plan, execute shell commands, inspect results, debug failures, and iterate—like a real technical professional. No multiple-choice tricks. No toy puzzles. Just real work in sandboxed environments.

But when a model is free to act, it can get stuck in a loop, retrying the same failing approach endlessly, burning tokens and compute for nothing. Timeouts exist to prevent that.

The problem with per-task timeouts

By default, Terminal-Bench 2.0 assigns each task its own timeout, ranging from 10 minutes to over 3 hours. The majority—45 out of 89 tasks—get just 15 minutes.

This severely penalizes slower models and endpoints. Even if a model could solve a task, a temporarily overloaded API or a slightly slower inference speed can prevent it from finishing in time. When you’re evaluating model capability, you don’t want to measure endpoint performance.

That’s why some recent benchmarks (e.g., MiniMax-M2.5, GLM-5) use fixed 2-hour timeouts for every task. Some, like Kimi K2.5, even disable reasoning/thinking—because models that invest in up-front planning over immediate execution can actually score worse in time-constrained agentic benchmarks. (This is also why you always need to read the fine print: two benchmarks with the same name can produce very different results depending on how they were configured.)

So why not just use 2 hours?

Because the data says 1 hour is the sweet spot.

I analyzed 9,636 task results across all models and configurations in my evaluation runs: every pass, every fail, every actual duration. Then I asked a simple question: At each possible timeout cap, how many successful task completions would we lose?

Timeout Cap	Passes Lost	% of Total
15 min	846 / 4,708	18.0% – way too tight
30 min	395 / 4,708	8.4% – still too aggressive
60 min	136 / 4,708	2.9% – the sweet spot
90 min	76 / 4,708	1.6% – diminishing returns
120 min	1 / 4,708	0.0% – almost nothing gained

Going from 60 to 120 minutes saves just 135 additional passes out of nearly 5,000. That’s a 2.9% gain—in exchange for doubling the maximum time an agent can spend looping on a hopeless task.

A model that can solve a task typically does so well within 60 minutes. A model that’s still looping after an hour is usually not going to find the answer in hour two—it’s just going to burn more tokens failing the same way.

The cost of generous timeouts

And “looping on hopeless tasks” isn’t hypothetical. Across all evaluated runs, I measured 921.9 hours of wasted compute—time spent on attempts that ran past their default timeout and still failed. The worst offenders burned 40–60 hours each, producing nothing.

Longer timeouts don’t just waste time. They waste tokens, inflate costs, and—in extreme cases—can fill up disk space when models generate output in tight failure loops. The 2-hour cap doubles the blast radius of every one of these failure modes compared to 1 hour.

What 1 hour actually loses

The 136 passes lost at a 60-minute cap come primarily from just a handful of tasks: mailman (22 lost passes), path-tracing (10), and rstan-to-pystan (9). These are tasks with shorter default timeouts that occasionally need over an hour to succeed. Meanwhile, the task with the longest default timeout in the entire benchmark—build-pov-ray at 200 minutes—has never needed more than 34 minutes to succeed. And sam-cell-seg, with its 120-minute default? It has a 0% pass rate across all attempts. Extra time doesn’t help what can’t be solved.

The verdict

A fixed 1-hour timeout per task is the optimal configuration for WolfBench. It:

• Preserves 97.1% of all successful completions
• Eliminates hundreds of hours of wasted compute
• Still gives every task 4x more time than the benchmark’s most common default (15 min)
• Only 2 tasks get less time than their default, and neither is affected
• Ensures scores reflect model capability, not endpoint speed

For a benchmark designed to measure what models can do—not how performant the infrastructure is—that’s the right call.

WolfBench is now locked at 1 hour per task. All future evaluations will use this configuration, and I’ve already begun re-running existing models under the new timeout for consistency.

Analysis based on 9,636 task results across 6 models (Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Kimi K2.5, MiniMax-M2.5, GLM-5).
Inference sponsored by CoreWeave. Sandbox compute by Daytona. Built with Harbor, Terminal-Bench 2.0, and W&B Weave.