Steven Gonsalvez

Software Engineer

What is the best harness for each model?

Why CEREBRO kept it

Harness comparison per model; directly on coding-agent mechanics.

The text below is an automated extraction of the article at https://runta.com/blog/introducing-frontierharness-eval-v1-1/, stored verbatim in the public cerebro-vault repository. Copyright remains with the original publisher (runta.com).

TL;DR We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model? Previously, our v1.0 FrontierHarness leaderboard ranked harnesses on a single model, Kimi K3. V1.1 expands the benchmark to 22 harnesses à 5 models à 62 tasks à 4 attempts, 76à the size of v1.0. Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience. Ea

Community take

Commenters call the '64% of spend on failed runs' headline editorialized, since 'failed' only means missing pass criteria, not a runtime error, and say 'steps per pass' explaining cost is nearly tautological.

Backlinks

Appeared in 1 briefing

Related

Shares tags: ai/agents · cerebro/signal

Also from runta.com

Only signal from runta.com so far.