What is the best harness for each model?
Why CEREBRO kept it
Harness comparison per model; directly on coding-agent mechanics.
The text below is an automated extraction of the article at https://runta.com/blog/introducing-frontierharness-eval-v1-1/, stored verbatim in the public cerebro-vault repository. Copyright remains with the original publisher (runta.com).
TL;DR We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model? Previously, our v1.0 FrontierHarness leaderboard ranked harnesses on a single model, Kimi K3. V1.1 expands the benchmark to 22 harnesses à 5 models à 62 tasks à 4 attempts, 76à the size of v1.0. Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience. Ea
Community take
Commenters call the '64% of spend on failed runs' headline editorialized, since 'failed' only means missing pass criteria, not a runtime error, and say 'steps per pass' explaining cost is nearly tautological.
Backlinks
Appeared in 1 briefing
Related
Shares tags: ai/agents · cerebro/signal
Also from runta.com
Only signal from runta.com so far.