Benchmark Models on Broken Repos, Not LeetCode

Most AI coding benchmarks measure things working developers rarely care about. They test whether a model can invert a binary tree or write an optimal dynamic programming algorithm in an empty sandbox with zero context. In the terminal, that is almost never the problem.

Real development happens inside messy repositories. There are conflicting configuration files, half-migrated dependencies, stale READMEs, and broken tests left behind from last week. When you bring an agent into that environment, synthetic scores stop mattering immediately.

Clean sandboxes lie about real leverage

HumanEval and competitive coding benchmarks assume a blank slate. You give the model a clean prompt, clean inputs, and ask for a self-contained function. The model does not have to worry about whether a build tool expects plural .agents/skills or singular .agent/skills, or why an untracked lockfile is blocking execution.

In practice, agent harnesses matter more than models because real capability is about navigating environment friction. A model can score in the 90th percentile on standard benchmarks and still choke the moment a build tool exits with code 1 instead of code 0.

Repo archaeology is the actual job

The hardest part of pair programming with an agent is rarely generating code. Code generation is the easy part. The real work is repo archaeology.

You need the model to look at an unfamiliar repo, find where configuration actually lives, recognize when vendor compatibility flags are turning features off, and trace failure logs without hallucinating nonexistent flags.

When a benchmark evaluates an agent on greenfield code, it tests the easiest part of the lifecycle. It ignores the cognitive tax developers actually pay every day: figuring out what broke and why.

Error recovery is the only metric that counts

What separates a useful coding agent from an annoying demo is error recovery.

When a command fails, does the model panic, repeat the exact same failing command three times, or start inventing files that never existed? Or does it stop, inspect the error output, check the directory tree, and adjust its plan?

If you want to know how good a coding model actually is, do not run it against algorithmic puzzles. Hand it a broken repository with three obscure build errors and see if it can get the test suite green without human intervention. That is the only benchmark that reflects daily work.