The agent era has a strange symptom: every new model release pushes benchmarks higher, leaderboards bristle with "superhuman" and "new SOTA," yet the moment you actually use the thing, the capability is nowhere near what the numbers promised. The problem isn't that models are weak; it's that evaluation itself has been gamed. Agent benchmarks like SWE-bench were built to answer "can a model independently fix a real GitHub issue," and have instead turned into an arena where everyone reports their own highs. This is worth taking more seriously than the chat-model era ever was: selection decisions increasingly ride on agent scores, and when the scores are inflated, money and engineering get misallocated. Agent eval is also harder to keep honest than chat eval: task chains are long, intermediate steps multiply, and scaffolding and prompts get amplified, so the same underlying ability can be packaged into several times the score. This piece names no vendor's specific score. It only dissects the mechanism.
How Leaderboard Gaming Happens
Gaming isn't one move; it's a combo. The most direct is overfitting: leaking the test set, or even its answers, into training data, so the model isn't "learning to fix bugs," it's memorizing them. SWE-bench was introduced by the Princeton NLP team at ICLR 2024, with tasks drawn from real issues in a dozen-plus open-source Python repos (Django, scikit-learn, matplotlib and the like). The intent was open and reproducible, but openness cuts both ways - once public, tasks are easy to scrape back into pretraining corpora, and newer models benefit disproportionately, which is exactly why some researchers now argue for private held-out sets to measure true generalization. More insidious is indirect leakage: even without the literal test set in training, if the corpus includes source code and issue discussions from these same repos, the model has effectively "seen" it, and the generalization score is inflated all the same. Next is prompt and scaffolding tuning: solving a SWE-bench task isn't a multiple-choice question. The agent has to localize the buggy file itself, read the surrounding context, write a patch, run the tests, and iterate on failure - drop any link in that chain and the whole task fails. Precisely because the chain is so long, the same task might pass one in ten with naive zero-shot, but bolt on a hand-tuned agent framework, multi-round self-reflection, tool calls, and maxed-out test-time compute, and the score can multiply several times. The catch is that this boost doesn't travel with you - call the same model through a bare API and you get nothing close, so a chasm opens between leaderboard score and real-world ability. Then comes selective reporting: the benchmark has a full set, a Verified subset, and various slices; some run on the full set, some on the subset, some cherry-pick the easy slices, all incomparable in methodology, all labeled "SWE-bench XX%." There's also an overlooked variable: stochasticity. Agent tasks are probabilistic - the same task passing this run and failing the next is normal, so reporting the high run and burying the failures is just taking the max of noise. The honest approach is to run many times and report mean and variance, but that drags the number down, and no one volunteers for it. Finally, weak test cases: early SWE-bench had tasks with insufficient test coverage, where a model could edit in a pass and be marked "resolved" - exactly the flaw that produced SWE-bench Verified, a human-vetted subset of roughly 500 instances meant to plug the hole.
Self-Reported vs Third-Party Independent Testing
To grasp the trust crisis, compare SWE-bench and Odysseys side by side. The SWE-bench official board (swe-bench.com) is, at its core, self-reported: anyone can submit, the scoring script and method sit with the submitter, and the board verifies the result number but doesn't force reproduction of your whole scaffolding. So the same model might score in the low teens under a bare call from vendor A, and fifties under a heavy scaffold from vendor B, both posted side by side - the user can't tell whether the gap is model capability or engineering wrapper. Odysseys (odysseysbench.com) takes the other road - third-party independent testing: 200 long-horizon real web tasks, run by the board under a unified environment and evaluation protocol, with contestants never touching the eval pipeline. That's why Browser Use sitting first on Odysseys at 87.4%, beating the computer-use agents of OpenAI, Anthropic, Google, and Microsoft, carries credibility - it isn't self-reported by Browser Use, it's independently measured by Odysseys. Same goes for LMArena, which has two models answer the same prompt and lets human votes rank them, and Artificial Analysis, which normalizes multiple benchmarks into a composite intelligence index - unified methodology, no vendor refereeing its own match. One line to separate them: a self-reported board measures "how high can you score," an independent board measures "how high do you score under one shared referee" - only the latter is cross-comparable. This is doubly true for agents: the heavier the scaffolding, the more inflated the score, and independent testing is exactly what pins the scaffolding variable down. There's a subtler effect too: self-reported boards drift upward over time, not because models genuinely improve but because scaffolding and prompts keep getting tuned against the benchmark - optimize a single metric long enough and it stops tracking what it was meant to measure, a textbook application of Goodhart's law to agent eval. Independent boards at least suppress that drift, because the methodology isn't in the vendor's hands.
How Ordinary Developers Can Spot a Trustworthy Benchmark
No need to study evaluation theory; four checks will do. First, who produced the score: self-reported gets a question mark; independent third parties (Odysseys, LMArena, Artificial Analysis, Vals AI) sit a magnitude higher in credibility, and a vendor's own launch-deck slide is the lowest of all. Add a gut-check heuristic: if a score looks implausibly high and clearly outruns the curve of peer models at the same tier, it's almost certainly the scaffolding talking, not the model. Second, is the methodology aligned: same test set, same eval script, same test-time limits and compute budget, whether tools are provided, how large the context window is - any score that's a bare percentage with no subset or conditions stated is basically skippable. Third, what's the reproduction cost: method public, code runnable, reproducible by others - that's a benchmark; a number with no method attached is marketing, not evaluation - SWE-bench still serves as an industry reference precisely because its tasks and scripts are open. Fourth, the sharpest move - run it yourself: pick your own real tasks, blind-test a few candidate models on a small sample across several runs and take the mean, and treat the leaderboard as a starting line for selection, not the finish line. Agent eval especially so: long tasks, many steps, high variance, and the gap between paper score and real performance is far wider than with traditional LLMs - a ten-point leaderboard bump might mean solving one or two more edge cases in your actual workflow. One closing line for every vendor publishing scores: release the weights, the prompts, and the scaffolding, and let an independent third party reproduce it - that's worth far more than yet another record-breaking percentage.
References
- SWE-bench official site (Princeton NLP, real GitHub issue resolution benchmark): https://www.swebench.com/
- SWE-bench GitHub repository (Verified subset and eval scripts): https://github.com/princeton-nlp/SWE-bench
- Odysseys leaderboard (200 long-horizon web tasks, third-party independent, Browser Use 87.4% first): https://odysseysbench.com/leaderboard
- LMArena (blind head-to-head, human voting): https://lmarena.ai/
- Artificial Analysis (independent composite intelligence index): https://artificialanalysis.ai/