Parsimony: Open Source Benchmark for AI Code Bloat
A look at Parsimony, a benchmark that ranks AI coding models by how little code they add per fix, what it counts, and the limits its author documents.

Solve rate and cost are the numbers most coding-agent benchmarks report. Neither penalises a model that fixes a bug by adding forty lines where ten would do. The extra code passes the tests, gets merged, and becomes something your team reads, reviews and maintains. Parsimony is an open source benchmark built around that gap: it measures how much code AI models add, not just whether they got the task right.
The obvious metric, lines of code, is easy to game and easy to distort. Formatting, line wrapping and one-liner style all move the count without changing how much the code does. Parsimony's answer is to count syntax elements instead, and to do it on patches that existing benchmarks have already published, so no new model runs are needed to score a model.
Counting coding units, not lines
The unit of measurement is a 'coding unit': a syntax element such as a call, a name or a literal. The headline figure is net units, defined as units added minus units removed. A patch that replaces a clumsy block with a tighter one can therefore score below zero.
The project's own example shows the idea. Two Python functions check whether a list contains a negative number, and both return the same result:
def has_negative(values):
found = False
for value in values:
if value < 0:
found = True
return found
def has_negative(values):
return any(value < 0 for value in values)The first version carries a flag variable, a loop and an assignment that the second does not need. Line counts would exaggerate or hide that difference depending on how the code is wrapped. Counting elements of the syntax tree is less sensitive to layout.
How a patch gets measured
The method is static. Each published patch is applied to the task's code, and both versions of every changed file are parsed. Nothing is executed. That is what lets the benchmark reuse other people's results instead of rerunning agents.
- Tests, docs and generated files are excluded, so a model is not charged for the tests it wrote or for regenerated output.
- Only Python is measured.
- Pass/fail comes from SWE-bench's published results. Tests are not rerun.
The trade-off is dependence. Parsimony's solved/unsolved labels are only as good as the upstream data, and the code-size numbers are only as comparable as the patches they came from.
Reading the leaderboard
Models are ranked by mean net units added on the tasks they solved, lowest first. A separate 'net units added (mean)' column also includes failed attempts. The solved flag comes from SWE-bench Verified, and a 'measured' column shows how many solved attempts could actually be parsed and counted.
Every model on the main board uses the same agent, mini-SWE-agent, which keeps the comparison fair in one respect. The project itself notes that the agent still influences how much code gets written, so the ranking describes a model inside one harness, not the model in isolation.
There is also an outside score, matched by model and reasoning effort. It does not affect the rank. The site lets you chart different axes, request a missing model, report a number that looks wrong, or contribute API credits to run a model on the tasks. You can also load your own code-size numbers from another benchmark: you fill in an example file, open it, and get the same chart and table, with the file staying in your browser.
The author's own runs
Some models on the board appear faded. The author ran those on the first few DeepSWE Python tasks: OpenAI models with pi, the GPT-6 family with Codex, and Anthropic models with Claude Code. They were graded with each task's own tests. These runs are not verified and not ranked.
One comparison in that data point is worth noting. GPT-5.6 Sol was run both ways, once by the author and once as a published run, and the author's run added 0.88 times the code of the published one. The same model, on different tasks and a different harness, moves by a margin that could matter when two models sit close together in the ranking. It is a useful reminder not to read small gaps on the leaderboard as real differences.
Where this approach does not fit
Smaller is not automatically better. Units measure size, not readability or design, and a dense one-liner can be harder to maintain than a longer, clearer function. Parsimony tells you which model changes the least; it does not tell you which change you would prefer to review.
The ranking has quirks the project documents itself:
- A model whose fixes mostly delete code can rank first with a negative net.
- Failed attempts that change nothing pull the all-measured mean toward zero, which flatters models that fail quietly.
- Models that write more code also tend to solve more tasks, with a Spearman correlation of 0.33 to 0.60 on the boards with 26 or more models. Minimal output can mean a model is doing less, not doing better.
Add the Python-only scope and the static, no-test-rerun method, and the sensible use is as a second axis next to solve rate. If you are choosing an agent for a non-Python codebase, or one where tests are the real gate, treat these numbers as a hint about diff size and nothing more.
- #AI coding benchmark
- #Open Source
- #Parsimony
- #SWE-bench Verified
- #code size
- #mini-SWE-agent



Comments