● Covered by 1 source · 1 reportMedium impact1 neutral
Parsimony: Open-Source Benchmark Measures Code Added by AI Models
Key points
- Parsimony measures net coding units added by AI models.
- It uses existing benchmark patches and counts syntax elements.
- Lines are not counted; deleted code is subtracted.
- The benchmark aims to address code bloat and maintainability.
Introducing Parsimony
Parsimony is an open-source benchmark designed to evaluate how much code AI models contribute to a codebase. Unlike existing benchmarks that focus on solve rate and cost, Parsimony specifically measures the quantity of code added, addressing concerns about long-term code maintainability and complexity.
Measurement Methodology
The benchmark analyzes patches published by other AI code generation benchmarks. It counts 'coding units,' which are syntax elements such as statements, calls, comparisons, names, and literals. Line counts are not used to prevent rewarding code cramming. Deleted code is subtracted from added code to calculate 'net units added'.
How Models are Ranked
Models are ranked based on their mean net units added across the tasks they successfully solved, with lower values indicating better performance. The benchmark uses pass/fail results from SWE-bench Verified without rerunning tests. An external 'Artificial Analysis Intelligence Index' is also provided but does not influence the ranking.
Limitations and Considerations
Parsimony acknowledges that smaller code size does not always equate to better readability or design. It also notes a correlation where models that write more code tend to solve more tasks. The benchmark uses a consistent agent (mini-SWE-agent) for all models, but the agent's influence on code generation is recognized.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Reporting from
Parsimony is a new open-source benchmark that quantifies the amount of code AI models add to projects, rather than just measuring solve rate and cost. It counts coding units like statements and calls in patches, subtracting deleted code, to assess code bloat and its long-term maintenance impact.