● Covered by 1 source · 1 reportMedium impact1 neutral
Collinear AI Releases CWE-bench v2 with 100 Public Security Vulnerability Tasks
Key points
- CWE-bench v2 offers 100 public tasks for AI model evaluation.
- Tasks involve finding and fixing security weaknesses in real codebases.
- Covers 91 distinct Common Weakness Enumerations (CWEs).
- Programmatic scoring ensures deterministic and reproducible results.
CWE-bench v2 Release
Collinear AI has launched CWE-bench v2, making available a public set of 100 tasks for evaluating frontier cyber capabilities of AI models. These tasks require an AI agent to identify and resolve security vulnerabilities within a real open-source repository, based on a short instruction, without introducing new regressions.
Evaluation Methodology
The evaluation process in CWE-bench v2 uses programmatic scoring, ensuring that results are deterministic and reproducible. This allows anyone to run and verify model performance without additional costs. The programmatic checks were curated with the assistance of LLM judge panels to ensure hardness and fairness in the evaluation criteria. A held-out set of tasks is also maintained for ongoing progress tracking, with access available upon request.
Comprehensive Vulnerability Coverage
The 100 tasks in CWE-bench v2 encompass 91 different Common Weakness Enumerations (CWEs), covering a broad spectrum of vulnerability classes. This comprehensive coverage aims to challenge models to understand various weakness surfaces that attackers might exploit, moving beyond simple pattern recognition.
Quality Assurance and Transparency
Collinear AI implemented a meticulous quality assurance process for each task, verifying that instructions are realistic, graded behaviors are inferable, verifiers check for correct behavior rather than specific fix structures, starter code fails while reference fixes pass, original functionality remains intact post-fix, and verifiers prevent memorized solutions. The public release of the task set is intended to further improve quality through community feedback and contributions.
Public Leaderboards
Results for CWE-bench v2 are tracked on two public leaderboards. An open-weight model leaderboard is available on the CWE-bench dataset page, updated via pull requests to model repositories. A separate leaderboard Space includes all evaluated models, including API-only models like Gemini, Claude, and GPT, with new models added through pull requests to the Space.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
GitHub huggingface/blogReporting from
Collinear AI has released CWE-bench v2, a public benchmark comprising 100 tasks designed to evaluate AI models' ability to find and fix security weaknesses in open-source repositories. This new version covers 91 different Common Weakness Enumerations (CWEs) and features programmatic scoring, quality checks, and a public leaderboard for tracking model performance.