Opus 5 Achieves 24% Pass Rate on SlopCodeBench, Outperforming Previous Models

SlopCodeBench Introduction

SlopCodeBench, released in March 2026 by UW Madison's @GOrlanski's lab, is a long-horizon coding benchmark designed to address limitations of existing benchmarks. Unlike traditional benchmarks that reveal the entire problem upfront, SlopCodeBench presents challenges with multiple checkpoints, requiring models to evolve a codebase as new requirements are disclosed incrementally.

Opus 5 Performance

Opus 5 achieved a 24% strict pass rate on a selected subset of the SlopCodeBench. This result represents a modest improvement over Opus 4.6, which previously scored 17% on the full benchmark. Other models tested, including Opus 4.8 and Sonnet 5, also participated in the evaluation.

Evaluation Methodology

The evaluation involved running three Claude models (Opus 4.8, Sonnet 5, and Opus 5) through a subset of SlopCodeBench problems: 'circuit_eval' (easy, 8 checkpoints), 'database_migration' (medium, 5 checkpoints), and 'dynamic_config_service_api' (hard, 4 checkpoints). Each model received the same prompts and operated within the Claude code harness, with a fresh context window for each checkpoint. The primary metric was the strict pass rate, ensuring all new code and regression tests passed.

Implications for AI in Software Engineering

The results suggest that while models like Opus 5 show progress, their current performance on long-horizon, incrementally revealed coding tasks indicates a limitation in fully autonomous software engineering. Models demonstrated a significant increase in verbosity and code complexity over the course of each challenge, with Opus 5 writing five times more functions than Opus 4.8 for the same set of challenges. This indicates that current AI models still require human steering for real-world software development workflows that involve building one issue at a time.