Big Pickle Model Achieves 50.8% Task Resolve Rate on SWE Atlas Codebase QnA Benchmark

Big Pickle's Benchmark Performance

The Big Pickle model, developed by OpenCode Zen, recorded a 50.8% task resolve rate on the SWE Atlas Codebase QnA benchmark. This evaluation involved 124 tasks and utilized the mini-swe-agent scaffold, a minimal bash-only scaffold used for non-first-party models on the leaderboard.

Comparison with Existing Models

Within its scaffold class, Big Pickle surpassed all other entries on the official SWE Atlas QnA leaderboard. It also outperformed GPT models running on the Codex-scaffold. Only two Claude models, operating on their native Claude Code scaffold, achieved higher scores.

Evaluation Methodology and Caveats

The evaluation adhered to Scale AI's published protocol, including the use of Harbor v0.18.0 with Modal sandboxes and the specified judge model, claude-opus-4-5-20251101. However, the run used a single trial per task instead of the official three, and reduced sandbox resources (4 CPU / 8 GB) compared to the declared 16 CPU / 16 GB. These adjustments were made to fit a personal budget and are noted as potential factors that could depress, but not inflate, the score. The evaluation was self-reported and not verified by Scale AI.

Model Availability and Cost

The Big Pickle model is currently available for free during its stealth period via OpenCode Zen's OpenAI-compatible endpoint. The total token consumption for the benchmark run was 674 million input tokens and 4.3 million output tokens, incurring no cost.