FrontierHarness Eval shows 17x cost variation for AI model evaluation

Evaluation Cost Discrepancy

FrontierHarness Eval revealed a significant disparity in the cost per pass when evaluating AI models. Across nine different harnesses using the same underlying model, the cost per pass varied by a factor of 17. This indicates that the choice of evaluation harness can substantially influence the financial outlay for AI model testing.

Impact of Failures and Cache on Cost

The analysis noted that excluding failed attempts can misrepresent the true cost; including them increased the cost per task to $3.24. Additionally, cache hit rates do not directly equate to cost savings, as a cached failure can still incur higher costs than a short cache miss. One instance showed a harness passing 19 tasks but reaching $18.34 in cost per task, illustrating that quality and cost can diverge.

Evaluation Environment and Scope

The evaluations were performed on Runta agent runtimes, ensuring consistent testing conditions. Each run started from a fresh restore with identical vCPU, memory, disk size, disk contents, and memory state. FrontierHarness v1.0 is specifically designed for software engineering contexts and terminal-based tasks, and its findings may not be generalizable to other areas of knowledge work.