BenchMIRT Introduced to Audit LLM Benchmarks at the Prompt Level

Introduction of BenchMIRT

BenchMIRT is a new method designed to audit large language model (LLM) benchmarks by examining individual prompts. This approach aims to clarify what specific abilities, such as safety or reasoning, are being measured by each question or task within a benchmark. Traditional benchmarks often combine diverse tasks into a single score, which can obscure the actual capabilities being assessed.

Addressing Benchmark Limitations

Current LLM benchmarks, like BBQ for age bias or WildJailbreak for safety, can contain prompts that test multiple abilities simultaneously or group disparate tasks. For example, a prompt designed to test age bias might also require complex reasoning. Similarly, a benchmark might mix harmful jailbreak prompts with benign ones, leading to an averaged score that does not differentiate between safety and general reasoning performance. BenchMIRT helps to disentangle these signals.

How BenchMIRT Works

BenchMIRT analyzes how models perform on each question or task within a benchmark to estimate which underlying capabilities are most closely associated with correct answers. It extends the concept of Item Response Theory (IRT), a psychometric technique, by using multidimensional IRT (MIRT). This allows it to separate multiple capabilities that may contribute to performance on the same questions, providing a more granular understanding of model strengths and weaknesses.

Application and Training

BenchMIRT applies IRT at both the model and question level. For each model, it estimates its strength across various capabilities reflected in selected benchmarks. For each question, it assesses difficulty and how well it distinguishes models based on those capabilities. The system was trained using benchmarking results from 100 LLMs across 16 different benchmarks, providing a broad dataset for its analysis.