Auditing LLM Evaluators: Addressing Bias and Improving Consistency

Subjectivity of LLM Judges

A common misconception is that AI offers objective evaluation compared to subjective human reviewers. However, Large Language Models (LLMs) also possess inherent biases, making them subjective evaluators. These biases can compromise the quality of evaluations if not addressed.

Identified Biases in LLM Evaluation

Several biases affect LLM judges. Verbosity bias causes LLMs to favor longer, confident-sounding answers, even if incorrect. Position bias can lead to different verdicts based on the order of options. The halo effect means clean grammar and formatting can mask factual errors. Agreeableness bias makes LLMs good at recognizing valid work but poor at detecting errors, missing up to 75% of incorrect or unsafe outputs. Self-enhancement bias causes LLMs to overrate outputs generated by themselves or similar models.

Pillar 1: Measuring Internal Stability

To ensure an AI judge is not guessing, its internal stability must be measured. LLMs are probabilistic and can be inconsistent, giving different scores to the same text. This can be tested by running multi-pass consistency checks on a single output and calculating the variance. High score fluctuation indicates an ambiguous rubric.

Pillar 2: Validating Human Alignment

An LLM judge might appear effective if evaluation datasets primarily contain valid responses, even if it misses many invalid or unsafe outputs. To confirm an LLM judge catches real errors, chance-corrected metrics are necessary. These metrics filter out agreements due to leniency or random chance, revealing how closely the evaluator aligns with expert human decisions.