Benchmark Evaluation

Benchmark Evaluation

LLM-as-Judge Evaluation Reliability and Systematic Biases

Using Cohen's kappa reveals LLM judges are reliably wrong, not reliably right.

September 23, 2026·10 min read