LLM-as-a-judge

Benchmark Analysis

5 Million Votes, 25 Elo Points: Inside the Chatbot Arena Leaderboard Nobody Can Shake

LMSYS Chatbot Arena ranks models by blind human preference on an Elo scale, with nearly five million votes now packing the top labs into a 25-point band. The LLM-as-a-Judge methods that scale this kind of evaluation carry their own documented biases toward verbosity and position.

2026-08-02