Glossary
What is LLM as a judge
In short
LLM as a judge (LLM-as-a-judge) is a way to evaluate the answers of neural networks with another language model, which gives scores by set criteria or picks the better of two answers.
Language models · LLM-as-a-judge
MMOOZZGG.com editorsChecked: 2 min read
For a child
Imagine a drawing contest with so many entries that the teacher can’t look at them all. So the teacher asks an older student to help and explains what to give stars for. The helper checks fast but sometimes makes mistakes, or likes drawings that look like their own. That is why the teacher still looks at some of the drawings personally.
In plain words
To know whether a chatbot answers well, someone has to check its answers. People do this slowly and at a high cost. So the answers are shown to another model with an instruction: rate on a scale from one to ten whether the facts are right, whether the tone is polite and whether the bot answered the question. Sometimes the judge sees two answers and is asked to choose the better one. In an hour it can check thousands of dialogs. But the judge model has habits: it may like long answers or the first option in a pair.
For business
An automatic judge makes quality control regular: every new version of a prompt, knowledge base or model runs through hundreds of typical questions, and you see at once whether things got worse. Without it, a company learns about a problem from customer complaints. The costs are the tokens for scoring and the time of experts who write the criteria and check a sample. The main risks: the judge makes its own mistakes, prefers wordy answers, and sometimes favors answers from its own model. So first compare its scores with the opinion of your experts on a small sample, and only then trust the automation. It is worth building this test set during the pilot, so you can measure the result in numbers.
For a specialist
LLM as a judge (LLM-as-a-judge) is the evaluation of open answers by a strong language model against a rubric. Its modes are pairwise comparison, single answer grading and reference-guided grading. Zheng and colleagues (2023) built MT-Bench and the Chatbot Arena platform to test the method. They showed that GPT-4 as a judge agrees with human preferences in more than 80% of cases, the same level at which people agree with each other. They also described systematic biases: position bias, verbosity bias and self-enhancement bias. In G-Eval (Liu and colleagues, 2023), the judge first reasons step by step and fills in an evaluation form; on summarization the method reached a Spearman correlation of 0.514 with people, and the authors noted that the judge leans toward texts written by models. Biases are reduced by swapping the order of options, using several judges and calibrating on human labels.
Example
An online store updated the knowledge base of its support bot. Before release, a judge model checked the answers to typical questions against the rubric “correct, complete, polite” and found a drop on the topic of returns, which was fixed before customers noticed it.
Related terms
Questions and answers
Can you trust the scores of an LLM as a judge?
Sources
Where companies use it
Not sure whether your company needs this?
AI solutions map