Can LLM-as-a-judge be trusted?

Only after calibration. LLM judges can be useful for scale, but their verdicts must be compared with careful human labels on your own task. Agreement between multiple judges shows consistency, not correctness, because models can share biases and make the same mistakes.

What biases do LLM judges have?

Common ones include position bias in pairwise comparisons, verbosity bias favouring longer answers, self-preference for outputs from the same model family, rewarding confident style over factual accuracy, inconsistent application of vague rubrics, and inability to verify facts without supplied evidence.

How do you calibrate an LLM judge?

Have domain experts carefully label a representative sample, run the judge on the same items, and measure agreement, examining where it fails. Refine rubrics, supply reference material, counter position and length biases, and repeat calibration whenever the model, prompt, or task changes.

Is using multiple LLM judges better than one?

It can reduce random noise and, with different model families, some correlated errors. It does not guarantee correctness, because judges can share training data and biases. Multiple judges still need calibration against human labels, and disagreements should go to human review.

When LLM Judges Agree, Should You Believe Them?

Not by itself. Agreement between LLM judges shows consistency, not correctness. Models often share training data and biases, so they can agree on the same wrong verdict. Trust judges only after measuring their agreement with careful human labels on your own task, and design the evaluation to counter known biases such as position, length, and self-preference.

Why agreement is weak evidence

LLM-as-a-judge means using a language model to evaluate outputs, such as grading answers, comparing two responses, or checking whether an agent completed a task. It is widely used because human evaluation is slow and expensive.

A common reassurance is that several judges agree. Agreement is useful for detecting noise: if judges disagree wildly, the rubric or task is probably unclear. But agreement is not independent confirmation.

Shared training. Many models learn from overlapping public data and similar alignment methods. They may hold the same misconceptions and reward the same surface features.

Shared prompt. Judges given the same rubric inherit the same ambiguities.

Correlated errors. When the answer being judged is fluent, confident, and wrong in a way that sounds plausible, several judges can be misled in the same direction.

Self-consistency is not truth. Asking the same model several times, or several similar models once, mostly measures stability.

The question to ask is not whether judges agree with each other, but how often they agree with a trustworthy reference.

Measuring agreement properly

Raw percentage agreement overstates reliability when most items fall in one category. Chance-corrected measures such as Cohen's kappa give a fairer picture, and reviewing the specific items where judges and humans disagree is more informative than any single statistic.

Known failure modes

Research and practitioner reports on LLM judges have identified recurring biases.

Position bias. In pairwise comparisons, judges may favour the first or second response regardless of content.

Verbosity bias. Longer, more detailed responses are often rated higher even when a shorter answer is more correct.

Self-preference. A model may rate outputs from itself or its own family more favourably.

Style over substance. Confident tone, formatting, and fluent prose can outweigh factual errors.

Rubric drift. Vague criteria such as helpfulness are applied inconsistently across items.

Limited verification. A judge without access to ground truth, tests, or source documents cannot verify factual claims; it can only assess plausibility.

Gaming. Systems optimised against a judge can learn to satisfy the judge rather than the underlying goal.

Where judges work well

Judges are more reliable when the criterion is concrete and checkable against provided material: does the answer cite the supplied source, does the output follow the required format, does the summary contain these facts. They are weaker on open-ended quality and on facts they cannot verify.

Judges and retrieval evaluation

For retrieval systems, judges are often asked whether a retrieved chunk is relevant to a query. This works better when the judge sees the full query intent and a clear relevance definition, and it should be checked against human relevance labels before being used to compare retrievers.

Using LLM judges responsibly

Calibrate against humans. Label a sample of items carefully with domain experts. Measure how often the judge agrees with those labels, including where it fails. Report agreement with humans, not only between models.

Write specific rubrics. Break judgements into concrete yes or no checks with examples of pass and fail.

Give judges evidence. Provide reference answers, source documents, test results, or tool access so judgements are grounded rather than plausibility-based.

Counter known biases. Randomise or swap response order in pairwise comparisons and average results. Control for length where it should not matter. Avoid judging a model's outputs with the same model when self-preference matters.

Use diverse judges deliberately. Different model families can reduce correlated errors, but still need human calibration.

Prefer programmatic checks where possible: tests, exact matching, schema validation, and citation checks are more reliable than model judgement.

Route disagreement and high stakes to humans. Use judges to screen at scale, and review cases where judges disagree, confidence is low, or consequences are serious.

Re-calibrate when models, prompts, or tasks change, because judge behaviour changes too.

Keep the judge prompt stable

Treat the judge prompt, model, and settings as part of the evaluation. Changing any of them changes scores, so version them and avoid comparing results produced by different judge configurations as if they were equivalent.

Budget human review deliberately

A practical split sends a random sample of judge-approved items to human review, all items where judges disagree, and all high-consequence items. The random sample is what tells you whether judge approval can be trusted at all.

Frequently asked questions

Can LLM-as-a-judge be trusted?: Only after calibration. LLM judges can be useful for scale, but their verdicts must be compared with careful human labels on your own task. Agreement between multiple judges shows consistency, not correctness, because models can share biases and make the same mistakes.
What biases do LLM judges have?: Common ones include position bias in pairwise comparisons, verbosity bias favouring longer answers, self-preference for outputs from the same model family, rewarding confident style over factual accuracy, inconsistent application of vague rubrics, and inability to verify facts without supplied evidence.
How do you calibrate an LLM judge?: Have domain experts carefully label a representative sample, run the judge on the same items, and measure agreement, examining where it fails. Refine rubrics, supply reference material, counter position and length biases, and repeat calibration whenever the model, prompt, or task changes.
Is using multiple LLM judges better than one?: It can reduce random noise and, with different model families, some correlated errors. It does not guarantee correctness, because judges can share training data and biases. Multiple judges still need calibration against human labels, and disagreements should go to human review.