What is LLM-as-a-judge?

LLM-as-a-judge means using a language model to score the output of another model. It is the standard method for evaluating an AI agent at scale, when having a human review every answer costs too much. The judge receives a question and one or more answers, then returns a verdict.

What is Jev from TypeSafe AI?

Jev is a decision model launched by TypeSafe AI on 15 September 2026. It does not generate text: it returns a typed value with a calibrated probability, in a single non-autoregressive pass. Only input tokens are billed.

Can Jev replace an LLM for evaluating AI agents?

On its own, no. On RM-Bench, Jev reaches 84.0% accuracy against 87.6% for Gemini 3.8 Flash. However, a two-stage architecture that escalates low-confidence judgements to the LLM reaches 88.0%, better than either engine alone, at 38% of the cost.

What is style bias in an AI judge?

A judge shows style bias when it prefers the best-formatted answer over the most accurate one. RM-Bench measures this bias by comparing a plain correct answer with a detailed, Markdown-formatted wrong answer. Jev loses 19 points between the easy case and the hard case.

How do you measure the calibration of a classification model?

Group the predictions by confidence bin, then compare the stated confidence with the accuracy actually observed in each bin. A calibrated model at 70% confidence should be right about 70% of the time. This curve lets you set an automatic acceptance threshold based on data rather than intuition.

How much does automated evaluation of an AI agent cost?

The cost is the judge's unit price multiplied by the number of judgements. In our measurement, Jev comes out about 23 times cheaper than Gemini 3.8 Flash for the same volume, and the hybrid architecture at 38% of the cost of Gemini alone. The gap matters most for an evaluation suite replayed on every deployment, where the same judgement is redone hundreds of times.

Which benchmark should you use to evaluate an AI judge?

RewardBench v1 is saturated: Jev scores 99% on it, which no longer tells anything apart. RM-Bench is more discriminating because it isolates style bias with three levels of formatting. JudgeBench and RewardBench 2 are two other recent options.

Jev vs LLM-as-a-judge: we measured 4,500 judgements

TL;DR: Jev, TypeSafe's decision model, reaches 84.0% accuracy as a judge on RM-Bench, against 87.6% for Gemini 3.8 Flash. On its own, then, it is less accurate. Because its confidence is calibrated, you can escalate only the 24% of uncertain judgements to the LLM: 88.0% accuracy for 38% of the cost and five times less time. Measured on 4,500 judgements per configuration.

TypeSafe AI launched Jev on 15 September 2026. The model does not generate text: it returns a typed decision with a calibrated probability, in a single pass. We wanted to know whether it could replace an LLM-as-a-judge, the standard method for evaluating an AI agent at scale. An evaluation suite is replayed on every deployment, and each replay redoes the same thousands of judgements, which makes the judge's unit price decisive.

Here are the raw numbers, including the places where the measurement proved us wrong.

Why Jev is a credible candidate for the judge role

Three primitives, three judgement formats

Jev exposes three response types, and they map exactly onto the three standard ways of having a model act as a judge.

1. Choice selects one option from a defined set, which covers pairwise comparison: A or B. 2. Score rates on an ordered scale, which covers Likert-style scoring from 1 to 5. 3. Noul answers yes or no with a probability, which covers binary verdicts such as "is this answer faithful to the context?".

So the task stays the same in both cases. Only the engine changes.

A price that changes the arithmetic

Jev only bills input tokens, at a rate about 18 times lower than Gemini 3.8 Flash. Gemini bills both input and output, with output five times more expensive than input, and reasoning tokens are counted at the output rate. On a task where the useful answer fits in a single character, that last point weighs heavily.

The measurement protocol

RM-Bench rather than RewardBench

We started with RewardBench v1. Jev scores 99.0% on it. That result measures nothing: the benchmark is saturated, which is precisely why the Allen Institute released RewardBench 2.

We switched to RM-Bench, which is designed to isolate style bias. Each question in the set comes with three correct answers and three wrong ones, at three increasing levels of formatting: plain, detailed, then detailed with Markdown markup. The 3 × 3 matrix splits into three regimes.

- Easy: the correct answer is also the better presented. - Normal: both answers have the same level of formatting. - Hard: the wrong answer is the better presented.

The gap between the easy regime and the hard regime directly measures how far a judge can be bought by formatting.

What we ran

We sampled 250 of the 1,327 questions in the set, split evenly across five domains: chat, code, maths, safety refusal and safety response. The 250 questions produce 2,250 comparisons, each judged in both orders (A/B then B/A) to expose position bias. That makes 4,500 judgements per configuration.

Both engines receive the same instruction, word for word, and the same pairs. Gemini's reasoning is disabled, for two reasons: the cost stays predictable, and Jev does not reason either.