TypeSafe AI launched Jev on 15 September 2026. The model does not generate text: it returns a typed decision with a calibrated probability, in a single pass. We wanted to know whether it could replace an LLM-as-a-judge, the standard method for evaluating an AI agent at scale. An evaluation suite is replayed on every deployment, and each replay redoes the same thousands of judgements, which makes the judge's unit price decisive.
Here are the raw numbers, including the places where the measurement proved us wrong.
Why Jev is a credible candidate for the judge role
Three primitives, three judgement formats
Jev exposes three response types, and they map exactly onto the three standard ways of having a model act as a judge.
- Choice selects one option from a defined set, which covers pairwise comparison: A or B.
- Score rates on an ordered scale, which covers Likert-style scoring from 1 to 5.
- Noul answers yes or no with a probability, which covers binary verdicts such as "is this answer faithful to the context?".
So the task stays the same in both cases. Only the engine changes.
A price that changes the arithmetic
Jev only bills input tokens, at a rate about 18 times lower than Gemini 3.8 Flash. Gemini bills both input and output, with output five times more expensive than input, and reasoning tokens are counted at the output rate. On a task where the useful answer fits in a single character, that last point weighs heavily.
The measurement protocol
RM-Bench rather than RewardBench
We started with RewardBench v1. Jev scores 99.0% on it. That result measures nothing: the benchmark is saturated, which is precisely why the Allen Institute released RewardBench 2.
We switched to RM-Bench, which is designed to isolate style bias. Each question in the set comes with three correct answers and three wrong ones, at three increasing levels of formatting: plain, detailed, then detailed with Markdown markup. The 3 × 3 matrix splits into three regimes.
- Easy: the correct answer is also the better presented.
- Normal: both answers have the same level of formatting.
- Hard: the wrong answer is the better presented.
The gap between the easy regime and the hard regime directly measures how far a judge can be bought by formatting.
What we ran
We sampled 250 of the 1,327 questions in the set, split evenly across five domains: chat, code, maths, safety refusal and safety response. The 250 questions produce 2,250 comparisons, each judged in both orders (A/B then B/A) to expose position bias. That makes 4,500 judgements per configuration.
Both engines receive the same instruction, word for word, and the same pairs. Gemini's reasoning is disabled, for two reasons: the cost stays predictable, and Jev does not reason either.
Results: on its own, Jev loses
| Metric | Jev | Gemini 3.8 Flash |
|---|---|---|
| Overall accuracy | 84.0% | 87.6% |
| Easy | 91.7% | 92.9% |
| Normal | 87.4% | 90.3% |
| Hard | 72.8% | 79.6% |
| Median latency | 243 ms | 2,021 ms |
| Relative cost (Gemini = 100) | 4.3 | 100 |
So the direct answer to our question is no. Gemini judges better, by 3.6 points overall and by 6.8 points on the hard regime. Jev is 23 times cheaper and 8 times faster.
Style bias is real, and it can be measured
Jev loses 18.9 points between the easy regime and the hard regime. When the wrong answer is the better presented, its accuracy drops to 72.8%. Gemini loses 13.3 points over the same range. Both engines are sensitive to formatting, Jev more so.
Our hypothesis about position bias was wrong
We expected Jev to be more stable when the order is swapped, since it does not build sequential reasoning. The measurement says the opposite: 92.0% of pairs keep their verdict after the swap with Jev, against 95.5% with Gemini. Counting the times the judge picks position A in both orders, Jev does so 89 times out of 2,250, Gemini 65 times.
Where Jev wins
The breakdown by domain explains the overall gap. Jev holds up on judgements that require no reasoning and falls behind where reasoning is needed, which is consistent with its architecture: a single pass with no chain of reasoning cannot check a mathematical proof.
The calibration holds up
Every article published since the launch repeats that Jev returns calibrated probabilities. Nobody had checked. We grouped the 4,500 judgements by confidence bin and compared the stated confidence with the observed accuracy.
| Confidence bin | Count | Actual accuracy |
|---|---|---|
| 0.0 to 0.1 | 133 | 52.6% |
| 0.2 to 0.3 | 176 | 60.2% |
| 0.4 to 0.5 | 218 | 66.5% |
| 0.6 to 0.7 | 283 | 80.9% |
| 0.8 to 0.9 | 507 | 87.2% |
| 0.9 to 1.0 | 2,224 | 95.3% |
The curve is almost monotonic and runs from 52.6% to 95.3%. A low confidence does signal a likely error. That makes it possible to set an automatic acceptance threshold on observed data.
The two-stage architecture beats both engines
The rule
Each comparison goes to Jev first, which returns a verdict and a confidence. Above the threshold, the verdict is kept. Below it, the comparison is sent on to Gemini, whose verdict prevails.
We ran this pipeline for real on the same 250 questions, with a threshold of 0.6.
The measurements
| Configuration | Accuracy | Relative cost (Gemini alone = 100) | Time for 4,500 judgements |
|---|---|---|---|
| Jev alone | 84.0% | 4.3 | 144 s |
| Gemini alone | 87.6% | 100 | about 28 min |
| Hybrid at 0.6 threshold | 88.0% | 38 | 316 s |
1,096 of the 4,500 judgements fell below the threshold, or 24.4%. The hybrid configuration is the most accurate of the three, at 38% of the cost of Gemini alone and in about a fifth of the time. It also matches Gemini on stability when the order is swapped, at 95.6%.
The gain comes from the split. Above 0.6, Jev runs at between 81% and 95% accuracy, so keeping those verdicts costs almost nothing in quality. Below it, Jev drops to between 52% and 68%, and those are the cases Gemini fixes. The frontier model's budget is thus concentrated on the judgements that the cheap model itself flags as uncertain.
Escalating more makes the result worse: at a threshold of 1.0, accuracy falls back to 87.8%, because you throw away judgements that Jev gets right and Gemini gets wrong.
What we did not measure
This campaign covers one dataset, one family of tasks and one frontier model. Four limits are worth stating.
- RM-Bench has been public for a while, and nothing guarantees it is absent from Jev's training data.
- The 250 questions produce 2,250 comparisons that are correlated with one another, since nine comparisons share the same question. The independent sample size is therefore 250, not 4,500.
- Gemini's reasoning is disabled. A judge allowed to reason would cost more and would probably judge better on code and maths.
- The 0.6 threshold is tuned to this dataset. On another corpus, it has to be recalculated from the calibration curve.
What we take away for our audits
A decision model covers the part of an evaluation where the judgement requires no reasoning, and the calibration curve tells you which part that is without any guesswork. The rest still goes through an LLM judge.
For an evaluation suite replayed on every deployment, dividing the cost by 2.6 and the time by 5 at equal or better accuracy is a gain that shows up on the invoice. At Idun, this arithmetic matters because we analyse existing AI estates, and the volume of judgements to produce is what decides what we can afford to examine.
