My reward model beats a model 2.4× its size. It also scores at the random floor.

I trained a preference reward model for EvalForge, my LLM evaluation platform, because I wanted a judge that could rank two answers without a golden answer to check against. It came out of the run with a number I liked: 0.7026 pairwise accuracy, against 0.6009 for a well-known public reward model with 2.4× the parameters, on the same split through the same harness. That is the number I would have led with a year ago. This piece is about why I no longer think it means anything on its own, and what I had to publish next to it before it did.

What the model is

microsoft/deberta-v3-base — 184M parameters — with a one-dimensional regression head, trained with a hand-written Bradley-Terry loop rather than a library trainer, so the objective is exactly one line and I can unit-test it against hand-computed values:

L = -log sigmoid(r_chosen - r_rejected)

One epoch on UltraFeedback train_prefs (60,700 pairs) at a 512-token budget. Raw Bradley-Terry logits are arbitrarily scaled, so afterwards I fit a single scalar temperature on the held-out split by minimising the NLL of sigmoid(margin / T), landing at T = 1.1668 and storing it in the checkpoint config alongside the sequence budget it was fit under.

That last detail matters more than it sounds. T was fit on margins, so the thing it calibrates is the probability that A is preferred to B for the same prompt — not the quality of a single answer. Bradley-Terry only ever sees the difference between two rewards, which means the objective is invariant to adding a constant to every score. The zero point is arbitrary. A single reward is not a 0-1 quality score, and no temperature makes it one.

The flattering number

On UltraFeedback test_prefs (N = 1,987 after a truncation audit), this model gets 0.7026. The public baseline, OpenAssistant/reward-model-deberta-v3-large-v2 at 435M, gets 0.6009 on the identical split through the identical evaluation function. Chance is 0.5000, and a failed run from my own learning-rate sweep sat at 0.5098, so I know what a collapsed model looks like on this harness. 0.7026 is real signal, not a collapsed number dressed up.

It is also not a claim that mine is the better reward model, and I want to say why in the specific rather than as a disclaimer. The comparison is in-distribution for me and out-of-distribution for them. I trained on UltraFeedback and I am testing on UltraFeedback: same annotator (an LLM), same prompt mix, same conventions about how much elaboration counts as good. The OpenAssistant model was fit on a completely different preference mixture — WebGPT, summarize-from-feedback, synthetic-instruct, Anthropic HH — so UltraFeedback is a distribution shift for it and not for me. A ten-point win under those conditions is what you would expect from home-field advantage alone.

The number that makes the first one believable

So I ran the official RewardBench 2 harness — allenai/reward-bench at commit 05a9005, dataset at 7ff0885, 1,865 prompts, best-of-4, scripts unmodified — on CPU, 1h18m, on 2026-08-13. Best-of-4 means the random baseline is 25% for the five accuracy domains.

My unweighted six-domain average was 25.3.

25.3 is the floor. Out of distribution, this model is guessing. The 435M OpenAssistant model — the one I beat by ten points in-distribution — officially scores 32.0 here, and the strongest sequence classifier on the leaderboard scores 84.1. So the ordering flips completely the moment we leave my training distribution, and the gap to the actual state of the art is roughly sixty points.

One domain survives: Math at 47.1, against the 435M baseline's 50.3 — within three points at 40% of the parameters. That is the only place I would defend this model in public, and I only know it because I ran the benchmark that embarrassed the other five rows.

Why the unflattering number is the point

A single in-distribution win is not evidence about a model. It is evidence about an overlap between a training set and a test set. On its own, 0.7026 is unfalsifiable marketing — the reader has no way to tell whether I found a good model or a favourable split, and the honest default is to assume the latter.

Publishing 25.3 next to it changes what 0.7026 is worth. It tells you I ran the test that could have destroyed the claim, and it draws the boundary of the claim precisely: this model predicts UltraFeedback-style preferences, cheaply, locally, on a CPU, with no API key, and it does not transfer. That is a narrow and useful thing to be. It is much more credible than a broad claim with one supporting number.

The lesson that cost me the most: accuracy can sit still while the model changes

The sharpest version of this came from a sibling model in the same repo — the DeBERTa hallucination judge. It encoded each example as Q: … C: … A: … and truncated from the right, which meant that on a 200-example RAGTruth sample the answer — the only span being classified — was deleted entirely on 50.5% of examples and partially on another 15.5%. The model was being asked an unanswerable question and answering it anyway.

I fixed the encoding to budget the context and keep the question and answer whole, and re-measured on the identical sample. The weights never changed.

ROC-AUC went 0.444 → 0.603. F1 at the best-accuracy operating point went 0.092 → 0.615. Best achievable accuracy went 0.605 → 0.605 — unchanged, and still below the 0.610 you get by labelling everything faithful.

Note the starting AUC: 0.444 is below 0.5. The model was ranking slightly worse than a coin flip, which is a signature of a broken input rather than a weak model, and accuracy never showed it. Accuracy collapses the whole score distribution to one side of one threshold, so a model that is ordering examples backwards and a model that is ordering them well can look identical if the majority class carries them both. AUC is threshold-free, so it saw the ordering change immediately. The fix bought ranking quality, not accuracy — and if I had been tracking accuracy alone, my conclusion would have been that a real bug fix was worthless.

Limitations, stated plainly

In-distribution accuracy is 0.70, which is modest before any shift at all — this is a base-size encoder trained for a single epoch. It inherits UltraFeedback's length and elaboration bias, so longer answers are systematically favoured regardless of correctness. It models no individual human: a 15-vote blind A/B probe by one rater put it at 0.400, which at N=15 is statistically indistinguishable from chance and is reported as a probe, not a result. There is a hard 512-token cap. And there is no absolute scale, so a single score cannot be thresholded, averaged across a dataset, or compared across prompts.

If you need a general-purpose reward model, take one off the RewardBench 2 leaderboard. If you want to argue that the in-distribution comparison is still unfair to the baseline in some way I have not accounted for, the harness is one file and both eval scripts share the same evaluate_pairs function, so you can attack the exact step you think is wrong.