Building an Eval Stack That Catches Regressions: Judges, RAG Metrics, and CI Gates (Part 2 of 2)

How to grade what code can't assert, evaluate RAG without debugging blind, and wire a gate that survives production.

Key Takeaways

• Deterministic first, always: If your pass condition contains parses, equals, contains, or compiles, it is not a judge task. Assertions are free, reproducible, and more accurate than a model.
• A judge is a model you're training: What makes one trustworthy is measured agreement with human labels, reported as true-positive and true-negative rates separately, not raw accuracy.
• RAG is two problems: Measure the retriever independently or debug blind. A low faithfulness score can't tell you whether the model ignored good context or never received it.
• Instrumentation is the only part with a deadline: Trace schema changes don't backfill, and you can't replay your way out of it because the index drifts underneath you. Every other eval task merely gets more expensive if you defer it.
• You're filling five slots, not picking one tool: Tracing, the label store, offline eval, online eval, and the CI gate are different layers. The free stack covers all five with three parts.
• Gate on deltas, not absolutes: Infrastructure configuration alone moves agentic scores by 6 percentage points, which is why differences under 3 points deserve skepticism.

Table of Contents

• LLM-as-Judge Without Fooling Yourself
• Evaluating RAG: Two Problems, Not One
• Instrumentation Is the Part That Expires
• The Tooling Landscape: Five Slots, Not Five Products
• The Gate That Survives Production

In Part 1 I argued that agent evaluation is a different discipline from prompt evaluation: reliability compounds against you, you grade the environment's end state rather than the transcript, and pass^k tells you what pass@k never will. All of that rests on assertions you can write in code.

But some questions don't reduce to an assertion. Is this answer grounded in the document we retrieved? Does it answer the question the customer asked, or a nearby one? Is the tone right for a support channel?

You can't regex your way to those. So people reach for a language model as the grader - and this is the exact point where most eval efforts quietly stop measuring anything real.

LLM-as-Judge Without Fooling Yourself

The technique traces back to Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, which reported that a strong judge reaches "over 80% agreement" with human preferences - described in the paper as "the same level of agreement between humans."

That result gets quoted constantly and almost always misread. The claim is not that the judge is correct 80% of the time. It's that the judge disagrees with a human about as often as two humans disagree with each other. When your ground truth is itself only 80% self-consistent, 80% is the ceiling, not a passing grade.

The same paper names the failure modes: position bias, verbosity bias, self-enhancement bias, limited reasoning. In practice your judge may prefer whichever answer you listed first, prefer the longer one regardless of content, and prefer output from its own model family.

Here's how this actually works when you build one that holds up.

1. One criterion per judge. Don't score helpfulness and factuality and tone in one grader. You can't debug a composite score, or tell which dimension regressed.

2. Binary labels. Pass or fail. A 1-10 scale invites the judge to cluster everything around 7, destroying your ability to discriminate between versions. Better still, don't ask for the verdict at all: have the judge emit atomic booleans - is each claim supported by the retrieved context, was the question actually answered - and compute the verdict from them in code. Keep the model's own score as metadata you can inspect, never as the number you report.

3. Reasoning before the label. Analysis first, verdict last. If the label comes first, the reasoning is post-hoc rationalisation.

4. Give it a reference, and an escape hatch. A hand-verified correct answer is the highest-leverage change in the literature, because it lets the judge check rather than know. And let it return "unknown" rather than forcing a verdict on insufficient information.

5. Never judge with the model that generated. Self-enhancement bias is measurable and large.

6. Align it against human labels. The step people skip, and skipping it makes everything above theatre. Hand-label a hundred examples and report true-positive and true-negative rates separately. Raw accuracy lies under class imbalance: a judge catching 95% of good answers but only 40% of bad ones scores well while being nearly blind to the defect you built it for.

And the hundred is per criterion and per segment. If your product answers in two languages, a judge aligned on one tells you nothing about the other: separate judges, separate label sets, roughly double the budget. Almost nobody plans for that, and a single blended score is exactly what hides the segment that's much worse.

Which brings us to the rule that saves the most money. If you can write the pass condition as a sentence containing "parses," "contains," "equals," or "compiles," it is not a judge task. Teams routinely ask a model whether JSON is valid or whether an account number matches. Those are json.loads and == - cheaper, faster, and strictly more accurate than any judge.

Evaluating RAG: Two Problems, Not One

RAG is where this pays off fastest, and where the most common mistake is structural rather than technical. (For the architecture itself, see my RAG series.)

The mistake is evaluating RAG as one system. It isn't. It's a search problem bolted to a generation problem, and they fail differently.

Ask yourself what a low faithfulness score tells you. It could mean the model ignored perfectly good context. It could mean the right chunk was never retrieved and the model filled the gap from memory. Those have opposite fixes - one a prompt problem, the other a chunking or embedding problem - and one end-to-end score cannot distinguish them. Measure the retriever separately or you are debugging blind.

On the retrieval side, classic information-retrieval metrics still work: recall@k, precision@k, MRR, nDCG@k. Recall@k is the most actionable, because it sets a hard ceiling. If recall@5 is 0.4, no amount of prompt engineering gets you past roughly 40% answerable questions. Compute it per segment as well: each language you support has its own independent ceiling, and the blended number is nobody's.

The usual objection is having no labelled data. There's a neat way around it: extract key facts from your corpus, generate the question each fact answers, and pair it with its source document. That gives you query-to-gold-document pairs with zero manual annotation.

On the generation side, the vocabulary the industry settled on is worth knowing even though its original implementation has stalled (more on that shortly). Faithfulness is the cleanest: decompose the response into claims, verify each against the retrieved context, divide supported claims by total claims. Answer relevancy asks whether the response addresses the question. Context precision measures whether relevant chunks ranked near the top; context recall, whether everything needed was retrieved at all.

One warning on the synthetic test sets every RAG framework offers to generate for you. Recent work found they rank retriever configurations reliably but do not reliably rank generator architectures, partly because generated questions reuse the source chunk's own vocabulary. Tune chunking and top-k on synthetic data; choose the model on human-labelled data.

Instrumentation Is the Part That Expires

Everything above assumes your traces contain what the measurement needs. Most don't, and you discover it at the exact moment you sit down to compute a number.

Go back to recall@k. To reconstruct it from real traffic, the trace has to carry the retrieved chunk IDs, the parent document IDs, the similarity scores, the k actually used, and the query as the retriever received it, after any rewriting. While you're in there, the fields you'll wish you had for everything else: model ID with version, the prompt or rubric version, session and user IDs, turn index, tokens, latency, cost, and whatever dimension you segment on - language, tenant, brand, channel.

Two properties make this urgent rather than a backlog item.

Trace schema changes don't backfill. A field added in September describes September onward. It tells you nothing about launch week, which is precisely the period you will most want to understand.

And you can't replay your way out of it, because the index drifts. Re-running last month's queries against today's retriever doesn't recover last month's retrieval - documents were added, re-chunked, re-embedded. You get a confident answer to a different question, with nothing to flag the substitution.

That combination is rare. Nearly everything else on an eval roadmap merely gets more expensive when you defer it. This one becomes impossible, which is why instrumentation is the one piece of eval work worth prioritising by deadline instead of by impact.

If you can only get two fields onto the trace before you ship, take the chunk IDs and the k. With those, retrieval quality stays reconstructable later. Scores and parent IDs sharpen the diagnosis; without IDs and k there is no diagnosis to sharpen.

The Tooling Landscape: Five Slots, Not Five Products

Now the question everyone actually asks: which framework?

It's the wrong question, and that framing is why teams end up with three overlapping tools and no gate. You're not picking one product. You're filling five distinct slots.

Here's where the honest verdicts land in 2026.

Promptfoo owns the CI gate. One YAML file, MIT, no account, no database - npx promptfoo eval gives you an exit code and a GitHub Action that comments the diff on your PR. Telling detail: OpenAI's own deprecation page points users migrating off its Evals platform (read-only 31 October 2026, shut down 30 November) straight at Promptfoo. What it doesn't do is production tracing.

Langfuse is the strongest free answer for tracing and online eval - MIT core, self-hostable, judges that score sampled production traffic. Its annotation queues quietly fill the label-store slot too, and they run in the browser even when your harness has no read path back to the traces, so human labelling never has to wait on infrastructure you haven't built. What it isn't: no metric library, so you bring your own faithfulness implementation, and no CI gate contract, so you write the threshold assertion yourself.

DeepEval is the metric library worth adopting, and it's pytest-native. Real agent metrics (task completion, tool correctness, argument correctness), not just text-similarity scores.

LangSmith covers all five slots coherently, annotation included, and is the lowest-friction single-vendor answer if SaaS is acceptable. Closed-source platform with an open client SDK, and self-hosting is Enterprise-tier only - the objection that kills it for many teams.

Two corrections to claims repeated everywhere. Arize Phoenix is not open source. Its core ships under the Elastic License 2.0, which forbids providing the software "to third parties as a hosted or managed service." Good tool, genuinely self-hostable, just not OSI open source - and if your legal team asks, that matters.

And Ragas has stalled. It defined the vocabulary the whole industry uses and is still the default in most tutorials. But its last release was v0.4.3 in January 2026, its last commit to main was February 2026, and pull requests keep piling up unmerged. No maintainer has declared it dead, so I won't either. Keep Ragas's vocabulary, take someone else's implementations.

The practical picks. Solo dev: Promptfoo, nothing else, until it hurts. Team shipping RAG: Langfuse for tracing, online eval and the labels, DeepEval for metrics, Promptfoo in CI - three parts, all free, all five slots covered. Just want a regression gate: Promptfoo, and it isn't close.

The Gate That Survives Production

The last mile is making this run automatically without producing noise everyone learns to ignore.

Gate on deltas, not absolutes. Pin a baseline and compare per-test-case against it. "Pass rate is 82%" means nothing alone; "four cases that passed last week now fail" is actionable. And don't require 100% - a suite you always pass isn't stressing anything.

And never gate on a composite. Inherited frameworks almost always contain one: several criteria collapsed into a single weighted score with a threshold, usually 0.5, usually named something like quality_status. It fails the same way a composite judge does. When it moves, nobody can say which dimension moved, so nobody can act on it, and within two sprints the team is overriding it by reflex. Gate per criterion, per case.

Know your three sources of flakiness, because they're independent and constantly conflated. Sampling randomness, fixed by temperature 0. Inference nondeterminism at temperature 0, because servers batch requests dynamically and the arithmetic isn't batch-invariant. And silent provider drift, where a model behind a stable alias changes underneath you with no changelog in your repo. Pin versions, never aliases, and remember that a cloud deployment name is an alias too. Record the version you ran against alongside every score, or last quarter's numbers quietly stop being comparable to this quarter's.

Then there's a fourth that surprised me. Anthropic ran the same model, same harness, same tasks across six resource configurations in a February 2026 study. Infrastructure error rates fell from 5.8% under tight limits to 0.5% uncapped, and success rate moved 6 percentage points on infrastructure alone. Their conclusion is the sentence I'd tape to every eval dashboard:

"leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched"

One security point deserves flagging. Your eval pipeline ships production traces - real user data - to whichever judge model and platform you picked. That's a data-processing decision, not a tooling preference. It's the strongest practical argument for self-hosted tracing, and at minimum it means redacting PII before traces leave your infrastructure, not after.

Note the second path, because it's the one teams miss: self-hosting your tracing doesn't make you self-hosted. A hosted judge API is egress on every single call, and judges run against exactly the content you were careful about. Either serve an open-weight judge on your own infrastructure, or use a commercial API under an agreement your legal team has actually seen, with redaction applied before the call rather than after.

Measurement Is the Part Nobody Can Copy

Anyone can wire up a model and a vector store this year. Prompts get copied, architectures get blogged, frameworks get swapped every six months.

What doesn't transfer is a test set built from your users' real failures, a judge calibrated against your domain expert, and a baseline that tells you whether last Tuesday's prompt change helped. That's why two teams on identical models ship products of very different quality.

Non-deterministic software has no compiler to tell you when you broke something. Either you build that instrument, or you find out from your users - and by then you're guessing which of the last twenty changes did it.

So start with the cheap rung. Write the deterministic assertions, add one aligned judge for what code can't check, put it behind promptfoo eval in CI, and turn on tracing - with the retrieval fields on it - so you can see production. That's an afternoon, not a quarter.

Build the instrument. Everything else is a demo with good lighting.

Frequently Asked Questions

How many labelled examples before I can trust a judge?

Around 30 gets you started and surfacing failure modes. A judge you're willing to gate releases on needs 100 or more, plus ongoing maintenance as your product changes. If a quality property isn't worth that, assert it in code or don't measure it.

Can I use the same model as both generator and judge?

You can, but measure it before you trust it. Self-enhancement bias is documented and material - judges score their own family's output higher. A different model family is cheap insurance; measured agreement against human labels is what actually settles it.

My team already uses Ragas. Do we rip it out?

No emergency for a pipeline that works today. But stop adding to it, and when you next touch that code, port the metric definitions somewhere maintained. Unmaintained code in a quality gate is a strange risk to carry.

We already shipped without the retrieval fields. Now what?

Add them now and treat today as day one of a corpus worth analysing. What you've lost is historical retrieval quality, not future retrieval quality, and the sooner the clock starts the smaller the hole. Meanwhile the synthetic query-to-gold-document set built from your own corpus still measures the retriever, needs no traces and no other team's time, and is the one piece of all this you can start this afternoon.

Where does my eval data actually go?

Wherever your platform lives, which for most hosted tools means a US cloud region. Traces routinely contain customer names, emails, and document contents. Redact before egress, prefer self-hosted tracing when data is sensitive, and treat your judge model provider as a subprocessor in your data-protection paperwork.

P.S. If you read one primary source from this article, make it Hamel Husain's LLM Evals FAQ - it's the densest practitioner writing on the subject and it will save you months.

Don't miss out on future posts and exclusive content—subscribe to my free newsletter today.

Ready to connect or explore more? Head over to my LinkedIn profile