Case Study: Eval-Gated CI/CD for an AI Product
A 28-engineer AI product team replaces post-merge regression hunts with eval-gated CI: every PR runs golden sets, LLM-as-judge with statistical correction, and failure-mode taxonomies before a merge button appears.
The Business Problem
An AI-first SaaS company ships a customer-facing answer-bot built on a RAG pipeline plus an agent loop. Six months ago the team shipped a "small" prompt change that regressed answer-quality on questions about a specific contract type, and lost a $4M renewal when the customer noticed. The post-incident review found three things: the change had not been evaluated against the contract-specific test set; the LLM-as-judge metric used in spot checks had drifted by 11 points and no one noticed; and a fix that needed a 2-day rollback took 9 days because no one had a safe-to-revert baseline.
Constraints:
- 28 engineers across 4 teams; about 50 PRs per week touch the AI surface
- Customers in regulated industries refuse to accept regression on their domain-specific queries
- Per-PR eval budget: under 40ofmodelspend;totalevalbudget:about15K per month
- p95 PR-to-merge time goal: under 90 minutes including eval
- Quarterly auditor signoff on the eval methodology
Eval-gated CI is no longer a nice-to-have. Hamel Husain's eval blog series, Eugene Yan's writings (evals), and the judgy library for statistical correction have all converged on a playbook. Phoenix, Langfuse, Braintrust, and Galileo (now Splunk Agent Observability, after Cisco's acquisition) all support running evals from CI. The question is no longer "should we do this" but "how do we do this without doubling cycle time."
The tooling underneath churns, so own the assets. OpenAI's hosted Evals goes read-only on October 31, 2026 and shuts down November 30 (OpenAI points users to Promptfoo, the open-source tool it agreed to acquire in March), and Dynatrace agreed in August to buy Arize, Phoenix's developer. Golden sets, judge prompts, and calibration labels live in our repo, so the runner is replaceable in a sprint.
Architecture
Diagram
Components
| Layer | Tech | Purpose |
|---|
| Golden sets | YAML in repo, 1,200 to 4,000 cases per surface | Stable test base |
| Code evaluators | Pytest with custom assertions | Cheap, deterministic checks |
| LLM judges | Claude Sonnet 5.5 (pinned ID and effort), GPT-6 Sol as the calibrated backup | Subjective quality |
| Statistical correction | judgy | Convert judge scores to estimates with CIs |
| Pipeline | GitHub Actions plus custom runner | CI orchestration |
| Trace store | Langfuse | Per-PR observability |
| Annotation | Argilla self-hosted | Human re-labeling for judge calibration |
Data flow
- PR opens; GitHub Actions fires; Stage 1 (lint, unit, type checks) runs in 2 minutes.
- Stage 2 starts the golden-set eval against a representative subset (10 to 25 percent of the full set by default; 100 percent on protected branches or when labels say
full-eval).
- Each golden-set case runs through the new build, generates an output, and is scored by (a) code-based evaluators where deterministic checks apply (JSON schema, regex, factual lookups) and (b) an LLM judge for quality dimensions.
- Stage 3 corrects the judge scores using
judgy with the train/dev/test split for the judge prompt.
- The corrected estimate (with confidence interval) is compared against
main's last green build; if the lower bound of the CI is within tolerance, the PR merges; otherwise it blocks with a detailed report.
Key Design Decisions
1. Golden-set construction and rotation
Each golden set is built from three sources: production trace samples from the last 90 days (stratified by failure modes from error analysis), synthetic adversarial cases generated by a separate red-team LLM, and curated edge cases from customer support tickets. We rotate 10 to 15 percent of cases quarterly; we never delete cases (cases get archived to a frozen "historical regressions" set that runs only nightly). This avoids the over-fitting trap where the eval set drifts with the product.
Sizing: 1,200 cases per surface is the floor; below this, the corrected-score CI is too wide to detect a 2-point regression at 95 percent confidence. Eugene Yan covers this sizing math; we re-derived it for our metric.
2. Train/dev/test split for the judge
The LLM judge is itself a model with prompt parameters and few-shot examples. We treat the judge prompt as a model and apply train/dev/test discipline: 60 percent of human-labeled cases tune the judge prompt, 20 percent select the best prompt variant, 20 percent are a hold-out we only consult before a major judge-prompt change. This pattern is the core of the judgy methodology and Hamel's eval posts.
Re-calibration cadence: every 30 days, 50 fresh cases get re-labeled by 2 humans (Cohen's kappa over 0.7 required); if the judge's accuracy on dev set drops below 80 percent, we re-tune.
3. Statistical correction with judgy
Naive LLM-as-judge accuracy on subjective categories is around 75 to 88 percent in our domain. A raw judge score is biased. judgy computes a corrected estimate of the true pass rate using the judge's confusion matrix on the held-out set, and returns a confidence interval. We gate on the lower bound of the CI being within tolerance. This means we never block a PR on judge noise alone, and we never approve a regression that the judge merely failed to catch.
The math: the correction uses the judge's true positive rate (it passes a good answer) and true negative rate (it fails a bad one), estimated on the held-out set: corrected = (observed + TNR - 1) / (TPR + TNR - 1). If the judge has a 99 percent TPR but only a 75 percent TNR (it rarely fails a good answer but passes a quarter of the bad ones), a judge-reported 89 percent pass rate corrects to about 86.5 percent, with a 95 percent CI of roughly 82 to 91 percent. Lenient judges inflate pass rates; the correction takes that back out. We allow merge if the CI lower bound is at most 2 points below main. (Reference: judgy README math).
4. Failure-mode taxonomy as the assertion surface
We do not score "quality" as a single number. We score along axes drawn from our failure-mode taxonomy: hallucination, retrieval-miss, format violation, refusal, persona break, citation error. The taxonomy is the output of error analysis (Hamel's open-coding + axial-coding pipeline) applied to 800 production failures over 6 months. Per-axis scores let us block on a hallucination regression even if overall quality improved.
5. Per-PR eval budget
A full eval-set run costs 80to200 depending on model spend. At 50 PRs per week, the naive cost is 4Kto10K per week. We bound this:
- Default PR runs 10 to 25 percent of the golden set, stratified by failure mode (so all failure modes are represented).
- The
full-eval label triggers 100 percent.
- Nightly cron runs 100 percent on
main to catch any drift we missed.
- A new judge-prompt change triggers a 100 percent run on a frozen historical set.
This bounds per-PR cost to under 40anddefaultPRtraffictoabout1,250 per week (see the cost model below).
For binary axes there is now a cheaper judge tier: decision models that return a typed yes/no or category with a probability instead of generated text. A September 2026 comparison (arXiv 2609.29769) found LLM rubric judges cost 16 to 325x as much as TypeSafe's Jev decision model (early access, $0.042 per 1M input tokens), with significant accuracy differences in at most 8 of 27 paired comparisons; Jev was strongest on binary checklist criteria and behind on ordinal ones, and cascading the two gained at most 2.7 points. We are piloting one on the format-violation and refusal axes, behind the same calibration gate as any judge. Graded axes, and anything where reviewers need a written rationale, stay on the LLM judge.
6. Judge-prompt drift detection
Even with calibration, judge prompts drift: the underlying model updates, the few-shot examples become less representative, the prompt's vocabulary feels dated to the model. We monitor drift by:
- Re-running the held-out set monthly and reporting accuracy delta vs the previous month.
- Tracking inter-judge agreement (we run two judge prompts in parallel; divergence over time signals drift in one).
- Versioning the judge prompt in git; rolling back is a 1-commit operation.
When drift exceeds 3 points or kappa drops below 0.65, we open a maintenance ticket.
7. Caching the eval pipeline
A typical golden-set case generates an output, which is then judged. The output is not deterministic (current Claude models reject non-default temperature, and GPT-6 Astra accepts no temperature at all), but for caching we treat one recorded output per key as the reference. We cache (prompt-hash, model ID, effort, provider epoch) to (output, judge-score) so that re-running the same eval is nearly free. The provider epoch is a counter we bump whenever a vendor changelog reports a serving change under an unchanged model ID. On September 25, 2026 OpenAI fixed an image-encoding bug in GPT-6 Sol and Luna without changing the IDs and told customers to rerun image evaluations; a cache keyed on model ID alone would have kept serving launch-week results from the broken path. Cache hit rate on PRs that touch only orchestration code (not prompts) is around 70 percent; this is a 3x cost reduction on this class of changes.
8. PR-level instrumentation
Each PR's eval report includes: per-axis pass-rate vs main, per-axis examples that newly failed, per-axis examples that newly passed, judge-correction CI bounds, total cost, and a link to the trace store so engineers can replay any failing case. The report is posted as a GitHub comment within 3 minutes of the run completing.
CI Pipeline Sequence
Diagram
Failure Modes and Mitigations
F1: Judge prompt drift goes unnoticed
The judge gradually under-detects hallucinations after a model upgrade. Mitigation: monthly held-out replay; inter-judge agreement tracking; a "freeze judge" mode for protected branches that pins the judge model version and its effort level even when newer models are available. Pin effort explicitly, because defaults move between versions (Claude Opus 5 defaulted to high, Opus 5.5 to medium), and a silent default change is a judge change. The drift incident that broke us before was caused by exactly this; we now catch drift within one cycle.
F2: Eval set becomes overfit
A few cases get debugged repeatedly; the prompt is implicitly tuned to them. Mitigation: quarterly rotation; reserved adversarial cases that are never shown to engineers in failure reports (only outcomes). A separate red-team team owns the held-out set.
F3: Single PR runs a corner of the eval that misses regressions
Stratified sampling: we ensure each PR's 10-percent sample includes at least 1 case from each of the 12 failure modes. The full nightly run still happens on main. Per-PR coverage is bounded but not zero.
F4: Cost overrun from accidental full-runs
A full-eval label on every PR triples cost. Mitigation: the label requires an approval from a CODEOWNERS file; an automated reminder pings whoever applies it. We also cap monthly eval spend with a hard ceiling at $17K (about 13 percent above plan) and refuse to start a job that would exceed it.
F5: Block-rate too high; developers learn to ignore
If 35 percent of PRs are blocked, developers stop reading reports and look for ways around. Mitigation: we tune the gating tolerance to keep block-rate at 5 to 12 percent; we treat block-rate as an SLI; when it spikes we investigate why (often the judge is too strict on a new failure mode). The goal is to surface real regressions, not act as a gatekeeping toy.
F6: Holdout set leakage into training or prompts
A held-out case ends up as a few-shot example. Mitigation: the held-out set is stored in a separate repo with a separate access list; engineers cannot read it; only the eval runner has a deploy key. Failure reports include hashes, not raw cases, for held-out failures.
Leakage also runs the other way once the system under test has tools: an agent that can search, read files, or run git can find the answers. A September 2026 audit of SWE-Bench Pro v1.0 (arXiv 2609.34262) found that 24% to 73% of passing runs, depending on the model, were unearned, mostly through access to reference solutions in git history. Our eval sandbox runs the agent against an index and filesystem that contain neither the eval repo nor its history.
F7: Judge model deprecation
The vendor announces end-of-life for the judge model. Anthropic's minimum notice is 60 days; OpenAI's is 6 months for GA models but can be 2 weeks for previews, so a preview model never judges a protected branch. Mitigation: we keep at least two judge models from different vendors calibrated in parallel; when a deprecation lands, we swap with kappa thresholds preserved. The git history of judge prompts plus the calibration data make this routine.
F8: Eval runner queue saturation
A surge of PRs around release time queues evals 30 minutes deep. Mitigation: dedicated eval-runner GPU pool with autoscaling; priority lanes for protected branches; if queue depth exceeds 20, we automatically downgrade non-protected PRs to a 5 percent sample to clear the backlog faster.
Operational Considerations
Monitoring
| SLO | Target |
|---|
| PR-to-merge p95 | under 90 minutes |
| Eval-cost per PR p95 | under $40 |
| Block-rate (false negatives + true regressions) | 5 to 12 percent |
| Judge inter-rater kappa | over 0.7 |
| Holdout-set replay accuracy delta month-over-month | under 3 points |
| Production regression escapes (post-deploy) | under 1 per quarter |
Cost model
At 50 PRs per week:
- Default sampling: 25perPRaverage;1,250 per week
- Full-eval runs (about 8 per week): 100each;800 per week
- Nightly cron: 200each;1,400 per week
- Judge re-calibration: $50 per month
- Total: about 3,450perweek,orabout15K per month
This pays for itself with one prevented regression. Our post-incident estimate of the $4M renewal we lost suggests this is well-bounded by even one save per year.
On-call playbook
- Block-rate spike: check if any recent change to judge prompt or golden set drove this; compare per-axis scores to baseline.
- Eval cost spike: check sample rate config; rate-limit
full-eval label.
- Judge drift alert: trigger calibration cycle; rotate judge to backup model if drift is severe.
- Holdout breach (hash collision): immediately quarantine, regenerate the affected cases.
- Eval runner outage: PRs queue with a clear "eval pending" status; we never auto-merge while the runner is down; SRE pages within 15 minutes.
Quarterly review
Every quarter the AI team reviews: failure-mode taxonomy (do the categories still match real production errors?), golden set rotation (which 10 to 15 percent are stale?), judge calibration history (is drift accelerating?), and block-rate trend (is the gate becoming theater?). This review feeds into the next quarter's eval roadmap. We use the Hamel field-guide ritual: open-coding sessions on the most recent 50 failures, then axial coding to update the taxonomy.
Auditor pack
The eval pipeline produces a quarterly auditor pack: methodology document (versioned in git), golden set summary (counts per failure mode), judge calibration results (Cohen's kappa over time), block-rate histogram, and a sample of failing PRs with the rationale. The pack is generated automatically and signed by the head of engineering.
Why we do not use a single composite quality score
The temptation is to roll all axes into one number and gate on it. We do not. A composite hides regressions: a hallucination regression can be masked by a format-compliance improvement. We gate on per-axis scores so that each axis has its own confidence interval and its own block. The cost is more report noise; the benefit is that we never silently regress on a critical dimension.
What Strong Interview Candidates Cover
- They distinguish code-based evaluators (cheap, deterministic) from LLM-as-judge (expensive, subjective) and use both in different stages.
- They name statistical correction explicitly; they understand that a raw judge score is a biased estimate and that confidence intervals are the right abstraction for gating.
- They define a failure-mode taxonomy from error analysis and gate on per-axis scores, not a single composite.
- They specify the train/dev/test discipline for the judge itself, including kappa thresholds for re-calibration.
- They bound eval cost explicitly; they know full runs cost too much for every PR and that stratified sampling is the lever.
- They have a story for judge-prompt drift: they monitor it, they version-control the prompt, they have a roll-back plan.
- They protect the held-out set with hashes and a separate access list to prevent leakage.
References
Related chapters: Evaluation and Observability, Reliability and Safety, AI Evals Comprehensive Guide.