How We Test Every Plan Before It Ships

In [the last post](/blog/how-the-intake-workflow-works) we said evals are a tripwire, not a leaderboard. This one shows what that means in practice: how the fixtures are built, which properties we actually check, how a failing eval blocks a prompt bump before it ever sees a client, and what we deliberately don't measure. No leaderboards, no aggregate LLM-judged scores, no dashboards that go up and to the right for reasons no one can name.

Why not just score with another LLM?

LLM-as-judge is fine for taste-testing prose. It is a bad primary signal for a plan generator. Judges drift with model version, they smuggle in their own preferences, and they compress five different failure modes into one squishy number. We want the opposite: a small number of narrow, deterministic properties that each fail loudly and independently. When something breaks, we want to know which thing broke, not that the vibes are down 3%.

If your eval can't tell you which specific thing regressed, it's not an eval. It's a mood ring.

The fixture set

A fixture is one anonymised intake: a target role, a normalised CV, a weekly hours budget, and an optional deadline. Fixtures are curated by hand and versioned in the repo. Two rules govern what gets in:

• Every fixture has a canonical benchmark it should ground to (or an explicit "expected: ungrounded" marker). We don't add a fixture we can't answer about ourselves.
• The set is stratified across grounding tiers (exact / alias / semantic / expected-ungrounded), across seniority (junior → senior → staff), and across role families (platform, data, ML, product-adjacent). No single cluster dominates the average.

The fixture file is small on purpose — dozens, not thousands. A tight set that we understand end-to-end beats a huge set nobody reads. When we find a real intake that broke in production, we minimise it, strip the PII, and add it to the fixtures with a note pointing at the incident. That is how the set grows.

What the harness actually does

For each fixture the harness runs the full pipeline — ground, retrieve, generate — against a target prompt version, captures the structured output and the sources record, and evaluates a fixed list of properties. It does not call the mentor queue; evals stop at generation. Everything is deterministic given the prompt version and the retrieved context; when the model is not, we pin sampling parameters and record the seed.

fixture ──▶ [ ground ] ──▶ tier + position + similarity
                    │
                    ▼
              [ retrieve ] ──▶ top-k (lexical ⊕ vector, RRF)
                    │
                    ▼
              [ generate ] ──▶ plan draft + sources
                    │
                    ▼
       ┌──────── property checks ────────┐
       │  grounding-hit                  │
       │  skill-coverage                 │
       │  hallucination-guard            │
       │  timeline-realism               │
       │  schema-conformance             │
       └──────────────┬──────────────────┘
                      ▼
              pass / fail per property
                      │
                      ▼
              CI gate: any fail → block

The five properties, in detail

1. Grounding hit-rate

For each fixture we assert which tier should resolve the target. Exact- and alias-expected fixtures must not fall through to semantic. Semantic-expected fixtures must resolve above the 0.65 cosine gate. Expected-ungrounded fixtures must not be silently forced into a false positive. The aggregate is a per-tier confusion matrix, not a single accuracy number — an alias regression looks different from a threshold regression, and we want to see which.

2. Skill coverage

The benchmark row for the resolved role carries a weighted skill list. For each fixture, the generated plan must cover at least a threshold share of the high-weight skills for that role, expressed as concrete milestones or exercises — not as a passing mention in a paragraph. Below threshold, the plan is doing prose, not planning. The check is a set-membership test with normalised skill names; it takes microseconds and it catches the failure mode that mattered most in early versions of the prompt.

3. Hallucination guards

This is the strictest check. Every named technology, certification, tool, or framework version in the plan is cross-referenced against two allow-lists: the benchmark row and the client's extracted CV text. Anything cited that is not in one of those two sources is a hallucination — and one violation fails the fixture. No soft thresholds; the model is not allowed to invent a certification the client then goes and pays for.

• Name extraction is a small deterministic pass over the structured output — we control the schema, so the fields where names can appear are known.
• Normalisation folds obvious variants ("AWS SA-A" ↔ "AWS Solutions Architect – Associate") before the set check.
• A separate allow-list captures generic terms we accept regardless ("REST", "SQL", "CI") so the check doesn't flag vocabulary.

4. Timeline realism

The plan carries a total estimated hours figure and a weekly cadence. The check verifies internal consistency (total ÷ weekly ≈ weeks-to-deadline, within tolerance) and, when a deadline is present, that the reality-check block correctly labels the deadline as On track, Stretch, or Ahead against the client's stated weekly hours. Off-by-2x is a bug, not a rounding error, and this is the eval that catches the prompt drifting back toward a hard-coded 12-week roadmap.

5. Schema conformance

The output must validate against the plan Zod schema. If it doesn't parse, it doesn't ship — and the harness treats a parse failure as a category-1 regression, not a soft signal. We also assert schema stability across prompt versions: adding a required field is a breaking change and needs a version bump on the schema, not a silent edit.

Shadow runs before a bump goes live

A new prompt version doesn't take over live traffic on merge. It first runs in shadow: for every real intake, both the current live version and the candidate version generate a plan. Only the live version's output reaches the mentor queue; the candidate's output is written to a shadow table with its sources record and its property-check results.

After a bounded window (a fixed number of real intakes, not a wall-clock duration — traffic is bursty) we compare the two versions on the same properties as the fixture harness, plus one production-only signal: mentor first-pass approval rate on the live version's outputs. A candidate that improves the properties without regressing approval gets promoted. A candidate that trades one for the other doesn't ship, no matter how much prettier its prose is.

The most dangerous prompt bump is the one that reads better in three cherry-picked examples. Shadow runs exist to make cherry-picking irrelevant.

The CI gate

The fixture harness runs on every change to the prompt file, the plan schema, the resolver, or the retrieval code. The rule is boring: if any property regresses against the last-known-good baseline, the PR is red and the prompt bump does not merge. There is no override in the tool; the override is a human writing down, in the PR, which regression they are accepting and why. That comment becomes a fixture.

• Baselines are pinned per prompt version, not per branch. A new baseline is only written when a version is promoted after a passing shadow run.
• Runs are cached by (fixture hash, prompt version, retrieval-code hash). Rerunning is cheap; changing anything upstream invalidates the cache automatically.
• The report is a plain markdown diff posted to the PR — per-property pass/fail, per-fixture deltas, and a link to the raw outputs. It is designed to be readable at a glance during review.

Scheduled runs against the live model

Prompts are versioned; models are not always in our control. A model provider can silently change behaviour behind an alias. So the same harness runs on a schedule against the current live prompt version and whichever model alias production is pinned to, and posts a diff when properties move. If the model shifted under us, we see it in the fixture set before we see it in a bad plan.

What we deliberately do not measure

• Aggregate quality scores from an LLM judge. See above — they compress independent failure modes and drift with the judge.
• BLEU / ROUGE-style surface similarity to a reference plan. A plan is not a translation; there is no single right answer.
• Latency, at eval time. Latency is a production concern with its own budget; mixing it into quality evals corrupts both.
• Client satisfaction, at generation time. Satisfaction is measured after the mentor checkpoint, against the reviewed plan, and belongs in a different report.

What breaks most often, and why we care

Two failure modes dominate the history of this harness. First: the semantic tier quietly starts doing work the alias list should be doing, because a new role got embedded before its aliases got written. Grounding hit-rate catches this — the exact/alias share drops, semantic share climbs, and we know to grow the alias list rather than tune the threshold. Second: a prompt edit aimed at brevity accidentally drops the reality-check block, and timeline realism regresses on the deadline-present fixtures. Both are small bugs. Both are exactly the kind of thing that would ship silently without the harness, and both are the kind of thing a client would notice.

Why this is the interesting engineering

There is nothing exotic in the harness. Fixture files, deterministic property checks, a CI gate, a shadow table. The interesting part is the discipline of saying no to metrics that feel comprehensive and aren't, and yes to a handful of narrow checks that fail loudly and point at the code that broke. The generator gets better in a straight line because the harness makes "better" a specific claim, and "worse" an immediate red build.

Next in the series: [Your CV, Rewritten for the Role You Want](/blog/introducing-the-cv-review-service) — the first client-facing service built on the same grounded corpus this harness protects.