How the Build & Ship intake workflow works: LLM analysis, pgvector retrieval, and why every plan passes a human checkpoint

A career plan is a small artefact with a large blast radius. If it drifts, a mentee spends months on the wrong skills. This post walks through how our intake workflow turns a booking, a CV, and a target role into a grounded, mentor-approved plan — the retrieval design, the prompt discipline, the evals that catch regressions, and the human checkpoint that sits between the model and the mentee. No code, just decisions.

The pipeline, end to end

Intake is a five-stage pipeline. Each stage has a single job, a typed contract with the next, and its own failure mode we can observe in isolation:

• Capture — booking, CV (or LinkedIn PDF export), target role, weekly hours, deadline. Files land in object storage; text is extracted server-side.
• Ground — resolve the mentee's stated target to a canonical role in our benchmark catalogue via a three-tier resolver (exact → alias → semantic).
• Retrieve — pull the benchmark's skills, tasks, and adjacent role neighbours using hybrid search (lexical + vector) over pgvector with HNSW.
• Generate — a versioned prompt takes the grounded context, the mentee profile, and a structured schema, and emits a plan draft plus a sources record.
• Review — the draft lands in a mentor queue. Nothing reaches the mentee until a human approves or edits it.

The important property is that every stage writes evidence: which tier grounded the target, which benchmark rows were retrieved and at what similarity, which prompt version generated the draft, and who reviewed it. That evidence is what turns a black box into something we can debug, evaluate, and improve.

Architecture at a glance

Booking + CV/LinkedIn PDF
        │
        ▼
   [ Capture ]  ── object storage + text extraction
        │
        ▼
   [ Ground ]   exact → alias → pgvector (0.65 cosine)
        │              │
        │              └── miss → flagged "ungrounded"
        ▼
   [ Retrieve ] hybrid: FTS  ⊕  HNSW vector search
        │              │
        │              └── reciprocal-rank fusion → top-k
        ▼
   [ Generate ] versioned prompt + structured schema
        │              │
        │              └── writes: sources, prompt_version
        ▼
   [ Review ]   mentor queue: approve / edit / reject
        │
        ▼
     Mentee

Grounding: three tiers, first hit wins

The resolver runs exact-name, then alias, then a semantic pgvector search gated at 0.65 cosine similarity. First hit wins; a miss on all three is recorded as ungrounded and the plan is flagged rather than silently fabricated. The cheap deterministic paths are cheap and deterministic for a reason — the semantic fallback exists to catch near-neighbours the alias list will never cover, not to be the default.

Retrieval: why HNSW over IVFFlat, and why hybrid

pgvector offers two ANN indexes. We chose HNSW; the reasoning is boring and worth stating.

• Recall at low latency: HNSW is graph-based and gives strong recall at small k without a training step, which matters when the catalogue grows incrementally and we can't afford to rebuild an IVFFlat index every time we add roles.
• No cluster tuning: IVFFlat requires you to pick a list count that tracks dataset size; get it wrong and recall collapses. HNSW's m / ef_search are easier to reason about and tune per query.
• Write-friendly: benchmark rows and embeddings get back-filled continuously. HNSW handles incremental inserts cleanly; IVFFlat wants a full rebuild for best results.

Vector search alone is not enough. Embeddings collapse rare, precise tokens (product names, specific certifications, framework versions) into their neighbourhood — exactly the tokens a career plan must not blur. So retrieval is hybrid: a lexical pass (Postgres full-text over weighted role name / alias / skill fields) runs alongside the vector pass, and results are fused with a reciprocal-rank blend before being handed to the generator. Lexical catches the exact matches; vector catches the paraphrases; the union catches both.

Embeddings are great at similar-in-general and terrible at exactly-this. Hybrid retrieval is how you get both without pretending one is the other.

The embedding choice

We embed with a 1536-dimension model via the Lovable AI Gateway, with Matryoshka truncation so we can trade dimensionality for storage without re-embedding. Requests go through a small in-process limiter — bounded concurrency, minimum inter-request spacing, exponential backoff on 429 / 5xx that honours Retry-After — because the fastest way to get rate-limited is to burst. Nothing exotic; the point is that the ingest job is a good citizen so the query path stays predictable.

Prompt versioning: prompts are code

The generator prompt is a first-class, versioned artefact. Every prompt has a semantic version, a changelog entry, and lives next to the generation function it drives. When a plan is written we persist which version produced it. That gives us three properties we refuse to give up:

• Reproducibility — a plan generated last month can be regenerated with the exact prompt it saw, not the current one.
• Attribution — a regression in plan quality can be traced to the version that introduced it.
• Safe rollout — new prompt versions can be shadowed against a hold-out set before they take over live traffic.

The prompt itself is boringly structured: system role, retrieved context block (grounded benchmark + top-k neighbours), mentee profile block (target, hours, deadline, extracted CV text), and an explicit output contract. The model doesn't get to invent the shape of the answer; the shape is the contract, validated on the way out.

What the evals catch

Evals are not a leaderboard. They are a tripwire. Ours run against a curated set of intake fixtures (real anonymised profiles, canonical target roles) and check specific properties, not vibes:

• Grounding hit-rate — the share of fixtures resolving via exact / alias vs semantic vs fell through. A drop in exact-match rate usually means the alias list needs an entry, not that the model got worse.
• Skill coverage — of the benchmark's high-weight skills for the target role, how many appear in the generated plan. Below threshold, the plan is not doing its job.
• Hallucination guards — no cited technology or certification that isn't in either the benchmark row or the mentee's CV. Deterministic post-check; a single violation fails the eval.
• Timeline realism — the plan's estimated total hours must be internally consistent with the roadmap's weekly cadence. Off-by-2x is a bug, not a rounding error.
• Shape conformance — the output validates against the plan schema. If it doesn't parse, it doesn't ship.

Evals run on every prompt change and on a schedule against the live model version. A failing eval blocks the prompt bump; it doesn't page anyone at 3am, because plans don't leave the queue without a human anyway. Which brings us to the checkpoint.

The human checkpoint is not decoration

Every generated plan lands in a mentor review queue. A mentor sees the plan, the mentee's profile, the grounding sources (which tier, which benchmark, what similarity), and the prompt version. They can approve, edit-then-approve, or reject with a reason. Only approved plans reach the mentee.

This is a design decision, not a limitation we haven't automated away yet. Three reasons we keep it:

• Accountability — a human name is on every plan a mentee acts on. That is the product.
• Feedback signal — reject reasons and edits are the highest-quality training signal we have for prompt improvements. An automated pipeline throws that away.
• Failure containment — the worst-case output of the pipeline is a plan a mentor doesn't approve. There is no path where a bad generation reaches a mentee unmediated.

The point of the LLM is not to remove the mentor. It's to make sure the mentor spends their time on judgement, not on typing.

Observability: what we look at on a Monday

Every plan writes a sources record — tier, matched position, similarity, skill and task counts, retrieved neighbour ids, prompt version. Every review writes an outcome — approved, edited, rejected, with reason. Together those two tables answer the questions that actually move the product:

• Where is the semantic tier doing work the alias list should be doing? (Add aliases.)
• Where are we falling through entirely? (Add benchmark rows.)
• Which prompt version has the highest first-pass approval rate? (Promote it.)
• Which reject reasons cluster? (That's the next prompt change, or the next eval.)

What this doesn't do

It doesn't scrape the web at generation time. It doesn't call a general-purpose agent loop. It doesn't personalise from anything the mentee didn't hand us. The scope is narrow on purpose — a narrow scope is what lets a small system be trustworthy.

Why we're writing this down

The decisions above — HNSW over IVFFlat, hybrid over pure vector, versioned prompts, deterministic post-checks, a mandatory human checkpoint — are the interesting part of the system. The code that implements them is the boring part, and the code that implements them will be rewritten several times before this product is done. Publishing the decisions is how we hold ourselves to them, and how anyone considering the Academy can see the standard the mentorship is built on.

More architecture posts to follow: the evaluation harness in detail, the ingest pipeline for the benchmark catalogue, and how we version the plan schema without breaking historical plans.