Two planes of quality: making meal-plan generation both evaluable and observable

Two planes of quality: making meal-plan generation both evaluable and observable

Melio generates a week of meals for real people with real dietary restrictions. "Looks plausible" is not a bar you can ship on — a plausible-looking plan can still miss its calorie target badly, or hide anchovy paste in a fish-free week. So the quality work splits into two separate problems, and I built two separate planes for them: online validation (is this plan correct, right now, before the user sees it?) and offline evaluation (is the system systematically getting better or worse over time?). Then a third concern — observability — ties them together so that when something breaks I can say which day, which attempt, and which layer.

Here's the stack, by layer.

Layer 1 — Online validation: a cheapest-first cascade

Every generated day runs through a three-layer validate_day cascade, and the ordering is the whole point:

1. Programmatic checks (free, always run): calories present and non-zero, restriction compliance via token-boundary matching, duplicate dishes, missing participant meals.
2. LLM-as-judge (Claude Haiku at T=0.0, only if layer 1 passed): catches the disguised violations rules can't express — "anchovy paste" for someone avoiding fish, recipe incoherence.
3. USDA fact-check (blocking, only if 1+2 passed): compares generated nutrition against ground-truth USDA data, with a coverage-adjusted relaxation so exotic ingredients with no USDA entry don't trigger false-positive retries.

Cheapest-first is a cost decision: don't spend a Haiku call on structurally broken output, and don't spend a DB + compositor round on content that isn't even coherent. On failure it isn't a retry-blind loop — build_retry_corrections() feeds structured, per-participant feedback back in, an oscillation detector bails to the best attempt when three retries circle the same wrong answer, and best_day_attempt ships the best result rather than the last one. This is Anthropic's evaluator-optimizer pattern, made production-shaped.

Layer 2 — Offline evaluation: measure the system, not the plan

The online cascade catches individual failures. It can't tell me whether last week's prompt change quietly made things worse across a hundred plans. That's the offline harness (run_all.sh → MLflow with a champion registry).

Two decisions mattered here.

What to evaluate — failure-mode-first. The metric ordering mirrors the ways the system actually hurts a user: safety first (restriction_violation_rate must be 0, hard), then nutrition accuracy (mean_calorie_deviation_pct, the north star), then the generation loop (first_pass_rate, retries, force-commits), then completion, then retrieval quality, and only last the subjective stuff (variety, coherence). How to evaluate — pick the cheapest method that's trustworthy. Deterministic checks where the rule is expressible in code; reference metrics against USDA ground truth for retrieval and nutrition; LLM-as-judge only where the thing is hard to codify but easy to describe — and those judge scores are advisory and never gate a deploy, because cross-provider judges drift.

The sharp edge: there are 18 tracked metrics, but the deploy gate uses exactly two — mean_calorie_deviation_pct ≤ 15 and first_pass_rate ≥ 0.85 (on ≥20 rows). A gate you can't reason about is a gate people route around. Keep the blocking contract tiny and legible; track the other sixteen for diagnosis.

Layer 3 — How to read the handbook (the optimization playbook)

The handbook isn't a spec, it's a loop. To improve one number you: read the MLflow run in a fixed priority order (did the gate pass? is it safe and complete? is the loop efficient? is retrieval healthy? is it good?), pick the single weakest gate-relevant metric, map it to a subsystem lever via the metric→lever table (e.g. high force-commits → add batching guidance to the prompt; low recall → fix the FTS query and the Haiku reranker), change one thing, re-run, and compare against the champion.

The anti-patterns are the load-bearing part: don't raise first_pass by loosening validation thresholds (you'd trade a visible retry for an invisible bad plan shipped to a paying user), don't lift one metric while regressing a green, don't tune on a noisy 3-case smoke run, and never accept an absolute number without the champion delta. That's how someone extends the system without fooling themselves.

Layer 4 — Observability: make the two planes converge

The observability plane is self-hosted Langfuse v3, instrumented manually across a Trace → Span(day) → Generation(attempt) hierarchy. Manual spans, not LangChain callbacks — callbacks double-count tokens when streaming, so I read usage_metadata exactly once after each call and keep the callback handler wired but off.

The part that matters for this story is how it closes back onto the eval decisions. The same per-day scores the CI gate cares about — first_pass, max_calorie_deviation_pct, restriction_violation — are emitted online to Langfuse, so online scoring is the CI gate, not a parallel approximation of it. And the online LLM-as-judge (server-side, ~5% sampling, off the request hot path) pins the same gpt-4o snapshot the offline DeepEval judges use, so online and offline variety/coherence scores are directly comparable. Dietary and health data never leave the box: raw prompts and PII are omitted by default, and tracing is never on the critical path — if Langfuse is down, generation continues unobserved but never fails.

The thread that ties it together

Online validation catches the failure in front of the user. Offline evaluation catches the regression across the fleet. Observability lets me walk from "retries spiked after the model bump" to the exact day-span that oscillated — and because both planes score the same things the same way, the numbers actually agree. That convergence, more than any single metric, is what makes the pipeline trustworthy.