The Prompt Cache That Silently Did Nothing

Prompt caching in the Melio ai-service Reasoning and Model layer — the LLM call sites of our LangGraph meal-generation pipeline — was done. cache_control: ephemeral breakpoints on the system and user prompts, cache-aware cost accounting, unit tests green. Shipped.

Then a Claude Console alert fired: "your prompt cache hit rate is low; caching repeated content like system prompts could save up to 37% of API spend."

The code was fine. The cache was silently doing nothing.

Root cause #1: the model-specific cache floor

Below the model's minimum cacheable prefix, prompt caching is a silent no-op — no error, just zero cache writes. cache_creation_input_tokens: 0. Haiku 4.5's floor is 4096 tokens; Sonnet 4-6's is 2048. And the prefix that counts is the cumulative sequence tools → system → messages up to the breakpoint, so your bound tool schemas count toward it.

Static token analysis of our real prompt builder: bound tool schemas ≈1,353 tokens, system block ≈1,462, cached user prefix ≈815–967. Total ≈3.7–4.3k — straddling the Haiku 4.5 floor. At typical English density it's under: zero cache writes. Only the largest, JSON-dense prompts marginally cleared it — exactly the low, sporadic hit rate the Console reported. The system-prompt-only cache the alert suggested could never fire on that model at all.

The uncomfortable part: model choice and caching payoff are coupled. Production runs Sonnet 4-6 (the prefix clears its 2048 floor comfortably), so caching fires there — but nobody had connected the two decisions until the alert forced us to.

Root cause #2: per-day content ahead of the breakpoint

Even where the floor was cleared, reuse only happened within one day's retry loop — never across the days of a plan. A Day {day_number} header sat in the system prompt, and the calorie targets carried a day-seeded ±5% jitter. Both ahead of the breakpoint, both changing every day, both silently invalidating the cache.

The fixes

• Day {day_number} removed from the system block (it stays in the user prompt), so the system block + bound tools are byte-identical across every day of a plan.
• The user prompt reordered: stable plan context (profiles, restrictions, targets) forms the cached prefix; per-day content follows it.
• The ±5% daily variation moved out of the prompt entirely, into the portion solver. The prompt shows stable targets; the server applies the variation deterministically.
• TTL raised to 1 hour: a 7-day plan generated day-by-day outlives the 5-minute default, so early-day cache writes expired before late days could read them. 1h writes bill at 2× (vs 1.25×), reads stay at 0.10× — break-even at ≥2 reads.

Every one of those is locked with byte-stability regression tests: assert the prompt bytes are identical across days, not just that the code runs.

What we deliberately did NOT do

Cache everything else. We measured every other Anthropic call site first: the USDA reranker (~320 tokens), the LLM validator (~582), the ingredient translator (~46). All far under even the lowest 1024-token floor — adding breakpoints there would do literally nothing.

Honest status

The flags are on in the rollout, but the live confirmation — cache_read > 0 in Langfuse on a real multi-retry generation — is still pending. Green unit tests already fooled us once here; the wire format was correct while the cache never fired.

The lesson, now written into our layer docs: caching only pays off if the static prefix clears that model's minimum cacheable length. Measure the tokens. Don't trust the green checkmark.