Kishan K

Cache your LLM like a database: bucket the inputs, template the output

LLMsCachingSystem Design 2026-11-03 · 2 min read

LLM calls are slow and cost money per token. If a million users ask nearly the same thing, generating a fresh answer for each is waste. The trick is to decide what the model actually needs to vary on.

Step 1: turn continuous inputs into a few buckets

Many apps feed an LLM numbers: sensor readings, scores, satellite indices. Rather than keying a cache on raw values (which never repeat), I bucket each number into a few named categories, for example low / medium / high. Five bucketed inputs give a small, finite key space. Similar situations now share a key.

Step 2: cache a template, not the final text

The first time a key is seen, call the model once and ask it for a report with placeholders for the exact values. Store that template. For every later request with the same key, fill the placeholders with that user’s real numbers. Personalisation survives; the model call does not repeat.

Step 3: a second level for translations

If the product speaks many languages, cache the translation of each template separately, keyed by (template_id, language). Translate once per language, not once per user.

Rules that keep it safe

What to measure

The trade-off

Bucketing is lossy by design. Coarse buckets mean more reuse but less specific advice. Audit the bucket boundaries with someone who knows the domain, and sample outputs for quality. Caching saves money; it does not make the content correct.

← All posts