Prompt Caching

A feature where the provider stores the KV-cache of a stable prompt prefix and reuses it across requests. Your next call with the same prefix skips the prefill computation for those tokens and is billed at a steep discount, with lower latency.

Whenever you repeatedly send the same long prefix — a big system prompt, tool definitions, a knowledge-base document, codebase context, or long few-shot examples — followed by varying user messages. It's a near-free win for chatbots with rich system prompts or RAG-style workflows that reuse context.

Anthropic charges ~1.25x normal input for the first write and ~0.1x for reads. OpenAI discounts cached input tokens by about 50%. Google's Gemini has a separate context caching API with hourly storage + cheap reads. Exact numbers change — check provider docs, but the economics favor heavy reuse.

Any change to the prefix — even a single-token edit — invalidates everything after the change, because subsequent KV values depend on the prefix. Put static content first (system prompt, docs, tools) and varying content (user messages) last to maximize hit rate.