Prompt caching is an inference optimization that allows an LLM provider to reuse computation for input tokens it has already processed. It primarily applies to input tokens, not newly generated output tokens, and its value becomes easier to understand when considered alongside context-window size and the way long-running agents accumulate context.

An LLM request consists of input tokens and output tokens. The input may contain system instructions, tool definitions, retrieved documents, conversation history, source files, previous tool results, and the user’s latest message. The model first processes that input and then generates the response token by token.

Without caching, repeated input must be processed again on each request. In a conversation or coding-agent session, much of the input is often identical to the previous turn. System instructions, tool schemas, earlier conversation history, and previously loaded files may all remain unchanged while only a relatively small amount of new material is appended.

Prompt caching allows the provider to reuse the model state associated with that unchanged prefix rather than recomputing it from the beginning.

For example, a request might contain:

system instructions
tool definitions
80,000 tokens of existing conversation and files
2,000 tokens of new material

On the following turn, most of that sequence may be identical, and if the model provider has retained the relevant cache, the previously processed prefix can be reused and only the newly appended portion needs full input processing.

What is cached

It is slightly misleading to think of this as caching the conversation itself. The application may already store the conversation text independently. What the inference system is interested in reusing is the computation produced while processing that text. Transformer attention layers maintain key/value state, usually referred to as the KV cache, for tokens that have already been processed. We think of the model as follows:

[cached prefix][new input][new output]

The cached prefix is input whose previous computation can be reused. The new input must still be processed, and the new output must still be generated. Caching generally works on a matching prefix, and stable content near the beginning of the request is therefore especially valuable. If a large system prompt, tool schema, or body of reference material remains identical across requests, the provider may be able to reuse its cached state. Changing something near the beginning of the prompt can invalidate reuse for everything that follows it. Changing only the most recent user message usually preserves much more of the reusable prefix. For that reason, systems that care about cache efficiency generally benefit from keeping stable material early and variable material later.

Why cached input is cheaper

Processing a large prompt requires substantial transformer computation. If the same 100,000-token prefix appears repeatedly, recomputing every layer over those tokens on every turn wastes work. With caching, the provider can retain and reuse the intermediate state produced by the earlier request. Fetching and extending that state is cheaper than performing the full prefill computation again. The provider still incurs memory, storage, bandwidth, cache-management, and serving costs. This is why inference APIs commonly distinguish among uncached input tokens, cached input tokens, and output tokens, with cached input priced below ordinary input. The economic benefit can be significant with agents, because their requests are highly repetitive and quick in succession. A coding agent may repeatedly send the same conversation state, and previous actions while adding only a small amount of new information on each iteration. Caching removes much of the cost of recomputing the stable portion.

Output tokens

New output cannot usually benefit from prompt caching in the same way because it obviously has not yet been generated. LLMs generate output autoregressively. Each new token depends on the tokens that precede it, so generation requires additional model computation for every token produced. An assistant response can, however, become cacheable on a later request. Once that response is included in the next request as part of the conversation history, it is now input. If the surrounding prefix matches a previous request, that formerly generated output can become part of the cached input prefix. The distinction is therefore based on the role a token is playing in the current request, not on where the text originally came from.

Conversation persistence and prompt caching

A persisted conversation does not imply a persisted inference cache. An application may retain a conversation indefinitely, but when the user returns, it can reconstruct the relevant history and send it back to the model. Prompt caching is a separate optimization maintained by the inference provider. A conversation opened the following day may contain exactly the same history while the corresponding KV cache has already expired or been evicted. In that case, the application still has the conversation, but the inference system has to process the input again.

Cache lifetime varies by provider and product. Some caches last only a relatively short period, while other systems expose explicit caching mechanisms with defined retention periods. Unless the provider guarantees otherwise, a sensible operational assumption is that rapid consecutive requests are likely to benefit from caching while requests resumed after a long gap may not.

Caching and large contexts

Modern models may support hundreds of thousands or even millions of tokens of context window. A larger context window gives the system permission to process more tokens and accumulate more into the window, which may not be efficient or even useful. Suppose a coding agent grows from 60,000 tokens of context to 200,000, then 400,000, then 800,000. Even if much of that context receives cached-input pricing, the system is still maintaining and transferring a much larger KV cache. Newly appended tokens attend over a larger history, cache storage grows, memory pressure increases, and cache misses become more expensive.

We could say that caching makes repeated context cheaper, but does not remove the cost of carrying it unnecessarily. This is why a well-designed coding agent should normally prefer relevant context over maximal context. A useful working set is usually a combination of the current task, relevant source files, important architectural information, recent actions, unresolved state, and selected historical context. Older tool output, irrelevant source files, obsolete reasoning traces, superseded plans, and repeated information should generally be removed, summarized, or retrieved again when needed.

Million-token context windows are often unnecessary

For many coding workloads, a practical ceiling around a few hundred thousand tokens is already extremely generous. A configuration such as:

maxInputTokens: 245760
maxOutputTokens: 16384

provides enough capacity for a very large amount of source code, instructions, conversation history, tool output, and generated work in a single request. A million-token limit can still be useful for exceptional workloads such as analysing an unusually large repository snapshot, consuming very large logs or generated artifacts in one request, or performing analysis where aggressive retrieval and compaction are undesirable. Such workloads are not necessarily representative of normal coding-agent operation.

For much of the routine software work, an agent that repeatedly needs close to one million tokens of active context may have a context-management and attention problem rather than a context-window problem. Retrieval, file selection, structured state, summarisation, and periodic compaction are usually better tools than indefinitely appending everything the agent has seen. There is also no reason to assume that a larger advertised context window automatically improves model quality. Relevant information can become harder to isolate when surrounded by large amounts of unrelated material. A model capable of processing one million tokens is not necessarily best used by filling all one million tokens.

Does a smaller context window save money?

For a hosted API that bills based on tokens processed, merely allowing a one-million-token maximum normally does not immediately mean paying for one million tokens. If two requests each contain 80,000 input tokens, they will generally cost the same regardless of whether the application permits a maximum of 250,000 or one million tokens, assuming the same model and pricing schedule.

A smaller application-level limit can nevertheless reduce real expenditure because it changes agent behaviour. If the agent is allowed to accumulate nearly unlimited history, its requests accumulate and may steadily become larger. A lower ceiling forces the system to compact, summarize, discard irrelevant material, or retrieve context selectively before reaching that point.

If the one-million-token configuration never results in larger requests, there may be little or no billing difference. If it encourages the agent to carry hundreds of thousands of unnecessary tokens on every turn, the smaller limit can reduce costs substantially over time. For self-hosted inference, very large maximum sequence lengths can increase KV-cache memory requirements and reduce the number of concurrent requests that fit on available accelerators. In that environment, limiting maximum context may improve infrastructure efficiency even when requests do not always reach the limit.

Design principles

A large context window answers the question of how much information the model is allowed to consider in one request. Prompt caching answers the question of how much repeated input computation can be avoided between similar requests. But neither removes the need for context management. Typically, agents should keep stable prefixes stable so they can be cached, retain enough context to preserve task continuity, and remove or externalize information that no longer needs to remain in active attention. Retrieval and summarisation should be used to bring older information back when it becomes relevant.

For coding agents in particular, a context limit in the region of a few hundred thousand tokens is often large enough for many demanding workloads. If extensive real-world use does not produce context-related failures at that size, increasing the ceiling to one million tokens offers limited practical value unless there are known tasks that genuinely require it.

The most useful cost model should consider how many input tokens are actually sent, how many of them are cacheable, how often the cache survives between requests, how much new context is added on each turn, and whether the agent is carrying information that could instead be retrieved or summarized. Prompt caching rewards stable, repeated context. Good context management ensures there is not too much of it.