Amazon Bedrock Introduces Prompt Caching to Reduce Costs and Latency

New Prompt Caching Feature in Amazon Bedrock

Amazon Bedrock has introduced a new prompt caching capability designed to optimize the cost and latency associated with using foundation models. This feature allows users to cache portions of their conversation context, such as system prompts, documents, and tool definitions, preventing redundant processing of these tokens on subsequent requests.

Without caching, sending a large context repeatedly with different user questions results in significant input token billing. For example, a 10,000-token contract sent with 50 user questions would incur charges for 500,000 input tokens, even though the core content remains the same.

Cost and Latency Benefits

The primary benefit of prompt caching is a potential reduction in input token costs by up to 90% for cached inputs, based on Amazon Bedrock's prompt caching pricing. Additionally, it lowers the time-to-first-token (TTFT) by allowing the model to skip reprocessing previously cached tokens and begin generation from a stored state.

This approach addresses the trade-offs of other optimization methods like shortening prompts or reducing context windows, which can compromise context quality or the model's ability to reason over complete information.

How Prompt Caching Works

When a request includes a `cachePoint` marker, Amazon Bedrock checks if the content preceding this marker matches an existing cache entry. If a match (cache hit) occurs, the model bypasses reprocessing those tokens and starts generation from the cached state. If no match is found, the content is processed normally.

This mechanism stores a snapshot of partially processed input, enabling subsequent requests with the same prefix to utilize the stored computation.

Practical Caching Scenarios

The new feature supports various practical scenarios, ranging from basic to advanced patterns. These include caching long documents for multi-question analysis, storing persona definitions and instructions across conversations, and caching tool schemas for agentic workflows. More advanced uses involve mixed TTL caching for different content tiers, tenant isolation for multi-tenant applications, and integration with the LangChain framework.