Prompt Caching Patterns for OpenAI Models on Amazon Bedrock to Reduce AI Agent Costs

The AI Agent's "Groundhog Day"

AI agents, similar to the movie "Groundhog Day" scenario, repeatedly process their entire history for each action. Every time an agent performs a step, such as calling a tool or deciding a next action, it must send the large language model (LLM) its complete story, including tool definitions, instructions, and all previous interactions. This is because LLMs lack internal memory between requests and must re-read all prior context to maintain continuity.

The Cost of Repetition

This repetitive sending of context means that for a coding session with 20 turns, the model re-reads turns 1-19 every time. Users are charged for every token sent, leading to increased operational costs as the session progresses. The model only predicts the next token based on the provided input, necessitating the re-submission of all prior information.

Prompt Caching as a Solution

Prompt caching offers a solution to this cycle. The service stores the agent's input prompt, which includes instructions, tools, and content. This stored prompt can then be reused in subsequent requests at a fraction of the original input price. This mechanism helps to keep costs down and ensures agents remain responsive and consistent.

Components of an Agent's LLM Invocation

A typical agent's initial invocation to an LLM includes several key components. These are the tool list, which defines all callable functions; instructions, comprising developer prompts, rules, and policies; the session history, detailing all past events, tool calls, and results; and the new turn, which is the latest tool result or user message. Without caching, all these components, especially the growing history, are sent repeatedly.