The feature works. A forty-turn session answers correctly, the logs look normal, and the invoice is five times what the token counter suggested. Caching is enabled. The bill says otherwise.
What separates those two facts is one line of documentation that the pricing page compresses away: a cache hit needs the same prefix, byte for byte, up to the point where you marked it. When part of that prefix changes, you do not fall back to the uncached price. You pay the write price, which is higher than the normal input price, and you pay it again on the next turn.
Key takeaways
- A cache hit requires an identical prefix up to your breakpoint, and a change at any level invalidates that level and everything after it.
- A prefix that changes every turn is worse than no caching: you pay the cache write price, which is above the normal input price, and never collect the read discount.
- On the same forty-turn loop, a stable prefix cost $0.43 where no caching cost $2.16 and a prefix that changed every turn cost $2.67.
- The transcript, not the system prompt, is where the money goes: it was 76% of the input tokens over forty turns.
- Total input cost grows roughly with the square of the turn count, so doubling the loop length multiplies the bill by about three to four.
What a hit actually requires
Three conditions, all of them cheap to check and easy to violate by accident.
The first is length. A prefix has to reach a minimum before it is cacheable at all: 512 tokens on the newer Claude models, and 1,024 tokens for GPT-5.6 and later, with the number varying by model and request settings for older ones (Anthropic, OpenAI). Below that, the question never comes up.
The second is identity. The cache is a hash of the prefix, so a single changed byte produces a different hash and a different entry. This is the condition that breaks in production, because the things people add to prompts for observability are exactly the things that change: a request id, a timestamp, the user's name, a shuffled list of tools, a retrieval result inserted above the conversation.
The third is where you put the marker. Anthropic's documentation describes the cache as a hierarchy, tools then system then messages, and states the consequence plainly: changes at each level invalidate that level and every level after it. Put a timestamp in the tools array and you have invalidated the system prompt and the entire message history with it. The cache does not keep the parts that did not change. It keeps the longest prefix that still matches.
One line in the wrong place
The usual version of this bug looks harmless in review:
{ "tools": [...], "system": "You are a support agent. Session started 2026-11-03T09:41:22Z." }The date is there for debugging, and it is genuinely useful. It is also, from the cache's point of view, a declaration that this prompt is new every time the clock ticks. Every turn rewrites the full prefix at the write premium and reads nothing back.
The arithmetic, with the assumptions written down
Take a loop with a 6,000-token static block (system prompt plus tool definitions), 1,000 tokens appended per turn (a tool result and the model's reply), 300 output tokens per turn, and forty turns. Price it at $2 per million input tokens, $10 per million output tokens, a cache write at 1.25x the input rate, and a cache read at 0.1x. Those multipliers are in the same range for both providers above; the absolute rates are not, and they move.
| Scenario | Input tokens sent | Cost |
|---|---|---|
| No caching | 1,020,000 | $2.16 |
| Stable prefix, cache hits | about 40,000 written, the rest read | $0.43 |
| Prefix changes every turn | 1,020,000, all written | $2.67 |
The third row is the one worth staring at. A prefix that never hits is 24% more expensive than turning caching off, and 6 times more expensive than getting it right, for the same conversation and the same answers.
That is not a rounding error, it is a sign error. The feature was enabled to save money and it added a surcharge to every request, because the write price sits above the ordinary input price. OpenAI charges 1.25x for the default 30-minute lifetime and 2.5x for a 12-hour minimum. Anthropic's table for a current Sonnet lists $2.00 per million input against $2.50 for a five-minute cache write and $4.00 for a one-hour write. Reads, in both cases, come back at a fraction of the input rate.
The curve is quadratic, and that is the real budget item
The per-turn cost is not the interesting number. The accumulated cost is, because the prompt grows: turn 40 in this model sends 45,000 input tokens to say one thing, against 6,000 on turn 1.
| Turns | No caching | Stable prefix |
|---|---|---|
| 10 | $0.24 | $0.09 |
| 20 | $0.68 | $0.18 |
| 40 | $2.16 | $0.43 |
| 80 | $7.52 | $1.16 |
Each doubling of the loop length multiplies the no-caching bill by about 2.8, then 3.2, then 3.5, heading toward 4 as the transcript term dominates. Caching does not change that shape. It divides the constant, roughly by five here, and leaves the exponent alone.
The practical consequence is that trimming the system prompt is the wrong lever once the loop is long. Over those forty turns the static block was 24% of the input tokens and the growing transcript was 76%. Making the system prompt 20% shorter saves about a twentieth of the bill; ending the loop four turns earlier saves considerably more.
The expiry nobody plans for
Caches expire. The default lifetime is five minutes on Anthropic and thirty minutes on OpenAI, with longer options that cost more to write. A reader who leaves a conversation open, reads the answer, thinks for ten minutes, and types a follow-up comes back to a cold cache. That turn rewrites the entire prefix at 1.25x, and the discount the pricing page advertised is nowhere in the invoice.
This is worth saying out loud because it is a product decision disguised as an infrastructure detail. If your interface invites people to pause, the pause is a cost.
What caching does not buy
OpenAI's documentation notes that cached input tokens still count toward tokens-per-minute limits. The win is money, not headroom. If the loop is hitting a rate limit, a better cache will not move it, and a shorter transcript will.
Measure your own loop
The table above is arithmetic on stated assumptions, not a quote. Change the numbers to yours and run it.
# loop_cost.py — edit the six numbers, then: python3 loop_cost.py
S, A, OUT, N = 6000, 1000, 300, 40 # static prefix, tokens per turn, output per turn, turns
IN_RATE, OUT_RATE = 2.0 / 1e6, 10.0 / 1e6
WRITE, READ = 1.25, 0.10
def no_cache():
tokens = sum(S + A * (k - 1) for k in range(1, N + 1))
return tokens * IN_RATE + N * OUT * OUT_RATE
def hits():
cost = S * IN_RATE * WRITE # first turn writes the prefix
for k in range(2, N + 1):
cost += (S + A * (k - 2)) * IN_RATE * READ # read what was written last turn
cost += A * IN_RATE * WRITE # write the new segment
return cost + N * OUT * OUT_RATE
def never_hits():
tokens = sum(S + A * (k - 1) for k in range(1, N + 1))
return tokens * IN_RATE * WRITE + N * OUT * OUT_RATE # writes only, no reads
print(f"no caching ${no_cache():.2f}")
print(f"cache hits ${hits():.2f}")
print(f"never hits ${never_hits():.2f}")Then replace the model with your actual numbers from the provider's response: the cached token count and the cache write count are both reported per request, which means the fraction of input tokens served from cache is a measurement, not an inference.
What this does not cover
The multipliers come from two providers' documentation in November 2026 and they differ by model, so treat the rates here as assumptions to replace rather than prices to quote. Output-side costs, reasoning tokens, batch discounts, and the interaction between caching and streaming are all outside this model.
Order the prompt by how often each part changes, put the marker at the end of the stable part, and measure the cache hit rate before measuring anything else. If the hit rate is near zero and the prompt looks static, the timestamp is almost always in the tools array or the first line of the system prompt, put there for a good reason.