[{"data":1,"prerenderedAt":395},["ShallowReactive",2],{"tag-labels":3,"site-header":4,"article-caching-that-never-hits-id":72},{},{"featuredPost":5,"navigations":29},{"id":6,"title":7,"title_id":8,"slug":9,"excerpt":10,"excerpt_id":11,"has_translation":12,"published_at":13,"reading_time":14,"hidden_from_listing":15,"tags":16,"thumbnail_image":21,"og_image":22,"series":23,"authors":28},110,"When You Pick a Framework Just to Look Different","Ketika Pilih Framework Cuma Biar Kelihatan Beda","ketika-pilih-framework-cuma-biar-kelihatan-beda","The last time you chose a new framework or library, was it because your problem needed that tool, or because it sounded cool and few people around you were using it? That question is uncomfortable. Technology decisions are often made for reasons that never appear in any technical document.","Terakhir kali kamu milih framework atau library baru, itu karena masalahmu butuh alat itu, atau karena alatnya kedengaran keren dan belum banyak yang pakai? Pertanyaan ini nggak enak dijawab. Soalnya keputusan teknologi sering diambil dengan alasan yang nggak pernah ditulis di dokumen teknis mana pun.",true,"2026-10-06T15:25:21.000Z",5,false,[17,18,19,20],"karier","framework","opini","pengalaman",null,"\u002Fog\u002Fketika-pilih-framework-cuma-biar-kelihatan-beda.png",{"id":24,"title":25,"title_id":26,"slug":27},9,"Articles","Artikel","all",[],[30,46,67],{"collection":31,"item":32},"series",{"__typename":31,"title":25,"title_id":26,"slug":27,"children_series":33},[34,38,42],{"title":35,"title_id":36,"slug":37},"Artificial Intelligence","Kecerdasan Artifisial","kecerdasan-artifisial",{"title":39,"title_id":40,"slug":41},"Programming Fundamentals","Konsep Dasar Pemrograman","konsep-dasar-pemrograman",{"title":43,"title_id":44,"slug":45},"Algorithms and Data Structures","Algoritma dan Struktur Data","algoritma-struktur-data",{"collection":31,"item":47},{"__typename":31,"title":48,"title_id":48,"slug":49,"children_series":50},"Tutorial","tutorial",[51,55,59,63],{"title":52,"title_id":53,"slug":54},"Learning Python","Belajar Python","belajar-python",{"title":56,"title_id":57,"slug":58},"Learning Go","Belajar Golang","belajar-golang",{"title":60,"title_id":61,"slug":62},"Learning Java","Belajar Java","belajar-java",{"title":64,"title_id":65,"slug":66},"Learning Pascal","Belajar Pascal","belajar-pascal",{"collection":31,"item":68},{"__typename":31,"title":69,"title_id":69,"slug":70,"children_series":71},"Lab","lab",[],{"article":73,"renderedBody":293,"renderedTakeaway":294,"latestArticles":295,"toc":370},{"id":74,"title":75,"title_id":21,"slug":76,"excerpt":77,"excerpt_id":21,"has_translation":15,"published_at":78,"reading_time":79,"hidden_from_listing":15,"tags":80,"thumbnail_image":21,"og_image":85,"series":86,"authors":285,"body_locale":288,"available_locales":289,"body":291,"takeway":292,"prerequisite":21},135,"Caching That Never Hits Costs More Than Not Caching at All","caching-that-never-hits","Same model, same conversation, same token count, and a bill that is five times higher. The difference is a prefix that changes, a write premium nobody reads about, and a cost curve that grows faster than the loop.","2026-11-04T01:10:00.000Z",7,[81,82,83,84],"llm","ai-agent","cache","api","\u002Fog\u002Fcaching-that-never-hits.png",{"id":87,"title":35,"title_id":36,"slug":37,"description":88,"description_id":21,"articles":89,"children_series":277,"parent_series":278},8,"Notes on AI that go past the demo: how the models work, where they break, and what changes once you put them in production.",[90,107,124,139,153,167,183,199,220,239,258,272],{"id":91,"title":92,"title_id":93,"slug":94,"excerpt":95,"excerpt_id":96,"has_translation":12,"published_at":97,"reading_time":98,"hidden_from_listing":15,"tags":99,"thumbnail_image":21,"og_image":104,"series":105,"authors":106},1,"Why Does AI Keep Hallucinating? Meet JEPA, the AI Brain With Common Sense","Kenapa AI Suka 'Halu'? Kenalan sama JEPA, Otak AI yang Punya Akal Sehat","kenapa-ai-suka-halu-kenalan-sama-jepa-otak-ai-yang-punya-akal-sehat","Generative AI hallucinates because all it does is guess word patterns. JEPA tries to change that by teaching machines how the real world works, the way a human brain does.","AI generatif sering 'halu' karena cuma jago nebak pola kata. JEPA coba ngerombak itu dengan ngajarin AI cara kerja dunia nyata kayak otak manusia.","2026-06-02T08:09:05.595Z",4,[100,101,102,103],"jepa","artificial-intelligence","yann-lecun","agi","\u002Fog\u002Fkenapa-ai-suka-halu-kenalan-sama-jepa-otak-ai-yang-punya-akal-sehat.png",{"id":87,"title":35,"title_id":36,"slug":37},[],{"id":108,"title":109,"title_id":110,"slug":111,"excerpt":112,"excerpt_id":113,"has_translation":12,"published_at":114,"reading_time":98,"hidden_from_listing":15,"tags":115,"thumbnail_image":21,"og_image":116,"series":117,"authors":118},2,"LLMs Are Going to Hit a Wall. What Then?","LLM Bakal Mentok, Terus Gimana Dong?","llm-bakal-mentok-terus-gimana-dong","LLMs like ChatGPT are great at stringing words together, but they never really grasp reality. Yann LeCun says this architecture will hit a wall and needs a new answer.","LLM kayak ChatGPT itu jago ngerangkai kata, tapi nggak beneran paham realitas. Yann LeCun bilang arsitektur ini bakal mentok dan butuh solusi baru.","2026-06-03T08:55:33.003Z",[101,81,102,100],"\u002Fog\u002Fllm-bakal-mentok-terus-gimana-dong.png",{"id":87,"title":35,"title_id":36,"slug":37},[119],{"authors_id":120},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},"Daffa Izzuddin","izzudd","Writer",{"id":125,"title":126,"title_id":127,"slug":128,"excerpt":129,"excerpt_id":130,"has_translation":12,"published_at":131,"reading_time":98,"hidden_from_listing":15,"tags":132,"thumbnail_image":21,"og_image":134,"series":135,"authors":136},3,"Meet JEPA: Where AI Starts Learning Common Sense","Mengenal JEPA: Saat AI Mulai Belajar Punya \"Akal Sehat\"","kenapa-ai-suka-halu-dan-gimana-jepa-beresin-masalah-ini","AI hallucinates because it only guesses data probabilities without understanding reality. The JEPA architecture tries to change that by building a world model in its head.","AI sering 'halu' karena cuma nebak probabilitas data tanpa paham realita. Arsitektur JEPA mencoba mengubah itu dengan membangun model dunia di kepalanya.","2026-06-03T09:32:50.446Z",[101,100,133,102],"world-models","\u002Fog\u002Fkenapa-ai-suka-halu-dan-gimana-jepa-beresin-masalah-ini.png",{"id":87,"title":35,"title_id":36,"slug":37},[137],{"authors_id":138},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":98,"title":140,"title_id":141,"slug":142,"excerpt":143,"excerpt_id":144,"has_translation":12,"published_at":145,"reading_time":98,"hidden_from_listing":15,"tags":146,"thumbnail_image":21,"og_image":148,"series":149,"authors":150},"Beyond Generative AI: JEPA Chases Human-Style Conceptual Understanding","Bukan Sekadar Generative AI, JEPA Coba Kejar Pemahaman Konsep ala Manusia","kenapa-ai-masih-gagal-paham-fisika-dasar-mengenal-arsitektur-jepa","AI usually only guesses pixels, while JEPA tries to learn from abstract concepts in latent space. That approach turns out to reach as far as reading brain waves.","AI biasanya cuma jago nebak piksel, tapi JEPA coba belajar lewat konsep abstrak di ruang laten. Ternyata, cara ini bisa sampai dipakai buat baca gelombang otak.","2026-06-03T09:34:14.840Z",[100,102,147,103],"machine-learning","\u002Fog\u002Fkenapa-ai-masih-gagal-paham-fisika-dasar-mengenal-arsitektur-jepa.png",{"id":87,"title":35,"title_id":36,"slug":37},[151],{"authors_id":152},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":14,"title":154,"title_id":155,"slug":156,"excerpt":157,"excerpt_id":158,"has_translation":12,"published_at":159,"reading_time":98,"hidden_from_listing":15,"tags":160,"thumbnail_image":21,"og_image":162,"series":163,"authors":164},"Inside JEPA's Kitchen: The Secrets of EMA, VICReg, and Compute Efficiency","Di Balik Dapur JEPA: Rahasia EMA, VICReg, dan Efisiensi Komputasi","di-balik-dapur-jepa-rahasia-ema-vicreg-dan-efisiensi-komputasi","Bored of AI that only guesses words or memorizes pixels? Meet JEPA, an architecture that teaches machines common sense very cheaply. Here is a look at its technical insides.","Bosen sama AI yang cuma jago nebak kata atau ngafalin pixel? Kenalan sama JEPA, arsitektur revolusioner yang ngajarin mesin punya \"akal sehat\" secara super efisien. Yuk intip jeroan teknisnya!","2026-06-07T16:14:13.723Z",[100,161],"artificial-intelligent","\u002Fog\u002Fdi-balik-dapur-jepa-rahasia-ema-vicreg-dan-efisiensi-komputasi.png",{"id":87,"title":35,"title_id":36,"slug":37},[165],{"authors_id":166},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":168,"title":169,"title_id":170,"slug":171,"excerpt":172,"excerpt_id":173,"has_translation":12,"published_at":174,"reading_time":87,"hidden_from_listing":15,"tags":175,"thumbnail_image":21,"og_image":180,"series":181,"authors":182},125,"MCP, Rebuilt: Good News If You're Tired of Writing Connectors","MCP dibongkar ulang: kabar baik buat kamu yang capek nulis konektor","mcp-dibongkar-ulang-kabar-baik-buat-kamu","You have a few internal APIs and one AI agent, but the code has to be rewritten every time the model changes. MCP ends that, and this year the protocol was rebuilt so servers can run like ordinary web servers. Here's what changed. 🔧","Kamu punya beberapa API internal dan satu agent AI, tapi kodenya harus ditulis ulang tiap kali modelnya diganti. MCP datang buat mengakhiri itu, dan tahun ini protokolnya dibongkar ulang supaya server-nya bisa jalan seperti web server biasa. Yuk lihat apa yang berubah. 🔧","2026-10-16T05:00:00.000Z",[176,177,82,81,178,179],"mcp","model-context-protocol","backend","python","\u002Fog\u002Fmcp-dibongkar-ulang-kabar-baik-buat-kamu.png",{"id":87,"title":35,"title_id":36,"slug":37},[],{"id":184,"title":185,"title_id":186,"slug":187,"excerpt":188,"excerpt_id":189,"has_translation":12,"published_at":190,"reading_time":79,"hidden_from_listing":15,"tags":191,"thumbnail_image":21,"og_image":194,"series":195,"authors":196},126,"AI Agent Memory: From Vector Store to Temporal Graph","Memori AI Agent: dari Vector Store ke Temporal Graph","memori-ai-agent-vector-store-ke-temporal-graph","Two contradictory facts make an agent answer nonsense. Here is how a vector store and a temporal knowledge graph differ at keeping long-term state.","Dua fakta yang saling bertentangan bikin agent menjawab ngawur. Ini bedanya vector store dan temporal knowledge graph dalam menjaga state jangka panjang.","2026-10-18T13:13:29.458Z",[192,81,178,193],"ai","system-design","\u002Fog\u002Fmemori-ai-agent-vector-store-ke-temporal-graph.png",{"id":87,"title":35,"title_id":36,"slug":37},[197],{"authors_id":198},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":200,"title":201,"title_id":202,"slug":203,"excerpt":204,"excerpt_id":205,"has_translation":12,"published_at":206,"reading_time":207,"hidden_from_listing":15,"tags":208,"thumbnail_image":21,"og_image":215,"series":216,"authors":217},127,"Why RAG That Leans on Cosine Similarity Alone Keeps Missing","Kenapa RAG yang cuma andalkan cosine similarity sering nyasar","kenapa-rag-cuma-cosine-similarity-sering-nyasar","RAG that leans only on embeddings returns chunks that look close in meaning, not the ones you need. Here is why, and the two layers that patch it.","RAG yang cuma andalkan embedding balikin chunk yang mirip maknanya, bukan yang kamu butuh. Ini penyebabnya, dan dua lapis penambalnya.","2026-10-19T02:04:06.718Z",6,[209,210,211,212,213,214],"rag","vector-search","bm25","hybrid-search","retrieval","embedding","\u002Fog\u002Fkenapa-rag-cuma-cosine-similarity-sering-nyasar.png",{"id":87,"title":35,"title_id":36,"slug":37},[218],{"authors_id":219},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":221,"title":222,"title_id":223,"slug":224,"excerpt":225,"excerpt_id":226,"has_translation":12,"published_at":227,"reading_time":207,"hidden_from_listing":15,"tags":228,"thumbnail_image":21,"og_image":234,"series":235,"authors":236},128,"Linear or Graph: How to Run Many Agents Without an Eternal Loop","Linear atau graf: cara mengatur banyak agent tanpa loop abadi","multi-agen-linear-atau-grafik","One agent with twenty tools gets messy fast. Once several agents revise each other, you need a graph that allows cycles and a number that makes it stop.","Satu agent dengan dua puluh tool gampang kacau. Begitu beberapa agent saling revisi, kamu butuh graf yang mendukung cycle dan angka yang membuatnya berhenti.","2026-10-21T16:05:00.000Z",[229,230,231,232,233,81],"multi-agent","langgraph","autogen","orkestrasi","checkpoint","\u002Fog\u002Fmulti-agen-linear-atau-grafik.png",{"id":87,"title":35,"title_id":36,"slug":37},[237],{"authors_id":238},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":240,"title":241,"title_id":242,"slug":243,"excerpt":244,"excerpt_id":245,"has_translation":12,"published_at":246,"reading_time":14,"hidden_from_listing":15,"tags":247,"thumbnail_image":21,"og_image":253,"series":254,"authors":255},129,"So Your RAG Stops Hitting 504: Split the Query Path From Ingestion","Biar RAG nggak kena 504: pisahin jalur query dan ingestion","rag-pisah-jalur-query-dan-ingestion","One RAG service that both parses uploads and answers queries will fall over the moment someone uploads a 200-page PDF. The cure is two paths and one queue.","Satu service RAG yang merangkap parsing dan menjawab kueri bakal tumbang pas ada yang upload PDF 200 halaman. Obatnya dua jalur dan satu queue.","2026-10-22T01:30:00.000Z",[209,248,249,250,251,252],"cloud-run","cloud-tasks","gcp","serverless","arsitektur","\u002Fog\u002Frag-pisah-jalur-query-dan-ingestion.png",{"id":87,"title":35,"title_id":36,"slug":37},[256],{"authors_id":257},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":259,"title":260,"title_id":21,"slug":261,"excerpt":262,"excerpt_id":21,"has_translation":15,"published_at":263,"reading_time":79,"hidden_from_listing":15,"tags":264,"thumbnail_image":21,"og_image":267,"series":268,"authors":269},134,"Your Stream Dies at 125 Seconds. No Timeout in Your Code Says 125.","your-stream-dies-at-125-seconds","Streaming does not remove the wait, it moves it into a connection that now outlives the request. Four layers sit between the model and the browser, their idle timeouts do not agree, and the smallest one ends your stream.","2026-11-04T01:00:00.000Z",[265,266,82,84],"streaming","sse","\u002Fog\u002Fyour-stream-dies-at-125-seconds.png",{"id":87,"title":35,"title_id":36,"slug":37},[270],{"authors_id":271},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":74,"title":75,"title_id":21,"slug":76,"excerpt":77,"excerpt_id":21,"has_translation":15,"published_at":78,"reading_time":79,"hidden_from_listing":15,"tags":80,"thumbnail_image":21,"og_image":85,"series":273,"authors":274},{"id":87,"title":35,"title_id":36,"slug":37},[275],{"authors_id":276},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},[],{"id":24,"title":25,"title_id":26,"slug":27,"parent_series":21,"children_series":279},[280,281,283],{"id":87,"title":35,"title_id":36,"slug":37,"cover_image":21},{"id":282,"title":39,"title_id":40,"slug":41,"cover_image":21},12,{"id":284,"title":43,"title_id":44,"slug":45,"cover_image":21},15,[286],{"authors_id":287},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},"en",[288,290],"id","\nThe feature works. A forty-turn session answers correctly, the logs look normal, and the invoice is five times what the token counter suggested. Caching is enabled. The bill says otherwise.\n\nWhat separates those two facts is one line of documentation that the pricing page compresses away: a cache hit needs the same prefix, byte for byte, up to the point where you marked it. When part of that prefix changes, you do not fall back to the uncached price. You pay the write price, which is higher than the normal input price, and you pay it again on the next turn.\n\n## What a hit actually requires\n\nThree conditions, all of them cheap to check and easy to violate by accident.\n\nThe first is length. A prefix has to reach a minimum before it is cacheable at all: 512 tokens on the newer Claude models, and 1,024 tokens for GPT-5.6 and later, with the number varying by model and request settings for older ones ([Anthropic](https:\u002F\u002Fdocs.claude.com\u002Fen\u002Fdocs\u002Fbuild-with-claude\u002Fprompt-caching), [OpenAI](https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fprompt-caching)). Below that, the question never comes up.\n\nThe second is identity. The cache is a hash of the prefix, so a single changed byte produces a different hash and a different entry. This is the condition that breaks in production, because the things people add to prompts for observability are exactly the things that change: a request id, a timestamp, the user's name, a shuffled list of tools, a retrieval result inserted above the conversation.\n\nThe third is where you put the marker. Anthropic's documentation describes the cache as a hierarchy, `tools` then `system` then `messages`, and states the consequence plainly: changes at each level invalidate that level and every level after it. Put a timestamp in the tools array and you have invalidated the system prompt and the entire message history with it. The cache does not keep the parts that did not change. It keeps the longest prefix that still matches.\n\n## One line in the wrong place\n\nThe usual version of this bug looks harmless in review:\n\n```json\n{ \"tools\": [...], \"system\": \"You are a support agent. Session started 2026-11-03T09:41:22Z.\" }\n```\n\nThe date is there for debugging, and it is genuinely useful. It is also, from the cache's point of view, a declaration that this prompt is new every time the clock ticks. Every turn rewrites the full prefix at the write premium and reads nothing back.\n\n## The arithmetic, with the assumptions written down\n\nTake a loop with a 6,000-token static block (system prompt plus tool definitions), 1,000 tokens appended per turn (a tool result and the model's reply), 300 output tokens per turn, and forty turns. Price it at $2 per million input tokens, $10 per million output tokens, a cache write at 1.25x the input rate, and a cache read at 0.1x. Those multipliers are in the same range for both providers above; the absolute rates are not, and they move.\n\n| Scenario | Input tokens sent | Cost |\n|---|---|---|\n| No caching | 1,020,000 | $2.16 |\n| Stable prefix, cache hits | about 40,000 written, the rest read | $0.43 |\n| Prefix changes every turn | 1,020,000, all written | $2.67 |\n\nThe third row is the one worth staring at. A prefix that never hits is 24% more expensive than turning caching off, and 6 times more expensive than getting it right, for the same conversation and the same answers.\n\nThat is not a rounding error, it is a sign error. The feature was enabled to save money and it added a surcharge to every request, because the write price sits above the ordinary input price. OpenAI charges 1.25x for the default 30-minute lifetime and 2.5x for a 12-hour minimum. Anthropic's table for a current Sonnet lists $2.00 per million input against $2.50 for a five-minute cache write and $4.00 for a one-hour write. Reads, in both cases, come back at a fraction of the input rate.\n\n## The curve is quadratic, and that is the real budget item\n\nThe per-turn cost is not the interesting number. The accumulated cost is, because the prompt grows: turn 40 in this model sends 45,000 input tokens to say one thing, against 6,000 on turn 1.\n\n| Turns | No caching | Stable prefix |\n|---|---|---|\n| 10 | $0.24 | $0.09 |\n| 20 | $0.68 | $0.18 |\n| 40 | $2.16 | $0.43 |\n| 80 | $7.52 | $1.16 |\n\nEach doubling of the loop length multiplies the no-caching bill by about 2.8, then 3.2, then 3.5, heading toward 4 as the transcript term dominates. Caching does not change that shape. It divides the constant, roughly by five here, and leaves the exponent alone.\n\nThe practical consequence is that trimming the system prompt is the wrong lever once the loop is long. Over those forty turns the static block was 24% of the input tokens and the growing transcript was 76%. Making the system prompt 20% shorter saves about a twentieth of the bill; ending the loop four turns earlier saves considerably more.\n\n## The expiry nobody plans for\n\nCaches expire. The default lifetime is five minutes on Anthropic and thirty minutes on OpenAI, with longer options that cost more to write. A reader who leaves a conversation open, reads the answer, thinks for ten minutes, and types a follow-up comes back to a cold cache. That turn rewrites the entire prefix at 1.25x, and the discount the pricing page advertised is nowhere in the invoice.\n\nThis is worth saying out loud because it is a product decision disguised as an infrastructure detail. If your interface invites people to pause, the pause is a cost.\n\n## What caching does not buy\n\nOpenAI's documentation notes that cached input tokens still count toward tokens-per-minute limits. The win is money, not headroom. If the loop is hitting a rate limit, a better cache will not move it, and a shorter transcript will.\n\n## Measure your own loop\n\nThe table above is arithmetic on stated assumptions, not a quote. Change the numbers to yours and run it.\n\n```python\n# loop_cost.py — edit the six numbers, then: python3 loop_cost.py\nS, A, OUT, N = 6000, 1000, 300, 40      # static prefix, tokens per turn, output per turn, turns\nIN_RATE, OUT_RATE = 2.0 \u002F 1e6, 10.0 \u002F 1e6\nWRITE, READ = 1.25, 0.10\n\ndef no_cache():\n    tokens = sum(S + A * (k - 1) for k in range(1, N + 1))\n    return tokens * IN_RATE + N * OUT * OUT_RATE\n\ndef hits():\n    cost = S * IN_RATE * WRITE                      # first turn writes the prefix\n    for k in range(2, N + 1):\n        cost += (S + A * (k - 2)) * IN_RATE * READ  # read what was written last turn\n        cost += A * IN_RATE * WRITE                 # write the new segment\n    return cost + N * OUT * OUT_RATE\n\ndef never_hits():\n    tokens = sum(S + A * (k - 1) for k in range(1, N + 1))\n    return tokens * IN_RATE * WRITE + N * OUT * OUT_RATE  # writes only, no reads\n\nprint(f\"no caching     ${no_cache():.2f}\")\nprint(f\"cache hits     ${hits():.2f}\")\nprint(f\"never hits     ${never_hits():.2f}\")\n```\n\nThen replace the model with your actual numbers from the provider's response: the cached token count and the cache write count are both reported per request, which means the fraction of input tokens served from cache is a measurement, not an inference.\n\n## What this does not cover\n\nThe multipliers come from two providers' documentation in November 2026 and they differ by model, so treat the rates here as assumptions to replace rather than prices to quote. Output-side costs, reasoning tokens, batch discounts, and the interaction between caching and streaming are all outside this model.\n\nOrder the prompt by how often each part changes, put the marker at the end of the stable part, and measure the cache hit rate before measuring anything else. If the hit rate is near zero and the prompt looks static, the timestamp is almost always in the tools array or the first line of the system prompt, put there for a good reason.\n","- A cache hit requires an identical prefix up to your breakpoint, and a change at any level invalidates that level and everything after it.\n- A prefix that changes every turn is worse than no caching: you pay the cache write price, which is above the normal input price, and never collect the read discount.\n- On the same forty-turn loop, a stable prefix cost $0.43 where no caching cost $2.16 and a prefix that changed every turn cost $2.67.\n- The transcript, not the system prompt, is where the money goes: it was 76% of the input tokens over forty turns.\n- Total input cost grows roughly with the square of the turn count, so doubling the loop length multiplies the bill by about three to four.","\u003Cp>The feature works. A forty-turn session answers correctly, the logs look normal, and the invoice is five times what the token counter suggested. Caching is enabled. The bill says otherwise.\u003C\u002Fp>\n\u003Cp>What separates those two facts is one line of documentation that the pricing page compresses away: a cache hit needs the same prefix, byte for byte, up to the point where you marked it. When part of that prefix changes, you do not fall back to the uncached price. You pay the write price, which is higher than the normal input price, and you pay it again on the next turn.\u003C\u002Fp>\n\u003Ch2 id=\"what-a-hit-actually-requires\">What a hit actually requires\u003C\u002Fh2>\n\u003Cp>Three conditions, all of them cheap to check and easy to violate by accident.\u003C\u002Fp>\n\u003Cp>The first is length. A prefix has to reach a minimum before it is cacheable at all: 512 tokens on the newer Claude models, and 1,024 tokens for GPT-5.6 and later, with the number varying by model and request settings for older ones (\u003Ca href=\"https:\u002F\u002Fdocs.claude.com\u002Fen\u002Fdocs\u002Fbuild-with-claude\u002Fprompt-caching\">Anthropic\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fprompt-caching\">OpenAI\u003C\u002Fa>). Below that, the question never comes up.\u003C\u002Fp>\n\u003Cp>The second is identity. The cache is a hash of the prefix, so a single changed byte produces a different hash and a different entry. This is the condition that breaks in production, because the things people add to prompts for observability are exactly the things that change: a request id, a timestamp, the user's name, a shuffled list of tools, a retrieval result inserted above the conversation.\u003C\u002Fp>\n\u003Cp>The third is where you put the marker. Anthropic's documentation describes the cache as a hierarchy, \u003Ccode>tools\u003C\u002Fcode> then \u003Ccode>system\u003C\u002Fcode> then \u003Ccode>messages\u003C\u002Fcode>, and states the consequence plainly: changes at each level invalidate that level and every level after it. Put a timestamp in the tools array and you have invalidated the system prompt and the entire message history with it. The cache does not keep the parts that did not change. It keeps the longest prefix that still matches.\u003C\u002Fp>\n\u003Ch2 id=\"one-line-in-the-wrong-place\">One line in the wrong place\u003C\u002Fh2>\n\u003Cp>The usual version of this bug looks harmless in review:\u003C\u002Fp>\n\n\u003Cdiv class=\"code-block-wrapper my-6 not-prose rounded-xl border-2 border-black dark:border-gray-600 shadow-neo overflow-hidden bg-[#24292e] text-[#e1e4e8] transition-all\">\n  \u003Cdiv class=\"code-block-header flex items-center justify-between px-4 py-2.5 bg-[#1f2428] border-b-2 border-black dark:border-gray-600 font-mono text-xs font-bold text-gray-300 select-none\">\n    \u003Cdiv class=\"flex items-center gap-2\">\n      \u003Cspan class=\"w-3 h-3 rounded-full bg-[#ff5f56] border border-black\u002F40 inline-block\">\u003C\u002Fspan>\n      \u003Cspan class=\"w-3 h-3 rounded-full bg-[#ffbd2e] border border-black\u002F40 inline-block\">\u003C\u002Fspan>\n      \u003Cspan class=\"w-3 h-3 rounded-full bg-[#27c93f] border border-black\u002F40 inline-block\">\u003C\u002Fspan>\n      \u003Cspan class=\"ml-2 font-mono font-bold text-xs uppercase tracking-wider text-gray-300\">JSON\u003C\u002Fspan>\n    \u003C\u002Fdiv>\n    \u003Cbutton type=\"button\" class=\"copy-code-btn flex items-center gap-1.5 px-3 py-1 text-xs font-bold bg-[#2f363d] text-gray-200 border-2 border-black dark:border-gray-500 rounded-lg shadow-neo-sm hover:translate-x-0.5 hover:translate-y-0.5 hover:shadow-none hover:bg-gray-700 transition-all cursor-pointer\" data-code=\"%7B%20%22tools%22%3A%20%5B...%5D%2C%20%22system%22%3A%20%22You%20are%20a%20support%20agent.%20Session%20started%202026-11-03T09%3A41%3A22Z.%22%20%7D\" title=\"Salin kode\">\n      \u003Csvg class=\"w-3.5 h-3.5 copy-icon-svg\" fill=\"none\" stroke=\"currentColor\" viewBox=\"0 0 24 24\">\n        \u003Cpath stroke-linecap=\"round\" stroke-linejoin=\"round\" stroke-width=\"2\" d=\"M8 16H6a2 2 0 01-2-2V6a2 2 0 012-2h8a2 2 0 012 2v2m-6 12h8a2 2 0 002-2v-8a2 2 0 00-2-2h-8a2 2 0 00-2 2v8a2 2 0 002 2z\">\u003C\u002Fpath>\n      \u003C\u002Fsvg>\n      \u003Cspan class=\"copy-text\">Copy\u003C\u002Fspan>\n    \u003C\u002Fbutton>\n  \u003C\u002Fdiv>\n  \u003Cdiv class=\"code-block-content p-4 overflow-x-auto text-sm font-mono leading-relaxed bg-[#24292e]\">\n    \u003Cpre class=\"shiki github-dark\" style=\"background-color:#24292e;color:#e1e4e8\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#E1E4E8\">{ \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">\"tools\"\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">: [\u003C\u002Fspan>\u003Cspan style=\"color:#FDAEB7;font-style:italic\">...\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">], \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">\"system\"\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">: \u003C\u002Fspan>\u003Cspan style=\"color:#9ECBFF\">\"You are a support agent. Session started 2026-11-03T09:41:22Z.\"\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> }\u003C\u002Fspan>\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n  \u003C\u002Fdiv>\n\u003C\u002Fdiv>\u003Cp>The date is there for debugging, and it is genuinely useful. It is also, from the cache's point of view, a declaration that this prompt is new every time the clock ticks. Every turn rewrites the full prefix at the write premium and reads nothing back.\u003C\u002Fp>\n\u003Ch2 id=\"the-arithmetic-with-the-assumptions-written-down\">The arithmetic, with the assumptions written down\u003C\u002Fh2>\n\u003Cp>Take a loop with a 6,000-token static block (system prompt plus tool definitions), 1,000 tokens appended per turn (a tool result and the model's reply), 300 output tokens per turn, and forty turns. Price it at $2 per million input tokens, $10 per million output tokens, a cache write at 1.25x the input rate, and a cache read at 0.1x. Those multipliers are in the same range for both providers above; the absolute rates are not, and they move.\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Scenario\u003C\u002Fth>\n\u003Cth>Input tokens sent\u003C\u002Fth>\n\u003Cth>Cost\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>No caching\u003C\u002Ftd>\n\u003Ctd>1,020,000\u003C\u002Ftd>\n\u003Ctd>$2.16\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Stable prefix, cache hits\u003C\u002Ftd>\n\u003Ctd>about 40,000 written, the rest read\u003C\u002Ftd>\n\u003Ctd>$0.43\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Prefix changes every turn\u003C\u002Ftd>\n\u003Ctd>1,020,000, all written\u003C\u002Ftd>\n\u003Ctd>$2.67\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Cp>The third row is the one worth staring at. A prefix that never hits is 24% more expensive than turning caching off, and 6 times more expensive than getting it right, for the same conversation and the same answers.\u003C\u002Fp>\n\u003Cp>That is not a rounding error, it is a sign error. The feature was enabled to save money and it added a surcharge to every request, because the write price sits above the ordinary input price. OpenAI charges 1.25x for the default 30-minute lifetime and 2.5x for a 12-hour minimum. Anthropic's table for a current Sonnet lists $2.00 per million input against $2.50 for a five-minute cache write and $4.00 for a one-hour write. Reads, in both cases, come back at a fraction of the input rate.\u003C\u002Fp>\n\u003Ch2 id=\"the-curve-is-quadratic-and-that-is-the-real-budget-item\">The curve is quadratic, and that is the real budget item\u003C\u002Fh2>\n\u003Cp>The per-turn cost is not the interesting number. The accumulated cost is, because the prompt grows: turn 40 in this model sends 45,000 input tokens to say one thing, against 6,000 on turn 1.\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Turns\u003C\u002Fth>\n\u003Cth>No caching\u003C\u002Fth>\n\u003Cth>Stable prefix\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>10\u003C\u002Ftd>\n\u003Ctd>$0.24\u003C\u002Ftd>\n\u003Ctd>$0.09\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>20\u003C\u002Ftd>\n\u003Ctd>$0.68\u003C\u002Ftd>\n\u003Ctd>$0.18\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>40\u003C\u002Ftd>\n\u003Ctd>$2.16\u003C\u002Ftd>\n\u003Ctd>$0.43\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>80\u003C\u002Ftd>\n\u003Ctd>$7.52\u003C\u002Ftd>\n\u003Ctd>$1.16\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Cp>Each doubling of the loop length multiplies the no-caching bill by about 2.8, then 3.2, then 3.5, heading toward 4 as the transcript term dominates. Caching does not change that shape. It divides the constant, roughly by five here, and leaves the exponent alone.\u003C\u002Fp>\n\u003Cp>The practical consequence is that trimming the system prompt is the wrong lever once the loop is long. Over those forty turns the static block was 24% of the input tokens and the growing transcript was 76%. Making the system prompt 20% shorter saves about a twentieth of the bill; ending the loop four turns earlier saves considerably more.\u003C\u002Fp>\n\u003Ch2 id=\"the-expiry-nobody-plans-for\">The expiry nobody plans for\u003C\u002Fh2>\n\u003Cp>Caches expire. The default lifetime is five minutes on Anthropic and thirty minutes on OpenAI, with longer options that cost more to write. A reader who leaves a conversation open, reads the answer, thinks for ten minutes, and types a follow-up comes back to a cold cache. That turn rewrites the entire prefix at 1.25x, and the discount the pricing page advertised is nowhere in the invoice.\u003C\u002Fp>\n\u003Cp>This is worth saying out loud because it is a product decision disguised as an infrastructure detail. If your interface invites people to pause, the pause is a cost.\u003C\u002Fp>\n\u003Ch2 id=\"what-caching-does-not-buy\">What caching does not buy\u003C\u002Fh2>\n\u003Cp>OpenAI's documentation notes that cached input tokens still count toward tokens-per-minute limits. The win is money, not headroom. If the loop is hitting a rate limit, a better cache will not move it, and a shorter transcript will.\u003C\u002Fp>\n\u003Ch2 id=\"measure-your-own-loop\">Measure your own loop\u003C\u002Fh2>\n\u003Cp>The table above is arithmetic on stated assumptions, not a quote. Change the numbers to yours and run it.\u003C\u002Fp>\n\n\u003Cdiv class=\"code-block-wrapper my-6 not-prose rounded-xl border-2 border-black dark:border-gray-600 shadow-neo overflow-hidden bg-[#24292e] text-[#e1e4e8] transition-all\">\n  \u003Cdiv class=\"code-block-header flex items-center justify-between px-4 py-2.5 bg-[#1f2428] border-b-2 border-black dark:border-gray-600 font-mono text-xs font-bold text-gray-300 select-none\">\n    \u003Cdiv class=\"flex items-center gap-2\">\n      \u003Cspan class=\"w-3 h-3 rounded-full bg-[#ff5f56] border border-black\u002F40 inline-block\">\u003C\u002Fspan>\n      \u003Cspan class=\"w-3 h-3 rounded-full bg-[#ffbd2e] border border-black\u002F40 inline-block\">\u003C\u002Fspan>\n      \u003Cspan class=\"w-3 h-3 rounded-full bg-[#27c93f] border border-black\u002F40 inline-block\">\u003C\u002Fspan>\n      \u003Cspan class=\"ml-2 font-mono font-bold text-xs uppercase tracking-wider text-gray-300\">PYTHON\u003C\u002Fspan>\n    \u003C\u002Fdiv>\n    \u003Cbutton type=\"button\" class=\"copy-code-btn flex items-center gap-1.5 px-3 py-1 text-xs font-bold bg-[#2f363d] text-gray-200 border-2 border-black dark:border-gray-500 rounded-lg shadow-neo-sm hover:translate-x-0.5 hover:translate-y-0.5 hover:shadow-none hover:bg-gray-700 transition-all cursor-pointer\" data-code=\"%23%20loop_cost.py%20%E2%80%94%20edit%20the%20six%20numbers%2C%20then%3A%20python3%20loop_cost.py%0AS%2C%20A%2C%20OUT%2C%20N%20%3D%206000%2C%201000%2C%20300%2C%2040%20%20%20%20%20%20%23%20static%20prefix%2C%20tokens%20per%20turn%2C%20output%20per%20turn%2C%20turns%0AIN_RATE%2C%20OUT_RATE%20%3D%202.0%20%2F%201e6%2C%2010.0%20%2F%201e6%0AWRITE%2C%20READ%20%3D%201.25%2C%200.10%0A%0Adef%20no_cache()%3A%0A%20%20%20%20tokens%20%3D%20sum(S%20%2B%20A%20*%20(k%20-%201)%20for%20k%20in%20range(1%2C%20N%20%2B%201))%0A%20%20%20%20return%20tokens%20*%20IN_RATE%20%2B%20N%20*%20OUT%20*%20OUT_RATE%0A%0Adef%20hits()%3A%0A%20%20%20%20cost%20%3D%20S%20*%20IN_RATE%20*%20WRITE%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%23%20first%20turn%20writes%20the%20prefix%0A%20%20%20%20for%20k%20in%20range(2%2C%20N%20%2B%201)%3A%0A%20%20%20%20%20%20%20%20cost%20%2B%3D%20(S%20%2B%20A%20*%20(k%20-%202))%20*%20IN_RATE%20*%20READ%20%20%23%20read%20what%20was%20written%20last%20turn%0A%20%20%20%20%20%20%20%20cost%20%2B%3D%20A%20*%20IN_RATE%20*%20WRITE%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%23%20write%20the%20new%20segment%0A%20%20%20%20return%20cost%20%2B%20N%20*%20OUT%20*%20OUT_RATE%0A%0Adef%20never_hits()%3A%0A%20%20%20%20tokens%20%3D%20sum(S%20%2B%20A%20*%20(k%20-%201)%20for%20k%20in%20range(1%2C%20N%20%2B%201))%0A%20%20%20%20return%20tokens%20*%20IN_RATE%20*%20WRITE%20%2B%20N%20*%20OUT%20*%20OUT_RATE%20%20%23%20writes%20only%2C%20no%20reads%0A%0Aprint(f%22no%20caching%20%20%20%20%20%24%7Bno_cache()%3A.2f%7D%22)%0Aprint(f%22cache%20hits%20%20%20%20%20%24%7Bhits()%3A.2f%7D%22)%0Aprint(f%22never%20hits%20%20%20%20%20%24%7Bnever_hits()%3A.2f%7D%22)\" title=\"Salin kode\">\n      \u003Csvg class=\"w-3.5 h-3.5 copy-icon-svg\" fill=\"none\" stroke=\"currentColor\" viewBox=\"0 0 24 24\">\n        \u003Cpath stroke-linecap=\"round\" stroke-linejoin=\"round\" stroke-width=\"2\" d=\"M8 16H6a2 2 0 01-2-2V6a2 2 0 012-2h8a2 2 0 012 2v2m-6 12h8a2 2 0 002-2v-8a2 2 0 00-2-2h-8a2 2 0 00-2 2v8a2 2 0 002 2z\">\u003C\u002Fpath>\n      \u003C\u002Fsvg>\n      \u003Cspan class=\"copy-text\">Copy\u003C\u002Fspan>\n    \u003C\u002Fbutton>\n  \u003C\u002Fdiv>\n  \u003Cdiv class=\"code-block-content p-4 overflow-x-auto text-sm font-mono leading-relaxed bg-[#24292e]\">\n    \u003Cpre class=\"shiki github-dark\" style=\"background-color:#24292e;color:#e1e4e8\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#6A737D\"># loop_cost.py — edit the six numbers, then: python3 loop_cost.py\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E1E4E8\">S, A, \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">OUT\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, N \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">=\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 6000\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">1000\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">300\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">40\u003C\u002Fspan>\u003Cspan style=\"color:#6A737D\">      # static prefix, tokens per turn, output per turn, turns\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#79B8FF\">IN_RATE\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">OUT_RATE\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> =\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 2.0\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> \u002F\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 1e6\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">10.0\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> \u002F\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 1e6\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#79B8FF\">WRITE\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">READ\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> =\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 1.25\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, \u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">0.10\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#F97583\">def\u003C\u002Fspan>\u003Cspan style=\"color:#B392F0\"> no_cache\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">():\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E1E4E8\">    tokens \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">=\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> sum\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">(S \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">+\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> A \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> (k \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">-\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 1\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">) \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">for\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> k \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">in\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> range\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">(\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">1\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, N \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">+\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 1\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">))\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#F97583\">    return\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> tokens \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> IN_RATE\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> N \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> OUT\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> *\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> OUT_RATE\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#F97583\">def\u003C\u002Fspan>\u003Cspan style=\"color:#B392F0\"> hits\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">():\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E1E4E8\">    cost \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">=\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> S \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> IN_RATE\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> *\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> WRITE\u003C\u002Fspan>\u003Cspan style=\"color:#6A737D\">                      # first turn writes the prefix\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#F97583\">    for\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> k \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">in\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> range\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">(\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">2\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, N \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">+\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 1\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">):\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E1E4E8\">        cost \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">+=\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> (S \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">+\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> A \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> (k \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">-\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 2\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">)) \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> IN_RATE\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> *\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> READ\u003C\u002Fspan>\u003Cspan style=\"color:#6A737D\">  # read what was written last turn\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E1E4E8\">        cost \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">+=\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> A \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> IN_RATE\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> *\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> WRITE\u003C\u002Fspan>\u003Cspan style=\"color:#6A737D\">                 # write the new segment\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#F97583\">    return\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> cost \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">+\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> N \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> OUT\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> *\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> OUT_RATE\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#F97583\">def\u003C\u002Fspan>\u003Cspan style=\"color:#B392F0\"> never_hits\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">():\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E1E4E8\">    tokens \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">=\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> sum\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">(S \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">+\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> A \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> (k \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">-\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 1\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">) \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">for\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> k \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">in\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> range\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">(\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">1\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">, N \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">+\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> 1\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">))\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#F97583\">    return\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> tokens \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> IN_RATE\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> *\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> WRITE\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\"> N \u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">*\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> OUT\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\"> *\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\"> OUT_RATE\u003C\u002Fspan>\u003Cspan style=\"color:#6A737D\">  # writes only, no reads\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#79B8FF\">print\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">(\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">f\u003C\u002Fspan>\u003Cspan style=\"color:#9ECBFF\">\"no caching     $\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">{\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">no_cache()\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">:.2f\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">}\u003C\u002Fspan>\u003Cspan style=\"color:#9ECBFF\">\"\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">)\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#79B8FF\">print\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">(\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">f\u003C\u002Fspan>\u003Cspan style=\"color:#9ECBFF\">\"cache hits     $\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">{\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">hits()\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">:.2f\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">}\u003C\u002Fspan>\u003Cspan style=\"color:#9ECBFF\">\"\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">)\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#79B8FF\">print\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">(\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">f\u003C\u002Fspan>\u003Cspan style=\"color:#9ECBFF\">\"never hits     $\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">{\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">never_hits()\u003C\u002Fspan>\u003Cspan style=\"color:#F97583\">:.2f\u003C\u002Fspan>\u003Cspan style=\"color:#79B8FF\">}\u003C\u002Fspan>\u003Cspan style=\"color:#9ECBFF\">\"\u003C\u002Fspan>\u003Cspan style=\"color:#E1E4E8\">)\u003C\u002Fspan>\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n  \u003C\u002Fdiv>\n\u003C\u002Fdiv>\u003Cp>Then replace the model with your actual numbers from the provider's response: the cached token count and the cache write count are both reported per request, which means the fraction of input tokens served from cache is a measurement, not an inference.\u003C\u002Fp>\n\u003Ch2 id=\"what-this-does-not-cover\">What this does not cover\u003C\u002Fh2>\n\u003Cp>The multipliers come from two providers' documentation in November 2026 and they differ by model, so treat the rates here as assumptions to replace rather than prices to quote. Output-side costs, reasoning tokens, batch discounts, and the interaction between caching and streaming are all outside this model.\u003C\u002Fp>\n\u003Cp>Order the prompt by how often each part changes, put the marker at the end of the stable part, and measure the cache hit rate before measuring anything else. If the hit rate is near zero and the prompt looks static, the timestamp is almost always in the tools array or the first line of the system prompt, put there for a good reason.\u003C\u002Fp>\n","\u003Cul>\n\u003Cli>A cache hit requires an identical prefix up to your breakpoint, and a change at any level invalidates that level and everything after it.\u003C\u002Fli>\n\u003Cli>A prefix that changes every turn is worse than no caching: you pay the cache write price, which is above the normal input price, and never collect the read discount.\u003C\u002Fli>\n\u003Cli>On the same forty-turn loop, a stable prefix cost $0.43 where no caching cost $2.16 and a prefix that changed every turn cost $2.67.\u003C\u002Fli>\n\u003Cli>The transcript, not the system prompt, is where the money goes: it was 76% of the input tokens over forty turns.\u003C\u002Fli>\n\u003Cli>Total input cost grows roughly with the square of the turn count, so doubling the loop length multiplies the bill by about three to four.\u003C\u002Fli>\n\u003C\u002Ful>\n",[296,301,306,325,340,355],{"id":74,"title":75,"title_id":21,"slug":76,"excerpt":77,"excerpt_id":21,"has_translation":15,"published_at":78,"reading_time":79,"hidden_from_listing":15,"tags":80,"thumbnail_image":21,"og_image":85,"series":297,"authors":298},{"id":87,"title":35,"title_id":36,"slug":37},[299],{"authors_id":300},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":259,"title":260,"title_id":21,"slug":261,"excerpt":262,"excerpt_id":21,"has_translation":15,"published_at":263,"reading_time":79,"hidden_from_listing":15,"tags":264,"thumbnail_image":21,"og_image":267,"series":302,"authors":303},{"id":87,"title":35,"title_id":36,"slug":37},[304],{"authors_id":305},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":307,"title":308,"title_id":309,"slug":310,"excerpt":311,"excerpt_id":312,"has_translation":12,"published_at":313,"reading_time":207,"hidden_from_listing":15,"tags":314,"thumbnail_image":21,"og_image":316,"series":317,"authors":322},133,"Python Strings: the Badge That Only Worked for One Name","String Python: Badge yang Cuma Jalan untuk Satu Nama","python-strings","The badge script printed the right name for the person who wrote it and the wrong name for everybody else. Slicing counts characters, and a name is words.","Skrip badge mencetak nama yang benar untuk orang yang menulisnya dan nama yang salah untuk semua orang lain. Slicing menghitung karakter, sedangkan nama itu kata.","2026-10-28T13:00:00.000Z",[179,315],"string","\u002Fog\u002Fpython-strings.png",{"id":318,"title":319,"title_id":320,"slug":321},30,"Collections and Strings","Koleksi dan String","python-collections",[323],{"authors_id":324},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":326,"title":327,"title_id":328,"slug":329,"excerpt":330,"excerpt_id":331,"has_translation":12,"published_at":332,"reading_time":207,"hidden_from_listing":15,"tags":333,"thumbnail_image":21,"og_image":335,"series":336,"authors":337},132,"Python Tuples: Why All Three Points Are the Same Point","Tuple Python: Kenapa Ketiga Titik Itu Titik yang Sama","python-tuples","Three points walk into a loop and come out identical. The list was never copied three times; the name was. A tuple makes that bug impossible.","Tiga titik masuk ke satu loop dan keluar identik. List-nya nggak pernah disalin tiga kali; yang disalin cuma namanya. Tuple membuat bug ini mustahil terjadi.","2026-10-28T12:00:00.000Z",[179,334],"tuple","\u002Fog\u002Fpython-tuples.png",{"id":318,"title":319,"title_id":320,"slug":321},[338],{"authors_id":339},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":341,"title":342,"title_id":343,"slug":344,"excerpt":345,"excerpt_id":346,"has_translation":12,"published_at":347,"reading_time":207,"hidden_from_listing":15,"tags":348,"thumbnail_image":21,"og_image":350,"series":351,"authors":352},131,"Python Sets: the Duplicate That Keeps Coming Back","Set Python: Duplikat yang Terus Balik Lagi","python-sets","A de-duplication loop that works on clean data leaves four tags where there should be two. The bug is case, and the fix is a `set`.","Loop de-duplikasi yang jalan mulus di data bersih ternyata menyisakan empat tag padahal harusnya dua. Biang keroknya huruf besar-kecil, dan obatnya `set`.","2026-10-28T11:00:00.000Z",[179,349],"set","\u002Fog\u002Fpython-sets.png",{"id":318,"title":319,"title_id":320,"slug":321},[353],{"authors_id":354},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},{"id":356,"title":357,"title_id":358,"slug":359,"excerpt":360,"excerpt_id":361,"has_translation":12,"published_at":362,"reading_time":207,"hidden_from_listing":15,"tags":363,"thumbnail_image":21,"og_image":365,"series":366,"authors":367},130,"Python Dictionaries: the Key That Isn't There","Dictionary Python: Key yang Nggak Ada","python-dictionary","A word counter dies the first time it meets a word twice. The fix is one line; what takes longer is knowing which of a dict's three lookups to reach for.","Program penghitung kata langsung mati begitu ketemu kata yang sama dua kali. Perbaikannya cuma satu baris; yang butuh waktu lebih lama adalah tahu mau pakai yang mana dari tiga cara lookup dict.","2026-10-28T10:00:00.000Z",[179,364],"dictionary","\u002Fog\u002Fpython-dictionary.png",{"id":318,"title":319,"title_id":320,"slug":321},[368],{"authors_id":369},{"id":91,"name":121,"slug":122,"role":123,"profile_picture":21},[371,374,377,380,383,386,389,392],{"text":372,"depth":108,"id":373},"What a hit actually requires","what-a-hit-actually-requires",{"text":375,"depth":108,"id":376},"One line in the wrong place","one-line-in-the-wrong-place",{"text":378,"depth":108,"id":379},"The arithmetic, with the assumptions written down","the-arithmetic-with-the-assumptions-written-down",{"text":381,"depth":108,"id":382},"The curve is quadratic, and that is the real budget item","the-curve-is-quadratic-and-that-is-the-real-budget-item",{"text":384,"depth":108,"id":385},"The expiry nobody plans for","the-expiry-nobody-plans-for",{"text":387,"depth":108,"id":388},"What caching does not buy","what-caching-does-not-buy",{"text":390,"depth":108,"id":391},"Measure your own loop","measure-your-own-loop",{"text":393,"depth":108,"id":394},"What this does not cover","what-this-does-not-cover",1793755531688]