Caching Context, Not Data

Here's a question nobody asked in 2024 that everyone's asking in 2026: "How much does it cost every time your AI thinks?"

Not "how much does the API cost per token." That's the wrong question. Like asking how much a gallon of gas costs when the real question is how far your car goes on a tank.

The right question is: how much of what your AI knows has to be rebuilt from scratch every single time a user asks it to do something?

If the answer is "most of it," your AI economics are broken. You're paying full price for the AI to re-learn its own identity on every call. That's like paying tuition every time you ask a college graduate a question. They already know this stuff. You already taught them. Why are you paying to teach them again?

---

The Tuition Problem

Let me put numbers on this so it hurts the right amount.

A well-configured workspace at DealiOS carries approximately 80,000 tokens of system context. That's the AI's identity, behavioral rules, feature capabilities, domain knowledge, workspace configuration, brand voice, pricing rules, and all the other stable context that makes the AI useful for this specific customer.

Eighty thousand tokens. That's who this AI is and what it knows about this customer.

Now imagine a 10-message conversation. Without caching, every single message resends the full 80,000-token system context plus the growing conversation history. By message 10, you've transmitted roughly 800,000 tokens. Eight hundred thousand. For a conversation that might have produced 5,000 tokens of actual new output.

That's a 160:1 ratio of context to output. You're paying to re-teach the AI its own name 10 times in a row.

The difference between caching and not caching isn't a performance optimization. It's the difference between a viable AI product and an API bill that makes your CFO start updating their resume.

With caching at the right boundary, messages 2 through 10 transmit the stable context at cached token rates, which major API providers (including Anthropic's Claude API) price at a significant discount compared to fresh tokens. ^[1] The exact multiplier depends on the provider and model tier, but at DealiOS's production scale, caching reduces per-conversation costs by roughly 5 to 8 times.

Multiply that across every workspace. Every user. Every conversation. Every day. The difference between caching and not caching isn't a performance optimization. It's the difference between a viable AI product and an API bill that makes your CFO start updating their resume.

---

The Volatility Gradient

So where do you draw the cache boundary? That decision sounds technical. It's actually philosophical.

The cache boundary is the line between what this AI IS and what this AI is DOING right now. Identity above the line, action below the line. Stable above, volatile below. Cached above, computed fresh below.

The companion paper, "Five Layers Deep," describes the full architecture. ^[2] Here's the summary, with the economic lens:

Layer	What It Contains	How Often It Changes	Cached?
1. Platform Expert	AI identity, personality, hard rules	Never (versioned with codebase)	Yes
2. Feature Module	Current capability set (chat vs. build vs. analyze)	Per feature switch	Yes
3. Industry Module	Domain knowledge (SaaS vs. services vs. cloud)	Per workspace config	Yes
4. Workspace Context	Company name, brand, pricing, tone, preferences	Per workspace config update	Yes (last cached layer)
--- CACHE BOUNDARY ---
5. Per-Call Context	Deal data, RAG results, calculator output, history	Every single call	No

Layers 1 through 4 form the AI's stable identity for this workspace. They change rarely. Layer 5 is the volatile action layer. It's different on every call because every call is a different task.

The boundary between 4 and 5 is where the money is. Everything above gets cached. Everything below gets recomputed. The placement isn't arbitrary. It follows a universal principle: the more stable a piece of context is, the higher it should sit and the more aggressively it should be cached.

This is the same principle that governs CPU cache hierarchies (L1/L2/L3), CDN architectures (edge vs. origin), and even how your brain works (long-term memory vs. working memory). The universal pattern: put the stable stuff closer to the processor and stop fetching it from the source every time you need it.

---

Three Independent Caching Strategies

At DealiOS, caching isn't one mechanism. It's three, running simultaneously, each targeting a different class of context.

Strategy 1: System Prompt Caching. Layers 1 through 4 are assembled into the system prompt with explicit cache control markers. The cacheUpTo position is dynamically computed based on the assembled layers. This tells the API provider: "everything up to this point is stable. Cache it. Don't charge me full price for it on the next call."

There's a technical constraint that matters here: Anthropic's API supports a limited number of cache control blocks per request. ^[1] This means you can't cache every layer independently. The architecture must be smart about where it places cache boundaries, consolidating stable layers into as few cacheable blocks as possible.

Strategy 2: Document Block Caching. Reference documents (company playbooks, brand guidelines, template libraries) are structured as synthetic conversation pairs. This exploits conversation-level caching mechanisms, so repeated references to the same documents don't incur full token costs on subsequent messages. The document content is stable across the conversation, so it benefits from the same caching economics as the system prompt.

Strategy 3: Tiered Token Budgets. Not every interaction needs the same context window. A quick chat message doesn't need 500,000 tokens of context. A full document build might. DealiOS runs three context tiers:

Mode	Total Budget	System Reserve	Output Reserve
Chat	~200K tokens	~80K	~16K
Build	~500K tokens	~80K	~16K
Analysis	~400K tokens	~80K	~16K

The system reserve (Layers 1-4) stays roughly constant. The difference is how much room Layer 5 gets for deal context, RAG results, and conversation history. Chat mode is lean. Build mode is expansive. Analysis mode is somewhere between.

This isn't about limiting the AI. It's about matching the economic cost to the task complexity. You don't fly first class to the corner store.

---

The Identity Question

Here's where it gets philosophical, and I think this is the part most AI teams skip entirely.

The cache boundary is an identity declaration. When you decide what goes above the boundary and what goes below, you're answering the question: "What is stable enough to be part of who this AI is?"

The cache boundary is an identity declaration. When you decide what goes above the boundary and what goes below, you're answering the question: "What is stable enough to be part of who this AI is?"

Most production AI systems don't think about this at all. They assemble one big string and send it. No layers. No boundaries. No distinction between identity and action. The AI doesn't know which parts of its context are permanent and which are temporary. It treats everything with equal weight, which means it treats nothing as sacred.

The result is the drift problem. By message 15, the AI has forgotten its own rules. Not because the rules aren't in the context window. They are. But they're competing with 14 messages of conversation, several thousand tokens of RAG results, calculator outputs, and user instructions. The rules drown in the noise.

Layered caching solves this structurally. The identity layers (1-4) are always at the top of the context. They're always the first thing the AI "sees." They're always at full fidelity, never truncated, never summarized, never competing with volatile content for attention.

The cache boundary is an attention boundary. And attention is the scarcest resource in any AI system.

---

The Compilation Problem Nobody Calls a Compilation Problem

What I've described is a compilation pipeline. The system takes source material (identity rules, feature modules, domain knowledge, workspace config, per-call context), assembles it into a structured prompt, optimizes it (cache placement, token budgets, document blocking), and outputs an artifact (the final API request) that the "processor" (the language model) executes.

That's a compiler. We just don't call it that because the industry decided that AI prompts are "creative" and compilers are "engineering" and never the two shall meet.

They've met. They should have met sooner.

The best software compilers in the world (LLVM, GCC, V8's JIT) all share a design principle: separate the front-end (language parsing) from the optimization passes (dead code elimination, register allocation) from the back-end (machine code generation). Each pass has clear inputs, clear outputs, and clear boundaries.

Production prompt systems need the same discipline. Layer assembly is the front-end. Cache optimization is the pass. API request construction is the back-end. Each stage is independently testable, independently optimizable, and independently debuggable.

Without this discipline, you're hand-writing assembly language. It works. It's unmaintainable. It doesn't scale. And when something goes wrong, you have no idea which layer broke because there are no layers.

---

What This Changes About the Business

The economics of caching aren't just about reducing API costs. They change what's possible.

When every AI call costs full price, you ration AI interactions. You make the AI generate one document at a time. You limit conversation length. You restrict how many times a user can iterate on a draft. The economics constrain the experience.

When cached calls cost a fraction of fresh calls, you can be generous. Long conversations. Multiple iterations. Cross-document generation. The AI can reference the proposal while writing the contract while checking the pricing, because the stable context (everything it knows about this workspace) is cached and only the volatile context (this specific deal) incurs fresh token costs.

The cache boundary is the line between "AI tool" and "AI platform." Tools ration. Platforms enable.

This is the difference between an AI that feels expensive and one that feels native. Users can tell. They don't know about token economics. They don't care about cache boundaries. But they can feel when a system is rationing their interactions versus when it's letting them work naturally.

The cache boundary is the line between "AI tool" and "AI platform." Tools ration. Platforms enable. The difference is architecture.

---

The Takeaway

The takeaway: The decision of what to cache is the decision of what constitutes your AI's stable self versus its volatile actions: get it wrong and you pay tuition on every API call while your AI forgets who it is between messages.

Caching isn't about speed. It's about identity. The decision of what to cache is the decision of what constitutes your AI's stable self versus its volatile actions. Get this wrong and you're paying tuition on every API call while your AI forgets who it is between messages.

Get it right and your AI wakes up every morning knowing who it is, what it knows, and who it's working for. The only question it needs answered is: what's the task?

That's the economics layer. Not a cost optimization bolted on after launch. A foundational architectural decision that determines whether production AI is financially viable at scale.

And nobody tells you about it until the API bill arrives.

---

References & Notes

1. Anthropic API caching and pricing: Anthropic's Claude API supports prompt caching with cached tokens priced at a reduced rate compared to fresh input tokens. Cache control blocks are supported with documented limits per request. Specific pricing and cache block limits vary by model tier and are subject to change. See: Anthropic API documentation at docs.anthropic.com; Anthropic pricing page.

2. "Five Layers Deep: Prompt Engineering as Systems Architecture." Architecture Series Library, DealiOS. Companion paper covering the full five-layer prompt architecture referenced throughout this paper.

3. CPU cache hierarchy analogy: The L1/L2/L3 cache hierarchy in modern processors follows the same principle described here. Frequently accessed data is stored in faster, smaller caches closer to the processor. See any computer architecture textbook, e.g., Patterson, D. & Hennessy, J., "Computer Organization and Design," Morgan Kaufmann.

4. Compiler architecture analogy: LLVM's multi-pass compilation pipeline (front-end IR generation, optimization passes, back-end code generation) is documented in Lattner, C. & Adve, V., "LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation," 2004. The analogy to prompt compilation is the author's.

5. The three-strategy caching architecture, tiered token budgets, and cache boundary computation described in this paper reflect DealiOS's production system design.

This paper is part of The Architecture Series by Ed, Founder & CEO of DealiOS. For the weekly series and more technical deep-dives, visit dealios.app/library.