Watch It Think
Why streaming AI output changes what AI can build.
The Architecture Series · 9 minute read · By Ed | Founder & CEO, DealiOS
Everyone demos AI agents. PowerPoint slides with arrows and boxes. "The agent reasons, then acts, then observes, then reasons again." Beautiful. Elegant. Completely silent on what happens when that loop runs over Server-Sent Events through a corporate proxy with a 30-second timeout and the first API token takes 12 seconds to arrive.
The demo works in a Jupyter notebook. Production works over SSE to real users with timeouts, cold starts, and a CDN that thinks your stream is dead if it doesn't hear from you for 15 seconds.
The demo works in a Jupyter notebook. Production works over SSE to real users with timeouts, cold starts, and a CDN that thinks your stream is dead.
That gap is five specific engineering problems. Nobody warns you about them because most teams haven't shipped enough AI volume to discover them. Here's the field guide.
Problem 1: Last-Iteration Text Isolation
The agentic loop looks clean on a whiteboard. The model thinks, calls a tool, gets a result, thinks again, maybe calls another tool, thinks again, and finally produces a response. Three iterations of reasoning with two tool calls. The output is whatever the model says after the last tool result.
Here's the problem: the model produces text on every iteration. Not just the last one.
On iteration one, the model says something like "Let me search for that information..." and then calls a tool. On iteration two, it says "I found the pricing data, let me now check the discount policy..." and calls another tool. On iteration three, it produces the actual response the user needs.
If you're streaming text to the user as it arrives (which you should be, because a 60-second silence is a death sentence for user trust), you're streaming all of it. Iterations one and two produce conversational filler that pollutes the final document. The user's proposal now starts with "Let me search for that information" because nobody isolated the text from the last iteration.
The fix: track the byte offset where each iteration's text begins. When a new iteration starts (signaled by the model requesting another tool call), mark the previous text as intermediate. Only the text produced after the final tool result is the actual output. Everything before it is process narration that the user should never see in the document.
This requires the streaming handler to distinguish between "the model is thinking out loud while working" and "the model is producing the deliverable." In a notebook, you read the whole thing. In production, you need surgical text extraction. The implementation is a running offset counter that resets when a new content_block_start event of type tool_use arrives. The text that accumulates after the last tool result is the output. Everything before it is scaffolding.
Problem 2: Time-Budget Pressure
AI agents love to research. Give a model access to search tools and the ability to reason iteratively, and it will happily spend 90 seconds investigating a question that the user expected answered in 20. The model isn't wasting time. It's being thorough. But thoroughness at the cost of responsiveness is a production failure dressed as a quality improvement.
The user doesn't want a perfect answer in two minutes. They want a good answer in 30 seconds.
The user doesn't want a perfect answer in two minutes. They want a good answer in 30 seconds. And they definitely don't want a spinner that says nothing for 45 seconds while the agent runs its fourth iteration of research on a question about discount thresholds.
The fix: a wall-clock timer that runs alongside the agentic loop. When the timer hits the budget (say, 45 seconds), the system injects a synthetic system message into the conversation: "You have exceeded the time budget. Synthesize the best answer from what you've gathered so far. Do not make additional tool calls."
This message doesn't crash the loop. It doesn't abort the stream. It changes the model's behavior from "continue researching" to "produce output now." The tools are disabled for this final iteration, so the model can't ignore the instruction and call another tool. It has to synthesize.
The result: the user gets the best answer the agent could produce within the time budget. Not the perfect answer it would have produced given unlimited time. But a good answer, delivered within a window that feels responsive.
This pattern, time-budget injection with tool disabling, is the difference between an agent that works in demos (where nobody watches the clock) and one that works in production (where everyone watches the clock, even if they don't know they're doing it).
Problem 3: Self-Canceling Heartbeats
The first token from the Anthropic API can take 5 to 15 seconds to arrive. That's not a bug. That's the model processing your system prompt, your conversation history, your tool definitions, and your context before it produces its first output token. For complex prompts with large system contexts, it can be even longer.
In a notebook, you wait. In production over SSE, you die.
Corporate proxies, CDNs, load balancers, and reverse proxies all have idle timeout configurations. Many default to 30 seconds. Some are as aggressive as 15. If no data flows through the connection for that window, the intermediary assumes the connection is dead and closes it. Your user sees a connection error. Your stream is gone. The generation that was about to start never reaches the client.
The fix: heartbeat events. While waiting for the first token, the server sends periodic SSE events (every 5 to 10 seconds) that carry no content but keep the connection alive through every intermediary in the chain. A heartbeat event is a comment or a typed "heartbeat" event that every SSE parser ignores but every proxy counts as activity.
The tricky part: the heartbeat timer must self-cancel the instant real content arrives. If the first token arrives at second 7, the heartbeat timer that was going to fire at second 10 must be cleared immediately. Otherwise, you get a heartbeat event interleaved with real content, which can confuse parsers or, worse, produce a visible artifact in the UI.
The implementation is a setInterval for heartbeats that starts when the API request fires and a clearInterval that triggers when the first text event arrives from the stream. Simple in concept. Easy to get wrong in practice, because the timing is tight and the failure mode (heartbeat arriving after real content has started) is a race condition that appears randomly in production but never in testing.
The heartbeat has a gap worth knowing about: during retries (see Problem 5). When the system receives a retryable error, it clears the heartbeat before entering the backoff delay. That means a long backoff wait can leave the connection silent long enough for a proxy to kill it. The current architecture accepts this tradeoff because restarting the heartbeat during backoff adds complexity to an already dense state machine. It's a known gap, not an unknown one, and the retry intervals are calibrated to stay within most proxy timeout windows.
Problem 4: Side-Effect Channels
The agentic loop isn't just producing text. It's calling tools that have side effects. A calculator tool runs a pricing computation and returns the result to the model. But the client also needs that result, right now, to update the pricing table in the UI.
In a notebook, you don't care. You read the final output. In production, the user is watching the document build in real time. The calculator result needs to appear in the pricing section before the model finishes its response. The engagement score update needs to reflect in the dashboard. The document status needs to change.
The fix: side-effect channels. When a tool handler executes and produces a side effect (a calculation result, a status change, a data update), that side effect is emitted as a separate, typed SSE event alongside the ongoing text stream. The client receives both: the text the model is generating and the structured side effects from tool execution.
The SSE stream becomes multiplexed. Text events carry the model's output. Side-effect events carry structured data from tool execution. The client has separate handlers for each. The text handler appends to the document. The side-effect handler updates UI components independently.
This means the calculator results appear in the pricing table at the moment the tool executes, not when the model finishes talking about them three paragraphs later. The user sees the numbers populate, then reads the model's explanation. The experience feels real-time and coordinated because it is: two channels, operating in parallel, both fed by the same agentic loop.
The architecture: each tool handler returns both a result (which goes back to the model as input for the next iteration) and an optional array of typed side effects (which get emitted as SSE events immediately). The streaming handler checks for side effects after every tool execution and emits them before continuing the model's response. The side effects are fire-and-forget from the model's perspective. The model never sees them. The client always does.
Problem 5: Retry with Heartbeat Continuity
API calls fail. Rate limits (HTTP 429) are common at scale. Service overloads (HTTP 529) happen during peak usage. Timeouts happen when the model takes longer than expected. These are not exceptional events. They're Tuesday.
In a notebook, you catch the error and retry. In production over SSE, you have a problem: the client is holding an open connection. If you pause for 30 seconds of exponential backoff before retrying, and the proxy timeout is 15 seconds, the connection dies before the retry fires. The user sees an error. The retry, which would have succeeded, never reaches them.
The fix: retry logic with interval calibration. When the system receives a retryable error, it enters a backoff loop. Rate limits (429) use longer intervals: 60 seconds on the first retry, 120 seconds on the second. Service overloads (529) use shorter intervals: 5 seconds, then 10 seconds. The intervals are calibrated to each error type because rate limits signal "come back later" while overloads signal "try again soon."
The gap: the heartbeat clears before the backoff wait, so the connection is silent during the delay. For the shorter 529 intervals (5-10 seconds), this fits within most proxy timeout windows. For the longer 429 intervals, the connection may drop before the retry fires. This is a known tradeoff. The client sees a retry status message before the wait begins, so they know the system is working even if the connection needs to reconnect.
Non-retryable errors (401 authentication failures, 400 bad requests) skip the retry loop entirely. There's no point retrying a request the server has definitively rejected. These get surfaced to the user immediately with a human-readable error message, not a raw HTTP status code. The error classification matters: retrying a 401 is wasting time. Not retrying a 429 is losing the user.
The implementation requires the error handler to inspect the status code, classify it as retryable or terminal, and either enter the backoff loop (with heartbeats) or surface the error (with a friendly translation). The friendly translation is its own small but important system: a function that maps API error codes to messages that make sense to someone who is not an engineer. "The AI service is temporarily busy. Retrying." Not "HTTP 529: Service Overloaded."
The Meta-Problem: Everything Happens Simultaneously
Each of these five problems is solvable in isolation. The real engineering challenge is that they all happen at the same time, in the same streaming handler, on the same connection.
The heartbeat timer is running while the model is thinking. The first token arrives and cancels the heartbeat. The model produces text (track the offset). The model calls a tool (emit side effects, reset the text offset). The tool returns, the model produces more text. The model calls another tool (more side effects, another offset reset). The model produces final text (this is the actual output). The stream ends.
Unless it doesn't end. Unless you hit a 429 on the second tool call. Now you're in the retry loop. The backoff timer is counting. The client is waiting through a silent gap. The retry fires. The model resumes. New text arrives. Side effects emit. The stream completes.
All of this is one function. One streaming handler that manages heartbeats, text offset tracking, side-effect emission, retry logic with error classification, and time-budget enforcement simultaneously. It's not conceptually difficult. It's operationally dense. Every piece interacts with every other piece.
The code that manages this is not glamorous. It's deeply procedural, full of state variables and timer management and conditional branching. It looks nothing like the clean "reason, act, observe" diagrams in the blog posts about agentic AI. It looks like production code. Because it is.
Why This Matters Beyond AI
These five problems aren't unique to AI agents. They're instances of a broader engineering pattern: what happens when a long-running, multi-step server process needs to communicate results to a client in real time.
Video transcoding pipelines face the same heartbeat and side-effect challenges. Complex database migrations face the same time-budget pressure. Distributed workflow orchestrators face the same last-step isolation problem. Anywhere a server does meaningful work over a period that exceeds the patience of proxies and users, these five patterns apply.
The AI-specific twist is that the work is nondeterministic. A database migration takes a predictable amount of time. An agentic AI loop takes however long the model decides it needs. You can't precompute the timeline. You can only manage the infrastructure around the unpredictability.
That management, the heartbeats, the side effects, the time budgets, the retry logic, the text isolation, is the actual engineering work of production AI. It's not the model. It's not the prompts. It's the unglamorous, stateful, timing-sensitive infrastructure that makes the model usable by real people on real networks through real proxies.
Nobody puts this in the demo. But it's the difference between an agent that works in a notebook and one that works in the world.
The takeaway: The actual engineering work of production AI isn't the model or the prompts: it's the unglamorous, stateful, timing-sensitive infrastructure (heartbeats, side-effect channels, retry logic, text isolation) that makes agents usable by real people on real networks.
This paper is part of The Architecture Series by Ed, Founder & CEO of DealiOS. For the weekly series and more technical deep-dives, visit dealios.app/library.
References
[1] Anthropic API documentation: Streaming Messages via Server-Sent Events. Error codes 429 (rate limit) and 529 (service overload) with recommended retry behavior.
[2] SSE specification: W3C Server-Sent Events standard. Comment lines (lines starting with :) as keepalive mechanism.
[3] Corporate proxy timeout behaviors: Common default idle timeouts of 15-30 seconds in enterprise proxy configurations (Nginx proxy_read_timeout, AWS ALB idle timeout, Cloudflare origin response timeout).
This paper is part of The Architecture Series by Ed, Founder & CEO of DealiOS. For the weekly series and more technical deep-dives, visit dealios.app/library.
See how these principles come together. Explore the platform →