Agent Systems
Agent loop optimization: harness, API, and inference.
An AI agent is not just a model. It is a loop around a model. The loop builds context, calls tools, observes results, updates state, and asks the model what to do next.
That loop can get expensive because it repeats work. The same instructions, tool schemas, conversation history, safety checks, tokenization, and cached state may be processed again and again unless the system is designed to reuse them.
Practical Takeaway
The best optimization is not one trick. It is removing repeated work at every layer.
The harness should send only what changed. The API should tokenize only what changed. The inference layer should route requests toward useful cached state and keep expensive GPU memory focused on active work.
System Anatomy
A request crosses three layers before the model answers.
The harness is the control plane closest to the user. It owns conversation state, tool definitions, approval rules, sandbox execution, and the loop itself.
The API layer validates requests, authenticates callers, handles rate limits, renders messages into model format, tokenizes text, starts safety checks, and streams events back.
The inference layer runs the model on accelerators. It manages model weights, KV cache, batching, routing, prefill, decode, and the raw generation work.
Harness Optimization
The harness should avoid resending the same world.
A simple agent can push the full prompt and full history through every step. That is easy to build and expensive to operate. A better harness keeps continuity with prior response state and submits only the fresh event, message, or tool result.
Stable prompt prefixes matter because prompt caching depends on exact token matches. If the first part of the prompt changes because a tool list serialized in a different order, the cache miss is invisible to the user but visible on the bill.
Tool discovery belongs on demand.
If an agent has hundreds of possible tools, the model should not receive every schema on every call. Keep core tools available, then let the model search for additional tool definitions when the task actually needs them.
{
"request_kind": "agent_turn",
"continue_from": "turn_123",
"delta": {
"event": "tool_result",
"tool_call": "lookup_456",
"summary": "appointment record found"
}
}Request Boundary
Stable context should not be rebuilt on every turn.
Tokenization is linear. If the whole conversation gets tokenized again on every loop step, the CPU cost grows with the context even when only one small tool result changed.
A stateful API path can keep the tokenized sequence in memory, tokenize only the new delta, and append it. Safety checks can also run in parallel with inference so harmless requests do not wait for checks that could have completed during time-to-first-token.
Delta tokenization
Tokenize the new input, not the whole transcript.
Parallel safety
Run classifiers during the model's unavoidable startup window.
Hardware-aware routing
Treat CPU generation and placement as real performance variables.
Inference Optimization
GPU work is saved when state lands where it can be reused.
The inference fleet has two jobs that fight each other: spreading load evenly and sending follow-up turns toward workers with useful cached state. Good routing balances both.
KV-cache management decides what conversation state stays hot, what moves to colder memory, and what gets evicted. Speculative decoding lets a smaller draft model propose tokens while the large model verifies them. Separating prefill from decode lets each workload run on hardware tuned for its bottleneck.
Product Lessons
Optimization should be tied to successful tasks, not isolated latency charts.
A faster tool call is useful only if the whole task finishes sooner and succeeds more often. Agent teams should measure cost per successful task, not only cost per model call.
The simplest optimizations are often the safest: keep prefixes stable, send deltas, avoid unnecessary tool schemas, compact old context, and cap loop iterations. Those changes reduce cost without changing the business behavior.
Deeper inference optimizations matter at scale, but they should follow evidence. Trace real sessions, find repeated work, and optimize the layer where the waste actually appears.
Trace the full loop
A single slow step may be harmless; repeated small delays can dominate long tasks.
Budget every turn
Loop limits, token budgets, and timeout rules protect both cost and user trust.
Optimize after observing
Production traffic shape decides which layer deserves attention first.
Related Reading