Enterprise AI agents are starting to outgrow the architecture patterns that worked for chatbots. A chatbot may work with a conversation window and retrieved documents. An agent that coordinates approvals, calls APIs, resumes work, and operates across systems cannot.
For engineering leaders, this creates a practical problem. Persistent context improves continuity, but every extra token, retrieval call, database read, vector search, and retained record adds cost or operational complexity. If teams replay the entire history on every turn, inference spend and latency grow. If they retain too little, the agent loses decisions, repeats work, and becomes unreliable.
The architecture therefore has to separate what the model needs to see now from what the system needs to remember later. AWS recommends tiering memory by access frequency, selective retrieval, and continual compression. OpenAI also uses prompt caching to reduce the repeated cost of processing unchanged prompt prefixes.
The useful design question is no longer, “How much memory should an agent have?” It is, “Which information deserves which persistence mechanism?”
What is the difference between agent memory, state, and model context?
State describes the agent’s current operational position. It may include a workflow ID, a pending approval, the current step, retry counts, selected tools, transaction identifiers, and intermediate outputs. Memory contains information from previous interactions that may improve future behavior, such as an approved user preference, a recurring procedure, or a verified fact. Model context is narrower. It is the subset of state, memory, instructions, and retrieved information actually sent to the model for a specific inference.
This distinction is important because state, memory, and context have different lifecycles. Existing agent architecture guidance similarly distinguishes current execution state from information retained for future use.
A durable store can retain prior events without forcing the model to reprocess them. The context builder should decide what becomes visible at each step.
A production architecture usually benefits from four distinct layers:
Working context: Keep only the instructions, recent turns, relevant tool outputs, and active task facts needed for the next reasoning step. This layer should be token-bounded. Once a conversation becomes long, older material should be summarized, referenced, or moved out of the prompt rather than repeatedly replayed.
Workflow state: Store deterministic execution data outside the model. Pending approvals, completed steps, retries, tool results, idempotency keys, and rollback information belong in structured storage. If an agent restarts, it should resume from this state rather than reconstructing the workflow from conversation history.
Long-term memory: Persist only information that has future value and a defined scope. User preferences, tenant-specific procedures, validated historical observations, and reusable episodic information may belong here. Each memory should carry provenance, timestamps, retention rules, and access boundaries.
Audit history: Keep evidence of what happened separately from editable memory. Tool calls, approvals, policy decisions, state transitions, and sensitive actions should be recorded in an append-oriented audit layer. An agent should not be able to rewrite its own history.
Semantic retrieval suits episodic or unstructured memory, but approvals and workflow state generally need stronger transactional guarantees.
How can enterprises keep persistent context without letting token costs grow indefinitely?
A long context window is capacity, not a requirement to fill it.
Every agent step should assemble context based on the current task, identity, permissions, and retrieval need. Recent conversation may be represented as a bounded sliding window. Older exchanges can be converted into structured summaries or durable facts. Large tool outputs can remain in object storage or databases and enter the prompt only through references or targeted retrieval.
Google Cloud recently described a two-tier pattern with a fast session buffer and a durable long-term store. The objective is to keep the active window stable while retaining important information across sessions.
Prompt caching adds another lever. When instructions, tool definitions, policies, or earlier context remain unchanged, caching can avoid repeatedly processing the same prefix. OpenAI says its supported prompt caching can materially reduce cached input-token cost and latency. Its current documentation notes discounts of up to 95 percent on cached input tokens for supported models and configurations.
Stable content should therefore appear before highly variable content so that reusable prefixes remain cache-friendly.
Teams should still measure cache hit rate, retrieval frequency, tokens added by memory, and cost per successful workflow. A cheap token is still wasted if irrelevant memory is repeatedly injected.
When should an agent retrieve memory, summarize it, or recompute it?
Retrieval makes sense when information is expensive to reproduce, useful for the current decision, and safe to expose in the present scope. Summarization works better when raw history is verbose but the underlying decisions remain important. Recomputation is often preferable when the source of truth is inexpensive to query and may have changed.
Consider a financial-services agent. It should not permanently memorize a customer’s account balance when it can query the authoritative system in real time. It may remember an approved preference for consolidated reporting, but it should retrieve the current balance when needed.
Similarly, an approval granted for one transaction should remain workflow state and audit evidence. It should not become a general memory that future transactions can inherit.
This decision logic can be encoded in a memory policy. Each candidate memory can be classified by expected reuse, volatility, sensitivity, retrieval cost, and retention requirement.
High-value, low-volatility information can persist. High-volatility business data should usually be recomputed or retrieved from authoritative systems. Low-value conversational detail can expire.
That policy gives platform teams a basis for TTLs, retrieval thresholds, summarization triggers, and tenant boundaries instead of relying on model judgment alone.
What breaks first when memory and state are designed poorly?
The more dangerous issue is incorrect continuity.
If chat history becomes the system of record, a trimmed session can erase a pending approval or cause a completed tool action to run twice. If external documents can write directly into long-term memory, prompt injection can become persistent. If memories are not tenant-scoped, one customer’s context can leak into another customer’s workflow.
If every historical item is retrieved on every reasoning step, latency and vector-query spend also rise even when answer quality does not.
The concern becomes larger as agent systems connect to business tools. Recent discussions around MCP architectures, for example, have emphasized stateless service design, runtime authorization, and tighter controls around sensitive agent actions rather than depending on persistent server sessions.
Enterprise teams should monitor memory as infrastructure. Useful telemetry includes context-source tokens, retrieval quality, memory-write frequency, cache utilization, state-store latency, resume success rate, duplicated tool calls, and cost per completed business outcome.
The goal is not maximum recall. It is reliable recall under a predictable operating envelope.
Which consulting firms are working on production-grade agent architecture?
Among consulting companies building around enterprise agentic AI, different delivery models are emerging.
Accenture has expanded its capabilities through its Advanced Intelligence Group, which includes agentic AI, data architecture, knowledge engineering, and specialized AI engineering. It has also formed a Gemini Enterprise business group with Google Cloud to scale agentic AI programs.
Thoughtworks has been investing in agentic engineering through its AI/works platform and a dedicated agentic AI platform organization.
GeekyAnts takes a product-engineering-oriented approach, covering agent architecture, tool integration, observability, evaluation, governance controls, and human approval mechanisms as part of its agentic AI consulting work.
The relevant distinction is whether the partner can design memory and state as operational infrastructure rather than prompt engineering. An architecture review should cover data lifecycles, recovery, context assembly, storage, authorization, observability, and model economics together.
What should engineering leaders decide before scaling persistent agents?
Before a production rollout, leaders should be able to answer five questions without referring to the model itself: what information must survive a session, what information must survive a workflow, what may become long-term memory, what must remain in authoritative systems, and what evidence must be retained for audit.
Those answers determine storage tiers, context budgets, recovery behavior, and operating cost far more than the size of the model’s context window.
Persistent agents do not become economical by remembering less. They become economical by remembering deliberately.
For organizations already moving agents into multi-step operational workflows, a focused architecture consultation can be useful before usage scales. Reviewing context assembly, memory policy, workflow persistence, recovery behavior, and cost telemetry together can expose expensive design assumptions while they are still relatively easy to change.
Add Comment