Agentic AI in Production: Architecture, Memory and Governance

Taking agentic AI from demo to production means building a planning layer, persistent memory, human-in-the-loop approval and a governance framework — the parts a prototype never needs. This guide covers the four-layer architecture, the confidence and cost thresholds that keep autonomous agents safe, and the seven metrics worth tracking once real work depends on them.

An AI agent that books a meeting in a demo and an AI agent that processes compliance documents unsupervised are not the same class of system. The first needs a model and a tool call. The second needs to plan, recover from partial failure, remember what it did last month, know when to stop and ask a human, and leave an audit trail that survives scrutiny. If you are earlier in the journey, start with our guide to what an AI agent actually is or the practical step-by-step guide to building one. This article is about the gap between those two systems.

The Gap Between Demo and Production

Across deployed agent systems — customer support, knowledge retrieval, and compliance processing — the same pattern holds: roughly 80% of cases are routine and consume almost none of the engineering effort. The remaining 20% — ambiguous inputs, partial tool failures, conflicting source documents — is where essentially all the work lives.

Demos are built against the 80%. Production is judged on the 20%. An architecture that treats edge cases as an afterthought will pass every internal review and then fail in front of a customer.

Agent Architecture: Beyond the ReAct Loop

The naive reason-act loop — think, call a tool, observe, repeat — is a fine mental model and a poor production architecture. It has no notion of a plan, so it cannot estimate cost before committing. It has no isolation between steps, so one failed tool call corrupts the whole run. It has no memory beyond the current context window.

What holds up in production is four distinct layers: planning, execution, tools, and memory, each with its own failure boundary. Separating them is what makes an agent debuggable.

The Planning Layer

Before acting, the agent produces a structured execution graph: discrete steps, their dependencies, an estimated cost, and a rollback strategy for anything with side effects. Modelling that plan as a typed schema — Pydantic works well — turns "what is the agent about to do?" into a question with a concrete, inspectable answer.

The plan also carries a confidence score. Below a 70% planning-confidence gate the agent does not proceed on its own; it routes to a human. Cost estimates gate the same way, with thresholds at $5.00 for mandatory review and $1.00 for logging and notification.

The Execution Layer

Steps run with isolation at the step level, resolved in dependency order via a topological sort, executed asynchronously where the graph allows it. Every action is recorded durably as it happens — not summarised afterwards.

Step-level isolation is what makes partial failure survivable. When step four of nine fails, steps one through three remain valid and the run resumes from the failure point rather than starting over. Without it, every transient API error costs a full re-execution and the money already spent.

The Tool Layer

Tools are the point where an agent touches real systems, so they are treated as strict API boundaries rather than convenient function wrappers. Every tool validates its inputs against a schema, declares an output schema, and enforces its own rate limit — 30 calls per minute for read operations, 10 for anything that writes.

Database access follows defence in depth: the agent queries through a restricted role with only the permissions the task requires, so a malformed or manipulated query cannot reach beyond its remit. Safety filters here are deliberately blunt — keyword blocking that occasionally produces a false positive is the correct trade against an agent executing something destructive.

Persistent Memory: The Real Differentiator

Memory is what separates an agent from a very capable one-shot prompt. Three kinds matter, and they behave differently:

Semantic memory carries a 90-day expiry window, and that constraint exists because of a specific failure: a compliance agent confidently cited a policy document that had been superseded six months earlier. It had remembered accurately and reasoned correctly from stale information. Unbounded memory is not a feature — it is a slowly accumulating liability. pgvector on PostgreSQL is sufficient for this; a separate vector database is rarely justified at agent-memory scale.

Human-in-the-Loop: Where the Agent Stops

Approval boundaries belong in a policy engine, not scattered through prompt text. The policy evaluates confidence thresholds, cost estimates, and novelty detection — has this agent seen a case like this before? — and routes accordingly.

The useful signal is the human override rate. Around 15% indicates healthy calibration. Much lower and the approval gates are theatre; nobody is really reviewing. Much higher and the agent is being asked to operate beyond its competence, and the honest response is to narrow its remit rather than tune the prompt.

AI Governance Is Not Optional

For any agent acting on real business systems, governance is a build requirement, not a compliance afterthought. Four properties carry the weight: auditability of every decision, containment of blast radius, observability of behaviour over time, and a scorecard that makes degradation visible.

What the Deployments Taught Us

Seven metrics are worth tracking continuously:

Three lessons recur. Autonomy is a dial, not a switch — the systems that succeed start narrow and earn scope as their metrics justify it. User expectations need managing explicitly, because an agent that is right 90% of the time is experienced as unreliable unless people know when to check it. And edge cases are the product: the 20% is where trust is won or lost, and it deserves the majority of the engineering budget.

None of this is exotic. It is ordinary systems engineering — schemas, isolation, rate limits, audit logs, metrics — applied to a component that happens to be probabilistic. Teams that treat agents as software rather than as prompts are the ones whose deployments survive contact with production.

We design and operate agentic systems with these boundaries built in from the start. Explore our AI agent development services, see agent use cases by industry, or get in touch to discuss what would need to be true for an agent to run safely in your environment.