Building Production Agent Systems: Lessons from the Trenches

Demos of AI agents look magical. They browse the web, write code, send emails, book meetings. Production agent systems look different. They time out, get stuck in loops, call tools with malformed arguments, and confidently complete the wrong task. I have been building agent-based systems at HCLTech for clinical workflows and have watched enough agent demos turn into production postmortems to have formed strong opinions about what actually works.

The Core Challenge: LLMs Are Not Deterministic Executors

Traditional software has deterministic control flow. You call a function; it either works or raises an exception. LLM-powered agents have probabilistic control flow. The same prompt, same tools, same input can produce different action sequences on different runs. Building reliable systems on top of probabilistic components requires a different set of engineering patterns.

Principle 1: Narrow Tool Surfaces

Every tool you give an agent is a degree of freedom - another place where the agent can make a wrong decision. The most reliable agent systems I have seen have small, narrowly scoped tool sets.

Bad tool design: one giant database_query(sql: str) tool that lets the agent write arbitrary SQL. Good tool design: get_patient_by_id(patient_id: str), search_patients_by_name(name: str), get_recent_labs(patient_id: str, days: int). The narrow tools constrain what the agent can do wrong. They also make the agent's actions more auditable.

Rule of thumb: if you cannot explain in one sentence what a tool does and when to use it, split it into two tools.

Principle 2: Build Explicit State Machines for Multi-Step Workflows

Do not let the agent maintain implicit state through conversation history for workflows that matter. Use explicit state machines.

from enum import Enum
from dataclasses import dataclass, field
from typing import Optional

class WorkflowState(Enum):
    INITIALIZED = "initialized"
    GATHERING_INFO = "gathering_info"
    VALIDATING = "validating"
    EXECUTING = "executing"
    CONFIRMING = "confirming"
    COMPLETE = "complete"
    FAILED = "failed"

@dataclass
class AgentWorkflow:
    state: WorkflowState = WorkflowState.INITIALIZED
    collected_data: dict = field(default_factory=dict)
    errors: list = field(default_factory=list)
    attempts: int = 0
    max_attempts: int = 3

    def transition(self, new_state: WorkflowState) -> None:
        allowed = {
            WorkflowState.INITIALIZED: [WorkflowState.GATHERING_INFO],
            WorkflowState.GATHERING_INFO: [WorkflowState.VALIDATING, WorkflowState.FAILED],
            WorkflowState.VALIDATING: [WorkflowState.EXECUTING, WorkflowState.GATHERING_INFO, WorkflowState.FAILED],
            WorkflowState.EXECUTING: [WorkflowState.CONFIRMING, WorkflowState.FAILED],
            WorkflowState.CONFIRMING: [WorkflowState.COMPLETE, WorkflowState.FAILED],
        }
        if new_state not in allowed.get(self.state, []):
            raise ValueError(f"Invalid transition: {self.state} -> {new_state}")
        self.state = new_state

The state machine makes invalid transitions impossible, makes the current state observable, and makes failures recoverable to a known state rather than an undefined one.

Principle 3: Human-in-the-Loop for Irreversible Actions

For actions that cannot be undone - sending emails, writing to production databases, making API calls that charge money, deleting records - require explicit human confirmation before execution.

This sounds obvious and is routinely ignored in prototypes that become production. Add a confirmation step. Log what the agent is about to do. Require human approval. The cost of one extra click is vastly lower than the cost of undoing a bad agent action.

In clinical settings, this is non-negotiable. Any agent action that could affect patient care gets a human review step, period.

Principle 4: Idempotent Tools

Make your tools idempotent wherever possible. If a tool call is made twice with the same arguments, the second call should have the same result as the first without causing duplicate side effects.

Use unique idempotency keys for API calls. Check for existing records before creating new ones. Design database operations as upserts. This is standard distributed systems practice, but many teams forget it when building agent tools because they are focused on the LLM integration, not the tool implementation.

Principle 5: Timeout and Recursion Limits

Set hard limits on everything. Maximum tool calls per task. Maximum iterations in a loop. Maximum total execution time. Maximum tokens in context. Without these, an agent can run for minutes consuming API tokens before you detect a problem.

I have seen an agent without a recursion limit enter a loop calling a search tool 47 times because the search results never satisfied its internal "have I found the answer" criterion. Hard limits convert open-ended failures into bounded, recoverable failures.

Principle 6: Observability is Not Optional

You cannot debug a production agent system without traces. Every tool call should be logged: what tool, what arguments, what result, how long it took. Every LLM call should be logged: the prompt, the response, the model, the token count.

Use structured logging from day one. LangSmith, Arize, Helicone, and Weights and Biases all offer agent tracing. Pick one and instrument your system before you ship to production - debugging an unobserved agent failure is close to impossible.

Common Failure Modes and How to Handle Them

  • Tool argument errors: The agent calls a tool with invalid arguments. Fix: strong JSON schema validation on tool inputs, return structured error messages the agent can interpret.
  • Overconfident completion: The agent marks a task complete when it has not actually succeeded. Fix: add an explicit verification step that checks success criteria before marking complete.
  • Context stuffing: Tool results grow the context until the agent loses coherence. Fix: summarize tool outputs rather than appending raw results; implement a sliding window over tool history.
  • Cascading errors: A wrong decision in step 3 causes all subsequent steps to fail. Fix: add validation checkpoints between major workflow stages.

The real point

Agents are powerful when the task genuinely requires dynamic decision-making across multiple tools. They are overkill when a deterministic pipeline would work. For tasks that need agents, build them with narrow tool surfaces, explicit state, hard limits on everything, and observability from day one. The demo takes a day to build. A production agent system that is reliable enough to trust takes significantly longer - and the difference is almost entirely in the engineering discipline around failure handling.

Related posts

Further Reading