Anatomy of an AI agent: from the trace to continuous improvement

Evaluation is the biggest unsolved problem in agent engineering: the output is non-deterministic, ‘correct’ is often subjective, and end-to-end tests are slow and expensive because every run costs real API calls.

MATTEO DI PAOLANTONIO
CTO Orbitant

This is the third and final part of the glossary series, dedicated to putting AI agents in production: how you observe them, evaluate them, and improve them once they’re already running. In the first post, I explained how an agent works on the inside: the loop, the tools, and the runtime. In the second post, I walked through the pieces you build around it: MCP, skills, limits, guardrails, multi-agent systems, and the spectrum of agency.

This installment closes the series with the operational layer: how you observe what the agent is doing, how you manage the model behaviors that show up in production (hallucination, non-determinism), how you evaluate when the output is no longer deterministic, when an agent is worth it and when a workflow is enough, and which frameworks dominate today’s ecosystem.

Observability, hallucination and non-determinism

Agents inherit three properties from the LLM that didn’t exist in traditional software and that completely change how you operate the system in production: the difficulty of observing what the agent is doing when it chains dozens of steps in a row, the model’s tendency to make things up with real consequences when those hallucinations reach the runtime, and the non-determinism that means the same input can produce different results on different runs.

All three share the same origin — the LLM at the core of the system — and understanding how each one manifests is what lets you operate agents in real environments.

Observability: agents are hard to debug

Debugging a normal API call is straightforward: seeing the input and the output is enough. But an agent that chains 30 turns through the loop, with its tool calls and intermediate results at each step, can’t be diagnosed without a dedicated tracking system.

For that problem, the industry’s standard solution is tracing: recording every step of the agent’s execution, with its inputs, its outputs, its latencies, and its costs.

Trace #4821
├─ LLM call 1: "I need to read the file" (2.1s, 1.2K tokens)
│   └─ Tool: read_file("main.py") → 200 lines (0.05s)
├─ LLM call 2: "I see the bug on line 42" (1.8s, 950 tokens)
│   └─ Tool: edit_file("main.py", ...) → success (0.02s)
├─ LLM call 3: "Let me run tests" (1.5s, 800 tokens)
│   └─ Tool: run_bash("pytest") → 2 failures (3.2s)
├─ LLM call 4: "Tests still fail, let me re-read..." (2.0s, 1.1K tokens)
│   └─ ...
Total: 12 steps, 45s, 28K tokens, $0.08

It’s the equivalent of the distributed tracing (Jaeger, Datadog) that the microservices world has been using for years: same problem, different domain. Every major agent framework already ships native tracing or connects to external observability platforms.

Hallucination and grounding

LLMs make things up: they cite a function that doesn’t exist, mention a library that was never imported, or claim that a file contains something it doesn’t. In a chatbot that gives you a wrong answer. In an agent, that same hallucination can turn into a real action, because if the agent hallucinates a bash command, the runtime will try to run it.

The techniques for keeping the agent anchored to reality are known as grounding, and they include three common practices:

• Read before editing: never let the agent modify code it hasn’t seen first.
• Verify before asserting: run the tests, check the output, don’t trust the agent’s self-assessment.
• Retrieval-Augmented Generation (RAG): pull data from a knowledge base instead of relying on the model’s training data.

Every tool call is a reality check against the world, and that’s the reason agents are built with a broad catalog of tools. Without them, what the agent does comes down to guessing; with them, it can read real files, receive the error messages that actually occur when running the code, and see the results of the tests that were genuinely executed.

Non-determinism

Agents are not reproducible by default: the same input can generate different results on different runs, with no bug involved. This behavior has three sources worth identifying before you try to test the system:

• Temperature in the LLM (controlled randomness in text generation).
• Dynamic tool results (files change, APIs return different data on each call).
• Branching decisions (the agent might choose to read file A first on one run and file B on another).

The practical consequences are three, and they directly affect how you test and monitor the system:

• You can’t write traditional unit tests for an agent’s behavior.
• You need an evaluation framework that runs the agent N times and measures the success rate.
• Agents in production need continuous monitoring, not just testing in CI.

Evaluation and continuous improvement

Once you’ve accepted non-determinism and the other LLM behaviors inside the system, you reach the hardest problem in agent engineering: how to decide whether what the agent produces is good enough, and what to do when it isn’t.

This section tackles the three axes that make up that problem: why evaluating agents is different from testing software, which evaluation strategies are working in production, and what concrete levers you have to improve the system when the evaluations show problems.

The evaluation problem

Evaluating agents is the biggest obstacle in engineering LLM-based systems. Where traditional software lets you write deterministic tests that pass or fail; with agents, reality gets complicated across three dimensions at once:

• The output is non-deterministic, so two identical runs can produce different verdicts.
• “Correct” is often subjective (is this a good commit message? is this refactor clean?).
• End-to-end tests are slow and expensive because every run costs real calls to the model’s API.

Evaluation strategies

There are four common strategies for evaluating an agent’s behavior in production, each with its own trade-offs:

Strategy	How it works	Pros	Cons
Benchmark suites	Standardized tasks (for example, SWE-bench for coding) that measure success rate	Reproducible, comparable	Narrow scope, risk of gaming
LLM-as-judge	Another LLM evaluates the agent’s output	Scalable, cheap	Can get it wrong, biases
Human evaluation	Humans score quality, flag failures	Gold standard for quality	Expensive, slow
Regression testing	Record successful runs, compare against future runs	Catches regressions	Fragile, high maintenance burden

How to improve based on evaluation

When evaluations show the agent is performing worse than expected, the first thing many people change is the system prompt, but that’s only one of the available levers. In practice, five of them are delivering results in production.

1. System prompt (the most common, the cheapest, ~70% of improvements). Human evaluators spot a failure pattern and you add or refine instructions to avoid it:

Before processing a refund, always check the shipment status first.
If the item shows as delivered, ask the customer for more details
before proceeding.

It’s the fastest feedback loop: you change text, run again, see if it improves. Other common changes to the prompt include adding examples of correct behavior (few-shot), restructuring the text for clarity, introducing explicit “don’t do X” rules, or refining the tool descriptions the model sees.

2. Tool design. Sometimes the problem isn’t the agent — it’s badly designed tools. Ambiguous APIs or ones with overloaded parameters produce incorrect agent behavior:

# Before: ambiguous, causes errors
def search_customer(name: str) -> Customer

# After: unique identifier, no ambiguity
def search_customer(email: str) -> Customer

3. Architecture or workflow changes. If the agent takes too many steps on simple tasks, it’s worth pinning the simple paths as a deterministic workflow and reserving the agentic loop only for the cases that truly need it:

# Before: fully agentic, the agent decides everything
Agent → (figures out what to do) → (multiple loops) → resolution

# After: route common cases to deterministic workflows
Router (LLM classifies the intent)
   ├─ "password reset"  → deterministic workflow (no agent)
   ├─ "refund request"  → deterministic workflow with one LLM check
   └─ "complex/unclear" → full agent (only when needed)

4. The model itself. Sometimes the agent’s problems aren’t solved by changing the prompt or the tools, and the lever left is to act on the LLM at the core of the system. There are three alternatives within this lever:

• Upgrade the model: move from a smaller, faster one to a larger, more capable one.
• Fine-tuning: train a custom version on domain-specific data.
• Different models for different steps: use a cheap, fast one for classification, a powerful one for complex reasoning.

5. Guardrails and post-processing. For safety-critical properties the prompt isn’t enough; it’s worth adding firm guardrails implemented in code that act as a final validation before the output leaves the system:

def output_guardrail(response):
    if contains_internal_data(response):
        return "Let me transfer you to a specialist."
    return response

The feedback loop

The continuous-improvement cycle takes the shape of an iterative loop: you deploy the agent, observe what fails, categorize the type of failure, apply the right lever, and evaluate again. Schematically:

Deploy agent
   │
   ▼
Evaluate interactions (rate quality, flag failures)
   │
   ▼
Categorize failures:
   ├─ Agent chose wrong action    → fix prompt or tool descriptions
   ├─ Agent used tool incorrectly → fix tool API or tool description
   ├─ Agent was too slow/costly   → change architecture (workflows)
   ├─ Agent misunderstood intent  → better model or fine-tuning
   └─ Agent leaked sensitive data → add guardrail
   │
   ▼
Make the change
   │
   ▼
Re-evaluate (did the fix help? did it break something else?)
   │
   ▼
Repeat

Workflows vs agents: the practical distinction

One of the most useful architectural distinctions in the ecosystem, published by Anthropic, is the separation between workflows and agents. Knowing which of the two models applies to a use case is the most relevant decision before you start building, because it shapes everything from development cost to how predictable the system is in operation.

Workflows: predefined control flow

In a workflow, the developer decides the control flow at build time:

Step 1: Generate code → Step 2: Run linter → Step 3: Fix errors → Step 4: Run tests

The LLM is used at each step, but the order and the logic are hard-coded. If step 2 fails, you always go to step 3. There’s no ambiguity in the behavior.

Agents: dynamic control flow

In an agent, the LLM decides the control flow at runtime. The developer gives it a goal and some tools, and the model figures out what to do:

Goal: "Fix the failing test"
Agent decides: read test → read source → think → edit source → run test →
  still fails → read error → try different fix → run test → passes → done

The path is emergent and not predetermined, so different runs can take completely different routes.

What turns a workflow into an agent

A workflow becomes an agent when it meets these five conditions:

1. It has a single entry point with a goal, instead of a pipeline of separate commands.
2. It makes its own decisions about what to do next at runtime.
3. It self-evaluates: it checks its own output and decides whether it’s good enough.
4. It recovers from errors: it tries alternative approaches when something fails.
5. It has selective human checkpoints: it only pauses on high-risk decisions and handles the rest autonomously.

The trade-off

The choice between workflow and agent implies concrete trade-offs across five operational dimensions:

	Workflow	Agent
Control	Total: the developer decides everything	Partial: the developer sets goals and reviews
Speed	Slower: human in every loop	Faster: runs steps autonomously
Predictability	High: the same steps every time	Lower: can take unexpected paths
Quality floor	Human catches every error	Needs good self-evaluation criteria
Build complexity	Simpler: linear pipeline	Harder: decision logic, error handling, eval

The practical takeaway

Start with workflows. Use agents only when the task genuinely requires dynamic decision-making. Workflows are cheaper, faster, more predictable, and easier to debug.

The industry’s trend at the start of 2026 is exactly that: structured workflows with a human in the loop, scaling up to full agents only when necessary.

Frameworks and SDKs in the ecosystem

The landscape of frameworks for building agents has grown fast between 2024 and 2026. Each one takes a different approach: some are model-agnostic, others are tied to a vendor; some are low-level and give granular control, others are high-level and hide the complexity of the loop.

This section walks through the most relevant ones of the moment, grouped by vendor.

Anthropic

Anthropic offers two main products for building agents around Claude, one ready to use and one meant for building custom:

• Claude Code: Anthropic’s CLI agent, a production-grade coding agent that uses Claude with tools for file I/O, bash, search, etc.
• Claude Agent SDK: a Python SDK for building custom agents. It includes a built-in agentic loop, native MCP integration, tool use with automatic schema generation, guardrails, and multi-agent orchestration via handoffs.

OpenAI

OpenAI’s official agent SDK is oriented toward multi-agent, handoffs and built-in tracing:

• Agents SDK (formerly Swarm): an open source Python SDK. Key concepts: agents configured with instructions and tools, handoffs for multi-agent, guardrails for validation, built-in tracing for observability.

LangChain and LangGraph

Two frameworks from the same team, at different levels of abstraction:

• LangChain: the original agent framework (2022). Abstractions for chains, tools, memory and retrieval.
• LangGraph: a newer, lower-level framework for building agents as state machines or graphs. Nodes (LLM calls, tool calls, logic), edges (transitions). It supports cycles, branches, parallel execution, persistent state, and a human in the loop at any node.

Other frameworks

Beyond the main ones, the ecosystem has several projects that are relevant for specific use cases:

• CrewAI: multi-agent collaboration. The developer defines “agents” with specific roles (“Researcher”, “Writer”, “Reviewer”) and “tasks”, and the agents collaborate to complete workflows.
• Microsoft AutoGen: a multi-agent conversation framework. Agents communicate by message passing, with complex group-chat patterns (round-robin, manager-directed, etc.).
• Google Agent Development Kit (ADK): integrates with Gemini models. Orchestration, tool use, multi-agent patterns, native Google Cloud integrations.
• Vercel AI SDK: web apps with agents (TypeScript/Next.js).
• Pydantic AI: type-safe agents with structured outputs.
• smolagents (Hugging Face): a minimalist framework, the code-agent pattern where the LLM writes Python as its action.

AI agents in production: how it all fits together

The three parts of the series have covered layers one at a time: the agent loop was the focus of the first article, the pieces you build around it centered the second, and this post tackles operation. This final section brings the pieces together in a single diagram and sums up what the ecosystem is converging on.

The architecture diagram

All the pieces that have shown up across the three posts fit into a single agentic architecture diagram:

                ┌──────────────┐
                │  USER GOAL   │
                └──────────────┘
                       │
                       ▼
                ┌──────────────┐
                │ AGENT RUNTIME│   (Claude Agent SDK, OpenAI Agents SDK,
                │ (Framework)  │    LangGraph, CrewAI, etc.)
                └──────────────┘
                       │
        ┌──────────────┼──────────────┐
        │              │              │
        ▼              ▼              ▼
┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│ REASONING   │  │   MEMORY    │  │ GUARDRAILS  │
│   (LLM)     │  │             │  │  (safety)   │
└─────────────┘  └─────────────┘  └─────────────┘
                       │
                       ▼
                ┌─────────────┐
                │    TOOLS    │   via function calling or MCP
                └─────────────┘
                       │
        ┌──────────────┼──────────────┐
        │              │              │
        ▼              ▼              ▼
     File I/O      API calls    Code exec, Web search
                       ▲
                       │
                ┌─────────────┐
                │ MCP SERVERS │   Standardized, plug-and-play
                └─────────────┘

The convergence

Despite the diversity of frameworks, the entire ecosystem is converging on the same set of pieces:

• Powerful LLMs as the “brain”.
• Tool use as the “hands”.
• MCP as the “nervous system” that connects everything.
• Skills as “reusable experience”.
• Agent frameworks as the “skeleton” that gives structure.
• Guardrails as the “immune system” that keeps things safe.
• Observability as the “senses” that let you understand what’s happening.