Anatomy of an AI agent: from the loop to the runtime

Every conversation about AI agents ends up, sooner or later, at the same question: what is an AI agent, and what exactly separates it from a chatbot with a clever prompt or a script with one LLM call? The short answer: how it works on the inside. The loop it runs, the tools it has access to, and one specific piece — the runtime — that orchestrates everything else.

This is the first of three posts in which I walk through the AI agent ecosystem. In this first one, I stay on the fundamentals: how an agent is defined, how it reasons, how it acts, and what program holds it all together. In the next, I get into how you build one in practice (MCP, skills, guardrails, multi-agent); in the last, how you operate it, evaluate it, and connect it with the ecosystem’s frameworks.

Without tools, an LLM is just a token generator. With tools, it becomes an actor capable of affecting the real world. That’s the deepest architectural difference between a chatbot and an agent.

MATTEO DI PAOLANTONIO
CTO Orbitant

What is an AI agent?

An AI agent is a system in which a Large Language Model (LLM) operates inside a loop, autonomously deciding what actions to take to reach a goal, instead of just responding to a single prompt.

The analogy is useful: a normal LLM call is like asking someone a question and getting an answer. An agent is like hiring someone to push a project forward: it plans, uses tools, reviews its work, adapts, and keeps going until it’s done.

Properties of an agent:

• Autonomy: it decides its own steps without human intervention at each one.
• Goal-oriented: it works toward a goal, not just toward a single question.
• Interaction with the environment: it reads files, calls APIs, runs code, browses the web. It acts on the world.
• Persistence: it operates across multiple steps or turns, holding state between them.

Terminology you’ll see in the market:

• Agentic AI: the broad category.
• Autonomous agent: emphasises the self-directed nature.
• AI agent: the general term.

All three refer to more or less the same idea.

The architecture: the agentic loop

Every agent runs an observe-think-act loop (also called the reasoning loop or agentic loop):

User Goal
   |
   ▼
┌─────────────────┐
│  1. OBSERVE     │  ← Reads the state of the environment, tool
│                 │     results, user messages, memory
└─────────────────┘
   |
   ▼
┌─────────────────┐
│  2. REASON/PLAN │  ← The LLM "thinks" about what to do next
│                 │     (chain-of-thought, reflection)
└─────────────────┘
   |
   ▼
┌─────────────────┐
│  3. ACT         │  ← Calls a tool, writes an output,
│                 │     asks the user, or declares the task done
└─────────────────┘
   |
   ▼
Goes back to step 1 (until it finishes or hits a limit)

The three phases of the loop rest on three components worth pinning down from the start.

Reasoning and chain-of-thought. The LLM “thinks out loud” to break down complex tasks. Sometimes visibly in the output; sometimes in a hidden scratchpad that some providers call extended thinking. Models like Claude or the GPT-4 class are trained specifically to be good at this kind of step-by-step reasoning.

Memory. Three different types coexist, and mixing them up is the most common cause of agents that forget what they should remember or load what isn’t needed:

• Short-term memory (the context window): the conversation history and tool results the model can “see” in a single call. Limited by the model’s capacity (200K tokens for most Claude models, 1M for Opus).
• Long-term memory: external storage (databases, files, vector stores) that the agent can read from and write to between sessions. Lets it remember things beyond a single conversation.
• Working memory: a scratchpad or intermediate state the agent keeps during a single task (a list of files it has already reviewed, a partial design doc, hypotheses to verify).

Planning. The most sophisticated agents plan before acting. Three dominant techniques:

• Task decomposition: breaking “build me a web app” into concrete sub-tasks like “create project structure”, “implement the backend API”, “write the frontend”, “add tests”.
• ReAct pattern (Reason + Act): the agent explicitly writes out its reasoning before each action. It’s the most common pattern in production agents today; the original paper is still the reference.
• Reflection: after receiving a result, the agent evaluates whether it’s on the right track and adjusts the plan.

Tools and function calling

A tool is a function the LLM can choose to call. The model doesn’t execute the function: it emits a structured request — usually a JSON object — saying “I want to call function X with arguments Y”, and the runtime runs it and returns the result. Without tools, an LLM is just a token generator. With tools, it becomes an actor capable of affecting the real world.

How it works step by step:

1. The system prompt declares the available tools to the LLM:
   • read_file(path: string) → string
   • run_bash(command: string) → string
   • search_web(query: string) → string

1. The LLM receives the user request: “Find the bug in my code”.
2. The LLM responds with a tool call (structured output):

   { "tool": "read_file", "arguments": { "path": "main.py" } }

1. The runtime runs the function and gets the result.
2. The result is returned to the LLM as a new message.
3. The LLM reasons over the result and decides: call another tool, or respond to the user.

The critical part is step 3: when the LLM emits { "tool": "read_file", "arguments": { "path": "main.py" } }, that’s just text. It’s a request, not an execution. The LLM is saying, “I’d like someone to read this file for me”. It’s the same logic as the body of an HTTP response: inert text until something parses it and acts on it. The official OpenAI guide on function calling covers the protocol in detail.

Common tool types:

• Code execution: Python, bash commands, etc.
• File read/write: reading, writing, editing.
• Search: web, code, vector DB.
• APIs: calling external services (GitHub, Slack, databases).
• Browser: navigating websites, clicking, filling out forms.
• Communication: sending messages, creating pull requests, posting to Slack.

The runtime: the piece that connects the LLM to the real world

The runtime is the program that orchestrates everything. The LLM on its own can’t do anything except emit tokens: it doesn’t read files, it doesn’t make HTTP requests, it doesn’t run code. It’s a pure text-in, text-out function. The runtime is what sits in between and makes things happen.

What the runtime does:

┌────────────────────────────────────────────┐
│                  RUNTIME                   │
│                                            │
│  1. Builds the prompt (system + history    │
│     + tools)                               │
│  2. Sends it to the LLM API                │
│  3. Parses the LLM's response              │
│  4. If there's a tool call →               │
│     executes → gets the result             │
│  5. Adds the result to the conversation    │
│     history                                │
│  6. Goes back to step 2                    │
│  7. If there's a final response →          │
│     returns it to the user                 │
│                                            │
│  Also handles: auth, rate limits,          │
│  retries, context management,              │
│  guardrails, logging                       │
└────────────────────────────────────────────┘

A concrete example. Imagine the user asks “What’s in main.py?”. Here’s what happens at the code level:

# THIS IS THE RUNTIME (simplified)

conversation = [
    {"role": "system", "content": "You have the tool: read_file(path)"},
    {"role": "user", "content": "What's in main.py?"}
]

while True:
    # Step 1: call the LLM API (network request to Anthropic/OpenAI)
    response = client.messages.create(
        model="claude-sonnet-4-6",
        messages=conversation,
        tools=[read_file_schema]
    )

    # Step 2: parse what the LLM returned
    if response.has_tool_call:
        # The LLM said: "I want to call read_file with path='main.py'"
        # This is just JSON text. Nothing has been executed yet.

        tool_name = response.tool_call.name       # "read_file"
        tool_args = response.tool_call.arguments  # {"path": "main.py"}

        # Step 3: THE RUNTIME actually runs the function
        # This is where the real work happens; the LLM didn't do it.
        result = execute_tool(tool_name, tool_args)
        # → opens the file, reads bytes, returns a string

        # Step 4: return the result to the conversation
        conversation.append({"role": "assistant", "content": response})
        conversation.append({"role": "tool", "content": result})
        # The loop continues; the LLM will see the file contents
        # on the next call.

    elif response.has_text:
        # The LLM decided to reply to the user instead of calling another tool
        print(response.text)
        break  # Done

What runtime means concretely. It’s not an abstract concept; it’s an identifiable piece of software.

Agent	The runtime is…
Claude Code	The Claude Code CLI app (Node.js process on your machine)
ChatGPT with tools	OpenAI’s server-side orchestration layer
A LangGraph agent	Your Python script using the LangGraph library
A custom agent	The while loop you wrote around the API

The LLM never touches the filesystem, the network, or any real resource. It only sees text representations of the results that the runtime chooses to show it.

With these four pieces — what an agent is, the loop, the tools, and the runtime — you have the mental model to open up any agentic system: Claude Code, a custom agent you wrote yourself around the API, or a LangGraph system. In the second post in the series, I move up a layer: how an agent connects to the world through MCP, how reusable work is packaged into skills, and which guardrails keep an autonomous agent from becoming dangerous.