What is harness engineering (and why you already do it)

If you don’t build your software project with agents, skills, and prompts, it looks like engineering isn’t needed anymore.

DANIEL ALARCÓN
Software Architect

In any AI project that makes it to production there’s a part of the work that never shows up in the demos or the headlines: the part that holds the system together on the inside once it stops being a proof of concept. That part has a name of its own, harness engineering, and it tends to be sold as a discipline you have to learn from scratch, as if traditional software engineering had gone obsolete overnight. So what is harness engineering, really? Before assuming it’s brand new, it’s worth looking at what it actually consists of.

That’s why I ran a Knowledge Sharing session: to walk through some of the pieces that make up that harness, and to argue something that sounds far less flashy put that way — the engineering is still there underneath, and it hasn’t changed as much as it seems. What it has done is evolve, to make room for the new trade-offs AI brings, while the underlying judgement is the same one we were already applying before, in other contexts, under other names. The title I gave the talk, “Learning harness engineering (again?)”, already gave away where the conclusion was heading.

What is harness engineering in practice

The concept of harness engineering refers to everything that surrounds the model: the system that makes it reliable, auditable, and sustainable in production. Break it down and something unexpected shows up: we already knew how to do a good chunk of that work, because this exact effect — the promise that a technology removes complexity, followed by the appearance of new roles to manage the complexity that was in fact still there — has happened before.

The cloud is a good example: in 2010 the promise was that with AWS you wouldn’t need to know about infrastructure anymore, and a decade and a half later we have roles that didn’t exist before — SRE, DevOps, platform engineer, FinOps, DevSecOps, MLOps — because someone had to deal with the complexity the cloud was hiding underneath that promise. The same thing is happening with AI, and harness engineering is precisely that complexity.

Here are some of those classic pieces, the ones you’ve almost certainly been working with for a while without putting that label on them:

• The model as an API. Stripped down to the essentials, an LLM is an API that is slow — seconds, not milliseconds — expensive — cost depends on usage, not on reserved capacity — and non-deterministic, because the same question can come back with two different answers. Having all of that live in a single call forces you to dust off the patterns you’d use for any other unreliable external dependency: queues, rate limiting, streaming, and routing to a cheaper model when you don’t need the expensive one. The alternative — a frontend talking straight to the LLM, no backend, no limits — is the fastest way to get a nasty surprise on your bill.
• Cost management. We’ve had years of FinOps in the cloud, reviewing spend before deploying. What changes in an AI project is the unit you budget against — per feature or per agent, not per database or per compute instance — which leads to decisions like a switch that drops down to a cheaper model if a spike in requests threatens the month’s budget.
• Observability. Same logic as always, except that on top of the classic metrics — latency, errors, request count — you now add the tokens of each call and the trace of which tool the model called. Langfuse gives you exactly that traceability, designed for a non-deterministic system.
• Security. Prompt injection is the evolution of SQL injection — instructions sneak in through a badly sanitised input — and it’s worth going through the OWASP Top 10 for LLM applications to know which vectors to cover first. Data leaking out through a prompt is the same old scope problem: the attack vector is different, but the threat model behind it is the one we’ve always had.
• Testing. Still very much alive, except that a unit test is now called an eval: input-output pairs you score to spot when a change to the prompt makes the answers worse. Without evals you have a pretty demo that nobody can take to production with any confidence.

So far, the part of the harness you probably already knew, even if you never called it that. The part you genuinely do have to learn from scratch starts with something that sounds so simple it barely looks like engineering.

Chunking, vector stores and reranking: RAG as harness engineering applied

If you want to see that new part in its purest form, the best example is RAG: the “hello world” of any enterprise AI project. You drop a client’s documentation into a vector database and let them ask questions about it in natural language. Sounds trivial until the very first decision: how do you split the documents before indexing them?

1. Fixed size: you cut the document into equal pieces. Simple, but it can slice a paragraph in half and hand you back half an answer.
2. Semantic splitting: an LLM divides the content by topic. More coherent, but less predictable — some chunks end up being a paragraph and others five pages — and it costs you a model pass over every document before indexing.
3. By structure: you split on real paragraphs or headings. Works beautifully with Markdown and considerably worse with tables or unstructured formats.

No option is universally better. The smaller the chunk, the more precise the retrieval but the less context; the bigger the chunk, the other way round. Making that call for a specific client — knowing their documents, knowing what they’ll be asked — is engineering, however much it sounds like “chopping text into pieces”.

Then come more decisions of the same kind. Which vector database to use depends on whether you need something fast to validate an idea or something that scales from day one: standing up a proof of concept is not the same decision as building a system that will take real traffic from minute one.

There’s no single way to retrieve the information either: you can search by vector similarity, which is the default approach, or combine it with keyword search when the content is full of identifiers or acronyms that the embedding alone doesn’t tell apart well. And once you have your candidates, you still have to reorder them (reranking) so the most relevant ones reach the model first. Deep down, it’s the same conversation we were already having years ago when choosing between Postgres, MongoDB, or Cassandra, only now the vector is the new variable on the table.

MCP, agents and the problem of designing tools for a model

RAG isn’t the only new piece of the harness: designing tools for an agent via Model Context Protocol is another, and this is the one where I had to correct course more than once. A tool is the equivalent of an endpoint: you expose a piece of functionality for something else to use. The difference is that this “something else” decides whether to use it by interpreting its natural-language description — something that used to be settled with a plain technical contract.

It happened to me with a very specific case. I’d implemented a tool on an MCP server whose description said, literally, that it “rendered an image”:

tool: render_image
description: "Renders an image"

In my case, the agent called that tool, received the data and considered the task done without showing anything on screen. It took me a whole afternoon to work out why: the model understood that if the tool “renders”, someone else had already taken care of displaying it. The fix was to change the description.

tool: render_image
description: "Returns an already-rendered image, ready to paint in the chat"

With an LLM in the middle, a tool’s contract stops being merely technical and becomes semantic too. Before, if an endpoint’s description wasn’t perfect, the developer would manage by looking at the endpoint itself; with an agent’s tool you want to be far more precise, because if the model can read your description another way, it’s fairly likely it eventually will.

Before getting into agent design proper, it helps to be clear on what an AI agent is and how the loop that drives it works. With that out of the way: designing agents is a lot like designing a state machine. You have to decide how many times a call can be retried before you break the loop, what minimum model each specific agent needs — not all of them are equally good at inferring which tool to use on complex tasks, however well written the prompt is — and how you’re going to audit afterwards what it did.

When one agent orchestrates other subagents, the problem multiplies rather than adds up: every level of the hierarchy can fail on its own, and working out where the chain broke demands real traceability over what each level attempted, beyond a simple error log. All of this is protected with input and output guardrails, so you don’t let private data or malicious instructions through and don’t return something out of place, bearing in mind that every layer adds cost and latency.

The decisions that make you a harness engineer

There’s no degree or certification that turns you into a harness engineer: you become one the moment you start resolving this kind of dilemma with a real client, like these:

• Do I stand up my own model on a reserved machine, or use an external API and accept depending on a third party?
• Does a good prompt solve the problem, does it need RAG because the client wants answers anchored to their documentation, or does it call for fine-tuning?
• Do I attack the task with one big model that solves everything, or break it down into small pieces with a specific model for each?

Each of these questions hides a business decision as well as a technical one. Standing up your own model, for example, delays the launch but gives you total control over the data; leaning on an external API speeds it up in exchange for handing part of that control to a third party. That reading — what you’re willing to give up and in exchange for what — is what separates someone who simply wires up an API from someone doing harness engineering.

Vibecoding has lowered the barrier to entry for writing code. Confusing that accessibility with the disappearance of engineering gets expensive: the judgement about how to chunk documents for a RAG, the semantic design of a tool, or knowing when a small model is enough comes from having been through the problem before, often by way of mistakes like the tool one above.

The difficulty has shifted from writing code to making these decisions, and that’s still where you need someone with the judgement to make them. Deep down, it’s the same idea as the cloud parallel at the start: engineering hasn’t changed as much as it seems; it has evolved to make room for the trade-offs AI brings.

In the full recording of this session, I go into more detail on how we trace the cost and quality of an agent at Orbitant with tools like Langfuse or Mastra, and I answer live the question that generated the most debate in the team: whether the model matters as much as the tool description when it comes to deciding whether an agent uses it or not.