Version control for AI agents: why your agent’s brain should be an npm package
Any output from an AI agent can be corrected, but the hard part is proving months later exactly what knowledge produced it. That question also tends to arrive at the worst possible moment: when a client spots a mistake in a text published under their brand and wants to know what went wrong. If nobody can answer it with evidence, the conversation shifts from fixing the problem to arguing over whose fault it was. That’s why I think any company that lets an agent write on its behalf will need that evidence sooner or later. To me, versioning an agent’s knowledge is clearly an investment for everything it brings, and most of that comes down to control.
Version control for AI agents solves that problem: it turns an agent’s knowledge (its instructions, its judgment, what it can and can’t do) into something you can pin, audit and roll back like any production dependency. We’ve spent months applying it to employents, our AI agents built to work as teammates, starting with the one that works in marketing. In its case, that discipline means publishing everything it knows as an npm package.
Version control for AI agents: proving what your agent did and why
For a while, we kept our first employent’s knowledge as a copy inside the project’s own repository. A control file recorded which version that copy came from, but nobody checked that the record was still accurate. A record nobody verifies works like a decorative label, because it can’t prove which instructions the agent was following on the day something went wrong. Without that proof, every mistake can end up as an argument about responsibility when it should be a quick fix.
That proof matters more as the agent takes on more exposed work. An internal draft gets fixed quickly, while a text published under your brand stays out there until someone spots it. The more visible the agent’s work, the more it costs not to be able to explain where a mistake came from.
That’s why the team decided to publish that knowledge as an npm package pinned by a sha512 hash, a kind of fingerprint that identifies each version. If what gets downloaded doesn’t match that fingerprint, npm ci fails and the change never reaches any environment. On top of that, every update goes through a pull request that a person reviews and that won’t merge if the automated evals fail, so there’s a trail of who approved each change and why.
None of this is new in software development. Behind the decision sits the DRY principle (Don’t Repeat Yourself), which in engineering means you don’t solve again what’s already been solved. We’ve spent too long maturing engineering processes not to use them now, so instead of inventing our own system to govern the agent, we decided to apply that discipline to its knowledge, which until then moved around without any verification. For anyone deciding whether an agent can write on a company’s behalf, this change turns a question of trust into a check.
When a client asks how you audit an agent that writes on their behalf, the answer is a specific version of the knowledge, with the pull request that approved it and the fingerprint that pins it. That control has a cost, because every change waits for a review and a set of evals before it reaches production. In return, when something fails, the team knows which version to roll back and which differences to review.
New capabilities without writing code: when marginal cost tends to zero
Until recently, an agent’s capabilities lived hand-written in the code that runs it. To add a new content format, someone on the team had to build it, test it and deploy it, so every format went through development even when its content was already defined. That coupling turned every marketing idea into an engineering task.
We changed that logic so the code would discover capabilities from what was already published in the agent’s knowledge package. Since then, what the agent can do depends on what that package contains. The team designed 5 content formats as skills inside the package, and the agent could use them without anyone writing a line of code to wire them up:
- LinkedIn posts
- Newsletter
- YouTube video descriptions
- Blog article reviews
- Blog article translations
The first package took work: migrating the knowledge, verifying it, publishing it and pinning it by its fingerprint. The formats that followed needed design and review but not a single line of code, so the number worth measuring is the cost of each next capability. When that marginal cost tends to zero because the building blocks already exist, the conversation with the business side changes too. The only question left is whether the capability is worth reviewing before you switch it on.
For anyone planning, the order of priorities changes: first you decide what the agent should know, then you review what it can do with that knowledge. Every new format still goes through the same reviewed pull request and the same automated evals. The agent can propose a format, but responsibility for what gets published still sits with a person.
This mechanism pays off differently depending on the agent’s stage. With only a few skills published, discovering them from the package adds little, and most of the value still comes from coding them by hand. That’s why I wouldn’t recommend it as a first step for a team starting out with its first agent. The savings show up later, once the package has built up skills that can be reused across formats thanks to the same DRY principle we apply to the rest of our software.
Every rule in the wrong place has a measurable cost
The separation between an agent’s knowledge and the code that runs it is fragile, and it tends to break under time pressure. All it takes is someone adding a rule to the code that belongs in the knowledge, because at that moment it looks like the faster route. With each of those rules, the code ends up deciding something that was already written somewhere else. Nobody notices until the two instructions collide.
To stop the code from piling up those rules, we documented a boundary with a one-sentence test: if whoever wrote the knowledge could also have written that line of code, the line doesn’t belong there and has to go. Even so, our own team crossed that line 4 times, each time for a one-off reason that seemed sensible at the moment. That’s why the rule is written down: without it, those 4 cases would have slipped through as isolated decisions.
When 5 models from 2 providers fail in exactly the same way, the bug is in something we wrote.
FELIPE POLO
CEO Orbitant
The most serious case shows it well. The skill that writes blog articles asks for 3 to 5 external links per article, but the code wrapped around the agent had its own rule banning it from including any URL. The agent was getting two incompatible instructions and the code always won: across 12 test articles not a single link appeared, with 5 different models from 2 providers. When we removed that rule from the code, the next articles included 1, 5 and 8 real links.
The fact that such different models failed in exactly the same way pointed at our own code as the source of the bug. For the business, the lesson is that a misplaced rule can look like a model failure and push you to switch providers without fixing anything. Before blaming the model, it’s worth checking what the code we’ve written around it is stopping it from doing.
The document that sets this boundary records 2 exceptions, each with its reason. Key points don’t have a skill behind them yet, so the code temporarily does that job until that skill exists. Descriptions are also cut to their first paragraph because the full version broke a test case that used to pass every time: it went from 5 out of 5 to 0. They’re practical rules that can be revisited when the situation changes.
A dependency that fits in one file is a reversible decision
Every time we choose a piece of technology, we also choose how much it’ll cost us to walk away from it. With AI agent frameworks that question weighs more than usual, because the market moves very fast and today’s safe option can fall behind within a year. In an earlier instalment we explained why we designed our employents’ stack so no single provider owns it. Separating knowledge from code is the piece that has strengthened that decision the most, because today our dependency on the framework comes down to a single file in the project and replacing it would be a job limited to that file.
A decision like that is only reversible if you can check that the change doesn’t break anything. When we moved the agent’s knowledge into the package, the risk was that something would change along the way without anyone noticing. So we measured the migration before signing it off, and the published package turned out byte-for-byte identical to the previous version. For anyone who has to approve a change like this, that check turns a bet into a verifiable formality.
The same logic applies when you choose who to hire. If I had to evaluate an AI agent provider, before talking about the model I’d ask them to prove three specific things:
- Who approved each change to what the agent knows, and when.
- What happens when a change makes results worse, and which tests stop it before it reaches production.
- How much of their system depends on a specific framework, and what it would cost to replace it.
This level of rigor has a cost and only pays off from a certain stage. I think that if your agent hasn’t left an internal environment yet and nobody else depends on its results, you probably don’t need half of this work. On the other hand, the investment in verification pays for itself as soon as a mistake reaches a client.
Other teams are working on how to package and distribute what an agent uses: Anthropic, for example, documented its own standard for connecting AI systems to tools and data. Version control for AI agents applies to an agent’s knowledge what the industry has been doing for years with every other software dependency. As more companies let an agent speak on their behalf, I think that discipline will end up as part of purchasing requirements.
My recommendation comes down to a single question. The next time someone pitches you an AI agent provider, before looking at the model they use, ask whether 6 months from now they could tell you exactly which version of the knowledge it was working with on the day an output went wrong.
