AI second brain: memory, agents, and scale from Passwd’s CEO system

Every time I opened a conversation with Claude, I had to explain from scratch who I am, what my company does, and who it works with. It was exhausting. So I decided the problem wasn’t the model: it was memory.

MAREK ELZNIC
CEO de Passwd

Marek Elznic has spent months using an AI second brain that combines Obsidian, Claude, and his work sources —Gmail, Slack, meeting transcripts— so that his assistant doesn’t start every conversation from zero.

He calls it his second brain, though I’d describe it differently: a structured external memory layer for an agent, and that distinction completely changes how you assess when it makes sense to build something like this. Marek presented it at one of our Knowledge Sharing sessions, and what follows isn’t a summary of what he explained, but the questions I was left with afterward.

What Marek built and how it works

The architecture is simple; it consists of just three pieces:

• Obsidian as the vault. A set of Markdown notes, linked to each other, stored locally and synced through iCloud. People, companies, meetings, active signals. The graph of connections between notes is what turns a flat file into something an agent can navigate.
• Claude as the engine. Not as a chat interface, but as a process that has access to that vault and can read it, update it, and link new content. The key is that it works with local Markdown files, with no third-party integrations or services in the middle, which simplifies the architecture and keeps the operating cost low in everyday use.
• The sources as a continuous signal. Gmail and Slack, mainly. Every night, Claude goes through those sources, updates the relevant notes, links people to projects and meetings, and generates a briefing for the next day: upcoming meetings, questions worth asking, tasks that might have slipped through.

What it solves is concrete: Marek doesn’t have to explain the context every time he opens a conversation. The agent already knows —to give an example— who the head of marketing is, that they’ve been working together for five years, and that the tone Marek uses with them is different from the one he uses with a new lead. In practice, that changes the quality of the output.

The Passwd team itself published an article about the system that captures the core logic well: knowledge only becomes valuable through the connections that accumulate over time, and most systems collapse before reaching that point because the maintenance becomes unsustainable. Delegating that layer to the agent is what makes the compounding effect achievable.

Andrej Karpathy described this same approach publicly: index raw sources —articles, papers, repos, images— and use an LLM to incrementally “compile” a wiki, which is nothing more than a collection of Markdown files with summaries, backlinks, and linked articles that the model itself writes and maintains, all visualized in Obsidian. Marek has landed this pattern in a real workflow —email, Slack, meetings— and sustained it over time, which is exactly where most of these systems fall apart.

AI second brain: when this pattern makes sense versus native context

The question I asked myself when I saw Marek’s system was: when does this pattern make sense, and when is the model’s native context enough?

The answer depends on three variables:

• Context persistence. If the agent needs to remember information across different sessions —people, past decisions, interaction history—, native context isn’t enough. The vault pattern gives you explicit control over what gets remembered and how.
• Volume and structure of the knowledge. Marek’s vault accumulates months of notes about people, companies, and projects. Putting all of that into every conversation as context would be prohibitive in cost and latency. The advantage of the vault is that the agent retrieves only what it needs at any given moment, like a database, not like an attached document.
• Update frequency. If the context changes little —a project with stable requirements, a well-documented domain—, it’s enough to pass it as context at the start. If it changes every day —interactions, decisions, business signals—, you need an automated update process. Marek’s nightly agent is exactly that: a synchronization process that keeps the memory up to date without manual intervention.

In client projects, the threshold for building this pattern appears when the agent has to operate on organizational knowledge that changes frequently, doesn’t fit in the context of a single conversation, and needs to persist across sessions. Below that threshold, native context is simpler to maintain and has fewer points of failure.

What changes when you stop being an individual user

Marek’s system is optimized for a single person. That’s not a criticism: it’s the right context for what he built. But when we think about replicating this pattern at team scale —or organization scale—, the architecture changes in non-trivial ways.

The first change is vault ownership. In the personal system, the vault is the representation of Marek’s context: how he thinks, who he works with, how he writes. At team scale, the vault has to represent shared knowledge —architecture decisions, client context, project history— without becoming anyone’s personal profile. That implies a different data model and, above all, a clear policy on what goes in and what doesn’t.

The second change is concurrent writes. A nightly agent updating a vault for one person is trivial to coordinate. Several agents updating a shared vault in parallel —or worse, one agent receiving signals from multiple users simultaneously— introduces consistency problems that don’t exist in the personal case.

The third is source governance. Marek connects his own Slack channels and his own Gmail. At organization scale, the sources are team conversations, internal documentation, tickets, PRs. Deciding what gets indexed, who has access to which fragment of the vault, and how sensitive information is handled stops being a personal preference and becomes a data architecture problem with legal and privacy implications.

The pattern is still valid. What changes is that the corporate version requires a redesign from the data layer up, not simply scaling the personal system.

What to keep in mind before taking it to production

You may be thinking of something Marek named explicitly: this system will be obsolete within a few years. The models themselves will solve the memory problem, and when that happens, building and maintaining an external vault will stop making sense. He built it because today the native solution doesn’t offer enough control, not because it’s the ideal long-term architecture.

It’s the right starting point for assessing the limitations that matter when you think about this for a client.

• Startup cost. The initial seeding phase —indexing all the existing knowledge and building the vault from scratch— is expensive in tokens and time. In personal use, you take it on once. In a corporate deployment, that cost multiplies and has to be planned for.
• Vault quality depends on source quality. If Slack is full of noise, if meetings aren’t transcribed well, if reference documents are out of date, the agent builds memory on poor signal. Garbage in, garbage out, with the added twist that here the garbage persists and propagates through the memory.
• Accumulated opacity. A vault that grows for months without manual review can develop inconsistencies that the agent perpetuates. Marek mentioned orphan files —notes with no connections— as a visible sign of that problem. In production, you need a vault auditing process, not just an update process.
• Reliability of the nightly agent. The whole system assumes that process runs reliably. If it fails one night, the memory goes stale. In production, that process needs observability, alerts, and error handling like any other critical pipeline.

The difference between Marek’s prototype and a production-ready version isn’t in the idea, which is solid, but in all the layers that make a system maintainable, observable, and scalable over time. Building those layers is the engineering work that turns a personal experiment into a tool an organization can rely on.