AI engineering · context systems · product design
Shared Memory Is the Next Team Interface for AI Coding Agents
Source: Boundary · Dex and Vaibhav · “Building a Shared Memory System for AI Coding Agents” · Video ID rTn8Vhdt-Jo
The hardest part of making AI coding agents useful across a team is no longer prompting. It is deciding which hard-won lessons should become shared context, who gets to approve them, and how to keep the system from turning into a junk drawer of stale advice.
Who Are Dex and Vaibhav?
Dex is building HumanLayer, a system for coordinating AI coding agents with human approval loops, design artifacts, diffs, and supervisor-style workflows. Vaibhav is building Boundary, described as “GitHub for design docs” and “the future of how software engineers collaborate on the production of software.” Their discussion matters because it is not abstract punditry about agent memory; it is a live product-design critique between two builders who spend their days shipping AI engineering tools.
Context Shards: Memory Without the Bad Habits
The feature under debate is called Context Shards: a HumanLayer system that watches agent sessions, finds knowledge teams keep teaching their agents, and promotes the best candidates into reusable context. Dex frames the goal plainly: “What are the things that people on your team are telling the agent over and over again?” Examples include repo-specific facts such as “you don’t have a main branch, Mary is main,” implementation preferences such as how to serialize from Rust data structures, or workflow norms that repeatedly show up in coding sessions.
Vaibhav’s first reaction gets to the core failure mode of memory systems. Claude “aggressively adds memories,” and once those memories exist, it tries to apply them in new codebases where they may be wrong. That is not merely annoying; it creates a dangerous sense of procedural certainty. A pattern that was correct in one worktree becomes an instruction in another. The system behaves as if repetition equals truth.
“What I really want is like one version of truth for my whole team.”
That line changes the product. Instead of treating memory as a personal scrapbook of everything an individual said to an agent, Context Shards becomes a pipeline for creating the right shared context. The hard problem is not storage. It is sourcing knowledge from smart people, distilling it, and making the distilled version useful to everyone without letting low-quality memories leak into every future session.
The User Does Not Care Whether It Is “Instruction” or “Information”
Dex initially separates memories into two categories: instructions, which steer behavior (“never do this,” “always run this command before implementing”), and information, which supplies background context (“a team over there owns this thing”). Vaibhav rejects the distinction as a product surface. As a user, he does not want to reason about whether a piece of context is an instruction or a fact. Both are simply things the agent may need to know.
That critique is important for anyone building agent infrastructure. Taxonomies that are useful to engineers often become friction for users. The person approving shared context wants to answer a simpler question: should this be available to the agent, for this team, under these conditions? The implementation can preserve richer types internally, but the interface should collapse the cognitive burden.
The Slop Problem: Purely Additive Memory Always Decays
The sharpest warning comes from Vaibhav’s experience with similar systems such as CodeRabbit. The first ten items look good. Two months later, the list has become purely additive: every plausible rule has been appended, almost nothing has been removed, and the system is bloated with guidance nobody trusts.
Dex’s proposed answer is staged adoption. A candidate shard can be discovered, reviewed, shared with the team, turned on or off, and eventually committed into the repo through a pull request when it is truly durable. That creates a ladder from ephemeral observation to canonical team knowledge. Vaibhav pushes for an even more opinionated version: one canonical team library, with personal enablement where necessary. His standard for repo-level material is high: “things in the repo must be extremely high standard.”
This is the central product insight: memory needs governance. Without human review, deletion, promotion, and scope control, it becomes a context landfill. With the right gates, it becomes a living team manual that agents can actually use.
Always-On Versus Task-Specific Context
The second axis is activation. Some shards should be global: every agent session should know them. Others should activate only when relevant to a task. Vaibhav accepts the distinction but immediately surfaces the engineering trade-off: if 60% of a team’s 100 shards are task-specific, the system must decide what applies before or during the main task. That decision can increase startup latency or inject stale context at the wrong time.
Dex’s answer is background inference. Instead of asking the user to choose relevant shards manually, the system can infer relevance from the research document, conversation, generated artifacts, or files being touched. Vaibhav points out that the classifier can be simple and fast, especially if it is running in parallel or asynchronously. The broader architectural principle is that context activation is itself a product surface: fast enough to disappear, accurate enough to trust, and cheap enough to run often.
Build a Supervisor, Not a Swarm
The architecture discussion demystifies the fashionable phrase “supervisor agent.” In HumanLayer, the supervisor is not a grand autonomous manager controlling a swarm. It is a set of focused inference pipelines that watch specific events. A post-tool-use hook may notice that an agent wrote a design document, parse front matter, ask a tiny model whether open questions remain, and then nudge the main loop only when useful.
For Context Shards, the same pattern applies. A supervisor whose mission is “help the user make progress” is different from one whose mission is “identify shared technical facts.” They may use the same model family, but they have different triggers, prompts, costs, and latency targets. Vaibhav frames the economics cleanly: if a main agent loop is worth $100, a supporting memory pipeline might be worth $10. It can add enormous value, but it should not cost as much as the primary work.
That suggests multiple execution modes. A nightly batch job can mine sessions slowly and cheaply. A Slack-triggered feedback flow may need to run instantly. A user saying a memory is good or bad can become an event that steers the supervisor toward a better candidate. The system improves not by remembering everything, but by learning from approval, rejection, and correction.
Product Discovery Before Product Construction
The Context Shards critique becomes a case study in building large AI features. Dex arrives with a detailed design, which lets Vaibhav challenge assumptions at the right level. Because many decisions are still cheap, the conversation can change the shape of the solution by 20% or 30% before implementation hardens around the wrong model.
Vaibhav calls out the workflow explicitly. One person should think deeply and write the doc. One or two collaborators should read it, then debate the user outcomes, probing points, and likely failure modes. Only after that should a trusted engineer—or an agent—implement. For a feature that may become 20,000 or 50,000 lines of code, spending 20 minutes to align on the product model is not overhead. It is leverage.
The prototype bar has also changed. Marty Cagan’s product discovery ladder used to run from napkin sketch to Figma mockup to high-fidelity prototype. With AI, Vaibhav says, “I don’t want to see a damn wireframe. I want to see the HTML mockup.” The modern discovery artifact should be concrete enough that customers can react to it, but cheap enough that the team can throw it away.
Launch Experimental AI Features Before They Are Perfect
Dex’s approach to experimental AI features is pragmatic: do not spend months building evals before anyone sees the system. First, run generations internally. Mark which candidates are good, which are bad, and which prompt changes improve the output. Over time, the team’s judgments become the eval set.
“Vibe based evals are 100% the way to go, man.”
That line sounds casual, but the underlying method is disciplined. Human review comes first because the team does not yet know the failure distribution. Once bad examples appear, they become test cases. Then the prompt or pipeline can be changed until the specific failure is fixed without regressing the other cases. For AI products where quality is partly subjective and partly contextual, the eval set is often discovered through use, not designed perfectly in advance.
A Practical Framework for Team Memory
| Decision | Good default | Failure if ignored |
|---|---|---|
| Source | Mine repeated human-agent interactions across the team. | Important local knowledge stays trapped in individual sessions. |
| Review | Require human approval before broad activation. | The system promotes accidental one-off advice into policy. |
| Scope | Prefer one canonical team library, with conditional activation. | Each user builds a private context universe that diverges from reality. |
| Promotion | Graduate durable shards into repo-backed docs or PRs. | Shared context never reaches the source of truth. |
| Deletion | Make dismissal, deny-listing, and correction first-class. | Memory becomes additive slop. |
| Triggers | Use cron, Slack/email feedback, post-tool hooks, and batch jobs based on latency needs. | Every improvement path is forced through one brittle workflow. |
Key Lessons
- Shared memory is a governance problem. The valuable unit is not “what did one user say?” but “what should the team and its agents rely on?”
- Users should approve meaning, not manage taxonomy. Instruction versus information may matter internally, but the product should ask whether context is useful, true, and appropriately scoped.
- Context quality decays without pruning. Dismissal, correction, and promotion are as important as capture.
- Supervisor agents are usually focused inference loops. A few tiny, well-triggered prompts can nudge a main agent more effectively than an expensive autonomous overseer.
- Prototype the product model before committing the implementation. AI makes HTML mockups, fake-data prototypes, and internal dogfooding cheap enough to use aggressively.
Why This Matters for Diffie
Diffie is building AI browser testing for frontend engineers, which means the product will repeatedly learn team-specific facts: preferred test selectors, design-system quirks, authentication flows, flaky routes, staging environments, accessibility standards, and “never do this in our app” rules. If those lessons stay inside one debugging session, Diffie will feel clever once and forgetful the next time. If they are automatically promoted without review, Diffie risks becoming noisy or confidently wrong.
The Context Shards lesson maps directly to Diffie’s GTM and product challenge. For an ICP of frontend teams, the wedge is not “AI remembers everything.” The sharper promise is: Diffie helps your team turn repeated browser-testing corrections into a reviewed, shared testing playbook that future runs can use. That is a concrete buyer story for engineering managers who care about consistency, onboarding, and reducing flaky QA labor.
A practical next move would be to design Diffie’s own memory ladder: candidate learning from failed or corrected test runs; human approval in a familiar inbox or PR-style review; team-level enablement; and eventual promotion into repo-visible test guidance. Keep the first milestone narrow. For example, mine repeated corrections around selectors and login flows, show them in an internal admin view, and let Anand approve which ones become team context. That would give Diffie a credible demo for ICP conversations: every manual correction makes the next automated browser test smarter, but only after the team decides it deserves to be remembered.