This Code Is Not for You (Human): Designing Software for Untrusted Builders
Why AI-assisted development forces discipline, context, and reasoning into the architecture itself
Most changes in software feel important at the time, but very few actually change how engineers think. New languages, new frameworks, new tooling — these tend to layer on top of an existing mental model rather than replace it.
There have been only a handful of moments when that mental model itself had to change. One of the last truly fundamental ones was the move from monolithic applications to distributed systems, microservices, and cloud-native architectures.
That shift wasn’t just about infrastructure. It forced a different definition of correctness.
From the Happy Path to “Everything Fails”
Prior to the early-2000s, most engineers designed software around the happy path, plus a limited set of known failure modes. You handled the errors you had personally encountered: invalid input, missing files, a database being temporarily unavailable…but not the ones you hadn’t. The underlying assumption was still that systems mostly worked, and that failures were exceptional.
Distributed systems broke that assumption.
Once systems were decomposed into independently deployed services, failure stopped being exotic and became constant. Networks dropped packets. Dependencies stalled. Services partially succeeded. Entire regions disappeared. None of this was surprising once you lived through it, but it required a major rewiring of how engineers reasoned about software.
Amazon’s early-to-mid-2000s push away from a central monolith into a service-oriented architecture made this impossible to ignore. That transition wasn’t merely technical; it forced a wholesale retraining of how engineers thought about dependency, ordering, and control. You had to assume that anything you depend on can fail at any moment, and design systems that continued to function anyway.
That shift was hard. Harder than people remember. Over time, patterns emerged (retries, idempotency, durable queues, workflow engines) and eventually platforms and managed services absorbed much of the operational burden. The mindset shift remained, but many of the sharp edges moved out of application code and into infrastructure.
The worldview changed, and it stuck.
A Different Kind of Breakdown Is Happening Now
I think we’re in the middle of another shift of similar magnitude. This shift, however, isn’t driven by infrastructure failures; it’s driven by who is writing the code.
Historically, software teams relied on more than documentation and best practices to stay aligned. They relied on consequences.
If someone repeatedly violated core invariants, ignored established constraints, or kept reintroducing the same class of problems, there was a feedback loop. Sometimes it was gentle: code review comments, one-on-one conversations, mentorship. Sometimes it was not. People were told clearly that certain behavior wasn’t acceptable. And if they didn’t adapt, there was a chance the carrot turned into a stick.
It’s why RFC-style language (MUST, SHOULD, MUST NOT) worked as well as it did. Those words carried weight not because of typography, but because everyone understood there were real consequences behind them. Humans remembered that. They internalized it. Over time, they learned where the hard boundaries were, even when those boundaries weren’t enforced directly by code.
AI coding agents do not participate in that social contract.
They don’t feel pressure. They don’t fear repercussions. They don’t remember being corrected last week or last year. No amount of prompting or carefully curated AGENTS.md files change that fundamental reality. You can influence their behavior, sometimes significantly, but you cannot rely on long-term discipline the same way you can with human teammates.
Even when rules are written down — RFC-style MUSTs and SHOULDs — they don’t carry the same force. To a coding agent, those words are signals, not constraints. They may be followed most of the time, but you must expect that they will not always be followed.
Not because the agent is “bad,” but because it has no persistent internal model of consequences.
When Code Is Locally Correct and Systemically Wrong
The failure mode this introduces isn’t broken syntax or failing tests. It’s systems that are locally correct and globally wrong; changes that make sense in isolation but quietly violate assumptions the system depends on to remain stable.
Many real-world systems work today because of invisible invariants. X must happen before Y. Z only works because Q guarantees durability. This component exists solely to prevent a rare but catastrophic issue. Often, these constraints are not enforced by code; they live in people’s heads and in shared team memory.
When humans work in a system long enough, they absorb these rules almost subconsciously. They learn what not to touch. They know where the sharp edges are.
AI agents don’t have that intuition. They see local opportunities for improvement and act on them, even when those changes violate global assumptions. The result is code that looks reasonable, tests that pass, and systems that quietly drift toward fragility.
Just as early distributed systems failed because engineers assumed the network was reliable, many modern systems are beginning to fail because they assume the builder has full context.
In both cases, the assumption was invisible — until it wasn’t.
Encoding Discipline Into the System
In the past, a system could rely on human discipline to preserve its invariants. In an AI-assisted world, that discipline must be encoded into the system itself.
This changes how I think about documentation and code comments. It’s no longer enough to explain what a piece of code does, or even why it exists. Increasingly, the most valuable documentation is about constraints — the boundaries within which future changes must operate.
This is different from intent documentation.
Intent explains purpose. Constraint documentation explains reasoning. It captures which assumptions must remain true, which invariants the system relies on, and which classes of changes are safe versus dangerous, even if they appear reasonable in isolation.
This kind of documentation isn’t about preserving the past. It’s about enabling safe evolution — especially when the person or agent making the change doesn’t have years of historical context.
Shared Context Becomes a First-Class System
Structural decomposition alone isn’t enough. One of the things humans have always relied on, often implicitly, is shared context that exists outside the codebase. Decisions made in meetings. Tradeoffs discussed verbally. Nuances that never quite make it into a pull request description.
When development is primarily human, that context spreads organically. When agents are involved, it does not.
This is where new mechanisms and tools matter. Systems need ways to capture and share context from both humans and agents. Not just what was decided, but why it was decided, what alternatives were rejected, and which future assumptions were being made at the time.
That shared context feeds better reasoning. It helps agents understand team expectations, architectural direction, and operational boundaries. And it helps humans reason about agent-generated changes with more confidence and less guesswork.
The Real Skill Shift: Conceptual Clarity
What this ultimately demands is a higher level of conceptual clarity than many teams are accustomed to.
Not just clean code or clever abstractions, but systems that can be reasoned about — by humans and machines — without years of tribal knowledge. Systems where relationships, failure modes, and invariants are evident from the structure itself, and reinforced by shared context that lives alongside the code.
You should be able to walk up to a system you’ve never seen before and form a reasonably accurate mental model of how it works and how it fails. Making a local change should feel safe, not because you’re confident in yourself, but because the system — and its surrounding context — are designed to guide you away from dangerous mistakes.
Historically, this kind of clarity was often a luxury for many teams. In an AI-assisted development world, it becomes a prerequisite.
A Trade-Wide Adjustment, Not a Silver Bullet
Just like the move to distributed systems required both new architectures and new tooling, this shift will require multiple changes working together.
Systems need to be decomposed in ways that reduce the blast radius of misunderstanding. Invariants need to be enforced structurally where possible. And teams need better ways to share forward-looking context — decisions, expectations, and reasoning — across both humans and agents. At the same time, individual engineers have to change how they approach their own work: relying less on personal context and intuition, and more on making assumptions, constraints, and reasoning explicit in the artifacts they leave behind.
None of these alone is sufficient. Architecture, tooling, and individual behavior have to evolve together. Together, they form a new way of building software.
The shift is already happening, whether teams and individuals acknowledge it or not. The question is not whether AI will be involved in construction, but whether our systems — technical and social — are designed to deliberately support that reality.