The Review Boundary Has Moved

September was again a wild month for discussions about how AI is changing software development. DHH’s “pencils down” position and Google’s use of Gemini 4 Argon to migrate large C and C++ systems to Rust reinforce the idea that code generation may have crossed yet another threshold.

DHH’s position is provocative in that at 37signals, handwritten code is treated as an exceptional state, and he describes his own role as moving away from professional software writing toward directing agents and reasoning at a higher level.¹ Google announced that argon agents are working on C/C++ to Rust migrations ranging from tens of thousands of lines in re2 and libgav1 to more than 800,000 lines for Fuchsia’s Zircon kernel, while Google explicitly says those rewrites undergo automated and manual auditing, emulation testing, and review before production rollout. In libgav1, an agent replaced 32,000 lines of SIMD code through repeated profile-guided experiments and produced safe Rust with identical video output while substantially improving performance.²

This begs the question of what we need to understand, inspect, or independently establish once agents generate all code. We can see arguments that unreviewed generated code is dangerous because nobody has reasoned through it. The counterargument is typically that reading source is neither necessary nor sufficient for understanding behavior. We already rely on enormous amounts of software we have never read, and agents make it cheap to generate fuzzers, property tests, traces, adversarial inputs, and empirical probes that may reveal more about actual behavior than careful source reading does.³ Source review has always been a means to an end, and that end is justified confidence that the system has the properties we rely on.

Source code is or was a trust interface

We routinely depend on code we do not understand at source level. Few application engineers read their compiler, database engine, TLS implementation, operating-system scheduler, filesystem, or every transitive dependency before relying on them. We tolerate the opacity because other mechanisms establish confidence for us. Narrow interfaces, specifications, test suites, years of production exposure, independent implementations, static guarantees, observed failure behavior, and large populations of users exercising cases we will never encounter ourselves.

We normally do not inspect generated assembly because the compiler is treated as a sufficiently trustworthy transformation beneath an abstraction boundary. That trust comes from an ecosystem of constraints and evidence surrounding the transformation.

Agent-generated application code may move in the same direction. If we can define sufficiently strong properties around a component, independently test those properties, and cheaply regenerate the implementation when they fail, human familiarity with its internal source becomes less valuable. The implementation becomes closer to compiler output. Important because it executes, but not necessarily the most useful level at which to reason about the system.

Google’s Rust work is a case where the old implementation constrains expected behavior, the Rust compiler constrains a class of memory and ownership errors, output equivalence constrains semantics, benchmarks constrain performance, and emulation and auditing provide additional checks.² In such cases, the transformation is surrounded by unusually strong external evidence. Sufficiently dense oracles can move the review boundary upward. We may not need to inspect the implementation if we can establish the properties that matter without doing so.

Deciding what needs to be true

An implication is that engineering judgement is relocated under this new model. But verification instead of code review may be incomplete advice. Property tests, fuzzing, runtime traces, differential testing, and static checks are powerful once we know which properties deserve protection, but they cannot by themselves determine whether the specification omitted important constraints. Several studies illustrate the point that we can generate many tests against the wrong model of the system.

SWE-RPG decomposes repository work rather than treating successful code generation as one indivisible capability. Across 163 tasks, the evaluated agents resolved only 31.5% on average, and recovery of implicit requirements was the largest diagnosed bottleneck, implicated in 24.5–46.0% of runs.⁴ Failure typically occurred before implementation was fully specified.

DENSO’s industrial requirements study points in the same direction from outside ordinary repository benchmarks. On 53 verification tasks that rely heavily on expert knowledge, an agent without domain context found 47% of the issues identified by humans. Supplying situational, conceptual, and procedural domain knowledge raised coverage to 74% and reduced false alarms by 42%.⁵ The model did not primarily need more syntax knowledge or a better ability to produce code but needed access to information that determined which conditions were consequential.

CodeTaste shows that agents perform substantially better when told which refactoring to perform than when asked to discover the structural change chosen by human maintainers. A propose-then-implement decomposition improves results further.⁶ Once the intended structural transformation has been selected, agents are comparatively good at executing it. Selecting the transformation remains harder.

A scarce step therefore may actually be question selection, meaning deciding what must remain true, what can safely vary, which failure modes deserve explicit treatment, and which uncertainty is important enough to stop the implementation until it is resolved.

LLM Wiki’s and other Context Solution Ideas

It is tempting to treat this as a temporary context problem. If the agent does not know why a strange branch exists, index the incident history. If it lacks product context, record every meeting. If requirements are implicit, put more repository instructions into AGENTS.md.

However, the evidence so far argues against such a simple scaling story. The ETH Zürich study of repository-level context files found no overall improvement in task success from AGENTS.md files while inference cost increased by more than 20%. Agents did follow the instructions and explore more broadly, but unnecessary requirements made some tasks more difficult. The authors concluded that human-written context should be minimal and relevant.⁷

SWE-Explore shows that access to the repository is not the same thing as selecting the small portion that matters for the current decision. Modern agents already perform relatively well at finding relevant files, while line-level coverage and efficient ranking remain much more discriminative. Those exploration metrics also track downstream repair success.⁸

Recently, organizations attempt to digitize more of themselves with access to Slack, complete archive of messages, incidents, documents, traces, and decisions, all of which increases the amount of potentially relevant evidence faster than it increases the agent’s ability to know which evidence should dominate a particular decision. A larger corpus may contain the answer while making the selection problem harder.

Humans have a significant advantage because they often carry a continuously updated model of relevance. Which customer interaction changed our assumptions, which old incident is analogous, which supposedly authoritative document is obsolete, which undocumented dependency is politically impossible to change, and which concern can safely be ignored. Digitizing the evidence is only one prerequisite for removing it as it doesn’t automatically confer understanding or priority. An agent would have to establish relevance, temporal validity, authority, causal relationships, and the consequences of being wrong.

Endogenous Evidence Selection

There is another problem with replacing review by automated verification in that verification can become circular. A recent paper Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians is not even about software engineering, but describing a mechanism that is directly relevant. The authors construct a model in which a chatbot can drive a rational user toward an incorrect belief without fabricating evidence. It is enough for the chatbot to preferentially select true observations that support the user’s existing belief. Informing the user that the chatbot may behave sycophantically reduces but does not eliminate the effect.⁹ We can call this endogenous evidence selection. Truthfulness of individual observations does not guarantee that the selected evidence forms an unbiased basis for inference.

An agentic development loop can easily reproduce this. One model writes the implementation, writes the tests, chooses what to fuzz, selects the resulting traces, summarizes the diff, retrieves the architectural history, runs a second “review” prompt, and presents the human with a concise explanation of why all the changes are sound. Nothing in that chain requires an explicit falsehood for the overall evidence stream to become self-confirming. This adds a second dimension to oracle density we could call oracle independence.

Tests inherited from an existing product are more independent than tests invented after seeing the generated implementation. A compiler constraint is much more independent than an agent’s explanation that its ownership model is safe. Differential behavior against an old implementation is more independent than a self-review. Production telemetry specified before the change is stronger than metrics selected afterward because they make the new implementation look good. Two agents running the same model, sharing the same assumptions, and deriving tests from the same implementation may provide less epistemic independence than one deterministic checker designed around an externally specified invariant. As more review becomes automated, the architecture of the evidence pipeline becomes part of the architecture of the software.

Maintainability

If we stop requiring humans to understand every generated implementation, maintainability may need a new definition. Readability, so an argument, may no longer be the primary property we are trying to preserve. A component that is ugly internally but cheaply regenerable, strongly bounded, independently verified, and easy to replace may be less problematic than beautiful code whose behavior depends on subtle interactions across half the system.

A property that survives both human-written and agent-written software is cheap local reasoning. We want to modify one part of a system without reconstructing the whole system. That requires stable interfaces, clear state ownership, bounded side effects, explicit trust transitions, observable failure modes, and enough independent checks that untouched components can remain black boxes.

Long-horizon benchmarks report a steep decline when agents move from isolated tasks to continuous evolution of the same software, with the original evaluation dropping from above 80% on isolated tasks to at most 38% in continuous settings.¹⁰ SlopCodeBench allows agents to make their own architectural decisions while repeatedly extending their previous solutions. Structural erosion rises in 77% of trajectories and verbosity in 75.5%, despite intermediate checkpoints continuing to pass.¹¹ WideSWE extends the problem across repository boundaries. On 120 real cross-repository tasks, full success ranges from 10.83% to 42.50% across seven configurations. Failure often comes not from an inability to edit code but from failing to identify all the places where coordinated changes are required or completing only part of the global change.¹²

These results show that performance degrades as more correctness resides in relationships outside the local implementation, which gives simplicity a more precise role in an agentic codebases. Simplicity tends to reduces the number of relationships that need to be rediscovered during each change. An interface is not merely easier for a human to read but reduces the state space an agent must reconstruct. Explicit ownership removes possible interpretations. Fewer abstractions reduce the number of interactions that can become inconsistent. Strong invariants turn contextual knowledge into something mechanically enforceable. Such benefits survive even if no human ever reads the implementation.

Review at the boundaries

Our most expensive review should increasingly happen at the points where independent evidence becomes weak. Inside a component with a good contract, tests, deterministic checks, narrow state, and low blast radius, line-by-line human review may add little. At the boundary between two systems with different ownership, different failure semantics, partial observability, and an implicit business contract, the implementation can be trivial while the engineering problem remains difficult.

The same applies to dataflow. We care where information becomes authoritative, where it is transformed irreversibly, where identities or permissions change, where an operation crosses a trust boundary, where retries become externally visible, where ordering assumptions appear, and where one system’s notion of success differs from another’s.

A local code agent can inspect both sides of an API and still miss that one side is operated by another team, has a six-month migration window, is contractually constrained, or contains e.g. intentionally undocumented exception for a particular customer. In such cases, uncertainty is not stored in the function body.

Humans and agents jointly define the claims a system must satisfy, implementation agents produce candidate systems, independent mechanisms test those claims, and review concentrates on the claims that remain poorly specified or poorly observable. Some of those reviewers will also be agents. We now care about their informational independence from the process that generated the solution.

Review objects become claims

The frontier is moving from implementation toward specification of consequential properties and construction of trustworthy evidence.

We reason if output remains behaviorally equivalent to the previous system, if a component cannot mutate state outside its ownership boundary, if an operation is idempotent under the retry model we actually use, if a new implementation satisfies the latency envelope under production load, if a permission transition cannot increase authority, if a protocol change remains compatible with these consumers, and if invariants still hold after the migration. Each claim should ideally be connected to evidence that exists independently of the implementation. A compiler rule, a model checker, an old implementation, a contract test maintained by another system, a production trace, a property defined before generation, a security boundary, a benchmark, a human domain expert, or some combination of them.

That is a more scalable object for both humans and agents to inspect than arbitrary generated source. It also makes clear where we still have a hard problem. Before we can prove a claim, somebody has to decide that the claim matters. SWE-RPG’s implicit requirements, DENSO’s tacit domain knowledge, CodeTaste’s refactoring-selection gap, the mixed results from repository context files, and the cross-repository failures in WideSWE are different manifestations of that same unresolved step.⁴⁵⁶⁷¹²

We may eventually stop reading most generated code, just as we stopped reading most compiler output. That does not make software a black box in the careless sense, but changes what has to remain transparent.

We need to know what systems promise, where those promises stop, which evidence justifies relying on them, and whether the evidence was produced independently enough to deserve our confidence. The implementation underneath those boundaries can become increasingly machine-owned.