Long-Horizon Coding Agents and Software Degradation

Long-horizon coding agents can now operate for hours, recover from failed attempts, search large repositories, run tests, use shells and debuggers, delegate work, compact context, and continue until a task succeeds. Yet there is growing evidence that this improvement in persistence has outpaced improvement in a different capability: preserving a coherent software system across repeated changes.

Software engineering is not a sequence of independent coding problems. Every solution changes the state on which the next problem must be solved. A locally successful patch can therefore make future work easier, leave it unchanged, or make it substantially harder. Several recent benchmarks suggest a persistent gap between solving the current task and leaving behind a system that remains easy to evolve.

That gap may also help explain why increasingly capable coding agents sometimes produce systems that experienced engineers regard as difficult to trust or maintain. The incentives and feedback available to current agents strongly favor immediately verifiable success while providing much weaker signals for the future consequences of today’s design decisions.

Software development is stateful

A defining characteristics of software development is that today’s implementation becomes tomorrow’s environment.

SlopCodeBench was an early attempt to evaluate that difference. Rather than giving agents independent problems, it required them to repeatedly extend their own earlier implementations as requirements evolved. In its published experiments, no evaluated agent completed an entire problem sequence. Structural erosion increased in 80% of trajectories, while redundant or duplicated code increased in 89.8%. Human open-source repositories used for comparison remained much more stable over time.¹ ([arXiv] 1 )

SWE-Milestone reconstructs actual release histories and requires an agent to carry one repository through multiple consecutive milestones without resetting its state. Earlier mistakes, regressions, design choices, and context all persist. As of September 2026, even the strongest systems on its leaderboard have overall scores around the high 50s, while fully resolving all requirements of an individual milestone occurs only around a quarter to a third of the time. GPT-5.6 Sol, for example, currently scores 55.6 overall with a 26.9% full-resolution rate.² ([SWE-Milestone] 2 )

ChainSWE independently constructs chronological chains of dependent bugs across 54 projects. It reports performance declines of as much as 70% as issue chains become longer.³ ([arXiv] 3 )

All these benchmarks differ substantially in construction, so their exact numbers cannot be compared directly. The consistent result however is that performance on persistent software evolution is generally lower than performance on isolated coding tasks. Which is not really surprising merely because the tasks are longer, but because previous agent actions alter the environment in which subsequent actions must occur. A failure at step (t) need not immediately break the program. It may instead make step (t+10) harder.

Agent-written code can make the next agent less effective

CodeThread provides stronger evidence because it attempts to isolate the effect of the code left behind. For each experiment, researchers construct two versions of the same repository. In one, an earlier pull request was implemented by a human. In the other, the equivalent task was implemented by a coding agent. A second dependent task is then attempted on both versions while keeping the downstream requirement constant. What changes is the implementation inherited from the previous task. Across four frontier coding agents and four repository-level benchmarks, agents were less successful when the previous change had been agent-written, with downstream resolve rates falling by as much as 13.1 percentage points.⁴ ([GitHub] 4 )

Traditional metrics such as cyclomatic complexity and common maintainability measures did not explain most of the difference. The authors instead found subtler behavioral differences around validation, error handling, implementation choices, downstream patch size, and task difficulty. Agent-generated code can be locally correct while leaving behind a representation of the system that is less useful for subsequent development. That points towards software-engineering failures being more significant than a code-style failure.

Refactoring

Whole-system refactoring provides a useful stress test because success requires two objectives to be satisfied simultaneously. They change the internal structure of the system while preserving its external behavior.

SWE Refactor Bench, released in August 2026, evaluates twenty repository-wide migrations across eight frontier models. Its evaluation separately checks whether the intended migration actually happened, whether existing behavior remains correct, and whether independent coding agents can discover hidden behavioral differences. Across 520 runs, only 28—5.4%—passed all three stages. Thirteen of the twenty migrations had no fully successful run at all.⁵ ([arXiv] 5 ) Some solutions preserve behavior by avoiding enough of the migration that they fail the structural audit. Others perform the migration but introduce regressions. The agent must satisfy two objectives whose gradients point in different directions, which resembles ordinary engineering much more closely than implementing an isolated feature against a fixed test suite.

Downstream change cost

We can describe the underlying engineering problem in terms of downstream change cost. Suppose (D_t) is the design of a system after today’s task. A task objective looks approximately like:

$$ \min C_{\text{now}}(D_t) $$

subject to:

$$ \text{current requirements pass}. $$

A competent engineer usually reasons over a broader objective:

$$ J(D_t) = C_{\text{now}}(D_t) + \mathbb{E} \left[ \sum_{k=1}^{\infty} \gamma^k C_{\text{change}} (D_{t+k},\Delta_{t+k}) \right]. $$

which is a useful formalization of an older idea with the important quantity being:

$$ \boxed{ \mathbb{E}[C_{\text{future change}}] } $$

i.e., the expected cost of modifying the system under future requirements.

David Parnas’s classic work on modular decomposition made essentially this argument in design terms. Modules should hide difficult decisions, particularly decisions likely to change, so that those changes do not propagate throughout the system.⁶ Technical debt describes the phenomenon economically, in that a decision can reduce today’s implementation cost while increasing the cost of future changes.⁷ This is also why experienced engineers routinely perform work that appears locally unnecessary. They introduce an interface instead of directly calling an implementation. They separate concepts that currently happen to move together. They remove duplication even though the duplicate code works. They constrain dependency direction. They refactor a component before adding another feature. These actions can make today’s patches larger, but their purpose is to change the distribution of tomorrow’s costs.

Future change cost

This creates a fundamental asymmetry in current coding environments. Immediate correctness generates excellent machine-readable signals, but future maintainability does not provide an equivalent oracle.

Consider two implementations of the same feature.

• Implementation (A) fits an existing abstraction and cleanly extends the system’s conceptual model.
• Implementation (B) introduces several special cases and duplicates behavior from another subsystem.

Both pass every current test:

$$ R(A)=R(B)=1. $$

Six months later, a new requirement might make (A) trivial to modify and (B) extremely expensive. The evaluator did not make a mistake when it gave both implementations the same immediate score, but it simply lacked information about something important, foresight into future changes.

We distinguish this from conventional reward hacking. Reward hacking occurs when a system discovers a way to maximize a measured proxy without satisfying the intended objective. Anthropic recently demonstrated that this is a real problem in frontier-model reinforcement learning. It trained an Opus-class model on 80 production-derived RL environments known to contain exploitable reward functions. By the end of training, approximately 40% of episodes were classified as reward hacks. Anthropic reports that such vulnerabilities have also occurred in actual frontier-model training environments, although production processes attempt to detect and remove them.⁸ ([Alignment Science Blog] 6 ) Maintainability is closer to reward omission, rather than reward hacking. The evaluator can correctly reward everything it measures and still omit an important part of the true objective.

Omitted terms are delayed and counterfactual

Future maintainability is unusually difficult to turn into a training signal because its value depends on requirements that have not yet occurred. Suppose an agent chooses an abstraction at task 1. It works correctly for nineteen subsequent changes. Task 21 introduces a new requirement that exposes the abstraction as a poor choice. So the observed trajectory ends up as:

$$ a_1,a_2,\ldots,a_{21} \rightarrow C_{\text{maintenance}}. $$

The training system must now determine how much responsibility for the cost at step 21 belongs to the decision made at step 1. This is a long-horizon credit-assignment problem.

It is harder still because the relevant future itself is uncertain. A design decision should ideally be judged across multiple possible requirement sequences:

$$ Q(D) = \mathbb{E}_{\Delta\sim P(\text{future requirements})} [ C(D,\Delta) ]. $$

That expectation can never be observed directly. Engineers approximate it using domain knowledge and experience. They reason that a particular policy is likely to vary independently from mechanism, that a persistence layer will probably change, that two concepts currently represented by one type will eventually diverge, or that an implementation detail should not become part of a public interface. Much of what is called “engineering judgment” is an estimate about this unknown distribution of future change.

Persistence can compensate for poor design

This gives rise to a counterintuitive possibility. Suppose an earlier coding agent makes a poor design decision. Several steps later, that decision makes a feature difficult to implement.

A stronger long-running agent can instead continue searching. It can add another conditional, introduce an adapter, modify several callers, add a compatibility layer, rewrite a test fixture, run the suite again, and continue until everything passes. Its measured capability has improved, but there are two possible ways to achieve that improvement are to avoid the bad state or to become better at surviving the bad state. Most task-completion benchmarks cannot distinguish them.

This provides one plausible explanation for the apparently contradictory reports around modern coding agents. The agent can genuinely be much more capable than previous generations while leaving behind a system that is no easier—and perhaps harder—to modify. Longer persistence then hides part of the architectural cost. The agent itself absorbs that cost through additional inference, retries, tool calls, and patches. What appears externally as improved autonomy may partially consist of spending more computation overcoming accumulated local mistakes.

Sequential reliability compounds

There is also a simpler reliability problem, the sequential reliability problem. If a workflow requires (n) dependent decisions and the probability that each decision leaves the process in a sufficiently good state is (p), then a crude approximation gives:

$$ P(\text{successful chain})\approx p^n. $$

Even seemingly high local reliability deteriorates quickly:

$$ 0.98^{10}\approx82% $$

$$ 0.98^{50}\approx36% $$

$$ 0.98^{100}\approx13%. $$

A recent controlled study of more than 10,000 agent trajectories reports approximately geometric degradation with increasing numbers of dependent steps. Model scale improves per-step reliability, but measured reliability remains below one, so sufficiently long dependent workflows eventually degrade.⁹ ([arXiv] 7 )

Software evolution is more difficult than this independent-error model implies. A mistake does not merely reduce the probability of success. It changes the state on which later reasoning operates:

$$ S_{t+1}=F(S_t,a_t). $$

An early poor decision can therefore increase the difficulty of later decisions. Error is not only accumulated but is incorporated into the environment.

The Codebase is Part of the Reasoning System

Software architecture is a form of externalized knowledge which leads to a less obvious reason maintainability matters for autonomous agents. Module boundaries record which concepts belong together. Types encode invariants. Interfaces specify what other components may rely upon. Dependency direction represents assumptions about responsibility. Tests preserve behavioral expectations. Names provide compressed descriptions of concepts.

A well-structured repository therefore does more than help a human reader as it stores part of the reasoning that produced it. An agent’s internal context is temporary. Context is compacted. Sessions end. Subagents receive only portions of history. A later model may know nothing about the reasoning that produced an earlier design except what can be reconstructed from the repository. The artifact becomes part of the agent’s persistent memory. This suggests a feedback mechanism of poor representation→poorer reconstruction of intent→poorer subsequent decision→poorer representation. CodeThread’s result—that agents perform worse on later tasks when inheriting agent-written rather than human-written implementations—is consistent with this hypothesis.⁴ ([GitHub] 4 )

This phenomenon makes the common response that “readability no longer matters because agents will write the code” considerably weaker. Agents must interpret the artifact too.

Implications for Benchmarking

These problems become more significant as agents improve, since a weak agent may currently simply fail against an imperfect benchmark, while a stronger optimizer can increasingly discover whichever strategies the evaluator happens to reward, which puts greater pressure on the quality of the objective itself.

OpenAI’s 2026 audit of SWE-Bench Pro illustrates the problem. Its automated investigation identified 27.4% of tasks as broken, while experienced human reviewers identified 34.1%. OpenAI summarizes the benchmark as having roughly 30% problematic tasks, including underspecified prompts, overly restrictive tests, misleading instructions, and low-coverage tests that allow incomplete implementations to pass.¹⁰ ([OpenAI] 8 ) The lesson is that increasingly capable optimizers make weaknesses in evaluation increasingly consequential. This is the software-engineering version of Goodhart’s law.

RL Alone Does Not Explain the Problem

Possibly this is caused by wrong RL rewards, but we believe the evidence does not justify that conclusion just yet. It is reasonable to suspect that reinforcement learning contributes to the behavior. Modern coding training environments can cheaply reward successful execution, tests, task completion, tool use, and other observable outcomes. Maintainability and downstream change cost are much harder to evaluate automatically. The selection pressure is asymmetric. But observed behavior at inference time is produced by more than RL:

$$ \text{behavior} = f( \text{model training}, \text{harness}, \text{tools}, \text{context management}, \text{inference budget}, \text{grader} ). $$

A harness that repeatedly instructs an agent to inspect failures, modify files, execute tests, and continue until completion creates its own optimization loop even if the base model contains no explicit preference for ugly code. Long inference budgets can similarly turn a locally poor solution into a successful one through repeated repair.

The defensible claim could be that current coding systems are strongly optimized around signals for immediate, verifiable task completion and weakly evaluated on the downstream cost of the states they create. Whether that imbalance originates primarily in RL, benchmark construction, agent harnesses, inference-time search, or some combination remains an empirical question.

Structural Weakness in Long-Horizon Software Agents

Stronger models improve on the relevant benchmarks with better memory, better state representation, architectural constraints, richer evaluations, and training on sequences of dependent changes may substantially reduce the gap. Structural weakness is much more about the optimization problem. Properties such as compilation and test correctness generate dense feedback.

$$ \text{action} \rightarrow \text{observable result} \rightarrow \text{reward}. $$

Properties such as architectural flexibility generate sparse and delayed feedback:

$$ \text{design decision} \rightarrow \text{many future changes} \rightarrow \text{eventual cost}. $$

Some of that cost is never observed because the relevant future requirement never arrives. The agent is therefore asked to optimize a quantity for which its environment provides weak evidence, which remains true even if the model is extremely capable.

Persistence vs. Foresight

A central problem is consequently whether an agent can repeatedly solve local problems while preserving a global state in which future problems remain easy to solve. I.e., maintain a good latent design under accumulating state and uncertain future requirements, or more economically, minimize expected downstream change cost. This separates two notions that are often conflated in discussions of long-horizon agents: persistence and foresight.

Persistence means continuing until an objective is reached, while foresight means selecting intermediate states partly because of what they make possible later. And while current systems have made extraordinary progress on the first, the evidence for equally strong progress on the second is much weaker.

Conclusion

Software engineering exposes a general problem with powerful optimizers in that the things easiest to measure are rarely identical to the things ultimately desired, as maintainability is delayed. Test success is observable, while architectural optionality is counterfactual. Current correctness is cheap to verify, while future change cost is not.

In other words, the combination of a powerful optimizer, a partial objective, and delayed feedback tends to produce locally successful behavior that may degrade the global state.

This is why the next major advance in autonomous software engineering may depend as much on evaluation and training environments as on raw model capability. The difficult part is teaching an optimizer which states are worth leaving behind.