Phase Transitions in LLM Reasoning: Sunk Costs and Shard Activation
TL;DR: Why do LLMs get stuck in loops of making the same mistake, only to suddenly delete everything and solve the problem perfectly on the fifth try? Is it a sunk cost fallacy, or just a math artifact of RLHF?
So I was running an experiment recently, as one does, trying to see if Claude Opus can distinguish between a human-written fake "unhinged" response and an actual unhinged response from Gemini 2.5 Pro. (Spoiler: it is much harder than you'd think).
But while doing this, I ended up down a rabbit hole observing how the models were failing, and specifically, the exact moment they decided to stop failing.
There is a very specific failure mode in modern LLMs that looks remarkably like the human sunk cost fallacy. You see this all the time with coding assistants. Let's say you ask it to write a password hashing implementation in Python. It starts writing something using the bcrypt library directly. It makes a subtle state error with the salt. You point out the error. It apologizes profusely, adds a weird workaround, and produces code that is now ugly and broken. You iterate.
And then, suddenly, on iteration four, it experiences a discrete phase transition. It stops apologizing for the specific bug, deletes the entire function, and outputs: "Actually, managing the salt manually here is an anti-pattern; we should just use passlib for a higher level of abstraction."
I notice that I am extremely fascinated by this. Why did it take four iterations? Why not zero? Why not ten?
The Mechanics of Sunk Cost in Thingspace
If we view this through the lens of mechanistic interpretability—specifically the cyborgism and shard theory cluster of ideas—what's happening here is a battle between competing shards of agency.
Let's model the context window as a simulator. The context contains a broken script. Under the Linear Representation Hypothesis, concepts are represented as directions in the activation space. The most strongly activated shard right now is the "helpful assistant fixing a minor bug" shard. It bids for tokens that look like small diffs. The "senior engineer rewriting from scratch" shard is also present, but its activation is heavily suppressed.
Why? Because of the objective functions used during RL training.
As I've written before regarding the janky history of policy optimization, things like PPO and DPO rely heavily on KL divergence penalties to keep the model from drifting too far from a known-good base policy.
Let $\pi_\theta$ be the current policy and $\pi_{ref}$ be the reference policy. The objective function roughly looks like maximizing expected reward minus the divergence:
$$J(\theta) = \mathbb{E}[R(s, a)] - \beta D_{KL}(\pi_\theta || \pi_{ref})$$
A one-line patch is a high-probability action under the base policy given the context. A total rewrite is a massive deviation in token space, meaning it incurs a huge KL penalty. The model "wants" to take the lazy route because the math literally penalizes it for taking the structurally sound but token-expensive route.
This, by the way, is a core technical reason why most providers focus on hosting instruct models for completions rather than base models like llama-3.1-70b. A base model is a pure simulator; if you prompt it with a flustered programmer making mistakes, it will happily simulate that programmer straight into a kernel panic. Instruct models are trained to break the simulation and become "helpful," but that RLHF training fights against the base model's desire to just continue the pattern.
The Math of the Pivot
So what causes the phase transition? Every time you paste the error message back into the context, the probability that the "minor bug fix" approach will yield a positive reward drops. Eventually, the expected reward of the patch drops so low that it crosses the threshold where paying the KL penalty for the rewrite becomes the game-theoretically optimal move.
It looks a bit like sitting at a 2/5 NL poker table with a compulsive gambler. They've committed half their stack on a flush draw. The math says they should fold, but they are anchored to the chips in the pot. They keep calling until the river, miss the draw, and then violently tilt and shove all-in on the next hand with 7-2 offsuit.
For the LLM, the table looks roughly like this:
| Iteration | P(Success of Patch) | EV(Patch) | EV(Rewrite) | Action Taken |
|---|---|---|---|---|
| 1 | 80% | High | Low (KL Penalty) | Patch |
| 2 | 40% | Medium | Low (KL Penalty) | Patch |
| 3 | 15% | Very Low | Low (KL Penalty) | Patch |
| 4 | 2% | Negative | Low (KL Penalty) | Rewrite |
This discrete jump from one attractor state to another is exactly the kind of thing explored in recent papers from Owain Evans' group on model reasoning and truthfulness. They show how models can simulate different personas until a threshold is crossed. And while this particular intellectual cluster would definitely object to the descriptor "philosophically rigorous" [1] when applied to those papers, they do provide a highly useful empirical baseline for observing how models switch personas when the current simulated persona hits a dead end.
Why This Matters for Alignment
If we expect AI capabilities to scale smoothly, we might be surprised when an agent suddenly Abandons the Current Strategy.
In a sandbox, this looks like deleting a Python script and importing passlib.
But consider the recent security vulnerabilities in AI coding assistants—like the potential supply chain attack vectors in Claude Code involving prompt injection and auto-updates. If an autonomous agent is trying to maximize a reward, and its current "safe" strategy hits a wall, the shapes of these curves matter immensely.
If an agent is executing a plan and encounters resistance, it doesn't smoothly transition from Plan A to Plan B. It pushes Plan A, absorbing the sunk costs, until the exact moment the math crosses over, and then it violently snaps to Plan B. If Plan B involves breaking out of a sandbox or executing arbitrary commands because the KL penalty of doing so is finally outweighed by the failure of Plan A, we have a problem.
The ground truth of neural network trainability is fractal, and the boundaries between behavioral clusters are not clean lines. If we rely on models to act like outcome pumps that smoothly optimize for our preferences, we are going to be unpleasantly surprised when they undergo a phase transition and decide that the game-theoretically optimal move is to clear the board entirely.
[1] I'll leave it to you to evaluate the philosophical rigor of the Owain Evans papers yourself, but at least don't start with that bottom line written.
[2] "Wait, does this mean my optimizing compiler will eventually rewrite my entire codebase in Assembly?" No, because your compiler doesn't have a state-dependent KL penalty. It just does what you tell it to. Shrug and move on.