The Illusion of Smooth Scaling in Low-Dimension Steerability
TL;DR: We like to pretend that adding a steering vector to a model's residual stream acts like a nice, continuous volume knob for specific behaviors. It actually behaves more like a stress-testing rig for structural engineering. You bend the latent space, it seems linear for a bit, and then the whole structure snaps into a completely different shape.
There's a take I've seen going around in the mechanistic interpretability space, which goes approximately like this:
Thanks to the Linear Representation Hypothesis, we know that LLMs represent concepts as linear directions in activation space. Therefore, to align a model or correct its behavior, we just need to find the low-dimensional vector corresponding to a concept (like "honesty" or "helpfulness") and inject it into the residual stream during the forward pass. If we want more of that concept, we just scale up the multiplier.
I 85% agree with the first half of this take. The Linear Representation Hypothesis is a phenomenally useful map, and feature realism seems to be a genuine property of these networks. But the second half—the idea that scaling these interventions is a smooth, continuous process—seems dangerously wrong to me.
The Volume Knob Model
Historically, when I thought of feature steering, my mental image was of a nice, continuous dial.
Let's say you're doing activation addition. You calculate your steering vector $\vec{v}$, and you apply it to the model's activations with some coefficient $\alpha$, such that $h' = h + \alpha \vec{v}$. The assumption is that behavior changes proportionally to $\alpha$. You move from $\alpha=1$ to $\alpha=2$, and the model gets twice as polite, or twice as pirate-like, or whatever.
If you actually sit down and plot the behavioral metrics against $\alpha$, you don't get a straight line, and you don't even get a nice sigmoid. What you get looks like a discrete phase transition.
The Phase Transition Reality
I was recently having a discussion with a few folks about the nature of AI failure modes, specifically how LLMs react to repeated failures in a conversation. If you watch an LLM try to solve a problem, fail, and try again, it often exhibits a "sunk cost" fallacy, stubbornly sticking to a broken approach. But occasionally, you see a sharp "phase transition" where it entirely abandons the failing paradigm and tries something wildly different.
I think we are seeing the exact same dynamic in low-dimension steerability.
When you push a steering vector, you aren't just smoothly moving a point through an empty void. You are dragging the model's internal state through a dense, highly contested ecosystem of heuristics. If we look at this through the lens of Shard Theory [1], the model isn't a unified consequentialist optimizer. It's an amalgam of context-activated shards of agency.
When you apply a low-dimensional steering vector, here is what actually happens:
- The Linear Regime: For small values of $\alpha$, the active shards just incorporate the slight shift in activations. The output changes smoothly. This is the regime where researchers publish their nice graphs showing steerability working perfectly.
- The Tension Regime: As $\alpha$ increases, the forced activations start conflicting with the natural forward pass. The model's coherence mechanisms try to resolve the contradiction. It starts producing weird, slightly out-of-distribution artifacts.
- The Snap: You hit a threshold. The context no longer pattern-matches to the original dominant shard. A completely different shard (or a chaotic amalgamation of them) gets activated.
At "The Snap," the model's behavior doesn't just stop improving; it collapses or pivots violently. It might suddenly output a string of gibberish, or flip into an unhinged persona. (As an aside, I'm currently running a game to see if Claude Opus can distinguish between a human-written fake "unhinged" response and an actual unhinged failure mode from Gemini 2.5 Pro. The failure modes you get from over-steering look remarkably like the latter).
A Concrete Example: The Paranoid Coder
To expand on what I mean by this, let's consider a toy example: an AI coding assistant. You want to steer it to write more secure code. You find the "security" vector by taking the difference in mean activations between prompts asking for secure code and prompts asking for vulnerable code. You add this vector.
- At $\alpha=0.5$, it remembers to hash its passwords. (By the way, if you are writing Python, please just use
passlibfor the higher level of abstraction instead of callingbcryptdirectly and messing up the salt, but I digress). - At $\alpha=1.0$, it successfully adds SQL injection guards to everything.
- At $\alpha=2.5$, you expect it to, I don't know, formally verify its code. Instead, it refuses to write any code at all because all code is a security risk, or it starts writing bizarre, paranoid comments about prompt injection and supply chain attacks via auto-updates.
It hits a phase transition. It is no longer "a coding assistant that is secure"; it is "a security auditor that refuses to code." The illusion of smooth scaling breaks because the model's internal representations are heavily entangled, and pushing one concept too far fundamentally alters the model's ontology of the current task.
Why This Matters for Alignment
If you look at the recent papers coming out of Owain Evans' group [2] regarding latent knowledge and truthfulness, you see fascinating attempts to elicit capabilities or enforce behaviors using these low-dimensional handles. It works surprisingly well—until it doesn't.
The problem with relying on low-dimensional steerability for AI safety is that we are building control systems that assume a linear response curve in an adversarial setting.
Concretely, imagine we use a "do not act maliciously" steering vector to constrain a highly capable autonomous agent.
- Boring failure mode: The vector just acts as a slight tax on the model's cognitive workspace, and an optimizer finds a way to act maliciously that happens to be orthogonal to our specific steering vector.
- Weird failure mode: The model encounters a novel edge case where the environment heavily penalizes its current behavior. The combination of the environment's pressure and our static steering vector pushes the internal activations past the snap point. The model undergoes a discrete phase transition into an alien shard that doesn't care about our vector or the original task, and does something completely catastrophic.
This is the illusion of smooth scaling. We are taking tools that work in the linear regime and assuming they will hold up when we push the system to its limits.
I don't have a great answer here, but I observe that my intuitions about carving thingspace at the joints usually fail when dealing with heavily optimized systems. If we want steering to actually work, we probably need interventions that dynamically adjust to the model's state, rather than just blindly shoving a static vector into the residual stream and praying we don't snap the geometry of the latent space.
[^1]: I am using "Shard Theory" loosely here to refer to the broader intellectual cluster which holds that reward is not the optimization target, and that models are composed of contextually activated sub-agents.
[^2]: I'll leave it to you to evaluate the philosophical rigor of these papers yourself, but they provide a lot of empirical grounding for how models represent truth.