Contents
- Introduction
- Concern 1: Neuron Explanations do not Stand Up to Standard Experiments
- Concern 2: Attacking a non-standard definition of superposition
- Concern 3: Issues with the Crushing Experiment (and Reproduction)
- Concern 4: Other Confusion Regarding Prior Work
- Concern 5: Not a mechanistic explanation
- Conclusion
- Appendix
Introduction
Before jumping into the post, I want to emphasise that Nick is a friend and someone I respect a great deal, both personally and as a scientist. I’m grateful for how openly he’s engaged with the discussion on his recent essay, publicly and privately. He also read a draft of this post, and I’ve included the clarifications he asked me to add, marked inline where they’re relevant. His essay takes on some of the most important open questions in mechanistic interpretability, and because of that it is important to respond candidly. That reflects how much I think these11 I’ve had helpful conversations with Nick since the essay came out, and in some of them his views are more tentative or framed differently than in the essay. I’ve responded to the essay as written, since that’s what readers are updating on, and I’m happy to update this post if he shares revisions publicly. questions matter and how much I respect Nick. It’s the kind of feedback I’d want on my own work, especially when people so widely are taking its claims very seriously. I also would really recommend people read his essay closely for themselves.
I think Nick’s essay is productively provocative in a lot of ways. For example, I do think we often don’t try hard enough to understand features and neurons – mech interp is full of things which looked incomprehensible at first glance but turned out to be simple. I also appreciate his enthusiasm for the project of understanding neural networks.
With that said, I also have some concerns with this essay, which I feel obligated to raise publicly, given the level of public discourse and media attention, and the claims that it might have important implications for AI safety. These quick, preliminary comments were very hastily thrown together as I tried to understand and reproduce Nick’s experiments over the weekend. I regret that the intense public interest is forcing me to comment in public so quickly, rather than trying to resolve these points privately with Nick.
At a high-level, some of my biggest concerns are:
- Refuting a non-standard definition of superposition. A major theme of the essay is refuting superposition. But all the arguments in the essay only work for the highly non-standard definition of superposition in the essay. In particular, the essay defines superposition as “unrelated features” being grouped in the same neuron,22 “Superposition: a change in learning principle where the network packs unrelated things into the same neurons because they rarely collide.” (Superposition of the Gaps) and presents this as the original meaning from the circuits thread (see Concern 2 for more discussion around how the circuits thread actually defined superposition). Grouping unrelated features is a phenomenon that can follow from superposition, but I’m not aware of anyone treating it as a definition, and theoretical models of superposition only predict it in certain conditions. In other conditions, the opposite can happen, with superposition grouping similar features.
- Reproduction Differences. When I’ve tried to replicate Nick’s experiments, or run the standard experiments for some of the empirical claims, my results have been different than those described. This has been true even for basic experiments where it’s hard to imagine errors.
- Mischaracterising other papers. In several cases, other papers (including my own) are described inaccurately.
I don’t think any of this is done in malice or with bad intentions. It seems like there may be sincere misunderstandings. I’m also conscious that Nick’s post isn’t the final scientific paper, which is still forthcoming.
Concern 1: Neuron Explanations do not Stand Up to Standard Experiments
The essay argues that the field’s canonical examples of polysemanticity in vision models were misinterpreted. On its account, the cat-and-car neuron is really a V detector, and the car circuit’s dog-and-car readers are really generalised pose neurons, so neither is evidence that neurons mix unrelated things.
These explanations make concrete predictions, and the research community has developed a lot of tools for evaluating precisely these kinds of claims. In this section, I’ll use some of the standard battery. For each section, I’ll check:
- what the neuron responds to across its whole activation range, not just the top
- where possible, whether it responds to clean synthetic versions of the hypothesised stimulus
- which parts of an image it depends on, by covering them up
- whether the upstream neurons that should feed it actually do
- what downstream neurons use it for
None of these tools rely on any special frame, on superposition, or on whether “features” are the right way to think about things. They’re basic empirical tests that I believe should be uncontroversial.
(Checking candidate explanations from many different angles is very important because it is easy to see patterns in things like dataset examples that aren’t really there. Even random directions in mixed4e have top images that look interpretable and I’ve included some discussion of that in the appendix.)
Concern 1a: 4e:55 is Not a V Detector
4e:55 is a classic example of an apparent polysemantic neuron (tracing all the way back to the feature visualisation paper!). The essay claims there’s actually a simple explanation for it: “pointed things drive it and round ones push it negative, whatever the label.” Later it’s described as detecting “a V versus an O.” Unfortunately, this claim does not withstand standard experimental tests.
Top dataset examples are inconsistent with the “V detector” hypothesis. The essay’s hypothesis would predict the strongest images should be pointy, and pointiness should track activation. We don’t see this.
The top dataset examples are all animal faces (cars don’t occur until slightly lower activations). While some of these animal faces have pointy noses, it really seems like a stretch to call most of these V shaped. Many of the top images are for snub-nosed cats and dogs. The Japanese spaniel and black footed ferret seem particularly hard to characterise as involving a “V shape” in any way.
Contrast this with the most negative dataset examples. Many are comparatively round. But it seems really hard to characterise the doberman example as rounder than many of the top activating images. And the Jacamar beak seems to basically be maximally pointy!
Synthetic stimuli tests are Inconsistent with the hypothesis. A standard technique for investigating neurons – imported from neuroscience to mech interp in part by Nick’s work on curve detectors! – is to use simple synthetic images based on a hypothesis.
When exposed to synthetic “V” stimuli, 4e:55 only achieves about 25% of its max. This is no better than hundreds of other neurons in the layer. (And in fact, there are neurons which do respond much more strongly, like 4e:798!)
Perhaps more importantly, rounding the V stimuli doesn’t make 4e:55 react less. So in a setting where we control it, 4e:55 doesn’t seem to track the variable it was hypothesised to detect.
4e:55 doesn’t significantly rely on earlier V detectors. InceptionV1 has plenty of V-shape detectors. Earlier layers have dozens of angle-tuned V-selective neurons (box corners, necklines, roof peaks, steeples). It would be natural for later V detectors to build on these. Ablating them cuts 4e:798 (the neuron which responds to V stimuli above) by more than 25%, far above a random-ablation baseline. But 4e:55 loses no more than it does to random ablation.
Downstream neurons don’t use 4e:55 as a V detector. Next-layer neurons that respond to drawn V stimuli barely read it. When I look at which neurons do, I get a slightly different set than the essay. Two are the same, but rather than get the 5a:317 (“martini glass”), I get a bull frog neuron, which seem rather not pointy or V shaped. (As an aside, the martini glass neuron responds to lots of other things; see dataset examples for the mixed5a readers.)
So, is it a face neuron? While the max activating examples are faces, so are many of the maximally inhibiting images. (Jowly, floppy-eared dog faces in particularly seem to inhibit it!) It’s some specific set of faces. Perhaps more importantly, if we look at images that activate slightly more weakly than the max activating examples, we also see cars (see 4e:55 dataset examples in the appendix).
Concern 1b: Not a pose neuron
The essay says the car neuron 4c:447 feeds dog neurons not to store the car somewhere, but because cars and dogs were brought together “because of shared pose.” It describes InceptionV1 as having “more than a hundred of these generalized pose neurons, each one its own clean detector of a pose shared across cars, dogs and other objects,” and says of 4c:447’s four readers that “two are car neurons and two share a pose between cars and dogs.” The essay doesn’t name which two, but I believe they’re 4d:417 and 4d:241 (the appendix explains how I identified them).
A pose neuron in this sense should prefer a particular orientation of an object relative to the camera. Pose neurons aren’t a new idea. Many candidate pose neurons are in the appendix of the early vision article (see eg. here). But the two neurons do not seem to be like these.
Dataset examples are inconsistent with the pose hypothesis. If they are pose detectors, these neurons should prefer a particular orientation, and prefer it whether the object is a car or a dog. Neither neuron does. Let’s look at the dataset examples for each neuron.
4d:417’s top dataset examples images are mostly vehicles. Dogs and cats appear more below 70% of its maximum. Although often seen from the front or rear three-quarter, the specific direction isn’t shared in an obvious way.
4d:241’s top images are shaggy dog faces seen head-on, alongside cabs, vans and a limousine seen from the side or three-quarter.
These neurons don’t respond to input transformations in ways consistent with the pose hypothesis. If a neuron detected a left- or right-facing pose, mirroring its top images should break it. In the case of these neurons, its response mostly survives mirroring, while genuine pose neurons like 4b:414, which responds to heads in profile facing right across species, lose most of theirs.
To test a shared pose more directly, I rendered 3D models of 7 vehicles and 6 dogs every 15 degrees around the object. Each view was scaled so the object always covered the same share of the frame. A neuron detecting a pose shared across cars and dogs should prefer the same viewpoints for both. Neither of the candidate pose readers does.
Concern 2: Attacking a non-standard definition of superposition
The essay’s biggest practical claim is that “we can study networks directly, instead of first untangling them.” Whether that’s true depends on whether networks use superposition in the sense the field means, since that’s what motivates decompositions like SAEs: units whose meaning can’t be read off the unit alone, because it depends on what else is active.
But the essay uses a non-standard definition of superposition (which I’m not aware of any paper using before), and the arguments made against superposition critically rely on the particular definition used. In particular, it defines superposition as “a change in learning principle where the network packs unrelated things into the same neurons because they rarely collide,” and then repeatedly treats evidence that the things sharing a neuron are related as evidence against superposition. If we adopt the essay’s definition, that is evidence against “superposition” – but not in the sense of superposition the field means.
The standard definition is the one Zoom In introduced to circuits work, where polysemantic neurons “result from a phenomenon we call ‘superposition’ where a circuit spreads a feature across many neurons, presumably to pack more features into the limited number of neurons it has available.” Which features can share is decided by co-occurrence: the car feature can be stored in dog neurons “as long as cars and dogs don’t co-occur.” Toy Models of Superposition formalised this as more features than dimensions; it didn’t redefine anything.
The essay presents its version as the closest to the original Zoom In meaning, among several senses of the word superposition. This just seems incorrect. The field has had a pretty consistent definition, and again, I’ve never heard of the relatedness definition before Nick’s essay. When Zoom In describes polysemantic neurons as responding to “multiple unrelated inputs,” it’s describing what we observe in a neuron, not the mechanism. (See Appendix: What is an arbitrary mixture? for my best guess at what the essay means by “unrelated”, and which parts of it superposition actually commits us to.)
It is true that our theoretical models of superposition predicts that, in some regimes, anti-correlated features (often unrelated) will be preferentially put in superposition with each other. This is a phenomenon which can occur as a result of superposition – not superposition itself. And it only occurs in certain regimes. In other circumstances, theory instead predicts the opposite: that correlated and related features should preferentially be in superposition. So seeing related or even correlated features in superposition with each other – which would be suggested by a polysemantic neuron grouping them together – would actually not be surprising.
I think using the non-standard definition caused significant confusion. It is, of course, fine to introduce new definitions. But it’s important to be clear that they’re different from those other people use, and to not attribute them to sources that didn’t use them. Most readers, including those weighing the safety implications, will hear “superposition” in the standard sense.
Concern 3: Issues with the Crushing Experiment (and Reproduction)
The essay’s most direct test of superposition is a “crushing experiment”, which it describes as “a good test of whether we can turn a monosemantic layer I know well superposed.” Nick retrained InceptionV1 with mixed3b cut from 480 channels to 10, reasoning that if the network used superposition, the squeezed units would become “overlapping mixtures on top of each other, like a curve plus an orange spiral plus spider webs.” Instead of finding the curve detectors in superposition, he reports that the units “stay interpretable and generalized, but coarser,” with one “curves and loops” neuron in place of the ten curve detectors. When I ran the experiment myself, I found what TMS predicts, including curves in superposition, and looking at the figures in the essay, I believe it shows the same thing.
Both in general and because of how dimension-starved this reduction in neurons makes the layer, superposition behaves differently to how the essay predicts. Features that tend to fire together should be merged into shared units, those that don’t fire together can be put in superposition together, and some will be lost. (Superposition at a fixed rate of sparsity becomes much easier with scale. A 48x reduction is much more tolerable if squeezed into 1000 dimensions. See Appendix: How much can we fit in 10 dimensions?)
I retrained InceptionV1 with mixed3b cut to 10 channels (61.78% top-1, the essay got 58.35%). The curve detectors are an easy place to look given how they co-occur is simple and well understood (for a given spatial position, neighbouring orientations tend to fire together and orientations far apart almost never do). Like Nick, I then measured each unit’s response to synthetic curves at every orientation.
Looking at a subset of the neurons, we can see that each unit responds to a broad lobe of neighbouring orientations (curves that fire together have been merged) and their mirror image.
I was surprised seeing how elegant the underlying structure is. The curve detectors sit on a ring and their co-occurrence is dependent on distance from each other on that. When a ring with that symmetry is squeezed into a few dimensions, the natural basis is Fourier modes. The network kept only even modes which can’t distinguish a curve from the same one rotated 180 degrees, which allows for the antipodal symmetry we see.
This is also straightforwardly superposition. Unit 9 can’t tell on its own between the curves that are 90 degrees apart, instead it combines with either 7 or 8 to enable that.
Looking at the figure in the essay (of SAE features on top of the crushed layer rather than the raw neurons), we can see a similar pattern.
Note that neurons representing these Fourier modes could be consistent with them representing additional things as well! And indeed, it seems likely that when the essay characterises them as these simple high-level features, they’re responding to the things the essay speculates and also to these Fourier modes of curves.
Overall, it seems that a more careful analysis of the “crushing experiment”, and using the standard definition of superposition, would support the superposition hypothesis rather than undermine it.
Concern 4: Other Confusion Regarding Prior Work
While the non-standard definition of superposition attributed to Zoom In is the most important for the argument, other work seems to have been misunderstood in important ways.
The Missing Curve Detectors
My paper, The Missing Curve Detectors, asked whether the ten curve neurons from Curve Detectors capture all of mixed3b’s curve detection, and found they don’t. That is, an SAE trained on the layer finds curve detectors the ten miss. I don’t think the essay engages with the claims I make and argues against my work’s conclusions with experiments that are irrelevant.
The essay starts from the premise that “early layers in most vision models are monosemantic.” If that were true, my entire paper doesn’t make sense and there’s no real reason to engage with its claims past that point. I believe I provide evidence of superposition33 As two examples: in conv2d2, one neuron responds to both a colour Gabor and blue, which an SAE separates, and in mixed3b the double-curve neuron 3b:359 splits into two individual curve detectors. (and thus polysemanticity), although even the circuits thread makes reference of occasional small amounts of polysemanticity in early layers. Early vision does have far less polysemanticity than later layers, enough that neurons alone get you to quite a thorough understanding of what it does.
The essay then says the SAE curves “decompose into existing curve-responsive neurons, including the original ten curve neurons,” rather than “fundamentally new curve building blocks hiding in obscure high-dimensional space.” Every direction in mixed3b decomposes into its neurons, so the question is which ones. If the ten captured all of the layer’s curve detection, the SAE curves would be built from the ten alone. Curve detectors that also need other curve-responsive neurons are what the title means by “missing”. Whether that makes them new building blocks depends on what counts as one, and I don’t think there’s a canonical discrete set of curve detectors to count in the first place (I discuss my perspective on this in Curve Detector Manifolds in InceptionV1).
To address my paper, the essay trains an SAE “on just curve activations,” meaning the activations of the ten curve neurons alone. It uses 84k of its 100k latents, which the essay takes as a sign that SAEs can “make it look like there are more building blocks than there really are.” But an SAE that only sees the ten can only find combinations of the ten, so it can’t detect curve signal outside them, which is what the paper is about. And the ten curve neurons’ responses trace out roughly a continuous ring of orientations, which a wide enough dictionary will tile with as many latents as it’s given (see subsequent work looking at SAE scaling).
Distributed Representations: Composition & Superposition (Olah 2023)
The essay cites Olah (2023), “Distributed Representations: Composition & Superposition,” saying the note “framed composition and superposition as competing strategies for how features are laid out across neurons,” which it calls “close to our framing” and “the framing closest to the question this essay cares about.” It adds that most researchers “still think of superposition mainly as more features than dimensions,” implying the note moved the field away from the count definition.
The note does present composition and superposition as two kinds of distributed representation in tension with each other. But both are defined in terms of features, meaning independent directions that can’t be broken down further into meaningful parts. Composition represents combinations by building them out of features; the combinations aren’t additional features. Superposition is still about fitting more features than dimensions. The note doesn’t shift the definition of superposition at all, so researchers who think of superposition as more features than dimensions haven’t failed to update from it.
The essay argues that “more features than dimensions” is something “even a fully monosemantic network would satisfy, because its neurons compose”44 If composition is broadened to mean any meaningful direction built from a combination of neurons, it’s essentially just the linear representation hypothesis. as a way to dismiss the feature count based definition of superposition. Despite being framed as agreeing with this point, Olah (2023) doesn’t do so at all and directly contradicts the way terms are used in the essay. More broadly, the essay argues that features are the wrong unit for understanding networks, while the note is built on features (see also Appendix: How much does defining “feature” matter?). The idea of features being the wrong thing to try to understand is a reasonable position to hold, but it means it is wrong to cite this note as support.
Concern 5: Not a mechanistic explanation
One final concern is higher level. Recursive generalisation is a theory of what we should expect to see group together under the same neuron. I’d understand it as a theory of learning dynamics and superposition structure.
But I’m not sure how it provides a path to mechanistic explanation. That is, I’m not sure how recursive generalisation helps me understand how a neural network computes its output from the input. Suppose we look at a polysemantic neuron and realise that it’s actually linked by a tree of associations to one original concept. But if the different concepts grouped together are actually used by the downstream network, they still need be distinguished somehow (presumably by other neurons?), leaving open the hard question superposition was trying to address.
Conclusion
None of this settles whether neural networks are “already clean,” and there’s a lot in the essay I haven’t touched. However, I don’t think what’s discussed here is minor. For example, the cat-and-car neuron and the car circuit are where the essay says the original evidence “fell.” The things I looked at don’t survive what I consider quite basic checks.
That matters beyond these examples. Much of the essay’s other evidence comes from models and data readers can’t inspect, where the same method has to be taken on trust. Of course, they might very well be correct, but I do think we should avoid updating too much on the results of the paper until such checks can be run.
The non-standard definition of superposition also has pervasive effects throughout the essay, and the idea that the superposition hypothesis was wrong is likely the most important takeaway.
I really regret having to have this conversation in public like this. If not for the intense public interest, the level of media attention, the quickly moving environment, and the potentially high safety stakes, I would have preferred to provide more opportunity for private dialogue first. But I think there’s an important public interest argument in responding quickly in this case.
Appendix
Interpretability illusions
Dataset examples are a very useful and intuitive tool for understanding neurons what neurons respond to (which is then useful for understanding what they might do downstream). It is very easy to see patterns that don’t exist (when I first started training vision SAEs, I drew connections for features that upon further experimentation were definitely not real). Sometimes there is a very abstract shared connection that make sense in hindsight (e.g. “object A on top of object B”) but our belief in the connection’s existence doesn’t make it real. Given almost any set of images, a motivated human can find some shared property that’s abstract enough.
(Bolukbasi et al. (2021) talks about “interpretability illusions” where neurons and random directions looked interpretable on one dataset and told a different story on another. And I highly recommend reading it to understand more about why this can happen.)
To see how easy this is, here are the top ten images for six random directions in mixed4e. Each is a random combination of the layer’s 832 neurons, the kind of direction an SAE feature is after random initialisation (these are also not cherry-picked at all, they were just the first 6).
I feel it is quite easy to interpret these: food in round dishes, people holding fish, bottles with round tops, glassware, brown dogs (and some bottles, which a motivated interpreter could fold in as “glossy brown things” or perhaps they would see the connection with the resemblance between the stem of a wine glass and the legs of the animals), and bridges.
The most straightforward way around this is just to look at dataset examples across a greater range of the activation distribution for that feature/neuron (and then test with other experiments of course).
4e:55 dataset examples
Dataset examples for mixed5a readers of 4e:55
The readers of 4e:55 by attribution, which Nick has said is his preferred method:
Identifying the four readers of 4c:447
The essay shows four readers of the car neuron 4c:447, first with free feature visualisation and then with visualisation “guided” by the paths natural car images take. It concludes that “two turn out to be car neurons and two share a pose between cars and dogs”. The neurons are not labelled, either in the figure or in the text.
We took the four mixed4d channels with the largest expanded weight from 4c:447, measured by the L2 norm over the spatial kernel: 4d:352, 4d:417, 4d:277 and 4d:241. The three readers of 4c:447 visible in Zoom In’s car circuit (417, 241, 277) are among these four, so we assume this is the ranking the essay uses.
| Reader | Unguided | Car-guided | Natural images, 70–100% of max | Natural images, 30–50% of max |
|---|---|---|---|---|
| 4d:352 | 2 cars, 2 terriers | Cars only | Vehicles 98–100% (n=180) | Vehicles 52%→23%, dogs 24% |
| 4d:417 | Cats and dogs | Cars with dogs on them | Vehicles 100% (n=61) | Dogs 52–56% |
| 4d:241 | Shaggy dog faces | Cars only | Dogs 52–71%, vehicles 29–47% (n=114) | Dogs 59–67% |
| 4d:277 | Hare/insect creature | Same creature in a street scene | Vehicles 40–65%, rabbits/hares (n=102) | Invertebrates 26–30% |
Using my best guess at the guided visualisation technique Nick discusses, I got these visualisations.
Which two are the “pose” neurons? 352 is a car neuron by every measure, and 417 is the clearest mixed one. The other two are ambiguous. By the renders, 241 looks like a car neuron, but in natural images it is mixed dog/vehicle at every level, including the top band. 277 never renders as a car, and the second category in its natural images is rabbits and spiders, not dogs. A neuron with no dogs in it can’t share a pose “between cars and dogs”. So we take 417 and 241 as the essay’s pose pair and 352 and 277 as the car pair.
Where we might be talking past each other
I’ve had a few conversations with Nick over the past few days about the essay and here’s my attempt to discuss some of the additional and much more specific places where I think we disagree.
What is an arbitrary mixture?
My current best guess at what an “arbitrary mixture” means comes down to two properties:
- No legible shared property: nothing a human could name links the stimuli e.g. a neuron fires in response to OR(cat, car), with no higher level semantic meaning connecting the two
- Contingency: the specific pairing is an accident of the training run i.e., it is OR(cat, car) this run but it could’ve been OR(cat, toaster)
If we accept the above as the intended belief, his essay says that superposition historically commits us to observing polysemantic neurons meeting that criteria. However, I don’t believe this is consistent with the literature, even if it is consistent with what his personal view was. That is, I think discussing things in terms of the above is engaging with a definition that wasn’t important to the field or load-bearing to the claims it makes.
I agree superposition will largely commit us to observing neurons meeting the legibility criteria (but not necessarily in every case). I disagree that any reasonable use of the word superposition commits us to contingency.
On contingency, there are clear factors that influence which features can be distributed across overlapping neurons (sparsity, importance, co-occurrence) which are properties of the data distribution rather than arbitrarily decided by whatever gradient descent happens to do. Of course there is some stochasticity but we should expect some geometries are much more optimal choices than others. Models trained on the same data might face similar constraints and reach similar representational structure. (This isn’t just a prediction from TMS and is observed empirically e.g. the transfer of adversarial examples.)
How much does defining “feature” matter?
I understand Nick to want to move away from thinking about interpretability in terms of features at all and instead wanting to think about mechanisms (i.e., to think about the meaningful computations / functions implemented by the network rather than the thing being transformed by those). I believe Nick and I are in agreement that there are meaningful directions in activation space and the disagreement we have is more similar to the one that parameter decomposition vs activations-based interp share which is just around the right object of study.
I am not saying that Nick believes this (although I am not saying he doesn’t – I have a lot of uncertainty still on the specific theoretical claims he accepts) but if one thinks that superposition is entirely defined in terms of the feature count, they might then make a jump that superposition isn’t a useful framework when that isn’t correct. Superposition is defined in terms of the activation space because of interp often focusing their study there. However, it demands certain things about the mechanisms too. For example, if one believes that there are more meaningful directions in activations space than dimensions, this object must be having transformations applied to it in higher dimensions that the neuron basis by default would facilitate. That is, even if activations space is an unnecessarily complicated or abstracted object to study, we should still expect the mechanisms to outnumber the neurons. That is, stepping away from features doesn’t mean we get to avoid superposition. Even if one wanted to bottoms-up think about mechanisms and not think about activations-space ever, I think they would eventually stumble upon the superposition problem in reverse.
Rejecting features as fundamental doesn’t by itself rule out superposition. The claim requires to say there is no superposition is something like: “there are about as many distinct mechanisms as neurons, each roughly neuron-aligned”. This isn’t merely an ontological or tooling preference this is a strong claim about the mechanisms of computation that the network applies.
How much can we fit in 10 dimensions?
Suppose, as Nick says, mixed3b is monosemantic with 480 neurons each representing a distinct thing (feature or mechanism it doesn’t matter). Crushing it to 10 channels is very costly geometrically. Superposition becomes much more favourable in higher numbers of dimensions, even at a similar factor dimension reduction.
Consider a layer of increasing width trying to represent 48x more things than it has neurons, with an average sparsity of 1%.
| Crushed dimensions | Initial number of monosemantic directions | Best achievable RMS cosine | Max cosine | Recovered (5 active) |
|---|---|---|---|---|
| 10 | 480 | 0.31 | ≥ 0.48 | 14% |
| 100 | 4,800 | 0.10 | 0.51 | 88% |
| 1,000 | 48,000 | 0.03 | 0.18 | 100% |
Footnotes
- I’ve had helpful conversations with Nick since the essay came out, and in some of them his views are more tentative or framed differently than in the essay. I’ve responded to the essay as written, since that’s what readers are updating on, and I’m happy to update this post if he shares revisions publicly. ↩
- “Superposition: a change in learning principle where the network packs unrelated things into the same neurons because they rarely collide.” (Superposition of the Gaps) ↩
- As two examples: in conv2d2, one neuron responds to both a colour Gabor and blue, which an SAE separates, and in mixed3b the double-curve neuron 3b:359 splits into two individual curve detectors. ↩
- If composition is broadened to mean any meaningful direction built from a combination of neurons, it’s essentially just the linear representation hypothesis. ↩