Neural Networks Are 'Already Clean' Says the Researcher Who Helped Prove AI Was a Tangled Black Box

On 15 October, Goodfire researcher Nick Cammarata challenged the view that artificial intelligence is an inscrutable black box, arguing that a preliminary language-model audit suggests most neuron poles perform one main job rather than combining unrelated concepts.

Nick Cammarata published 'Neural Networks Are Already Clean', arguing the messy evidence behind the black-box view was misread. His 2023 project at OpenAI judged only about 2% of neurons readable. Cammarata's preliminary audit of Qwen3-32B, a large open-source model, estimates that near 90% of neuron poles have one main job, with roughly 7% handling two related tasks and only 2% to 4% still unexplained.

The Man Who Built the Black Box

Cammarata is not an outsider to the idea he is now attacking. He co-authored the 2020 'Zoom In' circuits research that framed individual neurons as the building blocks of vision models, and he launched OpenAI's automated interpretability effort three years later. Those early tools could label only a tiny slice of neurons, which hardened the belief that models cram unrelated ideas into the same unit, a property known as polysemanticity.

Few people are better placed to question the theory, since his name sits on the papers that spread it.

The Cat and Car Neuron, Reread

The sharpest example sits inside InceptionV1, an early image model with a single neuron that fired for both cat faces and car fronts. For years, that read as proof of arbitrary mixing.

Cammarata now says both images share a pointed, V-shaped contour, and that the same neuron reacts to V-shaped flames and fox faces while rounded shapes push it the other way. He calls the pattern recursive generalisation, where a neuron sprouts related offshoots during training rather than fusing unrelated concepts. A language neuron, he notes, responds to 'iy', then to 'diy', then to 'do it yourself', a chain that drifts far from where it began.

A Public Fight Over 'Readable'

The claim has split the field. Neel Nanda, who leads interpretability at Google DeepMind, called the work interesting but said it was 'not a significant update to the big picture of interpretability', and that the essay defines a single-meaning neuron too loosely.

Several examples still look polysemantic to him. An OpenAI employee posting as Roon framed the moment as historic, while sceptics urged restraint until a technical companion paper lands. The sharpest scepticism came from Eliezer Yudkowsky, a longtime AI-safety voice, who pushed back with a pointed parable rather than accepting the essay's conclusions.

Why Legible Models Matter to Everyone

The stakes reach well past the lab. Debates over whether modern AI can be understood affect public trust, oversight, and regulation. If the interior is more legible than claimed, those arguments weaken.

Cammarata labels his essay a working hypothesis and calls the language-model audit preliminary, with a technical companion paper still to come. The lab has already pulled activations from a trillion-parameter model. Goodfire, the firm behind the research, was valued at $1.25 billion (£935 million) in February.

For now, the field has a fight, not a verdict, over whether AI was ever as opaque as everyone agreed.