Neural Networks May Be More Interpretable Than Previously Thought

Re-evaluating Neural Network Interpretability

After extensive study of neurons and circuits in image and language models, a new perspective suggests that neural networks' internal mechanisms are legible building blocks. This view contrasts with the previous belief that networks arbitrarily pack concepts, making them difficult to interpret.

The author posits that the outlook for mechanistic interpretability is more favorable than previously assumed, as networks may not be as complex or "tangled" as once feared.

Challenging Polysemanticity and Superposition

The canonical evidence for polysemanticity and superposition, which convinced many researchers that networks pack concepts arbitrarily, is now being re-evaluated. Polysemanticity describes neurons appearing to have multiple unrelated functions, such as a neuron responding to both cat faces and car fronts.

The leading explanation for this, superposition, suggested that networks fold unrelated concepts together when they run out of neurons. This implied a "knotted" structure requiring decomposition before study.

Introducing Recursive Generalization

What previously appeared as arbitrary packing is mostly attributed to a motif called recursive generalization. This mechanism is considered less problematic than superposition, as it suggests a more structured and understandable internal organization within neural networks.

This reinterpretation means that researchers may be able to study networks directly, rather than needing to untangle them first, simplifying the interpretability process.

Implications for Mechanistic Interpretability

The field of mechanistic interpretability initially focused on directly studying network neurons and circuits. However, it shifted towards reconstructing hypothesized features due to the perceived "tangled" nature of networks.

If networks are indeed "clean" and their mechanisms are legible, the original approach of direct study becomes more viable, potentially accelerating progress in understanding how neural networks function.