Mathematical Framework Developed to Reverse Engineer Small Transformer Models

Reverse Engineering Transformers

This paper introduces an initial approach to reverse-engineer transformer models by focusing on their detailed computations. The goal is to understand how transformers operate internally, similar to how software binaries are reverse-engineered into human-readable code. This method could help explain and identify safety issues in current and future AI models.

Focus on Small Models

Given the complexity of modern language models like GPT-3, the research began with the simplest possible transformer architectures: models with two layers or less, containing only attention blocks. This contrasts with larger models that integrate both attention and MLP blocks across many layers. The strategy is to identify fundamental algorithmic patterns in these smaller models that can be scaled up.

Discovery of Induction Heads

By re-conceptualizing transformer operations mathematically, the researchers gained insight into these small models. A key finding is the identification of "induction heads," specific attention heads that explain in-context learning. These induction heads were observed to develop only in models possessing at least two attention layers. Examples of these heads in action on specific datasets were also provided.

Future Implications

While this paper does not apply its insights to larger models, a forthcoming paper will demonstrate the partial relevance of this mathematical framework and the concept of induction heads to more realistic, larger transformer models. This research represents a preliminary step towards fully reverse-engineering complex AI systems.