Neural Networks Exhibit Emergent Symbolic Structure in Internal Representations

Symbolic Structures in Neural Networks

Modern AI systems, particularly neural networks, achieve high performance in domains traditionally associated with symbolic processing, such as language and logic. While intelligence has often been modeled as operations over structured combinations of symbols, neural networks represent information using continuous vectors. This research proposes that neural networks implicitly realize symbolic structure within their internal representations.

Approximating Representations with Symbolic Models

The study demonstrates that the vector representations of various neural networks can be closely approximated by symbolic structures. Researchers replaced the network's representation-generating process with a closed-form equation instantiating a symbolic structure, and the network's behavior remained largely unchanged. This approximation was effective for both small-scale neural networks trained on list manipulation and large language models (LLMs).

Applicability Across Domains

The findings extend to LLMs operating in four key symbolic domains: arithmetic, logic, computer code, and language. This broad applicability suggests a fundamental mechanism at play within these systems. The ability to approximate complex neural network behavior with simpler symbolic models provides insight into their internal workings.

Targeted Behavioral Modification

The symbolic approximation allows for precise interventions on an LLM's internal representations, leading to targeted modifications of its behavior. This capability indicates that the LLM's behavior is reliant on the identified symbolic structures. Such control over internal representations could lead to more interpretable and steerable AI systems.

Reconciling AI Paradigms

This work offers a potential bridge between longstanding symbolic conceptions of intelligence and the vector-based nature of modern AI. By showing that neural networks can implicitly realize symbolic structures, the research provides a framework for understanding how these seemingly disparate approaches to AI might converge or complement each other. This could lead to new hybrid architectures or improved understanding of existing ones.