Small Transformer Achieves 44% on ARC-AGI-1 for 67 Cents

New Model Performance

A small transformer model was trained from scratch, achieving a 44% score on the ARC-AGI-1 benchmark and 7% on ARC-2. This performance is comparable to other large language models (LLMs) and previous models like TRM/HRM. The training process was completed in 1.5 hours on a 5090 GPU, incurring a cost of 67 cents.

Research Motivation

The primary motivation behind this work is to address sample efficiency in AI, aiming to determine the limits of current deep learning methods and reduce research costs. The ARC benchmark is used due to its limited sample size, high-dimensional space, meta-learning nature, and accessibility for researchers.

Technical Approach

The model's approach involves converting input-output pairs into token sequences, which are then autoregressively trained by a small transformer. Cross-task learning is enabled through separate additive embeddings for each puzzle, and 3D RoPE embeddings are used for positional information. The sequences are augmented with color and dihedral permutations, with inverse augmentation applied during inference.

Architectural Upgrades

The current iteration includes several architectural and algorithmic improvements over previous versions. These upgrades were specifically implemented to enhance the model's sample efficiency, contributing to the improved scores on the ARC benchmarks.