360M-parameter language model trained to simulate a terminal interface

Project Overview

A developer trained a 360M-parameter language model to function as a terminal interface. This model, based on Qwen2.5-Coder-0.5B, predicts how the screen changes in response to user keypresses, rather than relying on an underlying shell or filesystem. The goal was to investigate if a model could maintain a consistent 'world' state.

The model was fine-tuned using recorded terminal sessions from real programs running within Docker containers. It can perform actions like creating files, running applications such as vim and tmux, and playing games like Snake, often remembering previous commands.

Motivation and Approach

The project was inspired by the idea of applications being learned by AI rather than explicitly coded, similar to how a model might predict game frames. Lacking resources for complex simulations, the developer chose a terminal environment due to its character-grid nature and the ease of generating perfectly labeled training data from real shells.

A key challenge was adapting the language model's token prediction method to the dynamic nature of a terminal. Instead of appending text, the model predicts changes to the screen grid. This involves processing a sequence of tokens describing the screen's contents and modifications, rather than a continuous text stream.

Technical Details

The model treats the terminal as a grid of characters. It receives a snapshot of the screen's state, including size, contents, and cursor position, as input. The model's ability to repair its own mistakes and decode screen rows in parallel were identified as significant improvements during development.

A demo is available in a browser, designed for desktop use with WebGPU, and is best experienced on Chrome or Edge. The demo allows users to interact with the simulated terminal, demonstrating its capabilities and limitations, such as revealing it is not a real shell when typing specific commands.