● Covered by 2 sources · 3 reportsMedium impact3 neutral
28.9M Parameter LLM Runs on $8 ESP32-S3 Microcontroller Using Flash Memory
Key points
- A 28.9 million parameter LLM runs on an $8 ESP32-S3.
- Model stores most parameters in flash memory, not RAM.
- Uses Google's Per-Layer Embeddings technique.
- Generates text at approximately 9 tokens per second.
- Previous models on similar chips had 260,000 parameters.
On-Device LLM Deployment
A 28.9 million parameter language model has been successfully deployed and run on an ESP32-S3 microcontroller. This microcontroller, which costs around $8, is capable of running the model entirely on the chip without requiring server communication. The model generates text at approximately 9 tokens per second, displaying output on a small screen connected to the chip.
Technical Achievement
This achievement represents a significant increase in model size for such devices; previous language models run on similar microcontrollers typically had only 260,000 parameters. The key to accommodating the larger model is a method that stores most of its parameters in flash memory instead of the limited SRAM available on the microcontroller. The ESP32-S3 has 512KB of SRAM and 16MB of flash memory.
Per-Layer Embeddings Technique
The technique used is inspired by Google's Per-Layer Embeddings, a concept also utilized in Google's Gemma models. This approach allows the bulk of the model's parameters, specifically the embedding table, to reside in slower flash memory. Only the necessary rows, about 450 bytes per token, are pulled into fast memory for computation, enabling the execution of larger models on devices with minimal fast memory.
Developer and Project
The project, named ESP32-AI, was developed by Slava S., known as 'slvDev' on GitHub. This development expands the possibilities for local AI applications on low-cost, resource-constrained edge devices, demonstrating a method for deploying substantially larger language models in such environments.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
GitHub cmc_internal/api GitHub github/collect GitHub slvDev/esp32-ai GitHub _private/browser GitHub get-started/accessibility GitHub open-source/sponsorsHow outlets covered it
A small language model was successfully trained from scratch on an ESP32-S3 microcontroller, demonstrating the feasibility of on-device model training without external computational resources. This development addresses scenarios where data is generated locally and cannot be pre-trained or transferred due to connectivity limitations, opening possibilities for adaptive edge AI applications.
A developer successfully ran a 28.9-million-parameter language model on an ESP32-S3 microcontroller by using Google's Per-Layer Embeddings technique and storing the embedding table on 16MB Flash memory. This achievement demonstrates a method for deploying larger AI models on low-cost, resource-constrained edge devices, expanding the possibilities for local AI applications.
A 28.9 million parameter language model has been successfully run on an ESP32-S3 microcontroller, which costs approximately $8. This achievement is significant because it demonstrates a method to deploy substantially larger language models on resource-constrained edge devices by storing most of the model in flash memory rather than RAM, a technique inspired by Google's Per-Layer Embeddings.