The DeepSeek-V3 architecture has been re-implemented in approximately 190 lines of pure JAX code. This implementation avoids reliance on higher-level frameworks such as Flax, Equinox, or PyTorch, presenting the model's components as functions operating on plain Python dictionaries of arrays.
The model was pretrained from scratch on a MacBook Air M5 (24 GB) using a dataset of public-domain detective fiction. During pretraining, the validation loss at step 2000 was 3.921 with expert balancing enabled, and 3.923 without, confirming the paper's claim that bias evens out experts without additional cost.
This project serves as an architectural demonstration rather than a production-ready language model. It highlights the core mechanics of DeepSeek-V3, including its expert balancing mechanism. A v1.0 release provides the exact checkpoint and data used in the published run for replication.
To run the implementation, users need an Apple Silicon Mac, Python 3.12, and the `uv` package manager. The published run consumed 24 GB of memory. Specific instructions are provided for installation, checking the setup, and building the dataset, which involves downloading and processing approximately 120 MB of text data.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
A developer has implemented the DeepSeek-V3 architecture in 190 lines of pure JAX, without using frameworks like Flax or PyTorch. The implementation was pretrained from scratch on a MacBook Air using public-domain detective fiction to demonstrate the architecture's core principles, including expert balancing.