Needle2: 14MB Agentic LLM for Edge Devices

Introduction to Needle2

Needle2 is a new agentic large language model (LLM) with 45 million parameters, specifically engineered for resource-constrained edge devices. It is designed to operate on hardware costing under $200, including budget phones, Raspberry Pis, microcontrollers, wearables, and smart home devices, which typically lack dedicated GPUs or NPUs.

Technical Specifications and Performance

The model boasts a compact 14MB file size and requires only 28MB of session RAM. On a Raspberry Pi 5, Needle2 achieves prefill speeds of over 800 tokens per second and decode speeds exceeding 500 tokens per second. This efficiency is attributed to its lossless 2-bit quantization, which is applied during the training process rather than post-hoc.

Functionality and Use Cases

Needle2 specializes in function calling and generating structured outputs. It maps natural language requests to device functions with typed parameters, making it suitable for controlling smart home devices or robots. The model's design prioritizes specific tasks over open-ended prose, allowing its smaller parameter count to be effective for device interaction.

Edge-Cloud Collaboration and Data Privacy

The model incorporates an edge-cloud collaboration strategy. Each response includes a learned confidence score, and off-topic requests result in an empty call. This allows devices to handle routine tasks locally for privacy and speed, escalating only complex or low-confidence requests to the cloud, thereby keeping most interactions on-device.

Quantization and Training Approach

Needle2 utilizes Cactus Quants, a lossless 2-bit quantization method applied from pre-training through post-training, covering weights, activations, and KV cache. This integrated approach ensures that the deployed 2-bit model retains the performance characteristics of the trained model, enabling its small footprint without compromising accuracy.