● Covered by 1 source · 1 reportMedium impact1 neutral
Petals enables running large language models on consumer GPUs via distributed network
Key points
- Petals runs large LLMs like Llama 3.1 (405B) on consumer GPUs.
- It uses a distributed network, sharing model parts among users.
- Supports fine-tuning and custom model interactions.
- Offers inference speeds up to 6 tokens/sec for Llama 2 (70B).
Distributed LLM Execution for Consumers
Petals is a project that allows individuals to run and fine-tune very large language models (LLMs) on consumer-grade hardware, such as a single GPU or Google Colab. It achieves this by adopting a distributed approach, similar to BitTorrent, where users load a portion of a model and connect to a network of others serving the remaining parts.
Supported Models and Performance
The platform supports a range of large models, including Llama 3.1 (up to 405B parameters), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B). For single-batch inference, Petals delivers speeds of up to 6 tokens per second for Llama 2 (70B) and up to 4 tokens per second for Falcon (180B), which is sufficient for chatbots and interactive applications.
Advanced Customization and Flexibility
Beyond standard LLM APIs, Petals provides users with greater control over model interaction. It enables the use of any fine-tuning and sampling methods, execution of custom paths through the model, and observation of hidden states. This offers the convenience of an API combined with the flexibility typically associated with PyTorch and Hugging Face Transformers.
Project Origin
Petals originated as a component of the BigScience research workshop, indicating its foundation in collaborative AI research efforts.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
GitHub bigscience-workshop/petalsReporting from
Petals allows users to run and fine-tune large language models, including Llama 3.1 (405B) and Mixtral (8x22B), on consumer-grade GPUs or Google Colab. This is achieved by distributing model parts across a network, enabling accessibility for interactive applications and custom research beyond traditional LLM APIs.