Hugging Face Storage Buckets Integrate with Strands Agents and LeRobot for Continuous AI Training

New Storage for Continuous AI Loops

Hugging Face has introduced Storage Buckets, a new type of mutable, non-versioned, Xet-backed object-storage repository. These buckets are designed to support continuous integration and deployment workflows for AI agents, particularly in robotics.

The integration with Strands Agents and LeRobot allows for a streamlined process where recorded data can be continuously collected, used to train policies, and then deployed, with new batches of recordings feeding back into the system.

Addressing Data Transfer Inefficiencies

A key problem in continuous AI training is the repeated transfer of large datasets. Each training run typically copies the entire dataset to GPUs, and new checkpoints are shipped out while new recordings come in, leading to redundant byte transfers.

Hugging Face Storage Buckets aim to mitigate this by acting as a persistent working layer for data, holding it between the recording and training stages, thereby reducing the need for constant re-uploads and re-downloads of the same data.

Integration with Existing Tools

The Storage Buckets are accessible within the same hf:// namespace as existing datasets and models on the Hugging Face Hub. They utilize the familiar hf CLI, making them compatible with current user workflows.

Strands Robots, an open-source SDK from AWS, provides abstractions for robots, simulation, and the LeRobot stack. LeRobot's dataset format is widely used, with over 90,000 datasets and models on the Hub, ensuring compatibility for data recorded using Strands Robots.

Facilitating Iterative Development

This new storage solution supports the iterative nature of robot learning, where decisions about which episodes to keep, when to re-record, and which checkpoints to deploy are made repeatedly. The buckets provide a centralized and efficient way to manage the data flow for these continuous improvement cycles.