SkyPilot integrates with Hugging Face for zero-egress AI workload storage

Integration with Hugging Face

SkyPilot has partnered with Hugging Face to allow users to utilize models and datasets directly from the Hugging Face Hub. Users can mount Hugging Face Buckets or any model/dataset repositories into their SkyPilot jobs using a simple `hf://` URL and their existing HF_TOKEN.

Zero Egress Fees

With this new integration, Hugging Face charges no egress or CDN fees. As a result, accessing models and datasets stored on Hugging Face incurs no cost regardless of the cloud where the computation is performed. This move aims to reduce data transfer costs for users running AI workloads.

Flexible Compute Options

SkyPilot can access computing resources from over 20 cloud providers, Kubernetes, Slurm, and on-premises setups. It dynamically finds available GPUs, allowing users to efficiently run jobs on the most suitable infrastructure without vendor lock-in.

Incremental Storage Efficiency

The integration employs Xet-backed deduplication, ensuring that only changed data chunks are stored and transferred. This is particularly beneficial for managing incremental checkpoints and model variants, optimizing storage requirements and transfer times.

No Migration Needed

For teams already using Hugging Face to store models and datasets, no migration is required. The `hf://` scheme facilitates complete lifecycle management, enabling easy reading, writing, and publishing of models directly within the workflow.