For most teams, models and datasets reside in buckets in one cloud and one region. Whether for development, training, or service purposes, the GPUs you get are increasingly located in a separate cloud from your data. The moment the two are separated, you’ll be paying cross-cloud transfer taxes just to read your data to your GPU.
With Hugging Face, we combined the two parts. The models and datasets remain on the hub, and SkyPilot performs the compute (development, training, or servicing) on clusters with GPUs. Mount the Hugging Face Bucket or any Hub repository to your SkyPilot job using a single hf:// URL and the HF_TOKEN you already own, and launch it if you have space. Hugging Face has no egress charges, so there is no cost to read data on these GPUs on any cloud.
New features include:
Hub data for every job. store: hf mounts a Hugging Face Bucket (read/write) or any model/dataset/space repository (read-only) to a SkyPilot task using a single hf:// URL and an existing HF_TOKEN via MOUNT or COPY. Run on any GPU, on any cloud. SkyPilot detects that your job computes across 20+ clouds, Kubernetes, Slurm, and on-premises, so the same run uses reserved or on-demand GPUs available from any vendor. There is no egress to read the data. Hugging Face Storage charges no egress or CDN fees, so no matter where SkyPilot runs your job, it reads models and datasets directly from the same bucket. There are no cloud-to-cloud copies or download fees for importing them. Deduplication using Xet. Buckets are built on top of Xet, so incremental checkpoints and model variants only save and forward changed chunks. Built together. Hugging Face and SkyPilot jointly shipped this, and the Hugging Face team upstreamed an hf-mount FUSE fix to make it work in unprivileged containers.
Hugging Face Storage is now the leading SkyPilot backend
SkyPilot tasks already read and write to cloud object stores (S3, GCS, Azure, R2, etc.) by mounting them to local paths. Hugging Face Storage is now added to the list as store: hf, reached through the hf:// scheme.
File mount:
/checkpoint:
sauce: hf://buckets/my-org/qwen-sft
shop: n/a
mode: mount
/Base model:
sauce: hf://Qwen/Qwen3.5-4B
shop: n/a
mode: mount
/data:
sauce: hf://datasets/my-org/my-dataset@main
shop: n/a
mode: mount
This one hf:// scheme covers the entire lifecycle. It reads models and datasets from the repository, writes checkpoints to buckets during training, publishes completed models to the repository, and pulls them to the inference server during service. Most teams already have models and datasets on the hub, so there’s no need for migration steps or creating new storage accounts.
MOUNT uses Hugging Face’s hf-mount FUSE backend, so your bucket or repository will appear as a local path next to SkyPilot’s other FUSE mounts (gcsfuse, blobfuse2, rclone, goofys). Fetching occurs at the file system layer. When your code issues a read(), the driver only retrieves those bytes from the Xet backend, so only the data you actually access goes over the network, and hf-mount maintains an on-disk cache so repeated reads stay local. This on-disk caching is the behavior that SkyPilot gives to other backends under MOUNT_CACHED, and instead plain MOUNT streams all reads from the bucket without holding anything locally. For hf stores, MOUNT and MOUNT_CACHED behave the same way, so both modes preserve the cache.
Reads are deferred, so the process can start processing large files before the entire file has been downloaded, rather than blocking the full copy first. This keeps the GPU busy almost immediately, training on the data being streamed rather than remaining idle (and charged) while a dataset or checkpoint is copied. It works best in the first epoch, when nothing is cached yet. COPY chooses the other route and downloads via Hugface Hub in advance, without any special requirements.
Authentication is a token you already own. Set HF_TOKEN in the environment and pass it to execution using –secret HF_TOKEN. SkyPilot uses this to mount to the cloud where your job lands. One token works whether your job hits AWS, GCP, Azure, Nebius, Lambda, or your own Kubernetes cluster, so you don’t have to deal with bucket keys for each cloud.
No Egress: Storage no longer decides where to run
GPU capacity is rarely available from one place anymore. To get enough H100s and H200s, teams simultaneously hold reserved and committed capacity across multiple vendors (blocks on hyperscalers, clusters on neocloud, perhaps on-premises racks) and run wherever they have allocations. SkyPilot is built for this purpose. A single job specification is scheduled across 20+ clouds, Kubernetes, and on-premises and lands on an available reserved cluster.
Object storage was the problem. Object stores exist per region and per cloud, so feeding a GPU or inference server in another vendor’s data center means either keeping a copy of the data in every vendor’s bucket or paying to pull the data. Most clouds charge egress charges (approximately $0.09/GB from AWS) the moment your data leaves the network, and often across regions within a cloud. Pulling a base model to all your inference nodes or iterating over a few epochs of a dataset from a cluster on a different cloud adds high fees on top of the GPUs you’ve already reserved. The team ends up locking each run to the vendor holding the data, leaving the remaining capacity idle.
Face Storage eliminates that cost from the table, the read side. There are no egress or CDN fees, storage is $12-18/TB per month (whereas AWS S3 is about $23/TB plus egress), you can reach the same bucket from all of these clusters, and reads from it are free regardless of where the GPU is running. Writebacks incur the usual compute cloud output costs, as do to off-cloud stores, but for most AI work, reads are preferred, such as datasets streamed over many epochs, or model weights pulled to every new training or inference node. So stop locking each run to a vendor that holds a copy of your data.
simple benchmark
To collect benchmark numbers, we performed a small tweak called Qwen/Qwen3.5-4B on the HuggingFaceH4/Multilingual-Thinking dataset using TRL’s SFTTrainer. I mounted the model read-only from the hub repository and wrote all checkpoints to a Hugging Face Bucket. The same SkyPilot YAML was run on AWS, GCP, and Lambda, with only -infra changed. SkyPilot placed each job where a GPU was available, and all three read and wrote to the same bucket.
resource:
Accelerator: H100:1
File mount:
/Base model:
sauce: hf://Qwen/Qwen3.5-4B
shop: n/a
mode: mount
/checkpoint:
sauce: hf://buckets/my-org/qwen-sft
shop: n/a
mode: mount
run: |
python train.py – Model /base model –output_dir /checkpoint
What we measured:
Models are loaded to all clouds for free. Lazy read only pulls what from_pretrained touches, so it’s ready for training in about 30 seconds (up to ~500 MB/s). Hugging Face has no egress fees, so there is no cost to pull. If your model resides in S3, you will be charged an egress fee (~$0.09/GB on AWS) for each read to a GPU on another cloud. Checkpoints were streamed directly to the bucket at ~170 MB/s (8.43 GB weight each) and persisted across GPU instances.
For each cloud, checkpoints are written to buckets at:
Cloud GPU Checkpoint Write AWS (us-east-2) L40S ~168 MB/s GCP (us-central1) L4 ~123 MB/s Lambda (us-west-3) H100 ~112 MB/s
Xet-based storage: Checkpoints and model variant deduplication
Hugging Face Buckets is built on Xet and uses content-defined chunking to split files into chunks of up to 64 KB and save each unique chunk once. Boundaries follow content, so editing only changes the touching chunks, and the rest is assumed to be already saved. This works in several places.
Incremental checkpoints and adapter checkpoints. If you freeze a layer, train an adapter, or leave most weights unchanged between saves, only the changed chunks will be uploaded, rather than the entire checkpoint. Variants of models that share a base. Since the fine-tuning and quantization of one base model overlaps considerably, shared chunks are saved once for all base models. Dataset to add. Logs such as conversation traces and inference output grow by appending lines to large Parquet files. Existing row groups remain byte-identical, so only new rows are transferred. In Hugging Face’s tests, adding 10,000 rows to a 100,000-row table moved about 10 MB instead of the full 106 MB. (If you want to edit or delete rows on the fly, write use_content_define_chunking=True to keep your changes local.) Re-uploads will skip anything that’s already saved. In our testing, re-uploading an 8.43 GB blob that was already in the bucket took about 8 seconds, compared to 24 seconds for the initial upload, since only the chunk hash is moved. The same mechanism allows server-side hf buckets to cp between the repository and the bucket, copying bytes by reference instead of re-uploading them.
Deduplication happens automatically, although the savings depend on how duplicate your artifacts are. If you write a checkpoint normally, only new chunks will be output from the machine.
Let’s get started
pip install “Sky Pilot (Hug Face)”
hf authentication login
Add the hf:// mount to your SkyPilot task and start it. MOUNT requires a base image containing glibc 2.34+ and /dev/fuse.
Building together: Hugging Face and SkyPilot
First store: hf support started as a contribution from Nikhil Jha. The Hugging Face team has pushed this forward and upstreamed an hf-mount FUSE fix that allows it to be mounted in unprivileged containers, which is the default for many Kubernetes clusters. The SkyPilot team connected it to a storage backend. The entire path is open source, including SkyPilot, Hugging Face’s hf-mount, and the huggingface_hub client.
resource
