Persistent Storage Guide

Keep your data safe across restarts and scaling operations.

Overview

Persistent storage allows your data to survive pod restarts and scaling operations. Unlike ephemeral storage (which is cleared when pods are stopped), persistent storage uses network-attached NFS volumes that retain data across all pod lifecycle events.

Key Features

Survives Restarts: Data persists when pods are restarted
Survives Scaling: Data persists through scale up/down operations
Survives Termination: Volumes are independent and can be attached to new pods
NFS-Based: Uses network file system for reliability
Automatic Mounting: Volumes are automatically mounted at /mnt/<volume-name>

Adding Persistent Storage

At Launch Time

When launching a new GPU pod:

Click Launch GPU
Select your GPU pool and instance type
Under Persistent Storage, choose one of:
- Use existing storage: Attach a volume you already own (preserves your data)
- Create new storage: Provision a new persistent volume
Click Launch

To an Existing Instance

You can add persistent storage to a running instance:

Find your GPU card in the dashboard
Click Add Storage
Select a storage size from available options
Confirm the addition

Note: Adding storage requires the pod to restart. Your ephemeral data will be lost, but the new persistent storage will be attached.

Using Your Storage

Accessing the Mount

Once your pod is running, persistent storage is mounted at:

/mnt/<volume-name>/

Verify the mount:

# List mounted volumes
ls -la /mnt/

# Check disk usage
df -h | grep mnt

Common Workflows

Storing HuggingFace Models

# Set HuggingFace cache to persistent storage
export HF_HOME=/mnt/your-storage/hf-cache

# Models will now be cached to persistent storage
python -c "from transformers import AutoModel; AutoModel.from_pretrained('bert-base-uncased')"

Storing Training Checkpoints

import torch

# Save checkpoints to persistent storage
checkpoint_dir = "/mnt/your-storage/checkpoints"
torch.save(model.state_dict(), f"{checkpoint_dir}/model_epoch_{epoch}.pt")

Using with vLLM

# Run vLLM with model cache on persistent storage
export HF_HOME=/mnt/your-storage/models

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000

Technical Details

How It Works

Persistent storage uses NFS (Network File System) provided by the infrastructure. When you add storage:

A shared volume is created in the storage pool
The volume is attached to your pod
NFS mounts the volume at /mnt/

Performance Tips

Network-attached storage has slightly higher latency than local SSD. For I/O intensive workloads:

# Copy data locally for processing
cp -r /mnt/persistent/dataset /tmp/dataset

# Work with local copy, then sync back
rsync -av /tmp/results/ /mnt/persistent/results/

Billing

Persistent storage is billed per hour:

Billing starts when the volume is created
Billing continues while the volume exists (even if pod is stopped)
Pricing varies by storage size - check the launch modal for current rates

Troubleshooting

Storage Not Visible

If you don't see your storage mount:

# Check mount status
mount | grep nfs

# Check if mount point exists
ls -la /mnt/

Permission Issues

If you encounter permission errors:

sudo mkdir /mnt/your-storage/new-folder
sudo chown $USER:$USER /mnt/your-storage/new-folder

Snapshots and Storage

What is a Snapshot?

A snapshot saves your pod's configuration for easy re-deployment:

GPU pool selection
Number of GPUs
Instance type
Reference to your persistent storage volume
Any HuggingFace model deployment settings

What Data Survives?

Storage Type	Location	Survives Termination?
Persistent Storage	`/mnt/<volume-name>/`	Yes
Ephemeral Storage	`/home/ubuntu`, `/tmp`, `/root`	No

Important: Ephemeral storage (local SSD) is wiped on termination. This includes pip packages, downloaded files, and anything not in /mnt/. Always store important data in persistent storage.

Snapshot Workflow

Save important data to /mnt/your-storage/ while running
Create a snapshot from the pod card menu
Terminate when done (your storage volume persists)
Restore from snapshot to launch a new pod with storage attached

Common Misconception

Snapshots do not capture ephemeral disk contents. They save a reference to your persistent storage volume. When you restore, you get:

A fresh container (new /home/ubuntu, no installed packages)
Your persistent storage mounted at /mnt/ (with all your data intact)

Think of it as "bookmark + persistent disk", not "full VM snapshot".

FAQs

What happens if my pod crashes?

Your persistent storage data is safe. The NFS volume exists independently of your pod. When the pod restarts, the storage is automatically remounted.

Can I access storage after terminating my pod?

Yes! Persistent storage volumes are independent and survive pod termination. When you launch a new pod, you can attach your existing storage volume and your data will be there. This makes it easy to:

Scale up/down while preserving your environment
Switch between GPU types without losing data
Save costs by terminating pods when not in use

Is ephemeral or persistent storage faster?

Ephemeral storage uses local SSD and is faster for I/O operations. Persistent storage uses NFS and has some network latency. For maximum performance, copy working files to local storage.

Persistent Storage