Mount storage in training jobs
Training jobs can mount PersistentVolumeClaims for input datasets, model checkpoints, and output artifacts. This guide covers attaching storage during training job submission.
Prerequisites
- A Kubernetes cluster in ready status
- At least one PVC created (see Work with PVCs)
- A training runtime configured (see Configure a training runtime)
Mount a PVC as input data
- Navigate to Workbench > Training Jobs and click
Submit Training Job. - Configure the runtime, sizing, and script as usual.
- In the Storage section, click
Add Volume. - Select the PVC containing your training data.
- Set the Mount path (for example,
/data/input). - Optionally mark it as read-only if the job should not modify the source data.
- Complete the rest of the form and click
Submit.
Your training script can then read data from the mount path:
data = load_dataset("/data/input/train.csv")
Mount a PVC for checkpoint output
Follow the same steps, but mount a writable PVC at a checkpoint path:
- Add a second volume mount with the path
/checkpoints. - In your training script, save checkpoints to that path:
torch.save(model.state_dict(), "/checkpoints/epoch_10.pt")
Using PVCs for checkpointing lets you resume training from the last checkpoint if a job fails or is pre-empted. Mount the same PVC on the resumed job and load the latest checkpoint file.
Difference from Slurm job mounting
Slurm jobs reference storage via file paths on the cluster's shared file system (typically NFS at /nfs). Training jobs use Kubernetes PVC mounts instead, which are namespace-scoped and attached at the pod level.
For mounting storage in Slurm jobs, see Mount storage in jobs.