NVIDIA has introduced Snapshot, an open-source component of the NVIDIA DSX AI Factory Platform, designed to significantly reduce GPU workload startup and recovery times on Kubernetes. By enabling initialized GPU workloads to be reused across inference replicas, Snapshot addresses the delays caused by cold starts, which can take several minutes, particularly during inference demand spikes or replica failures.
How NVIDIA Snapshot Works
Snapshot is a checkpoint and restore system for GPU workloads. It captures a fully initialized workload, including CPU and GPU memory, and restores it on compatible infrastructure. This process bypasses repeated workload initialization steps such as weight loading, kernel compilation, and CUDA graph capture, allowing inference capacity to come online in seconds instead of minutes.
The system integrates with Kubernetes through custom resources. Users can create a PodSnapshot to checkpoint a running workload or a SnapshotJob to checkpoint a workload immediately after initialization. To restore, a new Pod references the saved PodSnapshot, resuming from the captured state. Snapshot supports workloads spanning multiple GPUs on a single node, making it suitable for large-scale models using parallelism strategies like tensor parallelism (TP) and data and expert parallelism (DEP).
Performance Benchmarks
Benchmarks demonstrate that Snapshot delivers substantial performance improvements. For example, Crusoe, an upstream contributor, reduced GLM-5.3 Pod readiness time from approximately 8 minutes to 31 seconds on NVIDIA HGX B300 hardware. Similarly, Saturn Cloud reported vLLM startup times dropping from 2–8.5 minutes to 6–34 seconds on the same platform.
Across tested configurations, Snapshot restored replicas in 3.6 to 42.7 seconds, achieving speedups of 8x to 36x compared to cold starts. Even for the largest mixture-of-experts (MoE) models with 1 TB checkpoints across eight GPUs, restore times remained under 43 seconds, compared to cold-start times of 6 to 19 minutes.
Integration and Use Cases
Originally developed for NVIDIA Dynamo inference workers, Snapshot is now available for integration with other inference engines, schedulers, and platforms. It supports inference engines such as vLLM, SGLang, and NVIDIA TensorRT LLM, with ready-made examples provided for each. The system is particularly advantageous for AI factory operations requiring rapid scaling or recovery of inference workloads.
Snapshot's checkpointing process involves capturing the complete in-memory state of a GPU worker, while the restore process directly loads this state into a new container. This eliminates the need for repeated initialization steps, making it a valuable tool for reducing latency in AI inference pipelines.
