← PortfolioCompute and infrastructure

Cedana

GPU workload mobility for AI inference

Team

Neel MasterCEO

Niranjan RavichandraCTO

Founded

2023

Invested

2023

Links

Website · LinkedIn

The problem

Why do GPUs serving AI spend so much time producing nothing?

Click the machine the job is running on to take it away. The job is saved at the driver, moved to another machine and carries on where it stopped.A job runs on one of three machines, and every call it makes to the GPU passes through a layer at the driver on the way down. When its machine is taken back, the GPU memory and the changes to its files are saved as one checkpoint, moved to another machine and put back. The job carries on from where it stopped, while a job with no save has to start again from the beginning.An illustration, not real data.

How saving a program works

means saving a snapshot of a program's state, so it can restart from that point after a failure instead of from scratch. On Linux the standard tool is , short for Checkpoint/Restore in Userspace. It can freeze a running application, write it to disk as a collection of files, and later bring it back from the exact moment it was frozen. Think of a video game's save button, pressed from outside the game.

CRIU can do this because everything it saves, from memory and threads to open files, sockets and pipes, is defined by Linux and looks the same on any hardware. A GPU is another matter. It does things a standard Linux kernel knows nothing about, so CRIU can't save it alone. There are two ways in. Until recently the only one was : sit between the program and the GPU, and watch every call it makes. NVIDIA now offers a native route too, a cuda-checkpoint utility that locks the driver, lets work already sent to the GPU finish, copies the GPU's memory over to the host and lets go of the GPU. On restore, the memory goes back onto the GPU at its original addresses. Cedana doesn't use NVIDIA's utility. It built its own checkpointing on the interception route, by virtualising the GPU.

Why does that matter for serving models? Inference used to be cheap to throw away. Not anymore. Every turn of a conversation adds to the the model keeps in GPU memory, and on GLM-5.2 at FP8 a 100K-token context comes to roughly 4.5 GB of it. Agents lean on that hard. Across agent benchmarks, 84.6 to 99.5% of input tokens get reused from one turn to the next, so the cache is the session's accumulated work, not scratch paper.

Further reading Application checkpointing (Wikipedia)CRIU (Wikipedia)cuda-checkpoint (NVIDIA on GitHub)Checkpointing CUDA Applications with CRIU (NVIDIA Technical Blog)Cedana vs. CRIUgpu for GPU Checkpoint/Restore (Cedana Docs)The era of stateful inference (Cedana)

Why it is hard

i.
Waking a model up
Starting a frontier model isn't like starting a container. Placing it across GPUs, fetching the weights, building CUDA graphs and warming the cache takes 10 minutes, sometimes up to an hour, before the first token comes out. Nobody waits that long for an answer, so operators keep spare and GPU pools warm. That turns a scaling problem into an overprovisioning one: GPUs on standby, drawing power, serving nothing.
ii.
Failures get bigger
Big fleets break all the time. Production clusters report one failure per 7.9 hours per 1,024 GPUs. Models now span so many GPUs that one bad chip drags others down with it: on an NVL72 rack, a single GPU failure can hit the 71 bound to it. When a GPU dies mid-session the transcript survives. What's lost is the expensive part, the KV cache and the computation in flight. The retry re-runs rather than resumes.
iii.
A computer inside the computer
A GPU is harder to snapshot than a CPU because it is built differently, down to its memory system, its dynamic parallelism and the way its threads synchronise. It is also something of a sealed box. Getting at its state takes low-level access to the driver, and NVIDIA's drivers are proprietary.
iv.
Every route costs something
Interception isn't free. Logging every call and replaying it on restore adds performance overhead, and it needs hardware-specific code that is hard to test and maintain. The native route has its own catch. It waits for work already sent to the GPU to finish before a checkpoint completes.

Further reading The era of stateful inference (Cedana)CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads (arXiv)Performance of Cedana's GPU Interception (Cedana Docs)Checkpointing CUDA Applications with CRIU (NVIDIA Technical Blog)

What Cedana is after

Cedana's diagnosis is short: "AI workloads cannot move once running." A model pinned to one machine can't step off a failing node, can't be parked and woken quickly, and can't be shuffled to fill idle GPUs.

Cedana wants inference to move. It checkpoints, migrates and resumes running inference across a fleet, whatever the engine, model or orchestrator, to raise throughput, reliability and AI revenue per megawatt. Cost per token and revenue per megawatt are the same equation read from opposite sides: both come down to how many tokens each GPU actually delivers. Cedana's own name for where this goes is an OS for AI factories.

Further reading GPU Migration Infrastructure (Cedana)The era of stateful inference (Cedana)

How they go at it

Step 1: Listen at the driver
Cedana interposes at the while a process runs. From there it can capture and restore GPU state, even for multi-GPU jobs spread across several machines, and move workloads live. Its optimisations effectively turn it into a virtual machine for NVIDIA GPUs.
Step 2: Earn the overhead back
Watching every call costs something, so Cedana tries to pay for itself. Seeing every GPU operation lets it merge CUDA graph calls and drop redundant ones, a bit like a just-in-time compiler. And because the CPU often sits idle waiting on the GPU anyway, much of the extra work hides inside that wait. Its own benchmarks show a "minimal overhead of 3%" on raw GPU work.
Step 3: Save the whole server
A checkpoint covers the entire inference state at the OS and CUDA driver level: weights, KV cache, allocator state, CUDA graphs and whatever is in flight. Putting all that back is much faster than building it from nothing. Cedana restores multi-GPU frontier models, serving state included, in 57 to 70 seconds, 10 to 29 times faster than a native cold start. On the benchmark on its homepage, a restored model reaches first token 21.7 times sooner.
Step 4: More models per node
Fast restores change what one node can hold. In one benchmark, four coding models that needed 14 GPUs between them shared a single node, swapping in and out 29 times. With Cedana the run finished in 152 minutes. The same work on the same GPU without it took 317.
Step 5: No rewrites
None of this asks for changes to what's being served. It drops into Kubernetes with no rip-and-replace and no code changes, and it brings the container's files along, diffing the filesystem into the checkpoint.

Further reading Performance of Cedana's GPU Interception (Cedana Docs)A coding model portfolio on a single node (Cedana)GPU Migration Infrastructure (Cedana)Coding portfolio benchmark (Cedana)What We Checkpoint (Cedana Docs)

Still open

Which layer should do the saving?
NVIDIA's driver now checkpoints natively, and recent versions add GPU migration. The CRIUgpu researchers point out that this avoids interception's overhead and upkeep. The case for interception is that seeing every call opens up optimisations a native switch can't. Both routes are moving quickly, and it isn't settled which wins where.
What should happen to the state?
A whole industry layer now keeps the record of a session safe: transcripts, workspaces, sandbox snapshots. The computation itself is still the one uninsured asset in the stack. An offload tier can reload cache it already wrote out, but the in-flight computation has no copy anywhere. With DeepSeek-V4 shipping a 1M-token context the state at stake keeps growing, and whether to save it, offload it or recompute it is still being worked out.

Further reading cuda-checkpoint (NVIDIA on GitHub)CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads (arXiv)Cedana vs. CRIUgpu for GPU Checkpoint/Restore (Cedana Docs)The era of stateful inference (Cedana)

About Cedana

Cedana lets AI operators serve more inference from the GPUs and power they already have. Both are scarce, and a lot of both goes to waste: a GPU that is cold-starting a model, recovering from a fault or sitting idle as a warm spare uses capacity without producing a single token.

Cedana's software saves, moves and restores running GPU workloads at the CUDA driver layer, without changes to the code, the model, the inference engine or the orchestrator. A policy-driven control plane puts that to work across the whole fleet. Cold starts get 10-40x faster, so the same GPUs can serve a wider range of models with fewer idle replicas. When a GPU or node fails, work resumes on healthy hardware without losing progress. And because workloads can move to wherever capacity is free, more of every GPU-hour and every megawatt turns into finished work. Cedana is being deployed with inference providers and neoclouds.

Words used here

Checkpointing: Saving a snapshot of a running program's state so it can later resume from that point.
CRIU: Checkpoint/Restore in Userspace, an open-source Linux tool that freezes a running program to files and restores it.
KV cache: What a model keeps in GPU memory about the conversation so far, so it doesn't recompute every earlier token on each turn.
replicas: Copies of the same model running side by side, so traffic can be spread out or moved when one fails.
interception: Placing a layer between a program and the GPU so every call to the GPU passes through it and can be recorded.
Driver API: The low-level interface a program uses to talk to NVIDIA's GPU driver.

Sources

Why do GPUs serving AI spend so much time producing nothing?

Waking a model up

Failures get bigger

A computer inside the computer

Every route costs something