Case StudyHow Enkrypt AI ships its AI security platform into air-gapped KubernetesRead the case study
Skip to main content

Glossary

What Is GPU Fractioning? Sharing One GPU Across Many Jobs


What Is GPU Fractioning?

GPU fractioning is the partitioning of a single physical GPU into multiple logical slices, each with hard memory and compute limits, so several small inference or training jobs can safely share one accelerator instead of each monopolising a whole card. It exists because most inference jobs use a fraction of a modern GPU, and paying for a whole card per job is expensive and wasteful.

On 13 October 2026, NVIDIA open-sourced the full Run:ai platform under Apache 2.0, including the KAI Scheduler, a CNCF Sandbox project, and its GPU-fractioning capability. Self-hosted fractioning and bin-packing are now possible without the former enterprise licence, which is why the topic is suddenly live for every platform team running GPUs on Kubernetes. (HPCwire / AIwire)

Why does GPU fractioning matter now?

Because GPUs are the most expensive line in most AI infrastructure budgets, and most of that capacity sits idle. A typical inference workload uses a slice of a card's memory and compute, so running one job per GPU can leave the majority of an expensive accelerator unused. Fractioning turns that waste into density: more jobs per card, fewer cards, lower spend. With NVIDIA open-sourcing Run:ai and the KAI Scheduler in October 2026, the orchestration needed to do this safely is now available to self-hosted teams, not just managed-service customers.

How does GPU fractioning work?

There are three main approaches, and they trade isolation against flexibility.

ApproachHow it worksIsolationMemory guaranteesBest fit
MIG (Multi-Instance GPU)Hardware partitions the GPU into fixed instancesStrongest, hardware-enforcedHard, per instanceMulti-tenant, strict isolation, supported data-center GPUs
Time-slicingThe scheduler interleaves jobs on the whole GPU over timeWeakest, no memory isolationNone, jobs can contendBursty, trusted, low-stakes workloads
Software fractioningA scheduler enforces per-workload memory and compute limits in softwareMedium, software-enforcedSoft to hard, depending on toolFlexible bin-packing of many small inference jobs

MIG gives the hardest isolation but only in fixed, coarse sizes on supported cards. Time-slicing is flexible but offers no memory protection, so one job can starve another. Software fractioning, which is what the KAI Scheduler provides, sits in between: finer-grained than MIG and safer than raw time-slicing, which is why it fits inference bin-packing.

GPU fractioning vs MIG, what is the difference?

MIG is hardware partitioning; GPU fractioning is the broader practice, usually done in software. MIG carves a GPU into a fixed set of isolated instances at the hardware level, with strong guarantees but coarse, pre-defined sizes and limited card support. Software GPU fractioning lets a scheduler assign arbitrary memory and compute slices to many jobs, with more flexibility and finer packing, at the cost of software-enforced rather than hardware-enforced isolation. Teams often use both: MIG where strict multi-tenant isolation is required, software fractioning where density and flexibility matter more.

What breaks without GPU fractioning?

Without fractioning, you pay for whole GPUs to run jobs that need a slice, so utilisation stays low and GPU spend stays high. Teams either over-provision, buying far more accelerators than the workload needs, or they queue jobs behind each other on scarce cards, slowing everyone down. Both are expensive, and both get worse as the number of small inference services grows.

How CloudRaft helps with GPU fractioning

CloudRaft deploys and tunes GPU orchestration on Kubernetes. We stand up Run:ai and the KAI Scheduler, set fractioning and bin-packing policy so your inference workloads share accelerators without starving each other, and size the cluster so you stop paying for whole GPUs to run jobs that need a slice. For teams moving off managed GPU services to self-hosted, this is the orchestration layer we build and operate. Talk to our team about GPU orchestration.

Conclusion

GPU fractioning is how you make expensive accelerators earn their cost: by packing many small jobs onto each card safely, instead of dedicating a whole GPU to a job that needs a slice. With Run:ai and the KAI Scheduler now open source, self-hosted teams can run the same density the managed platforms do, if the orchestration is set up and tuned correctly.