LLM VRAM calculator

Pick one of 27 open models or enter your own, set the precision, context and batch, and get the GPU memory it needs. Then see every setup of 1, 2, 4 or 8 GPUs that holds it, cheapest first, at today's rental prices.

Last reviewed . Prices on this page are read live and are not part of the review date.

Workload

VRAM needed

80.6 GB

Weights
70.6 GB
KV cache
2.7 GB
Overhead (10%)
7.3 GB

GPU setups that fit

SetupCombined VRAMCost per hourFP8 figure publishedDecode ceiling
2x RTX A6000 48 GB96 GB$0.89 on LeaderGPUNo22 tok/s
1x Gaudi 2 96 GB96 GB$0.91 on LeaderGPUNo35 tok/s
2x A40 48 GB96 GB$0.98 on RunPodNo20 tok/s
4x RTX 3090 24 GB96 GB$1.07 on Vast.aiNo53 tok/s
4x RTX A5000 24 GB96 GB$1.08 on RunPodNo44 tok/s
4x L4 24 GB96 GB$1.33 on Vast.aiYes17 tok/s
2x A100 80 GB160 GB$1.35 on LeaderGPUNo58 tok/s
8x RTX 5060 16 GB128 GB$1.41 on Vast.aiNo51 tok/s
4x A10 24 GB96 GB$1.48 on LeaderGPUNo34 tok/s
2x RTX 6000 Ada 48 GB96 GB$1.56 on QuantaCloudYes27 tok/s
2x L40 48 GB96 GB$1.72 on Massed ComputeYes24 tok/s
1x RTX PRO 6000 96 GB96 GB$1.89 on VERDAYes25 tok/s
8x RTX 2000 Ada 16 GB128 GB$1.92 on RunPodYes25 tok/s
2x L40S 48 GB96 GB$1.94 on Massed ComputeYes24 tok/s
8x RTX A4000 16 GB128 GB$2.00 on RunPodNo51 tok/s

Worked example: Llama 3.1 70B at FP8 with 8,192 tokens

Llama 3.1 70B has 70.55 billion parameters, 80 layers, 8 KV heads and a head dimension of 128. At FP8 each parameter is one byte, so the weights are 70.6 GB. One 8,192-token sequence with a 16-bit cache adds 2.7 GB. Ten percent overhead on both is 7.3 GB, for a total of 80.6 GB. The cheapest setup that holds it right now is 2x RTX A6000 48 GB at $0.89 per hour on LeaderGPU.

SetupCombined VRAMCost per hourDecode ceiling
2x RTX A6000 48 GB96 GB$0.89 on LeaderGPU22 tok/s
1x Gaudi 2 96 GB96 GB$0.91 on LeaderGPU35 tok/s
2x A40 48 GB96 GB$0.98 on RunPod20 tok/s
4x RTX 3090 24 GB96 GB$1.07 on Vast.ai53 tok/s
4x RTX A5000 24 GB96 GB$1.08 on RunPod44 tok/s
4x L4 24 GB96 GB$1.33 on Vast.ai17 tok/s

Prices observed , USD, on-demand, in stock, per GPU times the GPU count.

Method and sources

  • Weights = parameters x bytes per parameter: 2 for FP16 or BF16, 1 for FP8, 0.5 for INT4. gpt-oss is published in MXFP4, so its MXFP4 option uses the size of the published checkpoint.
  • KV cache = 2 x layers x KV heads x head dimension x bytes x context length x concurrent sequences. Layers with a sliding or chunked window (Gemma 3, gpt-oss, Llama 4) stop at the window. DeepSeek V3 and R1 cache one compressed vector per token per layer.
  • Overhead = 10% of the above, for the CUDA context, activations and fragmentation.
  • LoRA mode = frozen base weights + 16 bytes per trainable parameter + batch x sequence x hidden size x layers x 2 bytes of checkpointed activations + 10% overhead. Trainable parameters default to 1% of the model, typical for adapters on every linear layer at a moderate rank. Treat the result as a floor.
  • Fit = the smallest of 1, 2, 4 or 8 GPUs whose combined VRAM is at least the total, assuming an even tensor-parallel split. 1 GB is 1,000,000,000 bytes and an 80 GB card counts as 80 GB.

Model presets and their sources are listed in full on LLM GPU requirements; every architecture number is from the model's Hugging Face config.json. GPU memory, bandwidth and the FP8 flag come from the datasheet-checked table behind the GPU specs chart. Prices are read live and never typed. The calculator runs in your browser; nothing you enter is sent anywhere.

Questions

How do you calculate VRAM for an LLM?

Add three things. Weights: parameters times bytes per parameter (2 for FP16 or BF16, 1 for FP8, 0.5 for INT4). KV cache: 2 x layers x KV heads x head dimension x bytes per value x tokens, where tokens is context length times concurrent sequences. Overhead: 10% of the first two. For Llama 3.1 70B at FP8 with a 8,192-token context that is 70.6 GB + 2.7 GB + 7.3 GB = 80.6 GB.

How much VRAM does the KV cache use?

It grows with every token you keep in context and every request you serve at once. Llama 3.1 70B stores 2 x 80 layers x 8 KV heads x 128 head dimension x 2 bytes per token, which comes to 2.7 GB for 8,192 tokens. Ten concurrent requests need ten times that. An FP8 cache halves it.

How much VRAM does LoRA fine-tuning need?

The calculator's rule: the frozen base weights at the precision you pick (INT4 is the QLoRA case), plus 16 bytes for every trainable parameter (16-bit weight and gradient, a 32-bit master copy and two Adam moments), plus checkpointed activations of batch x sequence length x hidden size x layers x 2 bytes, plus 10% overhead. It is a floor. Long sequences push real use higher because attention and MLP blocks are recomputed at full size.

Can I split a model across several GPUs?

Yes. Tensor parallelism splits the weights and the KV cache evenly across GPUs, so what matters is their combined VRAM. The calculator checks 1, 2, 4 and 8 GPUs of each type, which are the sizes providers rent. Splitting works best when the GPUs share a fast link such as NVLink; over plain PCIe it still fits but runs slower.

What is the decode ceiling column?

An upper bound on tokens per second for a single request. Generating a token reads every active weight once, so speed cannot exceed memory bandwidth divided by the size of the active weights. Real throughput is lower, and batching many requests raises total throughput well above it. It is there to compare GPUs, not to promise a number.

Where do the prices come from?

From live provider listings, read when the page is built and refreshed at least hourly: the cheapest on-demand price per GPU among offers that are in stock, from secure providers, seen in the last 15 minutes. The cost shown is that price times the number of GPUs. A provider may not sell that exact count in one machine, so follow the link to the rent page to check.