Pick one of 27 open models or enter your own, set the precision, context and batch, and get the GPU memory it needs. Then see every setup of 1, 2, 4 or 8 GPUs that holds it, cheapest first, at today's rental prices.
Last reviewed . Prices on this page are read live and are not part of the review date.
VRAM needed
80.6 GB
| Setup | Combined VRAM | Cost per hour | FP8 figure published | Decode ceiling |
|---|---|---|---|---|
| Nothing up to 8 GPUs fits this. It needs more than one node. | ||||
Llama 3.1 70B has 70.55 billion parameters, 80 layers, 8 KV heads and a head dimension of 128. At FP8 each parameter is one byte, so the weights are 70.6 GB. One 8,192-token sequence with a 16-bit cache adds 2.7 GB. Ten percent overhead on both is 7.3 GB, for a total of 80.6 GB. Live prices are unavailable right now, so the setups below carry no cost.
| Setup | Combined VRAM | Cost per hour | Decode ceiling |
|---|---|---|---|
| 1x H100 94 GB | 94 GB | No current offer | 47 tok/s |
| 1x Gaudi 2 96 GB | 96 GB | No current offer | 35 tok/s |
| 1x GH200 96 GB | 96 GB | No current offer | 57 tok/s |
| 1x RTX PRO 6000 96 GB | 96 GB | No current offer | 25 tok/s |
| 1x MI250X 128 GB | 128 GB | No current offer | 46 tok/s |
| 1x H200 141 GB | 141 GB | No current offer | 68 tok/s |
Model presets and their sources are listed in full on LLM GPU requirements; every architecture number is from the model's Hugging Face config.json. GPU memory, bandwidth and the FP8 flag come from the datasheet-checked table behind the GPU specs chart. Prices are read live and never typed. The calculator runs in your browser; nothing you enter is sent anywhere.
Add three things. Weights: parameters times bytes per parameter (2 for FP16 or BF16, 1 for FP8, 0.5 for INT4). KV cache: 2 x layers x KV heads x head dimension x bytes per value x tokens, where tokens is context length times concurrent sequences. Overhead: 10% of the first two. For Llama 3.1 70B at FP8 with a 8,192-token context that is 70.6 GB + 2.7 GB + 7.3 GB = 80.6 GB.
It grows with every token you keep in context and every request you serve at once. Llama 3.1 70B stores 2 x 80 layers x 8 KV heads x 128 head dimension x 2 bytes per token, which comes to 2.7 GB for 8,192 tokens. Ten concurrent requests need ten times that. An FP8 cache halves it.
The calculator's rule: the frozen base weights at the precision you pick (INT4 is the QLoRA case), plus 16 bytes for every trainable parameter (16-bit weight and gradient, a 32-bit master copy and two Adam moments), plus checkpointed activations of batch x sequence length x hidden size x layers x 2 bytes, plus 10% overhead. It is a floor. Long sequences push real use higher because attention and MLP blocks are recomputed at full size.
Yes. Tensor parallelism splits the weights and the KV cache evenly across GPUs, so what matters is their combined VRAM. The calculator checks 1, 2, 4 and 8 GPUs of each type, which are the sizes providers rent. Splitting works best when the GPUs share a fast link such as NVLink; over plain PCIe it still fits but runs slower.
An upper bound on tokens per second for a single request. Generating a token reads every active weight once, so speed cannot exceed memory bandwidth divided by the size of the active weights. Real throughput is lower, and batching many requests raises total throughput well above it. It is there to compare GPUs, not to promise a number.
From live provider listings, read when the page is built and refreshed at least hourly: the cheapest on-demand price per GPU among offers that are in stock, from secure providers, seen in the last 15 minutes. The cost shown is that price times the number of GPUs. A provider may not sell that exact count in one machine, so follow the link to the rent page to check.