Three numbers on a GPU datasheet decide almost every AI workload: memory capacity (VRAM), memory bandwidth, and dense throughput at the precision you will actually run. Most of the rest is either a core count that cannot be compared across generations or a headline figure that assumes a pruning technique your model probably does not use. This page covers the three numbers, the three traps around them, and what our own spec table got wrong until this month.
The three numbers that matter
1. VRAM decides whether the job runs at all
Work out the weights first. NVIDIA's inference guide (17 November 2023) gives the basic sum: a 7 billion parameter model in 16-bit precision takes about 14 GB. The bytes per parameter are 4 for FP32, 2 for FP16 or BF16, 1 for FP8 or INT8, and 0.5 for 4-bit formats. Runpod's pod selection docs (read 21 September 2026) turn this into a rule of thumb: about 2 GB of VRAM per billion parameters at 16-bit, so a 70B model needs about 140 GB and more than one GPU, while a 4-bit 70B model fits in about 35 GB.
Then add the KV cache. NVIDIA's formula is batch size × sequence length × 2 × layers × hidden size × bytes per value. For Llama 2 7B at FP16, one sequence of 4,096 tokens is about 2 GB. It grows linearly with both context and batch. Google DeepMind's scaling book (4 February 2025) notes that for LLaMA-13B a single 8,192-token sequence in BF16 needs 6.7 GB of cache, and four of them exceed the weights. Models with grouped-query attention need less, because the formula then counts KV heads, not attention heads.
Training is a different budget. Hugging Face's memory guide (read 21 September 2026) counts 6 bytes per parameter for mixed-precision weights, 8 for Adam states and 4 for gradients: 18 before activations, or about 126 GB to fully fine-tune a 7B model by our arithmetic. The LLM VRAM calculator and the LLM GPU requirements reference do these sums for you.
2. Memory bandwidth decides how fast tokens come out
Once the model fits, generation speed at small batch sizes is set by how quickly the GPU can read its own memory. Trap three explains why.
3. Dense throughput at your precision decides training and prefill speed
Training, prompt processing and large-batch serving are compute-bound. The figure to read is the dense TFLOPS row for the precision your software uses, and the hardware has to accelerate that precision. Which architecture accelerates which format is covered in FP8 vs FP16 vs BF16.
| Spec | A100 | L40S | H100 | H200 | B200 |
|---|---|---|---|---|---|
| FP32 | 19.5 TFLOPS | 91.6 TFLOPS | 67 TFLOPS | 67 TFLOPS | 75 TFLOPS |
| FP16 (dense) | 312 TFLOPS | 362.05 TFLOPS | 989 TFLOPS | 989 TFLOPS | 2,250 TFLOPS |
| FP16 (with sparsity) | 624 TFLOPS | 733 TFLOPS | 1,979 TFLOPS | 1,979 TFLOPS | 4,500 TFLOPS |
| FP8 (dense) | Not published | 733 TFLOPS | 1,979 TFLOPS | 1,979 TFLOPS | 4,500 TFLOPS |
| FP8 (with sparsity) | Not published | 1,466 TFLOPS | 3,958 TFLOPS | 3,958 TFLOPS | 9,000 TFLOPS |
| Memory bandwidth | 2,039 GB/s | 864 GB/s | 3,350 GB/s | 4,800 GB/s | 8,000 GB/s |
| VRAM | 40 to 80 GB | 48 GB | 80 to 94 GB | 141 GB | 180 to 192 GB |
| TDP | 400 W | 350 W | 700 W | 700 W | 1,000 W |
The table shows dense and with-sparsity figures as separate rows. "Not published" is not a zero. It means the vendor gives no such figure, which is why the A100 has no FP8 figure.
What is not on the list: core counts
CUDA cores and Tensor cores are different blocks of silicon and are counted separately. The L40S has 18,176 CUDA cores and 568 Tensor cores. The V100 has 5,120 CUDA cores and 640 Tensor cores. NVIDIA describes Tensor cores as the part that enables "mixed-precision computing, dynamically adapting calculations to accelerate throughput while preserving accuracy".
Neither count compares across generations. The V100's 640 first-generation Tensor cores deliver 125 TFLOPS of dense FP16. The L40S's 568 fourth-generation Tensor cores deliver 362.05 TFLOPS of dense FP16 and 733 TFLOPS of dense FP8. Fewer cores, 2.9 times the FP16 throughput, and 5.9 times with FP8. Use core counts to confirm which chip you were given, never to rank two chips. To work out which generation a chip belongs to, see our guide to NVIDIA GPU generations and names.
Trap one: the headline TFLOPS figure assumes sparsity
NVIDIA's definition of 2:4 structured sparsity (developer blog, 20 July 2021): "In each contiguous block of four values, two values must be zero." Half the weights are removed, and the workflow is to prune the network and then repeat the original training procedure to recover accuracy. NVIDIA's A100 datasheet says its Tensor Cores "can provide up to 2X higher performance for sparse models", and datasheets since then credit that as double throughput.
The measured gain in the same post is much smaller: "over 30% performance/watt gain compared to dense networks", and close to 20% for ResNeXt-101 on an A100 at large batch. If you have not pruned and retrained your model in the 2:4 pattern, none of it applies. Your number is the dense one, which is half.
The L40S shows how far a headline can drift. NVIDIA's product page leads with "Tensor Performance 1,466 TFLOPS". That is the FP8 figure with sparsity. Dense FP8 is 733 TFLOPS, dense FP16 is 362.05 and FP32 is 91.6. The headline is 16 times the FP32 figure. A factor of two of that is sparsity and the rest is lower precision.
The H100 is the same. On NVIDIA's H100 page (read 21 September 2026), the TF32, BF16, FP16, FP8 and INT8 rows are all marked "* With sparsity". Only FP64 (34 TFLOPS) and FP32 (67 TFLOPS) carry no footnote. So the honest answer to "how many TFLOPS is an H100" is 3,958 for sparse FP8, 1,979 for dense FP8, 989 for dense FP16 and 67 for FP32. Never quote one without its label.
Some pages print both. The A100 datasheet prints pairs, 312 dense and 624 with sparsity for FP16. NVIDIA's HGX Blackwell table is labelled "Specification in Sparse | Dense", with the note "Dense is ½ sparse spec shown"; per B200 that is 18,000 sparse and 9,000 dense TFLOPS at FP4. AMD's MI325X datasheet has a "PERFORMANCE with sparsity" row that doubles each figure, 1,307.4 to 2,614.9 TFLOPS at FP16. The RTX PRO 6000 Blackwell page prints only the large number and footnotes it as "Theoretical FP4 TOPS using sparsity".
Baselines also get mixed inside one comparison. A footnote on NVIDIA's HGX page reads "HGX Rubin NVL8 with Sparse NVFP4, HGX B200 with Dense NVFP4". Read the footnotes before the bar charts.
Trap two: H100 and H200 list identical compute
Put the two datasheets side by side and every compute row matches: 16,896 CUDA cores, 528 Tensor cores, 67 TFLOPS of FP32, 3,958 TFLOPS of sparse FP8, and a 700 W TDP. The H200 is a memory upgrade. The H100 carries 80 GB of HBM3 (94 GB on the NVL card) at 3,350 GB/s. The H200 carries 141 GB of HBM3e at 4,800 GB/s. That is 61 GB more than the 80 GB card and 1.4 times the bandwidth.
So the H200 is worth a premium in two cases: the model plus its KV cache does not fit in 80 GB, or decode is bandwidth-bound. For a compute-bound training run that already fits on an H100, it buys very little. The H100 vs H200 comparison shows the live price gap.
Trap three: bandwidth, not TFLOPS, bounds decode speed
Generating a token means reading every weight, plus the KV cache, out of GPU memory. NVIDIA's inference guide says of decode that the speed at which data is transferred from memory "dominates the latency, not how fast the computation actually happens". Databricks (12 October 2023) puts it the same way and gives a worked example: a 7B model at 16-bit producing a token every 14 ms is moving 14 GB in 14 ms, which is 1 TB/s and about half of an A100's peak bandwidth. The DeepMind scaling book gives the floor as a formula: minimum step time = (batch size × KV cache size + parameter size) / memory bandwidth.
This changes rankings. By dense FP8 throughput, an H100 is 2.7 times an L40S. By memory bandwidth, 3,350 GB/s of HBM3 against 864 GB/s of GDDR6, it is 3.9 times. For single-stream chat, the second ratio is the one you will feel. For prompt-heavy or large-batch work, the first matters more, because prefill is compute-bound. An L40S can still be the right choice for a small model; weigh that ratio against the live price gap, not the TFLOPS headline.
Lower precision helps on both fronts. FP8 weights are half the bytes of FP16, so there is half as much to read per token. The FP8 article linked above has the measured gains.
One qualifier on all three numbers: MIG. It lets a GPU from Ampere onward be split into up to seven isolated instances, each with its own path through the memory system, per NVIDIA's MIG guide (updated 11 September 2026). A listing sold as a MIG slice gives you a fraction of the card's VRAM and compute. NVIDIA's L40S page says that card does not support it, and GeForce cards are absent from the supported list. What is MIG covers slices in full.
What we found when we audited our own spec table
In September 2026 we audited the spec table behind this site. For 47 of our 60 GPU families, the stored FP16 figure was simply the FP32 figure, the result of a shortcut in a seeding script. The damage was wide: 1,492 comparison pages cited an FP16 number that their own spec table showed as "Not published", because the stored value was still passed to the software that generates those pages.
We fixed the data first. Dense and with-sparsity figures for FP16, FP8 and INT8 were entered for those 47 families from vendor datasheets, along with per-model TDP and memory bandwidth. The spec tables on this site now show dense and with-sparsity figures as separate rows, and a figure the vendor does not publish reads "Not published".
Then we added a validator that rejects any numeral in generated text that is not in that page's spec data, and regenerated all 1,711 GPU-family comparison pages. The first run rejected 390 of them for that reason. After three passes, 1,710 carried corrected content. The full spec chart is the result, and its blanks are deliberate.
Seven questions to ask of any spec sheet
- Is this figure dense or "with sparsity"? If the page does not say, look for an asterisk. Assume the headline is sparse.
- Which precision is it for, and does this architecture accelerate that precision or only emulate it?
- Is it per GPU, per eight-GPU board or per rack? HGX and NVL72 pages quote system totals.
- Which variant is it? SXM, PCIe and NVL versions of one chip can differ in memory, bandwidth and power.
- Do the weights plus the KV cache at your context length and batch size fit in VRAM, with headroom?
- What is the memory bandwidth, and is your workload decode-bound or compute-bound?
- Does each side of a comparison use the same baseline? Sparse against dense, or FP4 against FP8, is not a comparison.
The decision rule is short. For serving one model on one GPU, pick by VRAM first and memory bandwidth second. For training and batch work, pick by VRAM first and dense throughput at your precision second. Ignore core counts and any TFLOPS figure with an asterisk, then let the live price settle it. The best GPU for LLM guide applies the rule to current prices.
Sources
Source pages were retrieved with automated research tools on the dates shown and each figure was traced back to its source before publishing. Rental prices on this page are not typed: they are read live from GPUperhour's own data.
Every source was accessed on 21 September 2026. A date in brackets is the publication or last-update date.
- NVIDIA L40S product page
- NVIDIA H100 product page
- NVIDIA H200 product page
- NVIDIA A100 datasheet (PDF)
- NVIDIA V100 datasheet (PDF)
- NVIDIA HGX platform page
- NVIDIA Tensor Cores page
- NVIDIA RTX PRO 6000 Blackwell product page
- NVIDIA developer blog: Accelerating Inference with Sparsity Using Ampere and TensorRT (20 July 2021)
- NVIDIA developer blog: Mastering LLM Techniques, Inference Optimization (17 November 2023)
- NVIDIA MIG user guide: introduction and supported GPUs (updated 11 September 2026)
- AMD Instinct MI325X datasheet (PDF)
- Databricks: LLM Inference Performance Engineering Best Practices (12 October 2023)
- Google DeepMind scaling book: inference chapter (4 February 2025)
- Hugging Face: Model memory anatomy
- Runpod docs: Choose a Pod