What is the difference between BF16 and FP16?

Both use 16 bits, but BF16 keeps the same exponent range as FP32 while FP16 tops out at 65,504. FP16 training needs loss scaling to stop small gradients turning into zeros. BF16 does not, which is why it replaced FP16 for training on Ampere and newer GPUs.

FP8 is an 8-bit floating point format with two encodings. E4M3 has more precision and stores values up to plus or minus 448, and E5M2 has more range and stores values up to plus or minus 57,344. It halves memory against 16-bit formats.

Which GPUs support FP8?

On NVIDIA, FP8 compute needs Ada Lovelace, Hopper or Blackwell, which means cards such as the L40S, RTX 4090, H100, H200, B200 and RTX 5090. The A100 and other Ampere cards have no FP8 hardware. vLLM also names the AMD MI300X as an FP8 target.

Can an H100 run FP4 models?

It can load them, but it does not accelerate them. NVIDIA's TensorRT support matrix marks FP4 on the H100 and L40S as hardware emulation mode, and vLLM falls back to weight-only execution on GPUs without a native FP4 kernel. Native FP4 starts with Blackwell.

Does FP8 hurt model accuracy?

A study of more than 500,000 evaluations on the Llama 3.1 family found FP8 weights and activations effectively lossless at every model size, and INT8 within 1 to 3 percent. You should still run your own evaluation set before switching a production model.

Is FP16 faster than BF16?

Not in any way the vendor publishes. The A100, L40S and H100 datasheets each print one throughput figure for both formats. The reason to pick BF16 is stability in training, not speed.

FP8 vs FP16 vs BF16: Which Precision, Which GPU

Use BF16 to train, FP8 to serve on any GPU that accelerates it, and FP4 only on Blackwell. FP16 is now the fallback for older cards such as the V100 and T4 that have no BF16 support. The table gives the default for each job.

Job	Default precision	When to go lower
Pre-training	BF16 mixed precision on Ampere or newer	FP8 through Transformer Engine on Hopper, Ada or Blackwell, once you have checked it against a BF16 baseline. NVIDIA described NVFP4 training as research in August 2025.
Fine-tuning	BF16 on Ampere or newer. FP16 with loss scaling only on Volta or Turing.	Usually not needed. Fine-tuning runs are short, so the stability of BF16 is worth more than the speed of FP8.
Inference	FP8 weights and activations on Ada, Hopper or Blackwell	NVFP4 on Blackwell. On Ampere and Turing, INT8 or 4-bit weight-only quantization.

The formats in plain terms

All vendor pages in this section were read on 21 September 2026 unless a date is shown.

FP32 and TF32

FP32 is 32-bit single precision at 4 bytes per parameter. It is the reference every lower format is judged against, and mixed-precision training still keeps a primary copy of the weights in it.

TF32 is a compute mode, not a storage format, introduced with Ampere. NVIDIA's developer blog (27 January 2021) describes it as 8 exponent bits, 10 mantissa bits and one sign bit: the range of FP32 with the precision of FP16. The PyTorch CUDA notes say the TF32 flag for matrix multiplies defaults to off in PyTorch 1.12 and later, so FP32 code on an A100 does not get TF32 speed unless you turn it on.

FP16

FP16 is 16-bit half precision at 2 bytes per parameter. Its weakness is range. NVIDIA's mixed-precision guide (updated 1 February 2023) gives the largest normal value as 65,504 and the smallest as about 6.10e-5, and small gradients fall below that floor.

BF16

BF16, or bfloat16, also uses 16 bits but spends more of them on the exponent. Google's Cloud TPU documentation (updated 18 September 2026) states that "the dynamic range of bfloat16 and float32 are equivalent" while BF16 uses half the memory. You give up some precision and keep the range. There is no published speed difference: the A100, L40S and H100 datasheets print one throughput figure for both formats.

FP8: E4M3 and E5M2

FP8 is 1 byte per parameter and comes in two encodings, defined in a joint NVIDIA, Arm and Intel paper (12 September 2022). The names give the split: E4M3 has 4 exponent bits and 3 mantissa bits, E5M2 has 5 and 2. The vLLM documentation gives the ranges: E4M3 stores values up to plus or minus 448 and has no infinity, and E5M2 stores values up to plus or minus 57,344 with infinity. The paper tested models of up to 175B parameters and reported results effectively matching 16-bit training.

FP4: NVFP4 and MXFP4

Four bits cannot cover a useful range alone, so both FP4 formats store a shared scale factor for each small block of values. NVIDIA's developer blog (24 June 2025) describes NVFP4 as 4-bit values with 1 sign bit, 2 exponent bits and 1 mantissa bit, an FP8 E4M3 scale for every block of 16 values, and a second FP32 scale per tensor. MXFP4 uses one power-of-two scale per block of 32 values. With the scales included, NVFP4 works out at about 4.5 bits per value and MXFP4 at about 4.25 (our arithmetic from those block sizes).

OpenAI's gpt-oss-120b model card on Hugging Face (August 2025) says its mixture-of-experts weights are quantized to MXFP4, which lets it run on a single 80 GB GPU.

INT8 and INT4

INT8 and INT4 are integer formats used for inference. NVIDIA's TensorRT-LLM precision reference (updated 15 September 2025) lists the common recipes: INT8 SmoothQuant for weights and activations (W8A8), INT4 and INT8 weight-only (W4A16 and W8A16), and GPTQ and AWQ (W4A16). In weight-only recipes the activations stay in 16-bit, so the saving is memory.

Why BF16 replaced FP16 for training

FP16 training needs loss scaling. Without it, NVIDIA's mixed-precision guide reports that in one network 31 percent of gradient values became zeros. The fix is to scale the loss up, keep a primary copy of the weights in FP32, and unscale the gradients before each update. It works, but it is one more step to get wrong.

BF16 removes the step. The original bfloat16 study (arXiv 1905.12322, 29 May 2019) found that BF16 training needs "no changes to hyper-parameters", whereas FP16 "requires hyper-parameter tuning", and that BF16 reached the same results as FP32 in the same number of iterations.

The catch is hardware. NVIDIA's TensorRT support matrix (8 September 2026) shows BF16 as not available on the T4 and supported on the A100. From Ampere onward, use BF16.

Hardware support by architecture

The matrix combines SemiAnalysis's Tensor Core history (23 June 2025) and NVIDIA's TensorRT 11.3.0 support matrix (8 September 2026). "Yes" means the hardware accelerates the format. For what the architecture names mean and which cards belong to each, see NVIDIA GPU generations explained.

Architecture	Rentable examples	FP16	BF16 and TF32	INT8	FP8	FP4
Volta	V100	Yes	No	No	No	No
Turing	T4	Yes	No	Yes	No	No
Ampere	A100, A10	Yes	Yes	Yes	No	No
Ada Lovelace	L40S, RTX 4090	Yes	Yes	Yes	Yes	Emulated only
Hopper	H100, H200	Yes	Yes	Yes	Yes	Emulated only
Blackwell	B200, B300, RTX 5090, RTX PRO 6000	Yes	Yes	Yes	Yes, plus MXFP8	Yes: NVFP4 and MXFP4

The "Emulated only" cells are the ones that catch people. The TensorRT matrix marks FP4 as supported on the H100 and the L40S, but the footnote reads "Supported in hardware emulation mode (hardware does not accelerate FP4 linear operations)". An FP4 checkpoint will load on an H100 and save memory. It will not run at FP4 speed. NVIDIA attaches the footnote to those two cards. We apply it to the rest of Ada and Hopper, which is our assumption.

The Volta row comes from the V100 datasheet and SemiAnalysis, because TensorRT now requires compute capability 7.5 or higher.

SemiAnalysis also records that Hopper deprecated the INT4 data types Turing had added, and that Blackwell has lower INT8 throughput than Hopper. FP4 also reaches consumer Blackwell: NVIDIA's RTX 5090 page advertises fifth-generation Tensor Cores with FP4.

AMD and Intel

AMD's Instinct MI325X datasheet lists performance for TF32, FP16, BF16, INT8 and FP8, and has no FP4 or FP6 rows. The MI355X brochure (June 2025) adds "expanded FP6 and FP4 datatype support" with rows for MXFP8, OCP-FP8, MXFP6 and MXFP4. The vLLM documentation names the AMD MI300X next to the NVIDIA H100 as a hardware-accelerated FP8 target.

Intel's Gaudi 3 white paper lists FP8 and BF16 matrix throughput for both Gaudi 2 and Gaudi 3.

What the datasheets publish, and what native FP8 costs today

The table below comes from our datasheet-verified spec table, with dense and with-sparsity figures in separate rows. A "Not published" cell means the vendor publishes no figure for that format, not that the format is unsupported. Use the matrix above for support and this table for scale.

Spec	A100	L40S	H100	H200	B200	RTX 5090
FP16 (dense)	312 TFLOPS	362.05 TFLOPS	989 TFLOPS	989 TFLOPS	2,250 TFLOPS	209.5 TFLOPS
FP16 (with sparsity)	624 TFLOPS	733 TFLOPS	1,979 TFLOPS	1,979 TFLOPS	4,500 TFLOPS	419 TFLOPS
FP8 (dense)	Not published	733 TFLOPS	1,979 TFLOPS	1,979 TFLOPS	4,500 TFLOPS	419 TFLOPS
FP8 (with sparsity)	Not published	1,466 TFLOPS	3,958 TFLOPS	3,958 TFLOPS	9,000 TFLOPS	838 TFLOPS
FP4 (dense)	Not published	Not published	Not published	Not published	9,000 TFLOPS	1,676 TFLOPS
FP4 (with sparsity)	Not published	Not published	Not published	Not published	18,000 TFLOPS	3,352 TFLOPS
INT8 (dense)	624 TOPS	733 TOPS	1,979 TOPS	1,979 TOPS	4,500 TOPS	838 TOPS
INT8 (with sparsity)	1,248 TOPS	1,466 TOPS	3,958 TOPS	3,958 TOPS	9,000 TOPS	1,676 TOPS
Launch year	2020	2023	2022	2024	2024	2025
Architecture	Ampere	Ada Lovelace	Hopper	Hopper	Blackwell	Blackwell

Figures from the vendor datasheets: A100, L40S, H100, H200, B200, RTX 5090, checked 13 Sep 2026. With-sparsity figures assume 2:4 structured sparsity and are twice the dense figure, so compare dense with dense. "Not published" means the vendor gives no figure.

On the H100 the dense FP8 figure of 1,979 TFLOPS is 2.0 times the dense FP16 figure of 989. On the B200 each step down doubles the dense figure again: 2,250 at FP16, 4,500 at FP8 and 9,000 at FP4. These are datasheet ceilings for dense math. Vendors also print with-sparsity figures twice as high. How to read GPU specs for AI explains why to ignore them. Measured serving gains are smaller than either.

These are the cheapest GPUs with native FP8 today.

GPU	Cheapest $/GPU-hr	Provider	Providers in stock
L40S	$0.97	Massed Compute	5
H100	$2.59	Vast.ai	10
H200	$3.43	QuantaCloud	9
B200	$3.75	Packet.ai	2
RTX 5090	$0.53	Vast.ai	3

Cheapest in-stock on-demand price per GPU-hour, from providers with live stock tracking. Latest stock observation: 21 Sep 2026, 07:13 UTC.

The L40S and RTX 5090 are the small-memory way into FP8, at 48 GB and 32 GB. The H100, H200 and B200 add the memory and NVLink that large models need. Every listing is on the L40S, H100 and B200 rent pages, and B200 vs H100 compares the generations.

The accuracy evidence

The best independent evidence we found is a study of the Llama 3.1 family built on more than 500,000 evaluations (arXiv 2411.02355, 4 November 2024, revised 26 May 2026). It found FP8 weights and activations (W8A8) "effectively lossless across all model scales". INT8 W8A8 showed "surprisingly low (1-3%) accuracy degradation". INT4 weight-only (W4A16) was "more competitive than expected, rivaling 8-bit quantization".

On speed, the vLLM documentation says FP8 "allows for a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput with minimal impact on accuracy". Compare that 1.6 with the 2.0 on the datasheet.

For FP4 the evidence is vendor data, so treat it with more care. NVIDIA's blog (24 June 2025) reports that NVFP4 cuts memory by about 3.5 times against FP16 and about 1.8 times against FP8, with accuracy loss of 1 percent or less on DeepSeek-R1-0528 when moving from FP8 to NVFP4. For training, NVIDIA wrote on 25 August 2025 that "NVFP4 training is still in the research phase". We have not seen an independent NVFP4 study on the scale of the Llama 3.1 one.

Which inference stacks support FP8 and FP4

vLLM. FP8 W8A8 runs on Ada Lovelace, Hopper and Blackwell, at compute capability 8.9 or higher. Its hardware table lists INT8 W8A8 and AWQ from Turing up. On Turing and Ampere it offers weight-only FP8 (W8A16) through Marlin kernels, which saves memory without FP8 compute. For NVFP4 it loads NVIDIA Model Optimizer checkpoints. Without a native FP4 kernel it falls back to weight-only execution and logs a warning.
TensorRT-LLM. Its precision reference lists FP8 for Hopper and NVFP4 for Blackwell.
SGLang. Its documentation lists FP8, NVFP4, MXFP4, MXFP8, INT8 and INT4, the flags --quantization fp8 and --quantization modelopt_fp4, an FP8 KV cache option and AMD Instinct support.
Transformer Engine. For training, NVIDIA's library supports FP8 on Hopper, Ada and Blackwell, plus MXFP8 and NVFP4 on Blackwell, in PyTorch and JAX.

Halving the bytes changes the GPU you need

Weights take 4 bytes per parameter in FP32, 2 in FP16 or BF16, 1 in FP8 or INT8, and about 0.5 in a 4-bit format. Runpod's documentation uses the same rule of thumb: about 2 GB per billion parameters at 16-bit, so roughly 140 GB for a 70B model, and about 35 GB for the same model at 4-bit.

At 16-bit the weights need two 80 GB H100s, and even the 141 GB H200 has almost nothing left over. At FP8 the weights are about 70 GB (our arithmetic at 1 byte per parameter) and fit on one 80 GB H100 with little to spare. At 4-bit they fit on a 48 GB L40S, but not on a 32 GB RTX 5090. Each halving can move you down a GPU class, or from two GPUs to one, and one GPU means the interconnect stops mattering.

Weights are not the whole bill. The KV cache grows with context and batch size, and the specs article linked above shows how to size it. The LLM VRAM calculator does the sum for your model, context and precision. The LLM GPU requirements reference lists common models against the GPUs that hold them, and the GPU spec reference has every card.

What to check before you switch precision

The architecture gate. FP8 compute needs Ada, Hopper or Blackwell. BF16 needs Ampere or newer. An A100 will not accelerate FP8 however the checkpoint is labelled, and A100 vs H100 shows what that step costs.
Accelerated or merely supported. FP4 on an H100 or L40S is emulation. Weight-only FP8 on Ampere saves memory, not time. Check your server logs for a fallback warning on first load.
Your own evaluation. The published results are for Llama 3.1. Run your evaluation set at both precisions before you switch production traffic.
The KV cache. Quantizing weights does not shrink it. An FP8 KV cache is a separate setting.
For training, a baseline. Start in BF16. Move to FP8 only after a short run matches your BF16 loss curve.

The decision rule: train in BF16. Serve in FP8 on the cheapest Ada, Hopper or Blackwell card that holds your model. Go to 4-bit only when it lets you drop a GPU or a GPU class, with NVFP4 on Blackwell and weight-only INT4 everywhere else.

Sources

Source pages were retrieved with automated research tools on the dates shown and each figure was traced back to its source before publishing. Rental prices on this page are not typed: they are read live from GPUperhour's own data.

Every source was accessed on 21 September 2026. A date in an entry is the publication or last-update date.