GH200 vs RTX 4090

The FP16 dense rating of 989 TFLOPS on the GH200 exceeds the 165.2 TFLOPS dense rating on the RTX 4090 by a factor of six. This gap widens further when sparsity is applied because the GH200 reaches 1979 TFLOPS with sparsity while the RTX 4090 reaches 330.4 TFLOPS with sparsity. Training workloads that rely on dense FP16 therefore complete iterations faster on the GH200. Inference workloads that can exploit sparsity also favor the GH200. Memory bandwidth of 4000 GB/s on the GH200 versus 1008 GB/s on the RTX 4090 permits larger batch sizes before memory stalls occur. The FP32 rating of 67 TFLOPS on the GH200 trails the 82.6 TFLOPS rating on the RTX 4090 so single precision scientific kernels may favor the RTX 4090 in some cases.

The GH200 suits workloads that require the 96 GB HBM3 capacity and 4000 GB/s bandwidth together with 989 TFLOPS dense FP16. Large model training and scientific simulations that exceed 24 GB therefore run on the GH200 without partitioning. The NVLink C2C interconnect further supports multi GPU scaling that the PCIe 4.0 interface on the RTX 4090 cannot match.

The RTX 4090 suits workloads that fit within 24 GB GDDR6X and benefit from its 82.6 TFLOPS FP32 rating. Smaller inference jobs and local development tasks that value the 450 W TDP over the 900 W TDP of the GH200 therefore select the RTX 4090. The PCIe form factor also simplifies deployment in standard workstations.