H100 NVL vs RTX 3090

The FP16 dense figure of 989 TFLOPS for the H100 NVL family exceeds the FP16 dense figure of 71 TFLOPS for the RTX 3090 family by a ratio of approximately fourteen to one while the FP16 with sparsity figure of 1979 TFLOPS for the H100 NVL family exceeds the FP16 with sparsity figure of 142 TFLOPS for the RTX 3090 family by the same ratio. This gap translates into substantially larger batch sizes during training and inference because higher memory bandwidth of 3350 GB/s supports faster data movement than 936 GB/s. The FP32 figure of 67 TFLOPS for the H100 NVL family compared with 35.6 TFLOPS for the RTX 3090 family further widens the advantage in mixed precision workloads that rely on dense FP16 operations.

The H100 NVL variant suits workloads that require 94 GB HBM3 memory capacity paired with 3350 GB/s bandwidth and 989 TFLOPS dense FP16 throughput. Scenarios involving large model training or inference benefit from the 700W TDP allowance and NVLink bridge interconnect that scale across multiple cards more effectively than the 350W TDP and PCIe 4.0 interconnect of the RTX 3090 variant.

The RTX 3090 variant fits smaller scale tasks that operate within 24 GB GDDR6X memory and 936 GB/s bandwidth limits. Projects that prioritize lower 350W TDP consumption and PCIe 4.0 compatibility over the 989 TFLOPS dense FP16 performance of the H100 NVL family find this variant sufficient for modest inference or fine tuning jobs.