RTX 3060 vs V100

The V100 supplies 125 TFLOPS dense FP16 performance while the RTX 3060 FP16 performance remains unpublished by the vendor. This dense FP16 advantage on the V100 supports larger batch sizes during training and inference compared with FP32 only operation at 15.7 TFLOPS. The RTX 3060 FP32 performance of 12.7 TFLOPS limits throughput in precision sensitive tasks relative to the V100 FP32 performance of 15.7 TFLOPS. Memory bandwidth of 900 GB/s on the V100 versus 360 GB/s on the RTX 3060 permits larger batch sizes before memory capacity of 16-32 GB HBM2 becomes saturated. The RTX 3060 memory capacity of 8-12 GB GDDR6 constrains batch sizes sooner under the same workload. Power draw of 300W on the V100 exceeds the 170W TDP of the RTX 3060 and affects sustained dense FP16 operation.

Choose the RTX 3060 when the whole job fits in 8 to 12 GB of GDDR6: Stable Diffusion, small vision models, or serving a small quantized LLM at low batch. Its 12.7 TFLOPS of FP32 is close to the V100 at 15.7 TFLOPS, but with 360 GB/s of bandwidth against 900 GB/s it decodes tokens much more slowly, and it publishes no dense FP16 tensor rating. It is a PCIe 4.0 consumer card with no NVLink, so do not plan on multi-GPU training with it.

The V100 suits workloads that require dense FP16 throughput of 125 TFLOPS and memory bandwidth of 900 GB/s. Its 16-32 GB HBM2 capacity supports larger models during training and scientific computing. NVLink interconnect further benefits multi GPU scaling at 15.7 TFLOPS FP32.