How to Benchmark GPU Performance for Machine Learning Workloads – A Complete Guide

In the fast‑moving world of deep learning, raw GPU horsepower is only half the story—knowing how to measure it reliably is what separates hobbyists from production engineers. This guide walks you through a repeatable, science‑backed process for benchmarking GPU performance on typical machine‑learning workloads, from data‑loading to mixed‑precision training, using tools you already have on a Linux workstation.

What You’ll Need

• Ubuntu 20.04 LTS or later (or equivalent Linux distro)
• NVIDIA GPU with recent architecture (e.g., RTX 30xx, A100, H100)
• Supported driver (>= 525) and matching CUDA Toolkit (>= 12.0)
• Python 3.9+ with virtual‑env tools (venv or conda)
• PyTorch (>= 2.0) and/or TensorFlow (>= 2.12) compiled for your CUDA version
• Benchmarking utilities: nvprof, nsight‑systems, nvidia‑smi, torch‑utils‑benchmark
• Representative ML script (e.g., ResNet‑50 training on ImageNet)
• Access to a stable power supply and cooling (thermal throttling skews results)

Step 1: Set Up a Clean Environment

Start with a fresh virtual environment to eliminate hidden dependencies that could affect timing. For conda users:

conda create -n ml-bench python=3.10 -y
conda activate ml-bench
conda install pytorch torchvision torchaudio cudatoolkit=12.1 -c pytorch
pip install tensorflow==2.12.0
pip install torch-utils-benchmark

If you prefer venv, replace the first two lines with python -m venv ml-bench && source ml-bench/bin/activate. Verify the CUDA runtime:

python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
python -c "import tensorflow as tf; print('TF CUDA:', tf.test.is_gpu_available())"

Both should return True. Any mismatch here will invalidate later numbers.

Step 2: Install and Verify Drivers

Run nvidia-smi to confirm the driver version matches the CUDA toolkit you installed. Example output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12   Driver Version: 525.85.12   CUDA Version: 12.1 |
+-----------------------------------------------------------------------------+

If the driver is older, update it using the official .run installer or your distro’s package manager. After updating, reboot the machine and re‑run nvidia-smi to confirm the change.

Step 3: Warm‑Up the GPU

Modern GPUs ramp up clock speeds only after a brief warm‑up period. Run a lightweight kernel a few times before measuring real workloads:

python - <<'PY'
import torch, time
x = torch.randn(1024, 1024, device='cuda')
for _ in range(10):
    start = time.time()
    y = torch.mm(x, x)
    torch.cuda.synchronize()
    print('Iteration time:', time.time() - start)
PY

This stabilizes power limits and frequency governors, ensuring subsequent benchmarks reflect sustained performance rather than a cold start.

Step 4: Measure Raw Compute Throughput

Use a synthetic matrix‑multiply benchmark to capture peak TFLOPs. PyTorch’s built‑in torch.utils.benchmark makes this easy:

python - <<'PY'
import torch, torch.utils.benchmark as benchmark
size = 8192
A = torch.randn(size, size, device='cuda')
B = torch.randn(size, size, device='cuda')
stmt = "torch.mm(A, B)"
print(benchmark.Timer(stmt=stmt, globals=globals()).timeit(20))
PY

Record the average time, then compute TFLOPs:
TFLOPs = (2 * size³) / (average_seconds * 1e12). Compare the result to the vendor‑published peak; a gap larger than 10‑15 % often indicates sub‑optimal settings (e.g., power limit too low).

Step 5: Benchmark a Real‑World Training Loop

Now run a representative model. Below is a minimal ResNet‑50 training script using the ImageNet‑like synthetic data to focus on compute rather than I/O:

python - <<'PY'
import torch, torchvision.models as models, torch.nn as nn, torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

batch_size = 128
# Synthetic dataset: 128‑image batches, 3×224×224
x = torch.randn(batch_size, 3, 224, 224, device='cuda')
y = torch.randint(0, 1000, (batch_size,), device='cuda')
loader = DataLoader(TensorDataset(x, y), batch_size=batch_size)

model = models.resnet50().cuda()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

torch.cuda.synchronize()
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
for epoch in range(5):
    for inputs, targets in loader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()
end.record()
torch.cuda.synchronize()
print('Total time (s):', start.elapsed_time(end) / 1000)
PY

Divide the total time by the number of images processed to get per‑image latency, and by the number of floating‑point operations per forward‑backward pass (≈ 11.3 GFLOPs for ResNet‑50) to derive effective TFLOPs. Log both the raw timing and the derived efficiency.

Step 6: Profile Memory Bandwidth and Kernel Overheads

Use nvprof or nsight‑systems to capture kernel‑level metrics. Example with nvprof:

nvprof --metrics gld_throughput,gst_throughput,sm_efficiency 
       -f -o resnet50.nvvp python train_resnet50.py

Open the resulting .nvvp file in Nsight Systems to see which kernels dominate runtime, their occupancy, and whether memory throughput saturates the GPU’s bandwidth. If sm_efficiency stays below 70 % for compute‑heavy kernels, consider adjusting batch size or enabling Tensor Cores with torch.backends.cudnn.allow_tf32 = True (for RTX/Ampere) or mixed‑precision AMP in PyTorch.

Common Mistakes to Avoid

1. **Running benchmarks on a laptop with power‑saving mode enabled** – the GPU will throttle to 15 W, skewing results. Disable any “Battery Saver” or set the power limit to the GPU’s maximum using nvidia-smi -pl.
2. **Including data‑loading time in compute benchmarks** – unless you’re measuring end‑to‑end training, use torch.utils.data.DataLoader with num_workers set high enough to keep the GPU fed, or pre‑load synthetic data as shown.
3. **Neglecting to synchronize before timing** – CUDA calls are asynchronous; forgetting torch.cuda.synchronize() yields artificially low latency.
4. **Using mismatched driver/CUDA versions** – leads to fallback to CPU kernels or reduced Tensor‑Core usage.
5. **Running multiple benchmarks back‑to‑back without a cool‑down period** – thermal throttling can accumulate, making later runs appear slower.

Tips and Tricks

• **Set a fixed GPU clock** with nvidia-smi -lgc 1500,1500 for repeatable results (use cautiously; may void warranties).
• **Leverage Tensor Cores** by enabling torch.float16 or torch.bfloat16 where supported; this often doubles throughput for matrix‑multiply heavy models.
• **Automate the suite** with a Bash wrapper that logs driver version, GPU temperature, power draw (nvidia-smi --query-gpu=temperature.power.draw --format=csv), and benchmark numbers into a CSV for later analysis.
• **Compare against a reference** – NVIDIA publishes “Deep Learning Performance” tables; use them as a sanity check.
• **Use cuDNN benchmark mode** (torch.backends.cudnn.benchmark = True) when input sizes are constant; it lets cuDNN pick the fastest convolution algorithm.

Frequently Asked Questions

Do I need an internet connection to run these benchmarks?

No. All required packages are available via conda/pip mirrors, and the benchmark scripts use synthetic data, so the process is fully offline after the initial package install.

How many runs should I average to get a reliable number?

For synthetic compute kernels, 10‑20 iterations are sufficient. For full training loops, run at least three epochs and discard the first epoch (warm‑up) before averaging.

Can I benchmark multi‑GPU setups with the same steps?

Yes. Use torch.distributed.launch or tf.distribute.MirroredStrategy and repeat the same profiling commands. Remember to set CUDA_VISIBLE_DEVICES and capture per‑GPU metrics individually.

Conclusion

Benchmarking GPU performance for machine‑learning workloads is more than a single nvidia‑smi snapshot; it requires a disciplined workflow that isolates compute, memory, and I/O factors, validates driver‑CUDA compatibility, and records reproducible metrics. By following the six steps above, you’ll be able to quantify raw TFLOPs, gauge real‑world training efficiency, and spot bottlenecks before they become production headaches. Keep the checklist handy, automate the runs, and you’ll turn raw GPU specs into actionable performance insights every time you spin up a new model or hardware platform.

Photo by Mariia Berezovsky on Unsplash