How many GPUs do I need to train a 70B parameter LLM?

Training a 70B parameter model like Llama 3 70B typically requires 8-64 NVIDIA H100 or H200 GPUs depending on training time targets, batch size, and precision. Use GPUSIZE to calculate exact requirements based on your specific workload.

What is the difference between training and inference GPU requirements?

Training requires significantly more GPU memory due to optimizer states, gradients, and activations. Inference primarily needs memory for model weights and KV cache. GPUSIZE calculates both scenarios with NVIDIA benchmark data.

How does GPUSIZE calculate GPU requirements?

GPUSIZE uses NVIDIA NIM, MLPerf v6.1, and TensorRT-LLM benchmark data for throughput estimates. For training, it calculates total FLOPS, applies MFU and parallel efficiency. Results show whether they are benchmark-based or estimated.

How many GPUs do I need for DeepSeek R1 inference?

DeepSeek R1 is a 671B MoE model requiring all expert weights in GPU memory. On NVIDIA H100 (80GB), you typically need 8-16 GPUs for FP8 inference at moderate QPS. On GB300 NVL72, a single 72-GPU rack delivers about 690,000 tokens/sec and a 4-rack 288-GPU system reaches 2.7 million tokens/sec (MLPerf Inference v6.1, September 2026).

What is the difference between FP8, FP16, and FP4 for LLM inference?

FP16 is the standard precision offering full quality. FP8 halves memory and roughly doubles throughput with minimal quality loss, supported on H100/H200/B200/GB200. FP4 (NVFP4) is exclusive to Blackwell GPUs (B200/GB200/GB300) and offers up to 3.5x throughput over FP16 with some quality trade-off.

What is the difference between LoRA and QLoRA fine-tuning?

LoRA (Low-Rank Adaptation) trains only ~1% of model parameters using low-rank adapters, reducing memory by ~80% versus full fine-tuning while keeping the base model in FP16. QLoRA further quantizes the base model to 4-bit (INT4), enabling fine-tuning of large models on fewer GPUs at the cost of ~10-15% slower training.

How much does it cost to train a 70B LLM?

Training a 70B model on 300B tokens with 8 NVIDIA H100 GPUs typically takes 2-4 weeks and costs $30,000-$80,000 in cloud compute (H100 at ~$2.50/GPU/hr). Using FP8 precision or LoRA fine-tuning can significantly reduce cost. GPUSIZE calculates exact estimates based on your specific dataset size, batch size, and target training time.

What is KV cache and how does it affect GPU memory?

The KV (Key-Value) cache stores intermediate attention states during inference to avoid recomputation. For a 70B model serving 100 concurrent requests with 2K context, KV cache can consume 20-50GB of GPU memory. Using FP8 KV cache precision halves this memory requirement with negligible quality loss.

Which GPU is best for LLM inference in 2026?

For highest throughput: NVIDIA GB300 NVL72 (MLPerf v6.1: ~690K tokens/sec per 72-GPU rack for DeepSeek R1, 1.2M tokens/sec for gpt-oss-120b). For best price/performance on H100-class: H200 SXM adds 76% more HBM3e vs H100 at similar compute. For cost-sensitive workloads: A100 80GB or L40S offer good performance at lower cost. AMD MI355X (288GB HBM3e) is a new option with MLPerf v6.0 certification.

What is MFU (Model FLOPS Utilization) in LLM training?

MFU is the fraction of theoretical GPU FLOPS actually used for model computation. In practice, LLM training achieves 35-50% MFU due to communication overhead, memory bandwidth limits, and framework inefficiencies. GPUSIZE uses empirically-calibrated MFU values (35% for small clusters, up to 45% for 256+ GPU clusters) based on Megatron-LM and MLPerf data.

GPUSIZE - AI GPU Infrastructure Sizing Calculator | LLM Training & Inference GPU Sizing

Based on NVIDIA Official Benchmarks + MLCommons Standards 🏆

This calculator uses NVIDIA official benchmarks (NIM, 🥇 MLPerf™ v6.1, TensorRT-LLM) and 🏆 MLCommons Standards.

Select Workload Type

Training Parameters

Model Selection (Optional)

Model Size (Parameters)

GPU Type

Micro Batch Size (per GPU · micro_batch_size)

Tier 1: Megatron-Bridge official — LLaMA3-8B: micro_batch=2, 405B: micro_batch=1. Global batch = micro × GPUs × grad_accum.

Training Mode

All model weights updated. Highest memory usage.

Training Precision

Dataset Size (Tokens)

Epochs

Typical: 1–5 epochs

Target Training Time

Parallel Efficiency (%)

Typical: 75–90%

Context Length (tokens) Affects activation memory

Longer context = more activation memory

Tensor Parallelism (TP) Split model across GPUs

Reduces per-GPU memory. Requires NVLink for TP4+

Pipeline Parallelism (PP) Split layers across GPUs

Splits model layers across GPU groups. For large models (100B+) on multi-node clusters.

Inference Parameters

Import any model from Hugging Face

Model Selection (Optional)

Model Size (Parameters)

GPU Type

Inference Precision

Target QPS (Queries Per Second)

Peak traffic estimate recommended

Avg. Input Tokens Per request

Avg. Output Tokens

Target Latency (ms)

Real-time: 100–500ms recommended

KV Cache Precision Reduces KV cache memory

Speculative Decoding Boosts throughput with draft model

Requires a small draft model alongside main model

HA Replication Factor

VLM Inference Parameters

Vision-Language Models (VLM)

Multimodal AI models that process both images and text. Used for image Q&A, image captioning, chart understanding, and document analysis.

Source: NVIDIA NIM VLM Benchmarking

VLM Model

Llama 4 Scout: Official NVIDIA NIM benchmark data available

GPU Type

Only GPUs with official NVIDIA NIM VLM benchmark data. GB300/GB200 not available (no official data).

Input Sequence Length (ISL)

Image + text prompt tokens (default: 2500)

Output Sequence Length (OSL)

Response length (default: 300 tokens)

Inference Precision

BF16: best stability, FP8: best throughput

Tensor Parallelism (TP)

TP4/TP8 only (NVIDIA NIM VLM official)

Target Throughput (req/s)

Peak requests per second

HA Replication Factor

2x or more recommended for production

Why GPUSIZE?

Uses curated NVIDIA NIM, MLPerf, and TensorRT-LLM benchmark data — not just model-size estimates.

Calculates training time and GPU count from FLOPS, MFU, and parallel efficiency — not just VRAM sizing.

Shows transparent source badges so you know whether results are benchmark-based or estimated.

Separates MoE weight memory from active compute for accurate sizing of DeepSeek, Mixtral, and Qwen models.

GPU Comparison

Grouped by architecture and precision support. Sorted by performance within each group.

Blackwell Ultra (FP16 / FP8 / FP4) Blackwell (FP16 / FP8 / FP4) Hopper / CDNA4 (FP16 / FP8) Ampere / Ada (FP16 only or FP8)

GPU	VRAM	FP16 TFLOPS	Mem BW	FP16	FP8	FP4	$/hr (est.)	Architecture
⚡ Blackwell Ultra — FP16 / FP8 / FP4
GB300 NVL72 🚀 Top	288 GB/GPU 20.7 TB system	7,200	10,000 GB/s	✓	✓	✓	~$15.00	Blackwell Ultra
🔥 Blackwell — FP16 / FP8 / FP4
GB200 NVL72	192 GB/GPU 13.4 TB system	5,000	8,000 GB/s	✓	✓	✓	~$10.50	Blackwell
B300 DGX (8-GPU) 🆕 MLPerf v6.1	288 GB/GPU 2.1 TB system	4,500	8,000 GB/s	✓	✓	✓	~$10.75*	Blackwell Ultra
B200 DGX (8-GPU)	180 GB/GPU 1.4 TB system	4,500	8,000 GB/s	✓	✓	✓	~$8.00	Blackwell
RTX PRO 6000 Blackwell	96 GB	234	1,597 GB/s	✓	✓	✓	~$2.50	Blackwell
Hopper — FP16 / FP8
H200 SXM	141 GB	1,979	4,800 GB/s	✓	✓	—	~$4.50	Hopper
H100 SXM (default)	80 GB	1,979	3,350 GB/s	✓	✓	—	~$2.50	Hopper
H100 PCIe	80 GB	1,513	2,000 GB/s	✓	✓	—	~$2.00	Hopper
AMD CDNA4 — FP16 / FP8
AMD Instinct MI355X MLPerf v6.0	288 GB	3,840	8,000 GB/s	✓	✓	—	~$8.00	CDNA4
Ada Lovelace — FP16 / FP8
L40S	48 GB	733	864 GB/s	✓	✓	—	~$0.80	Ada
L4	24 GB	242	300 GB/s	✓	✓	—	~$0.35	Ada
Ampere — FP16 only
A100 80GB SXM	80 GB	1,248	2,000 GB/s	✓	—	—	~$1.80	Ampere
A100 40GB	40 GB	1,248	1,555 GB/s	✓	—	—	~$1.20	Ampere

* Prices are cloud on-demand estimates and vary by provider. GB300/GB200 TFLOPS and bandwidth are per-GPU figures; total system performance scales with 72 GPUs. Source: NVIDIA official specs + MLPerf v6.1.

AI GPU Infrastructure Sizing Calculator

Based on NVIDIA Official Benchmarks + MLCommons Standards 🏆

Select Workload Type

Training

Inference

VLM Inference

Training Parameters

Inference Parameters

VLM Inference Parameters

Vision-Language Models (VLM)

GPU Comparison

Detailed Sizing Report

Resource Analysis

Recommendations

Why GPUSIZE?

GPU Comparison