How much VRAM do I need to run an LLM?

Short answer: An LLM’s weights take about 2 bytes per parameter at FP16 (about 2 GB per billion parameters) and, as an estimate, about 0.6 GB per billion at 4-bit Q4_K_M (llama.cpp quantize README, read 2026-10-23). Published Q4_K_M weight sizes are about 5 GB for an 8B model, about 19-20 GB for a 32B model and about 40-43 GB for a 70B model (CloudGPU model pages, read 2026-10-23). The context window (KV cache) and runtime overhead come on top.

Rule of thumb

  • FP16 / BF16: about 2 bytes per parameter, so roughly 2 GB per billion parameters (the basis of the FP16 column on CloudGPU’s model pages, read 2026-10-23).
  • 4-bit (Q4_K_M): about 0.6 GB per billion parameters (estimate). Method: the llama.cpp quantize README (read 2026-10-23) lists Q4_K_M at about 4.89 bits per weight for Llama 3.1 8B, and 4.89 / 8 is about 0.61 bytes per parameter. It is an average and varies by model.
  • Context (KV cache): grows with context length and the number of parallel requests, and comes on top of the weights.

From CloudGPU’s model pages (VRAM table dated 2026-09-08; page re-read 2026-10-23):

Model FP16 Quantised
Llama 3.1 8B 16 GB about 4.9 GB (Q4_K_M)
Qwen3 14B 28 GB about 9 GB (Q4_K_M)
Mistral Small 3 24B 48 GB about 14 GB (Q4_K_M)
Gemma 3 27B 54 GB about 17 GB (Q4_K_M)
Qwen3 32B 64 GB about 19-20 GB (Q4_K_M)
Llama 3.3 70B 140 GB about 40-43 GB (Q4_K_M)
gpt-oss-20b 42 GB about 13 GB (native MXFP4)
gpt-oss-120b 234 GB about 63 GB (native MXFP4; the Hugging Face files total about 65 GB, read 2026-10-23)

Weights only. Add headroom for the context. The FP16 column uses nominal parameter counts (Hugging Face metadata, read 2026-10-23, shows slightly higher counts for some models, for example Qwen3 14B at about 14.8B, or about 29.5 GB at FP16), and the gpt-oss FP16 figures are computed equivalents, because those models are published in MXFP4.

Which VRAM class to rent

CloudGPU’s model pages list cards in 24, 32, 48, 80 and 96 GB classes. VRAM need depends on model, precision and workflow: add the weights from the table above to the memory your context needs, and check peak VRAM with nvidia-smi on a first run.

Individual cards come and go; see https://cloudgpu.app/deploy for what is available and https://cloudgpu.app/run for per-model pages.

Training needs more

Fine-tuning needs additional memory for gradients and optimizer states. Parameter-efficient methods such as LoRA/QLoRA reduce this a lot. See https://cloudgpu.app/blog/how-much-vram-to-run-llm-2026 for a longer table.

Before you rent

  • Instance storage is limited, so check the size of your model files first: a Q4_K_M 70B file is about 40-43 GB (CloudGPU model page, read 2026-10-23).
  • Startup takes about 2 minutes; billing is per minute with a 1-minute minimum. See how per-minute billing works.

Frequently asked questions

How much VRAM does a 7B or 8B model need?

About 16 GB at FP16 and about 5 GB at Q4_K_M (CloudGPU model pages, read 2026-10-23), plus memory for the context.

How much VRAM does a 32B model need?

About 64 GB at FP16 and about 19-20 GB at Q4_K_M (CloudGPU model pages, read 2026-10-23), plus memory for the context. VRAM need depends on model, precision and workflow.

How much VRAM does a 70B model need?

About 140 GB at FP16 and about 40-43 GB at Q4_K_M (CloudGPU model page, read 2026-10-23), plus memory for the context. VRAM need depends on model, precision and workflow.

Does a longer context need more VRAM?

Yes. The KV cache grows with context length and with parallel requests, and it comes on top of the weights.

Will a 70B model fit in instance storage?

A Q4_K_M 70B file is about 40-43 GB and the FP16 weights are about 140 GB (CloudGPU model page, read 2026-10-23). Check your instance’s storage size before you download, and use a quantised file unless you have confirmed more space.

Sources (read 2026-10-23, Asia/Shanghai)

The Deploy page lists the cards you can start. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-10-23 (Asia/Shanghai).