How Much VRAM to Run Each Open Model? (2026 Table, 24 to 96 GB Cards)

The question “what GPU do I need for model X” gets asked every day on r/LocalLLaMA, and the answers are usually a single number with no context length attached. This post is a table you can actually plan from: weights at FP16 and Q4, the KV cache you need for realistic context, and which rentable card that adds up to. The card column uses the GPUs listed on cloudgpu.app because that is what we can verify, but the VRAM numbers are independent of where you rent.

Two rules of thumb before the table:

1. Weights. Parameters × bytes per parameter. FP16/BF16 is 2 bytes, so a 32B model is 64 GB. Q4_K_M in GGUF averages about 4.4 to 4.8 bits, so call it 0.55 to 0.6 bytes per parameter and a 32B model is 19 to 20 GB. Add 5 to 10 percent for CUDA context and buffers.
2. KV cache. This is what people forget. For a dense model it scales with context length and is roughly 2 × number of layers × hidden size × 2 bytes per token at FP16. For Llama-3-8B that is about 0.5 MB per token, so 32k tokens of context costs about 16 GB on top of the 16 GB of weights. Models with grouped-query attention (most 2025 and 2026 releases) need 4 to 8 times less. Ollama and llama.cpp can quantise the cache to Q8 and halve it again.

The table

Weights are approximate file sizes of the common GGUF quantisations or the safetensors release; “fits” means weights plus a useful amount of context (at least 8k tokens) with headroom. When a row says “tight”, it runs, but you will be trimming context or batch size.

Model	Params	FP16 weights	Q4_K_M weights	24 GB (3090 / 4090)	32 GB (5090)	48 GB (A6000 / 4090 48G)	80 GB (A100)	96 GB (H20 / RTX PRO 6000)
Llama 3.1 8B, Qwen3 8B	8B	16 GB	4.9 GB	FP16 tight, Q4 fine with 64k ctx	FP16 fine	FP16 with long ctx	FP16, batch	FP16, batch
Qwen3 14B, Phi-4 14B	14B	28 GB	9 GB	Q4 fine, 32k ctx	Q4 fine; FP16 tight	FP16 fine	FP16, batch	FP16, batch
Mistral Small 3 24B, Gemma 3 27B	24 to 27B	48 to 54 GB	14 to 17 GB	Q4 fine, 16k ctx	Q4 fine, 32k ctx	Q4 long ctx; FP16 tight	FP16 fine	FP16 fine
Qwen3 32B, DeepSeek-R1-Distill-Qwen-32B, QwQ 32B	32B	64 GB	19 to 20 GB	Q4 tight, 8k ctx	Q4 fine, 16k to 32k ctx	Q4 with long ctx, or Q8	FP16 fine	FP16 fine
Llama 3.3 70B, Qwen2.5 72B, DeepSeek-R1-Distill-Llama-70B	70 to 72B	140 GB	40 to 43 GB	no (2 cards, Q4, slow)	no	Q4 tight, 8k ctx	Q4 comfortable, 32k ctx; FP16 needs 2 cards	Q4 comfortable; Q8 tight
gpt-oss-20b	21B MoE (3.6B active)	42 GB	native MXFP4 about 13 GB	MXFP4 fine	MXFP4 fine	fine	fine	fine
gpt-oss-120b	117B MoE (5.1B active)	234 GB	native MXFP4 about 63 GB	no	no	no	MXFP4 fine, 16k ctx	MXFP4 fine, 32k ctx
Qwen3-235B-A22B	235B MoE (22B active)	470 GB	about 130 GB	no	no	no	no (2 cards Q4)	no (2 cards Q4)
FLUX.1 dev / schnell (image)	12B + T5-XXL	about 24 GB (FP16), 12 to 16 GB (FP8)	n/a	FP8 fine; FP16 tight	FP16 fine	FP16 fine	FP16 fine, batch	FP16 fine, batch
Stable Diffusion XL	2.6B + encoders	about 7 GB	n/a	fine	fine	fine	fine	fine
Whisper large-v3	1.5B	3 GB	n/a	fine	fine	fine	fine	fine

Notes on the rows:

• gpt-oss ships in MXFP4 natively, which is why the 120B model fits an 80 GB card at all. Ollama and vLLM both load it directly; there is no FP16 path worth using.
• DeepSeek V4 and Kimi K3 are not in the table because they are too large to run on a single rented card at any practical quantisation. If you want them, call the hosted API instead; on cloudgpu.app they are on the same balance as the GPUs, priced at or under the vendor list, at https://cloudgpu.app/v1.
• Fine-tuning needs more than inference. Full fine-tuning at FP16 needs roughly 16 bytes per parameter with Adam (weights, gradients, optimiser states), so 8B is 128 GB and needs multiple 80 GB cards. LoRA and QLoRA bring that down to about 1.5 to 2 times the quantised inference footprint: QLoRA on a 32B model fits a 48 GB card, on a 70B model an 80 GB card, on an 8B model a 24 GB card with room to spare. LLaMA-Factory and Unsloth both default to these settings.

Which card to rent, by what you are doing

Prices are the live rates on cloudgpu.app on 8 September 2026, billed per minute. They move with supplier stock; the pricing page is authoritative.

You want to	Smallest card that is comfortable	Price on 8 Sep 2026
Run 8B to 14B at Q4 for chat, RAG, agents	RTX 3090 24 GB	$0.21/h (Spot $0.15/h)
Run 8B at FP16, or FLUX.1 dev at FP8	RTX 4090 24 GB	$0.33/h
Run a 32B at Q4 with 16k to 32k context, or FLUX.1 dev at FP16	RTX 5090 32 GB	$0.59/h
QLoRA a 32B, or run 70B at Q4 with short context	RTX A6000 48 GB or RTX 4090 48 GB	$0.44/h or $0.65/h
Run 70B at Q4 with long context, gpt-oss-120b, QLoRA a 70B	A100 80 GB	$1.49/h
Same, with more KV headroom or batch	H20 96 GB or RTX PRO 6000 96 GB	$1.29/h or $1.49/h

The 5090 is the sweet spot for most individual developers in 2026: 32 GB is exactly the size that turns “32B at Q4 with 8k context and no room to breathe” on a 24 GB card into a comfortable working setup, and it is a fraction of the price of an 80 GB card.

How to check your own numbers in two minutes

Rent a card, pick the Ollama template, and once the machine is up (about 100 to 120 seconds on cloudgpu.app, including the public HTTPS endpoint) run:

ollama run qwen3:32b "hello"
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

The second line tells you exactly what the model plus its default context took. Then raise the context:

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

and watch nvidia-smi again. If it spills over, Ollama will offload layers to CPU and get several times slower rather than crash, so the symptom of “not enough VRAM” is a slow model, not an error. That is the single most common cause of “this GPU is slower than I expected” posts.

Because billing is per minute and the unused part of the pre-authorised hour is refunded when you stop, checking three cards this way costs well under a dollar. If you find a row in the table that does not match what you measured, email support@cloudgpu.app with the nvidia-smi output and we will correct it.