How do I choose a rental GPU for Ollama?

Short answer: Pick the rental GPU by VRAM. Ollama usually runs 4-bit quantised models (Q4_K_M), so a model’s weights need roughly 0.6 GB of VRAM per billion parameters (an estimate based on the llama.cpp quantize README, read 2026-10-23), plus room for the context window. VRAM need depends on model, precision and workflow; see how much VRAM an LLM needs.

Step 1: estimate VRAM from the model size

For a 4-bit quantised model (Q4_K_M, the usual Ollama default), a rule of thumb is about 0.6 GB per billion parameters for the weights (estimate: the llama.cpp quantize README (read 2026-10-23) lists Q4_K_M at about 4.89 bits per weight, and 4.89 / 8 is about 0.61 bytes per parameter), then add headroom for the KV cache (the context window) and runtime overhead. Figures published on CloudGPU’s model pages (VRAM table dated 2026-09-08; page re-read 2026-10-23) are:

Model FP16 weights Q4_K_M weights
Llama 3.1 8B 16 GB about 4.9 GB
Qwen3 14B 28 GB about 9 GB
Gemma 3 27B 54 GB about 17 GB
Qwen3 32B 64 GB about 19-20 GB
Llama 3.3 70B 140 GB about 40-43 GB

These are weights only. Long contexts and several parallel requests add more. The FP16 column uses nominal parameter counts.

Step 2: pick a VRAM class

CloudGPU lists cards in VRAM classes of 24, 32, 48, 80 and 96 GB. VRAM need depends on model, precision and workflow: add the Q4_K_M weights from the table above to the memory your context needs, and check peak VRAM with nvidia-smi on a first run. See how much VRAM an LLM needs.

Check https://cloudgpu.app/deploy for the cards you can start now, and https://cloudgpu.app/run for one page per open model.

Step 3: use the Ollama template

CloudGPU has an Ollama template. Instances take about 2 minutes to start. The first model download takes extra time while the instance is running, so choose a model you actually plan to run.

Instance storage is limited, so check the size before you pull a model. A 70B model at Q4 is roughly 40-43 GB (CloudGPU model page, read 2026-10-23), so check the free space before you pull a second large model.

Step 4: call it like an OpenAI API

Ollama exposes an OpenAI-compatible API, so most OpenAI SDKs and tools work when you point the base URL at your instance. See the guide at https://cloudgpu.app/blog/ollama-cloud-gpu-public-api-2026.

Things to know

  • Billing is per minute with a 1-minute minimum; unused minutes of the pre-authorised hour go back to your balance when you destroy the instance. See how per-minute billing works.
  • Instances are automatically destroyed when your balance reaches zero, and the instance’s storage is deleted when an instance is destroyed, so copy out anything you need.

Frequently asked questions

How much VRAM do I need to run Llama 3.1 8B in Ollama?

About 4.9 GB for the Q4_K_M weights and 16 GB at FP16 (CloudGPU model pages, read 2026-10-23), plus memory for the context. VRAM need depends on model, precision and workflow.

Can one rented GPU run a 70B model with Ollama?

VRAM need depends on model, precision and workflow: add the 70B weights at your quantisation to the memory your context needs; see how much VRAM an LLM needs.

How long does an Ollama instance take to start?

About 2 minutes. The first model download takes extra time.

Will a 70B model fit in instance storage?

A Q4_K_M 70B model is about 40-43 GB and the 16-bit version about 140 GB (CloudGPU model page, read 2026-10-23), so check your instance’s storage size before you pull it.

Does Ollama on CloudGPU work with OpenAI client libraries?

Yes. Ollama serves an OpenAI-compatible API; set the base URL to your instance endpoint.

Sources (read 2026-10-23, Asia/Shanghai)

The Deploy page lists the cards you can start. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-10-23 (Asia/Shanghai).