How do I run Nemotron 3.5 Transcribe on a cloud GPU?

Deploy a Raw GPU instance with a card that appears in NVIDIA’s hardware table for the precision you want (the table lists the NVFP4 checkpoint for the RTX 5090 and the BF16 checkpoint for the A100 80GB), check nvidia-smi and nvcc --version, install vLLM 0.32.0 with the NeMo Speech main branch as the model card describes, start vllm serve, and send your audio to the server’s OpenAI-compatible /v1/chat/completions endpoint.

Updated 2026-10-30

Nemotron 3.5 Transcribe is a speech and audio model from NVIDIA. According to its Hugging Face model card, it was released on Hugging Face and through NeMo Speech on GitHub on 28 October 2026, under the OpenMDW-1.1 license. This page follows that model card and the NVFP4 model card, both read on 2026-10-30. Nemotron 3.5 Transcribe is not offered on the CloudGPU API, so you self-host it on a rented GPU.

See current rates on /pricing · Deploy a GPU

What Nemotron 3.5 Transcribe is (official figures)

Every figure below is NVIDIA’s own, from the model card (read 2026-10-30).

Figure	Value
Release date	28 October 2026 (Hugging Face; NeMo Speech on GitHub)
License	OpenMDW-1.1; the card says the model is ready for commercial or non-commercial use
Parameters	34B total, 4.3B active, including the audio encoder
Architecture	Mamba-2 and Transformer hybrid mixture-of-experts language model with a Transformer ASR encoder and a speaker encoder
Context	65,536 tokens; recordings up to one hour
Languages	40 for transcription
Tasks	Transcription, speaker-attributed transcription, word timestamps, speech translation, language identification, audio question answering
Checkpoints	BF16, FP8 and NVFP4 repositories, plus an optional DFlash2 draft model
Mode	Offline: the card says it is not a real-time streaming ASR service

NVIDIA’s official hardware table

The rows below are NVIDIA’s official figures, from the hardware compatibility table in the Nemotron 3.5 Transcribe model card (https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B, accessed 2026-10-30). We show only the rows that name cards also listed on CloudGPU’s /pricing page:

Row in NVIDIA’s table	Precision listed	Runtimes listed
A100 80GB SXM/NVLink (Ampere)	BF16	vLLM 0.32.0, Hugging Face Transformers
RTX 5090 (Blackwell)	NVFP4	vLLM 0.32.0, Hugging Face Transformers, GGML
RTX PRO 6000 Blackwell Server Edition	BF16, FP8, NVFP4	vLLM 0.32.0, Hugging Face Transformers, GGML, SGLang-Omni 0.2.0

Reader check: run nvidia-smi on your instance (Step 2) and confirm the card name it shows matches the row you are following. Variants of the same card differ (for example SXM and PCIe, or Server and Workstation Edition), and this table is NVIDIA’s, not a statement about the exact variants CloudGPU offers.

The table lists other data-center and workstation cards too. According to the NVFP4 model card, the NVFP4 checkpoint requires a Blackwell GPU, and its quantized kernels need nvcc from a CUDA 13 toolkit on PATH. The NVFP4 card’s single-GPU vLLM profile was tested on one RTX PRO 6000 Blackwell (96 GB).

Which CloudGPU card to pick

CloudGPU lists the RTX 5090 32G (Blackwell in our /pricing table), the A100 80G (Ampere in our /pricing table) and the RTX PRO 6000 96G on /pricing, read on 2026-10-30. NVIDIA’s rows name the A100 80GB SXM/NVLink and the RTX PRO 6000 Blackwell Server Edition; check nvidia-smi on the instance against the row you follow (see the reader check under NVIDIA’s table). VRAM need depends on model, precision and workflow; see How much VRAM do I need to run an LLM?.

Our own measurements on CloudGPU are coming; until then, verify with the nvidia-smi/vLLM checks below.

Step 1: Deploy a Raw GPU and connect

On /deploy, choose the Raw GPU option (JupyterLab with CUDA pre-installed, SSH access, root on a dedicated container), pick your card and connect over SSH. Do the long steps inside tmux so a dropped SSH session does not stop them.

Step 2: Check the GPU, driver and CUDA toolkit

Run these before any official command:

nvidia-smi
nvcc --version
which uv git tmux rsync

nvidia-smi shows the card name, its memory and the highest CUDA version the driver supports. nvcc --version shows the CUDA toolkit on PATH. For the NVFP4 checkpoint, the NVFP4 card asks for nvcc from a CUDA 13 toolkit; if your output shows an older toolkit or no nvcc, install the toolkit following NVIDIA’s CUDA installation docs, or use the BF16 checkpoint on a card the table lists for BF16. If uv is missing, install it as described in the uv installation docs.

Step 3: Install vLLM and NeMo Speech (official commands)

According to the model card, inference requires vLLM 0.32.0 and the latest NeMo Speech main branch, installed into one environment with uv:

tmux new -s nemotron
git clone https://github.com/NVIDIA-NeMo/Speech.git nemo-speech
cd nemo-speech
uv venv --python 3.13 .venv
UV_PROJECT_ENVIRONMENT=$PWD/.venv uv sync --extra asr --extra vllm --locked --no-install-project
uv pip install --python .venv/bin/python --no-deps -e .
uv pip install --python .venv/bin/python --no-deps av==19.0.2
uv pip install --python .venv/bin/python --no-deps opencv-python-headless==5.0.0.93
source .venv/bin/activate

Then download one checkpoint:

# BF16 (the table lists BF16 for the A100 80GB)
hf download nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B --local-dir ./NVIDIA-Nemotron-3.5-Transcribe-30B-A4B

# NVFP4 (Blackwell only; the table lists NVFP4 for the RTX 5090)
hf download nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4 --local-dir ./NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4

Step 4: Start the vLLM server

BF16 checkpoint. This is the model card’s single-GPU command:

export VLLM_PLUGINS=nemo_speechlm
export VLLM_MAX_AUDIO_DECODE_DURATION_S=3600
export VLLM_MAX_AUDIO_CLIP_FILESIZE_MB=180
export VLLM_USE_V2_MODEL_RUNNER=0
vllm serve ./NVIDIA-Nemotron-3.5-Transcribe-30B-A4B \
  --served-model-name nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B \
  --trust-remote-code --dtype bfloat16 --gpu-memory-utilization 0.94 \
  --max-model-len 65536 --max-num-batched-tokens 4096 --max-num-seqs 192 \
  --enable-chunked-prefill --no-enable-prefix-caching \
  --use-replayssm --mamba-backend flashinfer --mamba-cache-mode none \
  --mm-processor-cache-gb 0 --no-async-scheduling \
  --limit-mm-per-prompt '{"audio":{"count":6,"length":120000000}}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":8}' \
  --reasoning-parser nemotron_v3 \
  --trust-request-mm-kwargs \
  --middleware nemo.collections.speechlm2.vllm.salm.ctc_serving.ctc_timestamp_middleware

According to the model card, these settings target a GPU with 80 GB of memory, and other hardware or workloads may need a different --gpu-memory-utilization, --max-model-len or --max-num-seqs.

NVFP4 checkpoint. The NVFP4 card’s command adds --kv-cache-dtype fp8, uses --gpu-memory-utilization 0.85, --max-num-seqs 64 and one audio item per request, and sets VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel:

export PATH=/usr/local/cuda/bin:$PATH
export VLLM_PLUGINS=nemo_speechlm
export VLLM_MAX_AUDIO_DECODE_DURATION_S=3600
export VLLM_MAX_AUDIO_CLIP_FILESIZE_MB=180
export VLLM_USE_V2_MODEL_RUNNER=0
export VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel
vllm serve ./NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4 \
  --served-model-name nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4 \
  --trust-remote-code --dtype bfloat16 --kv-cache-dtype fp8 --gpu-memory-utilization 0.85 \
  --max-model-len 65536 --max-num-batched-tokens 4096 --max-num-seqs 64 \
  --enable-chunked-prefill --no-enable-prefix-caching \
  --use-replayssm --mamba-backend flashinfer --mamba-cache-mode none \
  --mm-processor-cache-gb 0 --no-async-scheduling \
  --limit-mm-per-prompt '{"audio":{"count":1,"length":57600000}}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":8}' \
  --reasoning-parser nemotron_v3 \
  --trust-request-mm-kwargs \
  --middleware nemo.collections.speechlm2.vllm.salm.ctc_serving.ctc_timestamp_middleware

That profile was tested by NVIDIA on a 96 GB card. On a 32 GB RTX 5090, if the server does not start with these values, adjust --gpu-memory-utilization, --max-model-len or --max-num-seqs, the settings the model card names for other hardware. The NVFP4 card asks you to confirm that the server log reports quantization=modelopt_mixed.

Both servers listen on 127.0.0.1:8000. To call it from your own computer, open an SSH tunnel instead of exposing the port:

ssh -L 8000:127.0.0.1:8000 root@<your-instance-host> -p <ssh-port>

Step 5: Transcribe a file

This request follows the model card’s quick start and uses the openai Python package against your own server:

import base64
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
audio = base64.b64encode(open("audio.wav", "rb").read()).decode()
response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Transcribe the following audio."},
        {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio}"}},
    ]}],
    temperature=0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}, "skip_special_tokens": False},
)
print(response.choices[0].message.content)

For the NVFP4 server, set model to nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4. According to the model card, a dense one-hour conversation can need about 20,000 output tokens, so raise max_tokens or leave it unset for long recordings. For speaker labels, the card’s prompt asks for tags such as <spk:0> <spk:1> and word timestamps are requested with mm_processor_kwargs={"capture_ctc_timestamps": True}; see the card’s “Word Timestamps and Speaker Diarization” section.

Before you destroy the instance

Copy transcripts and any audio you want to keep to your own computer, for example with rsync -avP root@<your-instance-host>:/root/out/ ./out/ -e "ssh -p <ssh-port>". Data on an instance is permanently deleted when the instance is destroyed. GPU time is billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance; see /terms.

Related guides

• Whisper large-v3 on CloudGPU
• How do I transcribe audio with WhisperX on a cloud GPU?
• How much VRAM do I need to run an LLM?
• How does per-minute GPU billing work?

FAQ

When was Nemotron 3.5 Transcribe released? According to NVIDIA’s model card, it was released on Hugging Face and through NeMo Speech on GitHub on 28 October 2026.

What license is Nemotron 3.5 Transcribe under? The weights are under the OpenMDW-1.1 license, and the model card says the model is ready for commercial or non-commercial use. Check the license and any third-party notices in the repository before you ship a product.

Which checkpoint goes with an RTX 5090? NVIDIA’s hardware table lists only the NVFP4 checkpoint for the RTX 5090. The NVFP4 checkpoint requires a Blackwell GPU and nvcc from a CUDA 13 toolkit. CloudGPU has not measured it on an RTX 5090 yet.

Does Nemotron 3.5 Transcribe work on live audio streams? No. The model card describes it as an offline model for recordings up to one hour, not a real-time streaming ASR service.

Will my transcripts stay on the instance after I destroy it? No. Data on an instance is permanently deleted when the instance is destroyed, so copy your files to your computer first. Billing is per minute, rounded up, with a 1-minute minimum, and stops when you destroy the instance.

Sources

• Nemotron 3.5 Transcribe model card (BF16): https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B (read 2026-10-30; release date, license, figures, hardware table, vLLM commands, request example)
• Nemotron 3.5 Transcribe NVFP4 model card: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4 (read 2026-10-30; Blackwell requirement, CUDA 13 nvcc, NVFP4 vLLM profile)
• OpenMDW-1.1 LICENSE file in the model repository (read 2026-10-30)
• uv installation docs: https://docs.astral.sh/uv/getting-started/installation/ (read 2026-10-30)
• CloudGPU /pricing, /deploy, /terms (read 2026-10-30)