How to run Qwen-Agent with a self-hosted model on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install Qwen-Agent with the pip install -U qwen-agent line the README prints, serve an ungated open-weight model on that machine, and set llm_cfg model_server to that server’s /v1 URL. Copy the text typewriter_print writes before you destroy the instance, because destroying the instance deletes its disk.

An agent can rent the GPU and run this library against a model served on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the Qwen-Agent install. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. An OpenAI-compatible engine whose CUDA 12 container tag is localai/localai:latest-gpu-nvidia-cuda-12 is How to run LocalAI on a cloud GPU. A gateway whose README prints uv tool install 'litellm[proxy]' is How to run LiteLLM Proxy in front of a self-hosted model on a cloud GPU. A framework whose 2.0.11 quickstart prints uv pip install "agentscope==2.0.11", and whose OpenAICredential field base_url is passed to the chat client, is How to run AgentScope agents with a self-hosted model on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. A library whose README prints uv add langchain is How to build LangChain agents with a self-hosted model on a cloud GPU. An AgentChat library whose OpenAI-compatible example sets base_url to http://localhost:11434/v1 is How to run AutoGen agents with a self-hosted model on a cloud GPU. A typed agent whose vLLM example sets base_url to http://localhost:8000/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A Python agent library whose OpenAIModel takes api_base is How to run smolagents with a self-hosted model on a cloud GPU. A library whose local-LLM page sets OpenAIGPTConfig api_base for a vLLM-style endpoint is How to run Langroid agents with a self-hosted model on a cloud GPU. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090.

The README header says “Copyright 2023 The Qwen team, Alibaba Group”, and the LICENSE file begins with “Apache License Version 2.0, January 2004”.

The commands below are copied from the README and from examples/assistant_qwen3.5.py at commit 31a4d36d123688581a9e9744427272b33ce940e0, except the download lines marked as added. They were not run on a CloudGPU instance for this page. Both files also contain a hosted llm_cfg block. This page does not copy those blocks.

What the docs print about memory

The README and examples/assistant_qwen3.5.py print no GPU GB figure. The Qwen/Qwen3.5-397B-A17B README fetched on 2026-11-14 prints no GPU GB figure. This page does not mark a catalog row as enough.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from GET /v1/public/gpu-catalog (generated_at 2026-11-14T21:12:05.060Z). The catalog response also includes an availability field. This page does not copy it. Every catalog row below has min_gpu_count 1. Which listing you can start is on Deploy.

The catalog template list had no Qwen-Agent template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as Qwen-Agent.

Catalog name vram_gb
RTX 2080 8
RTX 3080 10
RTX 2080 Ti 11
RTX 3080 Ti 12
V100 16G 16
A30 24
RTX 3090 24
RTX 4090 24
RTX 4090D 24
RTX 4080 SUPER 32G 32
RTX 5090 32
RTX 5090D 32
V100 32G 32
A100 40G 40
A800 40G 40
A40 48
RTX 4090 48G 48
RTX 4090D 48G 48
RTX A6000 48
Ascend 910B 64
NVIDIA RTX PRO 5000 72G 72
RTX PRO 5000 72G 72
A100 80G 80
A800 80G 80
H800 80
RTX 6000D 83
H20 96G 96
RTX PRO 6000 96G 96
RTX PRO 6000 Max-Q 96
H20-3e 141G 141

Install

The README installation section, at commit 31a4d36d123688581a9e9744427272b33ce940e0 (committed 2026-03-04T08:13:57Z), copied:

pip install -U "qwen-agent[gui,rag,code_interpreter,mcp]"
# Or use `pip install -U qwen-agent` for the minimal requirements.

The same section says the extras are gui, rag, code_interpreter, and mcp. qwen_agent/__init__.py on that commit sets __version__ = '0.0.34'. setup.py on that commit does not set python_requires. The news line dated Dec 3, 2024 says the GUI requires Python 3.10 or higher. PyPI qwen-agent info.version on 2026-11-14 was 0.0.34, requires_python empty, info.license empty, uploaded 2026-02-16T08:18:42.259321Z. The GitHub license field on 2026-11-14 was Apache-2.0. pushed_at was 2026-03-04T08:15:07Z. This page did not install the package.

Assistant.__init__ on that commit defaults function_list, llm, name, description, and files to None. system_message defaults to DEFAULT_SYSTEM_MESSAGE.

Point llm_cfg at the GPU server

The README’s “Developing Your Own Agent” example comments an OpenAI-compatible block. The comment says model_server is the base URL, also known as api_base. The three keys, with the comment marks removed on this page:

llm_cfg = {
    'model': 'Qwen2.5-7B-Instruct',
    'model_server': 'http://localhost:8000/v1',  # base_url, also known as api_base
    'api_key': 'EMPTY',
}

Qwen2.5-7B-Instruct has no repository owner in that comment. This page does not invent one. The same example’s active generate_cfg sets top_p to 0.8. That key is not inside the comment. The active model and model_type in that example are a hosted block. This page does not copy them.

A later FAQ comment uses the same model_server and api_key with model set to qwen3-32b. That string is not Qwen2.5-7B-Instruct.

examples/assistant_qwen3.5.py on the same commit comments a vLLM or SGLang block. The comment marks are removed on this page. The hosted block above it in that file is not copied.

llm_cfg = {
    # Use your own model service compatible with OpenAI API by vLLM/SGLang:
    'model': 'Qwen/Qwen3.5-397B-A17B',
    'model_type': 'qwenvl_oai',
    'model_server': 'http://localhost:8000/v1',  # api_base
    'api_key': 'EMPTY',

    'generate_cfg': {
        'use_raw_api': True,
        # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way
        'extra_body': {
            'chat_template_kwargs': {'enable_thinking': True}
        },
    },
}

Those two model strings are not the same. The README preparation section says a self-hosted OpenAI-compatible service can follow the Qwen2 README’s vLLM or Ollama section. This page does not copy that other README. It says that for QwQ and Qwen3, do not add --enable-auto-tool-choice and --tool-call-parser hermes, because Qwen-Agent parses the tool outputs from vLLM on its own. For Qwen3-Coder, it says to enable both of those parameters and combine them with use_raw_api. How to serve an OpenAI-compatible API with vLLM on a cloud GPU and How to run SGLang on a cloud GPU are the server installs.

The Qwen-Agent files this page read do not print hf download. How to download a Hugging Face model prints pip install -U "huggingface_hub" and hf download. The block below is added on this page for the repository id the vLLM/SGLang comment writes. This page did not download it.

# Added on this page. The vLLM/SGLang comment passes this id as model.
pip install -U "huggingface_hub"
hf download Qwen/Qwen3.5-397B-A17B

On 2026-11-14, GET https://huggingface.co/api/models/Qwen/Qwen3.5-397B-A17B reported gated false, cardData.license apache-2.0, pipeline_tag image-text-to-text, lastModified 2026-04-24T05:51:23.000Z, and sha 8472618112abcbd45acbcdc58436aff4233c23f7. The card README this page read prints no GPU GB figure. This page does not treat 397B in that id as vram_gb.

For a server you start on the instance, the SSH tunnel is How to use SSH port forwarding on a cloud GPU. The README and the example do not print an SSH command.

Assistant

The README example constructs Assistant after the llm_cfg block. Copied:

bot = Assistant(llm=llm_cfg,
                system_message=system_instruction,
                function_list=tools,
                files=files)

In that example, tools is ['my_image_gen', 'code_interpreter'] and files is ['./examples/resource/doc.pdf']. system_instruction is the string above that call. The constructor defaults function_list and files to None, so those two arguments are optional. The FAQ says the code interpreter runs in a local Docker container, and that Docker has to be installed and running before that tool is used. This page did not start Docker.

The run loop in that example, copied:

for response in bot.run(messages=messages):
    response_plain_text = typewriter_print(response, response_plain_text)

messages is a list of {'role': 'user', 'content': query} dicts. typewriter_print comes from qwen_agent.utils.output_beautify.

The Qwen3.5 file pairs its llm_cfg with this constructor. tools in that file is a list of MCP server commands. Copied:

bot = Assistant(
    llm=llm_cfg,
    function_list=tools,
    name='Qwen3.5 Tool-calling Demo',
    description="I'm a demo using the Qwen3.5 tool calling. Welcome to add and play with your own tools!")

Its test function defaults query to What time is it? and runs the same bot.run loop. This page did not run it.

When it fails

CUDA out of memory. The README and the Qwen3.5 example print no GPU GB figure. The Qwen/Qwen3.5-397B-A17B card README prints no GPU GB figure. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The README and setup.py do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this library on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

What to copy off, then destroy

bot.run yields responses. The example assigns the printed text to response_plain_text. Copy that text before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.

The pricing page FAQ states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Prices are not copied here.

Frequently asked questions

What memory figure do the Qwen-Agent docs print?

The README and examples/assistant_qwen3.5.py print no GPU GB figure. The Qwen/Qwen3.5-397B-A17B README fetched on 2026-11-14 prints no GPU GB figure. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.

Which install line does the README print?

The README prints pip install -U qwen-agent for the minimal requirements, and pip install -U “qwen-agent[gui,rag,code_interpreter,mcp]” for the extras. qwen_agent/init.py at commit 31a4d36d123688581a9e9744427272b33ce940e0 sets version to 0.0.34. PyPI info.version on 2026-11-14 was 0.0.34, requires_python empty, info.license empty, uploaded 2026-02-16T08:18:42.259321Z. setup.py on that commit does not set python_requires. The news line says the GUI requires Python 3.10 or higher. This page did not install the package.

How does llm_cfg point at a server on the GPU?

The README comment sets model to Qwen2.5-7B-Instruct, model_server to http://localhost:8000/v1, and api_key to EMPTY. The comment says model_server is the base_url, also known as api_base. A later comment sets model to qwen3-32b with the same model_server and api_key. examples/assistant_qwen3.5.py comments a vLLM or SGLang block that sets model to Qwen/Qwen3.5-397B-A17B, model_type to qwenvl_oai, model_server to http://localhost:8000/v1, and api_key to EMPTY. Assistant takes llm. The constructor defaults function_list and files to None. Those files do not print an SSH command. The active blocks in both files are a hosted service this page does not copy.

What license does the file print?

The README header says Copyright 2023 The Qwen team, Alibaba Group. The LICENSE file begins with Apache License Version 2.0, January 2004. The GitHub license field on 2026-11-14 was Apache-2.0. PyPI info.license was empty.

Sources (read 2026-11-14)

  • QwenLM/Qwen-Agent README at commit 31a4d36d123688581a9e9744427272b33ce940e0, committed 2026-03-04T08:13:57Z. Install lines, the OpenAI-compatible llm_cfg comment (Qwen2.5-7B-Instruct, http://localhost:8000/v1, EMPTY), the qwen3-32b comment, Assistant(...), bot.run, and the vLLM tool-parser sentences. The hosted llm_cfg block is not copied. GitHub license field Apache-2.0. pushed_at 2026-03-04T08:15:07Z
  • examples/assistant_qwen3.5.py on that commit: the commented vLLM/SGLang llm_cfg (Qwen/Qwen3.5-397B-A17B, model_type qwenvl_oai, http://localhost:8000/v1, EMPTY, use_raw_api) and the Assistant constructor. The hosted block in that file is not copied
  • qwen_agent/agents/assistant.py on that commit: function_list and files default to None
  • qwen_agent/__init__.py __version__ = '0.0.34'. setup.py does not set python_requires. PyPI qwen-agent 0.0.34, uploaded 2026-02-16T08:18:42.259321Z, requires_python empty, info.license empty
  • Hugging Face model API Qwen/Qwen3.5-397B-A17B: gated false, cardData.license apache-2.0, pipeline_tag image-text-to-text, lastModified 2026-04-24T05:51:23.000Z, sha 8472618112abcbd45acbcdc58436aff4233c23f7. The card README prints no GPU GB figure. hf download is added on this page and was not run
  • Public GPU memory: https://cloudgpu.app/v1/public/gpu-catalog (generated_at 2026-11-14T21:12:05.060Z). GET /v1/public/compute-capabilities returned feature flags and no GPU list. The catalog had no Qwen-Agent template

Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-14.