Company logos are used for identification purposes only and do not imply endorsement or official partnership unless otherwise stated.

Running NVIDIA Cosmos on the GB10: A Physical AI Field Guide

What works, what doesn't, and what nobody tells you about running Cosmos models on ARM64 Blackwell hardware

Written and prepared by Curtis Burkhalter, Ph.D. | Technical Product Marketing Manager, AI Solutions at HP

Last verified: July 2026

---

This guide covers what works, what doesn't, and what nobody tells you about running NVIDIA's Cosmos model family on GB10 Blackwell hardware (HP ZGX Nano, DGX Spark). It was written after building a grid infrastructure anomaly prediction demo that uses Cosmos-Reason1-7B for visual reasoning and Cosmos3-Nano for video generation, running simultaneously on a single GB10 with 128GB unified memory.

If you're trying to build Physical AI applications on ARM64 Blackwell and hitting walls that the official docs don't mention, this guide is for you.

Model Compatibility Matrix

This is the table I wish existed when I started. Every entry was tested on a GB10 with the specific versions noted.

Model	Works on GB10?	Serving Method	GPU Memory	Notes
Cosmos-Reason1-7B	✅ Yes	vLLM Docker	~41GB	Full VLM reasoning, equipment detection, anomaly classification
Cosmos3-Nano	✅ Yes	In-process diffusers	~37GB	Text-to-video generation, degradation visualization
Cosmos-Predict2.5-2B	❌ No	N/A	N/A	Hard dependency on transformer_engine — no ARM64 build. GitHub #120
Cosmos-Transfer2.5-2B	❓ Untested	N/A	N/A	Same dependency chain as Predict2.5, likely same outcome
Cosmos3-Super	❓ Untested	diffusers	~130GB+	64B model, would need layerwise offloading on single GB10

Setting Up Cosmos-Reason1-7B via vLLM

Cosmos-Reason1-7B is a 7B vision-language model based on Qwen2.5-VL. It handles image and video understanding with chain-of-thought reasoning. On the GB10, the cleanest path is serving it through vLLM's Docker container.

Download the model

pip install huggingface-hub hf_transfer
export HF_HUB_ENABLE_HF_TRANSFER=1
huggingface-cli download nvidia/Cosmos-Reason1-7B

About 16.6GB. Takes a few minutes with hf_transfer enabled.

Start the vLLM container

docker run -d \
    --name cosmos-reason-vllm \
    --gpus all \
    --shm-size=16g \
    -p 8091:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    nvcr.io/nvidia/vllm:26.01-py3 \
    python3 -m vllm.entrypoints.openai.api_server \
        --model nvidia/Cosmos-Reason1-7B \
        --dtype bfloat16 \
        --max-model-len 16384 \
        --gpu-memory-utilization 0.35 \
        --enforce-eager \
        --trust-remote-code \
        --host 0.0.0.0 \
        --port 8000

Three flags here that are not optional and not in the standard vLLM docs:

--gpu-memory-utilization 0.35 limits vLLM to about 45GB of the 128GB unified memory. The default is 0.9, which pre-allocates ~115GB for KV cache and leaves nothing for other models. If you plan to run Cosmos3-Nano alongside vLLM (and you probably do), you need this.

--enforce-eager disables CUDA graph capture. At low gpu-memory-utilization, graph capture runs out of contiguous memory during warmup and throws cudaErrorIllegalInstruction. Eager mode is slightly slower per-token but loads reliably. Without this flag, the container crashes at ~71% through graph capture with no useful error message.

--max-model-len 16384 sets the context window. The model supports up to 32768 but chain-of-thought reasoning with JSON output can consume 4000-6000 tokens per call. 16384 gives enough headroom without over-allocating KV cache.

Verify it's running

curl http://localhost:8091/v1/models

First request after startup will be slow (CUDA kernel compilation). Subsequent requests: ~13 tokens/second generation throughput, ~290 tokens/second prompt processing.

Send a test request

# Encode a test image
IMG_B64=$(base64 -w 0 test_frame.jpg)

curl -s http://localhost:8091/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Cosmos-Reason1-7B",
    "messages": [
      {"role": "system", "content": "You are an expert infrastructure analyst. Respond directly with JSON."},
      {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'$IMG_B64'"}},
        {"type": "text", "text": "Identify all equipment in this image. Return a JSON array."}
      ]}
    ],
    "max_tokens": 8192,
    "temperature": 0.6
  }' | python3 -m json.tool

Things that will bite you

httpx timeout. If you're calling vLLM from a Python backend (FastAPI, Flask), set your HTTP client timeout to at least 300 seconds. Chain-of-thought reasoning at 13 tok/s produces 2-3 minutes of generation before the JSON output. The default httpx timeout of 5 seconds will kill every request.

client = httpx.AsyncClient(timeout=300.0)

max_tokens vs max-model-len. If you set max_tokens in your API request higher than --max-model-len on the server, you get a 400 Bad Request with no explanation. Keep max_tokens below the context window.

Chain-of-thought wastes tokens for structured output. The <think> system prompt makes the model reason extensively before producing JSON. For equipment detection where you just need a fast list, strip the chain-of-thought instruction from the system prompt. For anomaly classification where reasoning quality matters, keep it.

Bounding box format varies. The model sometimes returns [x1, y1, x2, y2] as a list, sometimes {"x": ..., "y": ..., "width": ..., "height": ...} as a dict. Your parsing code needs to handle both.

Unknown equipment types. The model will return types like "metal_frame" or "power_line" that aren't in any predefined enum. Map unknown types to a catch-all rather than throwing a ValueError.

Setting Up Cosmos3-Nano via Diffusers

Cosmos3-Nano is a 16B unified model that handles both reasoning and generation. We use it specifically for text-to-video generation of equipment degradation scenarios. The key advantage over Cosmos-Predict2.5 is that it installs with standard pip packages.

Install dependencies

# CUDA-enabled PyTorch for ARM64
pip install torch --index-url https://download.pytorch.org/whl/cu130

# Match torchvision to the same torch version
pip install torchvision --index-url https://download.pytorch.org/whl/cu130

# Diffusers from git (Cosmos3OmniPipeline is not in the PyPI release yet)
pip install "diffusers @ git+https://github.com/huggingface/diffusers.git"

# Additional dependencies
pip install accelerate av imageio imageio-ffmpeg

Three things to watch for in this install:

PyTorch cu130 ARM64 wheels exist. pip install torch --index-url https://download.pytorch.org/whl/cu130 pulls torch-2.12.1+cu130-cp312-cp312-manylinux_2_28_aarch64.whl. This is not obvious from the PyTorch website, which doesn't list ARM64 CUDA builds prominently.

python3 -c "import torch; print(torch.__version__, 'CUDA:', torch.cuda.is_available())"

If CUDA shows False, force reinstall: pip install --force-reinstall torch --index-url https://download.pytorch.org/whl/cu130

Cosmos3OmniPipeline is only in the diffusers dev branch. The PyPI release (0.38.0 as of July 2026) doesn't have it. You must install from git. This will likely change in a future release.

Load and generate

import torch
from diffusers import Cosmos3OmniPipeline

pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Nano",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)

result = pipe(
    prompt=(
        "A high-definition surveillance camera view of a power transformer "
        "at an electrical substation showing progressive oil leak degradation. "
        "Dark staining at the base, drip marks on the side. Stationary camera. "
        "Clear daylight."
    ),
    num_frames=49,
    height=480,
    width=640,
    num_inference_steps=25,
    guidance_scale=6.0,
    generator=torch.Generator(device="cuda").manual_seed(42),
)

from diffusers.utils import export_to_video
export_to_video(result.video, "degradation_30d.mp4", fps=24)

First run downloads ~32GB of model weights from HuggingFace and takes 2-3 minutes to load. Subsequent runs skip the download. Generation itself takes about 60 seconds for 49 frames at 480p with 25 inference steps.

Running Both Models Simultaneously

This is the part nobody documents. Cosmos Reason via vLLM and Cosmos3-Nano via diffusers can coexist in the GB10's 128GB unified memory, but only with specific settings.

Memory budget

Component	GPU Memory
vLLM (Cosmos-Reason1-7B, 0.35 utilization)	~41GB
Cosmos3-Nano (diffusers, BF16)	~37GB
Total	~78GB
Remaining	~50GB

This leaves enough headroom for KV cache, intermediate tensors, and the OS. In practice, nvidia-smi shows 96% GPU utilization during inference with both models loaded.

The startup sequence matters

1. Start vLLM first (Docker container, port 8091)
2. Wait for vLLM to report healthy: curl http://localhost:8091/health
3. Start your application (which loads Cosmos3-Nano on first generation request)

If you start Cosmos3-Nano first and it grabs GPU memory, vLLM may not have enough for its KV cache allocation and will crash on startup.

Deferred loading for Cosmos3-Nano

Don't load Cosmos3-Nano at application startup. Load it on first use. This keeps your app startup fast and avoids GPU memory allocation when you might not need video generation for every request.

class CosmosPredictService:
    def __init__(self):
        self._pipe = None
        self._loaded = False

    async def generate(self, prompt, output_path):
        if not self._loaded:
            # First-use loading: ~2-3 minutes
            self._pipe = self._load_pipeline()
            self._loaded = True
        # Generation: ~60 seconds
        return self._generate_video(prompt, output_path)

Monitoring during dual-model inference

watch -n 1 nvidia-smi

You'll see two processes: VLLM::EngineCore (~41GB) and python3.12 (~37GB). GPU-Util will spike to 96% during active inference. Temperature typically runs 57-81°C depending on workload.

Why Cosmos-Predict2.5 Doesn't Work on GB10

The spec for our demo originally called for Cosmos-Predict2.5-2B with Video2World, which takes your actual input video and generates a future version of it (image-conditioned generation). This is architecturally superior to Cosmos3-Nano's text-to-video approach because the output is grounded in your specific footage.

Here's the dependency chain that blocks it on ARM64:

We got past cosmos-oss with --no-deps, stubbed cosmos-cuda with a fake package, installed decord via conda, but transformer_engine is the wall. It's a compiled C++/CUDA library designed for NVIDIA's internal Docker images (DGX, HGX). Building from source on ARM64 is not supported.

This is tracked as GitHub issue #120 on the cosmos-predict2.5 repo.

The practical impact

Cosmos3-Nano generates degradation videos from text prompts. The generated video shows what "a degraded transformer" looks like generically, not what your specific transformer will look like in 30 days. The detection, classification, health scores, and time-to-failure estimates from Cosmos-Reason1-7B are real analysis of your actual footage. The video visualization is illustrative.

For demo purposes, this works. For production deployment where pixel-accurate prediction matters, you'd need to either run Cosmos-Predict2.5 on x86 hardware or wait for transformer_engine ARM64 support.

Benchmarks

All measurements from the grid anomaly prediction pipeline on HP ZGX Nano.

Operation	Time	Notes
vLLM startup (cold)	~2.5 min	Model loading + kernel compilation
vLLM startup (warm, cached kernels)	~1.5 min	Cached CUDA kernels
Cosmos Reason: equipment detection	~2-3 min	Chain-of-thought + JSON output
Cosmos Reason: anomaly classification	~2-3 min	Per equipment item
Cosmos Reason: prompt throughput	~290 tok/s	Prompt processing
Cosmos Reason: generation throughput	~13 tok/s	Token generation
Cosmos3-Nano: first load	~2-3 min	Downloads weights on first run
Cosmos3-Nano: video generation	~60 sec	49 frames, 480p, 25 steps
Combined GPU memory	~78GB	Both models loaded
GPU utilization during inference	96%	Peaks during active generation
GPU temperature	57-81°C	Idle to sustained load

Full pipeline (upload to results)

Stage	Duration
Video upload + frame extraction	~5 sec
Equipment detection (Cosmos Reason)	~2-3 min
User confirmation	Manual
Anomaly classification (Cosmos Reason)	~2-3 min
Video generation (Cosmos3-Nano, 3 horizons)	~3-4 min
Morph rendering + annotation + encoding	~30 sec
Total	~8-10 min

What NVIDIA Needs to Improve for Physical AI on Edge ARM64

NVIDIA markets the GB10 for Physical AI workloads. The Cosmos model family is their Physical AI platform. These two products should work together seamlessly, but right now it appears they don't. Here's what's missing:

1. transformer_engine needs ARM64 support. This single dependency blocks Cosmos-Predict2.5, which is the model designed for the exact use case NVIDIA pitches (video prediction from real footage). Without it, the GB10 can't run NVIDIA's flagship Physical AI prediction model.
2. Cosmos3OmniPipeline needs to be in a stable diffusers release. Currently requires installing from git, which is fragile for production deployments.
3. The cosmos-oss package has broken version pins. pyproject.toml pins cosmos-oss==0.1.0 but the package ships as 1.5.0. Basic package management.
4. vLLM --gpu-memory-utilization + CUDA graphs interaction needs documentation. Low memory utilization settings crash CUDA graph capture with an unhelpful cudaErrorIllegalInstruction. The fix (--enforce-eager) is not documented anywhere in relation to memory utilization.
5. Image-conditioned Video2World in Cosmos3. The text-to-video path works. The image-to-video path (which would make degradation predictions grounded in actual footage) needs to be available through the same clean diffusers interface.

Resources

• Cosmos GitHub
• Cosmos-Reason1-7B on HuggingFace
• Cosmos3-Nano on HuggingFace
• Cosmos-Predict2.5 ARM64 issue
• vLLM on GB10 guide (companion guide)
• Grid Anomaly Prediction demo source (the application that produced these findings)