Video Generation - Cosmos3 Nano

<aside> 🛠

Text-to-Video and Image-to-Video — Powered by NVIDIA's Cosmos 3 World Model

Generate short cinematic video clips from a text description or a single image. Cosmos3-Nano is NVIDIA's lightweight 16B-parameter "world model" that has learned how the physical world looks and behaves. The studio exposes its video generation capability as a standalone feature first; other modalities (audio, action, reasoning) the model also supports may follow).

</aside>

What it does

Cosmos3-Nano generates short, realistic video clips from either a text prompt ("a wide shot of a winter forest, snow falling, soft golden hour light…") or a single starting image plus a text description of what should happen next. The output is an MP4 video, by default ~7.9 seconds long at 1280×720 and 24 fps. Optionally, the model can also generate synchronized audio muxed into the video — useful for ambient sound, footsteps, environmental noise, etc. Cosmos3-Nano comes from NVIDIA's Cosmos 3 family of world foundation models, designed with Physical AI in mind (robotics, autonomous driving, industrial simulation), which makes it especially strong on realistic motion, scene dynamics, and camera behavior. "Nano" is the lightweight variant of the family (16B parameters); a larger Cosmos3-Super variant (64B) also exists with higher quality at heavier compute cost.

Problem it solves

Two generation modes in one model – Text-to-video for fully imagined scenes, image-to-video for animating a starting frame
Optional matching audio – Generate ambient sound or environmental audio that's synchronized with the generated video, in one pass
Physical-world realism – Trained for Physical AI use cases (robotics, autonomous driving, industrial scenes), so motion and scene dynamics tend to look more physically plausible than general-purpose video generators
Cinematic 720p output – Native 1280×720 generation at 24 fps, supports multiple aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16)
Animate a single photo – Bring a still image to life with a short text instruction
Open-source and commercial-friendly – Released under OpenMDW 1.1, approved for both commercial and non-commercial use

Input/Output

Input:
- Text to Video:
  - Plain-language description of the scene you want
  - Text up to 4,096 tokens
  - English works best
- Image to Video:
  - One reference image (PNG / JPG / JPEG / WebP) at 256p, 480p, or 720p
  - Aspect ratios: 16:9, 4:3, 1:1, 3:4, 9:16
  - A text description of what should happen in the video starting from that image
Options:
- Resolution (W × H) - ****Pick the ratio that matches the input image (when using image-to-video)
- Frame count (must be a 4n + 1 value, e.g. 61, 121, 189, 241, 301) - default 189 frames (≈ 7.9s at 24 fps)
- Output FPS (24 is the native frame rate) - 16, 24, 30
- Sound generation (On / Off) - default Off
  - When On, describe the desired sound in the text prompt (e.g. "footsteps on gravel, wind in trees")
- Diffusion steps (20 - 75, default 35) - More steps = higher quality, slower generation
- Advanced Options:
  - CFG scale (default 6) - how strictly the output follows the prompt
  - Additional negative prompt (default empty) - extra description of what to avoid in the output. The model already ships with a built-in default negative prompt; this field is for anything you want to add on top
  - Random seed (-1 = random, default 1) - set a fixed integer to reproduce the exact same video
Output: Generated video
- Format: MP4
- Default size: 1280 × 720 at 24 fps
- Default length: 189 frames (≈ 7.9 seconds), configurable 5 – 400 frames
85afd189-e9af-409b-b080-a2c24cdc7719.mp4

Accuracy & Speed


Model size	16 billion parameters (the "Nano" variant of NVIDIA Cosmos 3)
Bigger sibling	Cosmos3-Super (64B) — higher quality, much heavier to run
Default video output	1280×720, 24 fps, 189 frames (≈ 7.9 seconds)
Output range	5 to 400 frames, multiple aspect ratios supported
Tested hardware	NVIDIA GB200 and H100
Recommended serving GPU	NVIDIA H200 (single-GPU, the documented vLLM-Omni configuration)
Multi-GPU scaling	Context parallelism (`--ulysses-degree`) or tensor parallelism (`--tensor-parallel-size`)
Lower-VRAM option	Layerwise offload (`--enable-layerwise-offload`) for smaller GPUs
Precision	BF16 — the only officially tested precision (FP16 / FP8 / FP4 not officially supported)
Strengths	Realistic motion and scene dynamics (Physical-AI training focus), optional synchronized audio, multi-aspect-ratio support

Technical Details


Architecture	Mixture-of-Transformers (MoT) — two complementary transformer towers in a unified model
Towers	(1) Autoregressive transformer for text; (2) Diffusion transformer for non-text (images, video, audio, actions). Video generation is produced by the diffusion tower through iterative denoising
Parameters	16B (Cosmos3-Nano)
Framework	NVIDIA Cosmos Framework
Default Inference Hyperparameters	`num_inference_steps=35`, `guidance_scale=6.0`, `flow_shift=10.0`, `max_sequence_length=4096`, `num_frames=189`, `fps=24`, size `1280×720`
Prompt upsampling	Text prompts should be JSON-upsampled (separate utility provided in the cosmos-framework repo) for optimal quality; NVIDIA's text-to-video example uses Claude Opus 4.6 as the upsampler
Training data (generation)	767M image, 348M video, 139M audio, 8M action samples — drawn from public datasets (OpenImages, Coyo700M, YouTube Video, UMI), private datasets (Egocentric, Nexar, AgiBot, HOI), and large synthetic augmentation (HiDream-I1, Qwen-Image-2512)
Supported Hardware Microarchitectures	NVIDIA Ampere, Blackwell, Hopper
OS	Linux
Supported Frameworks	PyTorch, vLLM-Omni (Cosmos3 container), vLLM (with `vllm-cosmos3` package), Hugging Face Diffusers (`Cosmos3OmniPipeline`)
Other capabilities of the underlying model	The same Cosmos3-Nano model also supports text reasoning over images and video (256K context), action-conditioned video generation for robotics, and action prediction from video — not exposed in the studio's video generation feature, but may follow in later releases
Release date	May 31, 2026
Paper	"Cosmos 3: Omnimodal World Models for Physical AI" — NVIDIA Cosmos Lab technical report

What it does

Problem it solves

Input/Output

Accuracy & Speed

Technical Details

Compliance & Provenance