What is new

NVIDIA has released Nemotron 3.5 Super VL, a new addition to its Nemotron line of open models that already includes Nemotron 3 Nano, Nano Omni, Super, Ultra, and Nemotron 3.5 Lightning. The distinguishing feature of this release is that it is a single model handling both language reasoning and visual understanding: it can read text, images, documents, and video, and it can use tools, while producing text output. Amazon Web Services has published a companion guide showing how to deploy it on Amazon Elastic Kubernetes Service (EKS), EKS Auto Mode, and SageMaker HyperPod EKS, alongside a method for measuring how it performs under load.

NVIDIA describes Nemotron 3.5 Super VL as distilled from the larger Nemotron 3 Ultra model, and says it is publishing the weights, the training datasets, and the post-training recipes. That openness matters for practitioners because it means the model can be fine-tuned or adapted for specific domains and run on infrastructure the user controls, rather than only being accessible through a hosted API.

How it works

The model has 120 billion total parameters but only activates 12 billion of them for any given token. This is possible because it uses a mixture-of-experts (MoE) design: instead of running every parameter for every piece of input, the model routes each token to a small subset of specialised sub-networks ('experts'). A learned router decides which experts handle which tokens. The benefit is that the computational cost per token looks like that of a much smaller, 12-billion-parameter model, even though the full 120-billion-parameter model has to be held in GPU memory.

Nemotron 3.5 Super VL combines this MoE approach with a hybrid architecture that interleaves three kinds of layers: Mamba-2 layers, standard attention (transformer) layers, and the expert layers. Mamba-based layers are a more recent alternative to the attention mechanism used in standard transformers; rather than keeping track of every previous token in a growing cache, they carry a fixed-size internal state as they process a sequence. In plain terms, this means memory use does not keep climbing as the input gets longer, which is one reason the model can support context windows up to one million tokens without the usual memory penalty that comes with transformer-only designs. A RADIO vision encoder handles the image, document, and video inputs so the same model backbone can reason over both text and visual material.

The model also includes a recursively shared Multi-Token Prediction (MTP) head, which enables what NVIDIA calls native speculative decoding. Speculative decoding is a technique where a lighter-weight mechanism guesses several tokens ahead and the main model verifies them in parallel, which can speed up generation compared to producing one token at a time.

Checkpoints are available in BF16 and NVFP4 precision formats, which trade off numerical precision against memory footprint and speed. NVIDIA reports the following benchmark results for the model, according to its technical report: 63.44 percent on Hermes Bench, 87.05 percent on LangChain agent evaluations, and 80.8 percent on the CharXiv chart-reasoning benchmark.

Why it matters

Agentic AI systems are software setups where a model does not just answer a single question but plans and carries out a sequence of actions, often calling external tools, reading documents, or browsing the web, and checking its own results before proceeding. Having one model that can both reason through such a plan and directly interpret images, documents, or video removes the need to stitch together separate vision and language models, which can simplify agent pipelines and reduce the number of moving parts an engineering team has to maintain.

AWS and NVIDIA describe several concrete applications for this combination:

  • Coding agents that test and validate code, analyze errors, and recommend fixes
  • Web research agents that cross-reference and verify sources returned by search tools
  • Document intelligence that reads tables, charts, and page layouts to produce summaries
  • Cybersecurity workflows that identify vulnerabilities and triage threats
  • Video understanding tasks such as search, content review, and incident analysis

The model is also built to integrate with existing open agent frameworks and orchestration systems through NVIDIA's reference implementations, cookbooks, and its Agent Toolkit, which should lower the integration effort for teams that already have agent pipelines built around other models.

Deploying and measuring performance

AWS's guide focuses on running Nemotron 3.5 Super VL (and Nemotron 3 Ultra) on Kubernetes-based infrastructure: standard Amazon EKS, EKS Auto Mode (where AWS manages scaling and GPU/network support automatically), and SageMaker HyperPod EKS. The same Kubernetes deployment manifests work across all three options. In the reference architecture, CPU nodes handle system and test workloads while GPU nodes run the model itself; container images are stored in Amazon Elastic Container Registry, model weights sit on a shared Amazon FSx for Lustre volume, and GPU nodes are connected through Elastic Fabric Adapter (EFA) networking for fast communication between GPUs.

NVIDIA Dynamo, the inference-serving component, can run the model either aggregated on a single node or disaggregated, meaning the prefill stage (processing the prompt) and the decode stage (generating output tokens) run on separate nodes; for vision-language workloads, the image or video encoding step can also be split onto its own node. The minimum number of GPUs needed depends on precision and hardware: for example, NVIDIA lists 8 B200/GB200 GPUs in BF16 versus 2 in NVFP4, and 16 H100 GPUs in BF16 versus 8 in NVFP4.

AWS validated this setup on p6-b200.48xlarge instances (8 NVIDIA B200 GPUs) and p5en.48xlarge instances (8 NVIDIA H200 GPUs). Cluster health and networking are checked using an NCCL all-reduce micro-benchmark, which confirms that GPUs and EFA adapters are all being used as expected. Once a model is deployed, its real-world performance is measured with NVIDIA's aiperf tool, which reports time to first token (TTFT), inter-token latency (ITL), and throughput in tokens per second.

AWS illustrates how to use these measurements for capacity planning: in one example concurrency sweep, under a latency budget of TTFT below 600 milliseconds and ITL below 100 milliseconds per token, a single aggregated or disaggregated replica on two nodes could serve around 320 tokens per second at a concurrency level the post labels c32. To scale capacity, teams would divide their target overall throughput by this per-replica figure to determine how many such replicas (prefill-decode pairs) are needed. Because TTFT and ITL are tracked separately, a rising ITL points to a need for more decode capacity while a rising TTFT points to a need for more prefill capacity.

Limits and open questions

The source material does not provide comparisons between Nemotron 3.5 Super VL and competing multimodal models beyond the three benchmark figures cited, so it is not possible from this announcement alone to say how it stacks up against other open or proprietary vision-language models. It also does not specify training data sources, licensing terms, or inference cost figures beyond the GPU count tables. AWS presents the deployment pipeline as a reference pattern validated on specific instance types, rather than a fully managed, one-click service, so teams adopting it will need to handle cluster setup, weight downloads, and benchmarking themselves, following the detailed walkthrough in the aws-do-eks project referenced in the post.

How to try it

AWS points readers to the aws-do-eks project for a full walkthrough covering prerequisites, cluster creation, downloading model weights, deployment, testing, benchmarking, and cleanup for Nemotron 3.5 Super VL and Nemotron 3 Ultra on Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS. The same deployment pattern is said to apply to other models in the Nemotron family. Additional inference blueprints are available through the AI on EKS project, and AWS mentions upcoming workshops for hands-on experience with inference on EKS.

Sources