使用 NVIDIA Topograph 优化负载平衡

NVIDIA Developer says AI factories are power-constrained systems that deliver maximum value when fully optimized. Load placement on graphics processing units (GPUs) is a key element of this optimization. Poor load placement fragments topological domains and forces traffic through shared links, which reduces throughput, increases task execution costs, and leaves the GPU consuming allocated power while waiting for data without advancing the load. GPUs constantly communicate during training and inference, so distributed workloads benefit from local communication. NVIDIA NVLink and NVLink Switch provide high-speed, scalable connectivity across GPU domains at the rack level, while NVIDIA Ethernet Spectrum-X delivers predictable, low-latency network connectivity at the system and rack level. The scheduler can only place workloads efficiently if it has an up-to-date and accurate view of the relationships between the GPU and the network infrastructure, and keeping this view up-to-date as the cluster changes is a challenge faced

are found in practice. NVIDIA Topograph solves this problem. It discovers the cluster topology through cloud APIs or on-premises systems, normalizes it into a common model, and publishes it in the format every load manager expects: Kubernetes node labels, Slurm topology configuration, or Slinky ConfigMaps. As part of the NVIDIA DSX OS cluster orchestration layer, Topograph works in conjunction with Dynamic Resource Allocation (DRA) and the KAI scheduler to provide topology-aware group scheduling across the entire AI fabric infrastructure. The main topology issue is how the cluster hardware is connected to allow schedulers to favor nearby resources. Think of the network as a road system: GPUs in one local domain have short, high-speed paths, while traffic between domains passes through more common links and switches. Distributing densely coupled workloads across remote domains can increase contention and latency, so Topograph helps place workloads in the most efficient locations and avoid these bottlenecks. Modern NVIDIA Quantum InfiniBand ports can reach speeds of up to 800 Gbps, while NVIDIA NVLink both

Delivers 1.8 TB/s of bidirectional throughput on fifth-generation GPUs (NVIDIA Blackwell, such as GB200/GB300) and 3.6 TB/s on sixth-generation GPUs (Vera Rubin) via dedicated NVIDIA NVLink Switch infrastructure. This non-blocking, shared system gives each GPU its own bandwidth rather than sharing bandwidth under load. Schedulers with an up-to-date view may favor GPUs in the closest topological domain. Slurm and Kubernetes both support topology-aware distribution, but the scheduler can only act based on the topology it observes. Topograph restores this view on demand and as changes occur in the cluster, so the scheduler works with up-to-date data rather than a manually maintained snapshot. Topograph is an open tool that identifies the network topology of a cluster, allowing load managers to make topology-aware scheduling decisions. It has two concepts: providers and engines. The provider discovers the topology through cloud APIs or on-premises systems and normalizes it into a canonical model. The engine translates this model into a Slurm configuration, Kuber labels

netes, Slinky ConfigMaps, Node Feature Discovery (NFD) resources, or instance-oriented topology JSON. Cloud providers that have working integrations with Topograph include Google Cloud, Lambda, Nebius, Nscale and OCI, with additional cloud and colocation providers in development. For on-premises use, use an InfiniBand provider with ibnetdiscover or NetQ for Spectrum-X or Multi-Node NVLink (MNNVL) domains. The provider interface is open so operators can add their own for their environment and contribute to the overall project. The material was prepared by a neural network based on the original publication. AI may make mistakes in details (numbers, names, dates, quotes) - for important information, check with the original source. Delivering Secure, High-Performance AI with NVIDIA Confidential Computing Claude Opus 5.5 Now Available on Amazon Bedrock Tata Elxsi has developed the IRIS platform to detect risks in factories in seconds. AstroForge is developing AI to control spacecraft, which will allow the company to conduct missions without constant monitoring from Earth. Startup uses neural patterns from rat brains to create AI fashion

lei.