AI Networking: RDMA in GB200 and GB300

If you know roughly what CPUs, memory, and networks do, you have all the background you need. We will start with why AI needs networking, then work our way into the three-layer architecture of NVIDIA GB200 and GB300 NVL72 racks and the key technology that makes 100,000 GPUs operate like one computer: RDMA.

1. AI’s Bottleneck Is Often Traffic, Not Compute

Discussions of AI chips usually focus on GPU speed and memory capacity. But here is a counterintuitive fact: in large AI training clusters, GPUs can spend a great deal of time waiting for the network instead of computing. One illustrative textbook model, based on a 70-billion-parameter workload and H100-class interconnects, estimates that communication consumes about 25% of each training step on a single eight-GPU node. At 4,096 GPUs, communication and synchronization together reach roughly 80%. The exact share is workload- and topology-dependent, and techniques such as overlapping communication with computation can reduce it, but the network remains one of the most critical bottlenecks.¹

Why? Today’s largest language models (LLMs) contain hundreds of billions or even trillions of parameters, and their full training state—including weights, gradients, optimizer state, and activations—does not fit on a single GPU. Training therefore requires partitioning the model and data across thousands or tens of thousands of GPUs. After each training step, those GPUs must exchange results through collective communication operations such as AllReduce and synchronize before they can begin the next step.

Think of it as an enormous group project:

• Each team member (GPU) is responsible for one part.
• Every few minutes, everyone must stop and meet to reconcile their answers. If one person is late, everyone waits.
• The slower the meeting (network transfer), the more time everyone spends idle—and the more expensive talent (GPU capacity) goes to waste.

The rhythm of distributed training: after every step, all GPUs wait for one another at an AllReduce synchronization point. The slowest component determines overall speed, and the larger the cluster, the more time is spent idle.

That makes the network the invisible foundation of an AI supercomputer. At the heart of that foundation is the subject of this article: RDMA (Remote Direct Memory Access).

2. What Is RDMA? Start With Why Traditional Networking Is Slow

2.1 Traditional TCP/IP: Every Package Passes Through Several Customs Checkpoints

The TCP/IP protocols we use for everyday networking move data through a process that looks roughly like this. This is a simplified conventional sockets path: modern NIC offloads and tuned zero-copy APIs can remove some of the work, but the host networking stack still remains in the data path.

1. The application hands its data to the operating system kernel.
2. The operating system copies the data into buffers in system memory, divides it into packets, adds headers, and calculates checksums.
3. The NIC sends the packets. On receipt, the other system does the process in reverse: CPU interrupts, packet processing, and another copy into the application’s memory.

This process is extremely robust—the internet runs on it—but it carries three costs:

• The CPU stays busy: protocol processing, interrupts or polling, and buffer management can consume substantial host CPU time at high transfer rates, even when NIC offloads are enabled.
• Repeated copies: data moves back and forth among application memory, kernel buffers, and the NIC.
• Added latency: the extra software and memory path adds delay; exact latency ranges from microseconds in highly tuned data center networks to milliseconds in less specialized environments.

That is more than adequate for everyday internet use. For a distributed AI job that may exchange hundreds of gigabytes of gradients and other training state per step across the cluster, however, it is like driving an F1 car on city streets.²

2.2 RDMA: Let the NIC Reach Directly Across the Network

The core idea behind RDMA fits in one sentence:

After the endpoints have registered memory and set up their queues, let one computer’s NIC read from or write to another computer’s memory directly on the steady-state data path, without routing each payload through the CPUs or operating-system kernels.

RDMA accomplishes three specific things:

1. Kernel bypass: after setup, the application posts work directly to NIC queues without taking each transfer through the operating system kernel.
2. Zero copy: payload data travels directly from registered source memory to registered destination memory without being copied through intermediate kernel buffers.
3. Hardware offload: dedicated NIC hardware handles packet segmentation, reassembly, and reliable transport, sharply reducing CPU work on the data path.

RDMA is not limited to one-sided Read and Write operations. It also supports two-sided Send/Receive, in which both endpoints post buffers. In either model, the NIC can access only memory that has been registered and authorized; RDMA does not expose arbitrary system memory.

By analogy, traditional TCP/IP is like writing a letter, taking it to the post office, having the recipient’s post office receive it, and then waiting for delivery. RDMA is like installing a pneumatic tube directly between two buildings: after the tube has been set up, the document shoots straight to the other person’s desk without either building’s reception staff (the CPUs) handling each delivery. Depending on the fabric, topology, and message size, latency can fall into the microsecond range while steady-state CPU overhead drops sharply.²

Traditional TCP/IP and RDMA data paths compared: TCP/IP passes through the kernel and copies data at each endpoint, while RDMA’s steady-state payload path lets the NIC access registered remote memory directly.

2.3 GPUDirect RDMA: Bypassing Host Memory Too

NVIDIA takes the concept one step further in AI servers. Traditionally, a GPU sending data over a network must first copy it into host system memory—a bounce buffer—and then hand it to the NIC. GPUDirect RDMA allows the NIC to use PCIe to read and write GPU memory (HBM) directly, changing the data path to:

Traditional: GPU memory → host system memory → NIC → network → remote NIC → host memory → GPU memory
GPUDirect RDMA: GPU memory → NIC → network → remote NIC → GPU memory

GPUDirect RDMA versus the traditional path: data moves directly between GPU memory (HBM) at both endpoints, bypassing host system memory entirely.

NVIDIA reports that this can improve data-transfer performance by up to 10×.³ It is the technical foundation that lets an RDMA-capable NIC move gradients directly between GPU memories in different racks, creating a tightly coupled distributed system—not one coherent GPU or a single-device programming model.

3. Meet the Main Players: What Exactly Are GB200 and GB300?

3.1 GB200 Superchip vs. GB200 NVL72 Rack

Many people assume that GB200 is a single GPU card. The name actually appears at two levels: GB200 is a Grace Blackwell Superchip, while GB200 NVL72 is the rack-scale supercomputer built from those Superchips:

• GB200 Superchip: one Grace CPU based on the Arm architecture plus two Blackwell GPUs, packaged together and directly interconnected with NVLink-C2C.⁴
• GB200 NVL72: one rack containing 36 Grace CPUs and 72 Blackwell GPUs—18 compute trays and nine NVLink switch trays—with full liquid cooling and power consumption of roughly 120 kW.⁵
• GB300 NVL72: the upgraded system launched in the second half of 2025, featuring Blackwell Ultra GPUs. Per-GPU HBM3e capacity increases from 186 GB (372 GB per two-GPU GB200 Superchip) to 288 GB, dense FP4 Tensor Core throughput rises by 1.5×, and the ConnectX-8 I/O module provides an aggregate 800 Gb/s of scale-out connectivity per GPU. It targets long-context inference and AI reasoning workloads.^{5 6}

3.2 Three Conceptual Network Domains

The key to understanding GB200 and GB300 NVL72 networking is to group their traffic into three conceptual domains, each with its own job.^{7 8}

These three layers complement rather than replace one another. NVLink makes 72 GPUs within a rack behave like one giant GPU. The RDMA network connects tens or hundreds of racks into an AI factory. The front-end network handles everything else. This is a workload-oriented simplification: production designs can divide these roles among additional physical fabrics, and out-of-band BMC management is separate from the BlueField-backed converged network. Let us examine each layer in turn.

4. Layer One—NVLink: Welding 72 GPUs Into One

GB200 and GB300 NVL72 racks use fifth-generation NVLink:

• Each GPU has 18 NVLink connections providing an aggregate 1.8 TB/s of bidirectional bandwidth, more than 14 times that of PCIe Gen5.⁹
• Nine NVLink Switch trays fully connect all 72 GPUs, with only one switch hop between any pair. Aggregate rack-wide NVLink bandwidth is 130 TB/s.⁵
• The rack backplane integrates more than 5,000 copper cables. Why copper instead of optical fiber? At these short distances, copper does not require power-hungry optoelectronic conversion modules such as DSPs and lasers. Industry analysis estimates that this saves roughly 20 kW per rack, while also reducing cost and improving reliability.¹⁰
• The NVLink fabric supports mapped peer access, so GPUs can load from and store to one another’s HBM. Separately, NVLink-C2C provides coherent CPU–GPU access within each Grace Blackwell Superchip. Across a GB200 NVL72 rack, the aggregate capacity is 13.4 TB of HBM3e and 17 TB of LPDDR5X. This is not one flat, uniform-latency memory pool; visibility and placement still depend on the software and partitioning model.⁴

NVLink scale-up: NVSwitch fully connects 72 GPUs within a rack—1.8 TB/s per GPU and 130 TB/s across the rack—making them function like one giant GPU.

In one sentence: within a rack, GPU peer traffic does not need the external scale-out fabric, because the dedicated NVLink scale-up network makes the GPUs behave like different regions of one giant accelerator. The next challenge begins outside the rack. That is where RDMA enters the picture.

5. Layer Two—The RDMA Scale-Out Network: The AI Factory’s Highway System

When a model is too large for even one 72-GPU rack, tens or hundreds of NVL72 racks must be connected. This high-speed inter-rack GPU highway has several important design features.

5.1 One Scale-Out Endpoint Per GPU (Logical 1:1)

In NVIDIA’s reference NVL72 compute fabric, every GPU is provisioned with a dedicated RDMA scale-out endpoint—400 Gb/s through ConnectX-7 for GB200 NVL72 and an aggregate 800 Gb/s through ConnectX-8 for GB300 NVL72. Exact device, port, and breakout packaging can vary by system, but the logical bandwidth allocation is 1:1, giving each GPU its own on-ramp. ConnectX-8 also integrates a 48-lane PCIe Gen6 switch and serves as an I/O hub for the compute tray.^{7 11}

A logical RDMA scale-out endpoint per GPU gives every GPU dedicated bandwidth—400 Gb/s on GB200 NVL72 and an aggregate 800 Gb/s on GB300 NVL72.

5.2 Rail-Optimized Topology: The Same Local GPU Position Shares a Rail

The inter-rack network uses a leaf-spine architecture with a rail-optimized topology. The same local GPU/NIC position across compute trays or nodes connects to the same rail: position 1 to rail 1, position 2 to rail 2, and so on. Collective libraries can then keep corresponding traffic within those rails, reducing cross-rail contention and unnecessary hops. The actual switch-hop count depends on deployment size: small systems may use a collapsed design, while larger clusters add spine or super-spine tiers.⁷

RDMA scale-out with a rail-optimized leaf-spine topology: the same local GPU position across compute trays or nodes maps to the same rail, reducing cross-rail traffic and unnecessary switch hops.

5.3 Two Technology Paths: InfiniBand vs. Spectrum-X Ethernet

This is the most consequential divide in AI networking today, and NVIDIA is backing both sides.

Path One—InfiniBand: A Purpose-Built Race Car for HPC

InfiniBand was designed specifically for supercomputers. RDMA is native to the architecture, and credit-based link-level flow control prevents receiver-buffer overruns during normal operation, allowing the fabric to operate losslessly without claiming that physical faults can never drop data:

• Quantum-2 (NDR): 400 Gb/s and the mainstay of the GB200 generation.
• Quantum-X800 (XDR): 800 Gb/s with 144 ports. Paired with GB300 and ConnectX-8, it is the flagship fabric for the next generation of AI factories.
• SHARP in-network computing: SHARP offloads supported aggregation and reduction stages into the fabric, reducing repeated endpoint traffic and GPU or CPU involvement; it does not move arbitrary computation into the switches or imply that one switch completes every AllReduce operation. Quantum-X800’s SHARP v4 provides up to 14.4 TFLOPS of in-network compute.¹²
• Very low fabric latency: switch port-to-port forwarding is designed for ultralow latency, although end-to-end application latency still depends on the adapters, topology, message size, and software stack. Supported NCCL or MPI integrations can deliver the benefit without major application changes, but the fabric and communication libraries still require correct configuration.

The tradeoff is that although InfiniBand is an open IBTA standard, today’s high-end commercial switch, adapter, and software supply is highly concentrated around NVIDIA. Deployments also tend to require specialized operational expertise; cost depends on topology, optics, support, and purchase scale.

Path Two—Spectrum-X: Turning Ethernet Into a Racetrack

Ethernet is standards-based, and its tools and operational skills are widely available; actual cost still depends on the design. But conventional Ethernet running RDMA through RoCEv2 (RDMA over Converged Ethernet) can encounter severe problems in large-scale AI deployments. AI traffic consists of a small number of extremely large elephant flows. Traditional ECMP routing can place several of them on the same path and cause severe congestion. In a specific set of NVIDIA-published comparisons, some conventional ECMP configurations delivered about 60% effective throughput; the result varies with topology, workload, and tuning.¹³

For the GB200/GB300-era data path, NVIDIA’s Spectrum-X platform pairs Spectrum-4/SN5600 switches with compatible BlueField-3 or ConnectX SuperNICs. Three core mechanisms are:

1. Adaptive routing and packet spraying: based on real-time congestion, the switch selects the least congested path for every packet, distributing an elephant flow across all available paths.
2. Receiver-side reordering and Direct Data Placement (DDP): packets that traverse different paths may arrive out of order. Hardware in the destination SuperNIC places payloads at their correct destination offsets in memory while preserving in-order delivery semantics for the application.
3. Hardware congestion control: the switch reports network conditions in real time through in-band telemetry. The NIC adjusts its sending rate through a hardware-speed feedback loop, helping prevent sustained congestion and improve predictability and isolation between tenants.

According to NVIDIA’s published results, Spectrum-X raises effective Ethernet throughput to roughly 95% while retaining familiar Ethernet tools and operational skills.¹³

Two RDMA paths: InfiniBand combines credit-based flow control with SHARP in-network computing, while Spectrum-X uses adaptive routing, packet spraying, and direct data placement to raise effective Ethernet throughput to roughly 95%.

Which Path Should You Choose?

The industry is currently pursuing both paths. CoreWeave’s GB300 instances, for example, are available with either Quantum-X800 InfiniBand or Spectrum-X RoCE. Oracle Cloud Infrastructure supports both RDMA networks for GB200 and GB300 as well.^{14 15}

6. GB200 to GB300: What Exactly Changed in the Network?

According to SemiAnalysis’s account of the GB200 architecture, next-generation networking silicon with 224G SerDes was not ready for the initial launch, so early systems reused the ConnectX-7 and Quantum-2 generation from H100 deployments. On that reading, end-to-end 800G networking became more complete with GB300.^{10 11} The broader point is that network-generation transitions are every bit as important as GPU transitions and can affect delivery schedules.

7. Real-World Examples

xAI Colossus: A 100,000-GPU Bet on Ethernet

Elon Musk’s xAI built the Colossus supercomputer in Memphis, connecting 100,000 NVIDIA Hopper GPUs with Spectrum-X Ethernet. When NVIDIA published its case study, expansion toward 200,000 GPUs was still under way. Only 122 days elapsed from the start of construction to the beginning of training. NVIDIA reports that the cluster sustained 95% effective data throughput with no packet loss caused by flow collisions. These vendor-reported results demonstrate the Ethernet path’s viability at enormous scale and make Colossus one of the clearest examples of the “AI factory” concept.¹⁶

What the Cloud Giants Chose

Oracle Cloud Infrastructure. OCI developed a dedicated API for GB200 and GB300 NVL72 that provisions an entire rack as “one supercomputer,” automatically optimizing NVLink plus either InfiniBand or Spectrum-X RoCE.¹⁵

CoreWeave. GB300 is available in both InfiniBand and Spectrum-X configurations, each providing 800 Gb/s per GPU.¹⁴

Meta and Oracle have announced that their AI data center networks will use Spectrum-X switches. Microsoft, CoreWeave, and OCI have also begun deploying GB300 NVL72 systems.¹³

Google Cloud. Google took a different route from Spectrum-X, integrating NVIDIA NICs into its own stack. Beginning with A3 Ultra (H200), Google moved to Titanium ML adapters with standard RoCE, replacing its earlier custom GPUDirect-TCPX approach.^{17 18} A4X (GB200 NVL72) provides 28.8 Tb/s per rack, while A4X Max (GB300 NVL72) doubles per-GPU bandwidth to 800 Gb/s. Google’s in-house Jupiter fabric connects tens of thousands of GPUs into a nonblocking cluster.

Google also opened Falcon’s design to ecosystem collaboration through the OCP; Falcon is a hardware-assisted multiprotocol transport rather than necessarily an open-source code release. TPU accelerator pods use Google’s own ICI plus OCS optical circuit switching rather than Ethernet or InfiniBand. A single Ironwood Superpod directly connects 9,216 chips. The TPU v4 paper reports lower cost and power, and higher performance, than a comparable contemporary InfiniBand design.^{19 20 21 22 23}

AWS. Amazon developed both its network adapter and transport protocol in-house. EFA (implemented in Nitro silicon) plus SRD sprays packets independently across as many as 64 paths, tolerates out-of-order arrival, and retransmits within microseconds. This resembles Spectrum-X’s adaptive-routing and direct-data-placement approach, but entered production in 2019. At the fabric layer, AWS uses its own 10p10u network, delivering tens of petabits per second and round-trip latency below 10 microseconds.^{24 25 26}

P6e-GB200 UltraServers still use NVL72 within a rack and EFAv4 between racks, providing 28.8 Tb/s per rack plus GPUDirect RDMA. Even NVIDIA’s own Project Ceiba—with 20,736 B200 GPUs—and Project Rainier, built for Anthropic with nearly 500,000 Trainium2 accelerators, use EFA rather than InfiniBand; Rainier uses NeuronLink within each 64-chip Trainium2 UltraServer and EFA between UltraServers.^{27 28 29}

Comparing Google Cloud and AWS with the InfiniBand and Spectrum-X approaches above reveals a striking convergence in engineering techniques, not one universal protocol. At the largest scales, these designs draw from the same toolkit: multipath packet distribution, tolerance or correction for out-of-order delivery, and hardware-offloaded congestion control. RoCE, Falcon, EFA/SRD, and TPU ICI expose different transports and semantics; the strategic choice is whether to buy the highway as an integrated platform or build more of it in-house.

8. The Next Step: When One Data Center Is Not Enough

As AI clusters grow, site power and capacity are becoming major scaling constraints. After scale-up within a rack and scale-out across racks, NVIDIA’s answer is a third dimension: scale-across between data centers.

• Spectrum-XGS Ethernet, announced in 2025, uses distance-adaptive congestion control to combine data centers at multiple locations into one “giga-scale AI superfactory.” NVIDIA reports roughly 1.9× the NCCL collective-communication performance across data centers, and identifies CoreWeave as an early adopter.³⁰
• Silicon photonics and co-packaged optics (CPO): optical engines are co-packaged with the switch ASIC rather than fabricated inside the switch die itself. As of August 2026, NVIDIA says Spectrum-X Ethernet Photonics is in production; the SN6800 is specified at up to 409.6 Tb/s. Shipment and deployment status still varies by system, partner, and region.^{31 32}
• The Rubin generation is arriving: the Vera Rubin NVL72 design provides 1.6 Tb/s of ConnectX-9 scale-out bandwidth per GPU, while NVLink 6 provides 3.6 TB/s per GPU and 260 TB/s across the rack. NVIDIA says the platform is ramping into full production, with production shipments scheduled to begin in fall 2026.³²

9. Conclusion

To recap, the AI race stopped being only a chip race long ago; it is now a networking race too. When a workload no longer fits on one GPU, one rack, or even in one data center, the network’s ability to coordinate tens of thousands of GPUs efficiently directly determines training speed and cost. On its configured steady-state data path, RDMA bypasses the operating-system kernel and avoids payload copies through kernel buffers while moving data directly between registered memory regions; GPUDirect RDMA extends that path to GPU HBM. Together, these technologies cut latency into the microsecond range and sharply reduce CPU overhead.

In GB200 and GB300 systems, these capabilities take the form of a three-layer network architecture. NVLink welds 72 GPUs within a rack into one giant GPU. Between racks, InfiniBand or Spectrum-X Ethernet—both carrying RDMA—connects tens or hundreds of racks into an AI factory. Spectrum-XGS extends the design across data centers. The next time you see news about an AI supercomputer, look beyond its GPU compute numbers to the invisible highway beneath them. That foundation determines how fast the AI factory can actually run.

Glossary

References