NVLink vs InfiniBand vs PCIe

Introduction

As the number of parameters of AI models grows exponentially and reaches hundreds of billions, the performance characteristics change qualitatively. The critical factor in model training and inference becomes not the amount of computing power, but the speed of data transmission between graphic processing units (GPUs), servers, and storage. The three technologies most popular in this area are NVLink, InfiniBand, and PCIe. Although all three are used in the same industry and often in the same companies, they are completely different technologies that solve different problems. While NVLink focuses on achieving maximum performance within a single server at the level of GPU modules, InfiniBand is designed for large-scale data center communications between servers. In turn, PCIe is a general-purpose standard that connects almost all computer devices internally. The differences and similarities of these three technologies are discussed below.

What Is PCIe?

PCIe (Peripheral Component Interconnect Express) is the standard universal bus, connecting GPUs, NICs, storage devices, and other computer equipment to the server’s CPU. It is used everywhere, is vendor-neutral, and affordable, but is not optimal for large-scale GPU interconnects typical for AI training. The latest PCIe generation’s bandwidth is significantly lower than the one provided by dedicated GPU interconnects, and the traffic between GPUs not connected directly to each other should pass through the CPU, adding to the latency. However, PCIe buses are flexible in terms of the connected hardware, consume fewer power per GPU than other options, and can be used in smaller AI inference configurations.

What Is NVLink?

NVLink is a proprietary point-to-point interconnect for linking GPUs (and CPUs) within a single chassis, which bypasses the PCIe bus architecture. It provides much higher bandwidth than PCIe by directly connecting all GPUs in a scale-out configuration with each other through an NVSwitch, which enables coherent memory access. This allows for large-scale distributed model training, as activation and gradient records can flow directly between accelerators. Bandwidth per generation is increasing substantially; for instance, NVIDIA’s SXM cards in Blackwell-based systems provide over ten times the bandwidth of PCIe. However, this comes at the cost of increased power consumption and a reduced ecosystem of compatible hardware – NVLink is only available in NVIDIA’s SXM-based server accelerators and high-end cloud GPUs like DGX/HGX.

What Is InfiniBand?

InfiniBand is an open standard, switched-fabric, point-to-point interconnect communications protocol initially developed to connect entire servers, storage, and clusters together. It has the ability to address tens of thousands of devices simultaneously by providing remote direct memory access (RDMA) capabilities. This technology is characterized by very high bandwidth, extremely low latency, and direct access to memory over high-speed networks and is implemented in a switched-fabric topology. These features have made InfiniBand the dominant technology for large-scale distributed AI training as well as the backbone of many supercomputer clusters for years. The latest generation of InfiniBand (NDR) provides hundreds of gigabytes per second of bandwidth per port, with future generations promising to deliver even more. In addition, unlike NVLink, InfiniBand is not proprietary to one particular GPU manufacturer and provides much more flexibility for end-users in terms of available configurations and cost-performance efficiency.

Key Differences by Technologies

Here are key characteristics of different technologies:

NVLink Characteristics

• In-node only: connects GPUs (and occasionally CPUs) within one server/chassis.
• Bandwidth: Highest of any interconnect by a significant margin, with modern SXM based systems often exceeding one terabyte per second per GPU, compared to PCIe.
• Latency : Submicrosecond range, with zero-copy capabilities in certain configurations.
• Topology : Utilizes NVSwitches which allows for full bisection bandwidth for all GPUs connected to the switch, rather than having to share a single bus.
• Standardization : Proprietary interconnect, only available in NVIDIA SXM based GPUs and switches.
• Power/Cost: Much higher power consumption for each GPU (often in the 400-700 Watt range), and only available as part of NVIDIA’s own DGX/HGX systems, which can be considerably expensive.
• Application: Ideally suited for large language models, and other applications that require extensive in-node (i.e., inside a single computer/server) communications between GPUs.

InfiniBand Characteristics

• Interconnect scope: End-hosts within a coherent fabric, supporting many thousands of nodes – servers, storage, or both, at the level of the enterprise data center.
• Bandwidth: Very high, compared to other networking standards; currently available NDR (next-generation interconnect) hardware provides hundreds of gigabits per second of bandwidth per port, with further forthcoming generations expected to provide higher performance.
• Latency: Extremely low due to the switched-fabric point-to-point topology and the ability to perform remote direct memory access (RDMA) at the hardware level, allowing for data transfers between nodes without involving the CPU.
• Topology: Fat tree, mesh, torus, and other designs can be implemented using InfiniBand’s subnet managers and switches. The use of subnets allows for easy horizontal scaling.
• Standards: An open industry standard, available across multiple vendors, not tied to a particular ecosystem.
• Cost: Lower cost and higher level of configurability due to the aforementioned competitive multi-vendor market.
• Typical use cases: Large-scale supercomputer clusters, multi-node AI training fabrics, high-performance distributed storage systems, and large-scale distributed high-performance computing (HPC) workloads.

PCIe Characteristics

• Scope: General-purpose, internal bus, connecting GPUs, network adapters (including InfiniBand adapters), storage, and other devices to the CPU. It is the primary interconnect in most computers/servers.
• Bandwidth: The narrowest of the three regarding bandwidth; PCIe’s latest generation provides significantly lower data transfer rates than NVLink.
• Latency: Generally higher than NVLink or InfiniBand, especially when connecting two GPUs through a CPU, which may cause substantial delays.
• Topology: Uses a shared bus topology. GPUs are not directly connected but rather linked through the CPU, which may limit performance.
• Standards: An industry standard, available in most computer hardware, not tied to any particular manufacturer.
• Cost-performance: The lowest cost and power consumption. Does not require specialized hardware for installation, and each GPU connected via PCIe tends to consume less electricity (about 300W).
• Preferred for: Less demanding tasks such as training, inference, or virtualization; or when there is a need to reduce costs and complexity.

Choosing the Right Fit

While in practice many large AI data centers are unlikely to adopt a strictly single-architecture approach, the choice is usually between using NVLink within individual servers (to achieve maximum GPU-to-GPU bandwidth) and InfiniBand interconnection between them (for global scaling), with PCIe providing the connection to attached peripherals such as storage and networking devices within each server. By contrast, smaller organizations that are focused on inference or that only perform relatively small-scale training are likely to find that standard PCIe-based GPU servers offer a much better value proposition, since the wide GPU interconnects offered by NVLink or InfiniBand are of much less value in a loosely-coupled system.

Conclusion

NVLink is not a competitor to InfiniBand, but another level in one of the high-performance interconnects for GPUs. First of all, PCIe is the go-to solution for low-cost and universal GPU connection, while NVLink ensures higher performance within one server, and InfiniBand extends the power of NVIDIA GPUs beyond one computer. The choice of the interconnect technology and the level of GPU integration are based on specific requirements and expectations from the team of scientists or engineers. To maximize performance, cost, and power efficiency, it is important to understand the capabilities of the technologies listed above.

FAQs

Is NVLink faster than Infiniband?

Yes, for intra-server communications, NVLink provides significantly higher bandwidth than Infiniband, enabling much higher per GPU throughput. This said, Inifiniband is intended for another purpose – to connect multiple servers, while NVLink can be used only in a single server.

Can we combine NVLink and InfiniBand?

Yes, and this is the most common scenario for large-scale AI applications. Within each server, NVLink is used to achieve maximum bandwidth between GPUs, and InfiniBand is used to connect servers in a cluster.

Why is NVLink so expensive?

NVLink requires specialized server platforms (DGX/HGX) and SXM modules. Compared to PCIe-based GPU cards, these technologies add significant cost and complexity.

What infrastructure technology should we pick for our small AI lab/start-up?

For smaller-scale applications with limited GPU resources (e.g., less than four accelerators), PCIe-based servers are typically more cost-effective. As the organization grows, larger-scale AI applications (i.e., spanning multiple servers) will benefit from the InfiniBand fabric connected with NVLink switches.