Ethernet vs InfiniBand for AI

Introduction

Training a large language model is not only a compute intensive task, but a networking intensive task. When you distribute the model across hundreds or thousands of GPUs, the GPUs spend a lot of time communicating with each other, for example, to reduce gradients or to perform all-reduce operations. This makes the quality of the interconnect critically important, as a slow or congested interconnect will cause the GPUs to wait for each other, causing wasted compute time and increased training time. This is why the choice of interconnect,  Ethernet or InfiniBand can be a vital decision during AI infrastructure design.

InfiniBand

InfiniBand is a proprietary high-speed low-latency networking communications technology. Initially designed for high-performance computing (HPC), it is now used as an interconnect in large AI training clusters. The technology is purpose-built to reduce the overhead of communication between compute nodes and has been adopted as a standard for GPU-based distributed AI training.

Key Features of InfiniBand

Some of the features of Infiniband are:

• RDMA (Remote Direct Memory Access): enables reading and writing of memory on one computer by another over a network without involving the CPU or Operating System kernel, thus reducing overhead. This is extremely beneficial for reducing communication overhead in distributed systems.
• Lossless, Credit-Based Flow Control: Uses a credits-based system at a hardware level to prevent messages from being dropped due to congestion (a sender cannot send more messages than what can be consumed by a receiver).
• Low Latency: Latency is routinely achieved within the range of 1-5 microseconds per hop between switches. Much lower than anything achieved by traditional Ethernet.
• High Bandwidth: Current and future generations (NDR: 400 Gb/s today with XDR to 800 Gb/s) will support bandwidth-hungry applications such as fastest GPU interconnects.
• In-Network Computing (SHARP): NVIDIA has developed software-defined switches that offload computation-heavy tasks such as reductions (summing) of distributed data during collective operations. This enables much faster execution of distributed operations that would otherwise have to be done on the endpoints.
• Collective Operations Support: Accelerates collective operations used in distributed training (all-reduce, all-gather, broadcast, etc.) through close integration with communication libraries such as NCCL.
• Subnet Manager Architecture: Improves management of large fat-tree/dragonfly networks by providing a single point of configuration.

Ethernet

Ethernet is the ubiquitous, general-purpose networking standard that underlies most home and enterprise routers, as well as public and private cloud data centers for the largest hyperscalers. Ethernet used to be seen as too lossy and latency-inducing for tightly coupled AI training, but recent advances in RoCE (RDMA over Converged Ethernet) have made it a viable, if not preferred, option for large AI clusters.

Key Features of Ethernet

Some of the features of Ethernet are:

• RoCE: It allows to perform RDMA operations over converged Ethernet network thus avoiding significant overhead imposed by traditional Ethernet software stack which is not suited for AI/ML applications.
• Priority Flow Control: The standard allows to stop transmission on a particular flow to avoid buffer overflow thus making Ethernet lossless as InfiniBand.
• ECN: Explicit congestion notification allows to notify the sender about impending packet drops thus avoiding congestion without dropping packets.
• High bandwidth and availability: Ethernet switches with 400 Gbit/s and 800 Gbit/s are already available on the market and provide the same level of bandwidth as InfiniBand.
• Multi-vendor: Broadcom, Cisco, Arista, Juniper, NVIDIA (Spectrum switches), and many other vendors provide switches and other infrastructure equipment for Ethernet networks. In contrast, InfiniBand is available from a very limited set of vendors.
• Convergence and development: Ultra Ethernet Consortium which is formed by Meta, Microsoft, AMD, Intel, Broadcom, NVIDIA, and many other vendors, is working on standardization of enhanced Ethernet specifications for AI applications. These specifications include enhanced congestion control, multi-path TCP/IP, improved collective communications, and more.
• Similar operational experience: It is based on common networking standards which are already familiar to network engineers.

Key differences between Ethernet vs InfiniBand

Here are the key differences between ethernet and Infiniband.

Conclusion

Neither of them are a clear winner. InfiniBand still offers the most reliable low-latency and lossless options for large-scale tightly coupled training workloads, but it has steep costs both in terms of required hardware and the associated expertise. Ethernet is quickly closing the gap, is far easier on the wallet and offers more flexibility in terms of vendors, and is often a more practical choice for inference, smaller scale training, or if you wish to avoid being tied to a specific vendor.

A similar trend is seen across the industry; networks are no longer an afterthought to be added on to the cluster, and there is a strong push towards tighter integration with the hardware stack. The network choice is becoming more and more important to evaluate for all types of machine learning workloads, but the majority of the challenges and opportunities lie underneath, where the transport layer algorithms, congestion control, topologies and software optimizations come together.

FAQs

Is one protocol generally faster, particularly in terms of latency?

InfiniBand provides lower latency out-of-the-box, with typical port-to-port latency being under a microsecond. Ethernet is rapidly catching up due to RoCE, but generally, InfiniBand wins in latency-critical applications.

Can Ethernet support RDMA, or is that a proprietary InfiniBand thing?

Yes, Ethernet can support RDMA, via RoCE, but it’s not as well supported, or as performant, requiring significant tuning at scale. But it’s definitely possible.

Are large organizations actually using Ethernet for AI training?

Yes, absolutely. Meta and Microsoft have both built large-scale GPU training clusters using Ethernet-based networking, capable of hosting tens of thousands of GPUs.

Is InfiniBand generally considered better for large-scale training jobs?

Yes, for now, because of its excellent properties for large-scale synchronous communication (all-reduce), making it a favorite for the heaviest lifting jobs. But it’s not an ironclad rule.

What’s the ultra Ethernet consortium, and what are they working on?

It’s a grouping of industry players (AMD, Broadcom, Meta, Microsoft, Intel, etc.) that are working on extending Ethernet’s capabilities for AI/HPC, particularly in terms of collective communication, by developing new flow control, multi-pathing and other optimizations.

Not that it matters much, but does Ethernet offer similar performance benefits for inference as InfiniBand?

For inference, it’s much harder to get significant performance gains from either Ethernet or InfiniBand, since inference is less latency-critical and often uses asynchronous operations. So in this domain, both options are roughly comparable.

What’s SHARP, and is there an Ethernet equivalent?

SHARP is the InfiniBand equivalent of RoCE, allowing for in-network aggregation of messages (e.g. summing gradients) without needing to involve the CPU.