RoCE vs InfiniBand: Quick Comparison
- RoCE (RDMA over Converged Ethernet): RDMA over commodity Ethernet infrastructure trade-off. More flexible and less expensive than IB, but requires careful configuration of PFC/ECN to avoid congestion-induced packet loss.
- InfiniBand: IB is a proprietary RDMA network with built-in flow control and congestion management capabilities. Lower latency and better behaved for large AI/HPC clusters, but more expensive and less flexible (NVIDIA/Mellanox ecosystem domination).
Introduction
As AI training clusters, High Performance Computing (HPC), and large-scale data centers push the limits of network infrastructure to the max, the choice of interconnect technologies becomes critically important. There are two interconnect technologies worth considering, namely RoCE (RDMA over Converged Ethernet) and InfiniBand, both of which provide high-performance networking based on Remote Direct Memory Access (RDMA). However, there are also significant differences between these two, and one has to make a choice about which one is better for their application, be it GPU clusters, High Performance Computing (HPC), or other scalable systems. This blog discusses the differences between RoCE and InfiniBand.
What is InfiniBand
Infiniband is a proprietary network connectivity technology made for Infiniband’s specialized network switching and protocol stacks and is not based on standard Ethernet technologies.
Key features of InfiniBand
Some of the key features of Infiniband:
- Native implementation of Remote Direct Memory Access (RDMA) (high-performance data transfer).
- Very low-latency and predictable latency.
- Credit-based flow control (no packet loss).
- Subnet manager for configuration purposes.
- Used extensively in supercomputers and large-scale AI training clusters.
As it was stated, InfiniBand is optimized for high-performance computing (HPC) and provides consistently excellent results. In addition, it often requires specialized switches, adapters, and cabling from a limited set of vendors.
What is RoCE?
RoCE (RDMA over Converged Ethernet) is a networking technology that provides RDMA (Remote Direct Memory Access) over Ethernet. RoCE version 1 specifies the use of RDMA on the link layer, while RoCE version 2 runs over UDP/IP and thus is routable at the Internet Protocol (IP) level. RoCE version 2 is the version that is now in wide-spread adoption in the industry, with RoCEv1 being rarely deployed.
Key features of RoCE
Some of the key features of RoCE:
- Ability to run on readily available Ethernet infrastructure.
- Requirement to run with Priority Flow Control and Explicit Congestion Notification to ensure low packet loss rates.
- Use of standard Ethernet infrastructure, tools, expertise and ecosystem.
- Adoption in hyperscale data centers and cloud computing platforms.
The fact that RoCE is based on Ethernet enables the use of readily available infrastructure, tools, expertise, and ecosystem that are well-established in the industry. This makes RoCE a cost-effective choice compared to using InfiniBand interconnects.
RoCE vs InfiniBand: Head to head comparison
Here are the key differences between RoCE and InfiniBand:
| Factor | InfiniBand | RoCE |
| Latency | High consistency | Higher and network based |
| Infrastructure | Specialized, proprietary | Standard ethernet |
| Cost | Higher hardware cost | Low, uses current ethernet gear |
| Scalability | Great within IB fabric limits | Excellent, uses Ethernet’s scale |
| Ecosystem | Narrow vendor base | Broad Ethernet vendor ecosystem |
| Ease of Management | Centralized subnet managed | Familiar ethernet tools |
Performance Characteristics
InfiniBand typically provides lower latency and jitter, due to its congestion control and flow control mechanisms, which are critical to prevent packet losses at the hardware level. For these reasons, it is the preferred interconnect for high-performance computing (HPC) clusters that host demanding applications requiring consistent performance at the level of microseconds.
RoCE, particularly its version 2, has been significantly improved and now provides similar performance characteristics in terms of throughput, jitter, and packet loss. As such, for AI applications and other similar purposes, RoCE can be considered virtually equivalent to InfiniBand but at a lower overall cost.
Cost and Complexity
The RoCE’s ability to run on the industry-standard Ethernet hardware is its significant advantage over InfiniBand. It implies that it can be deployed using readily available switches and other infrastructure components, making this technology more attractive in terms of cost-efficiency and ease of management. Moreover, deploying RoCE typically requires fewer specialized IT professionals to manage the network platform.
In contrast, while InfiniBand has superior performance characteristics, it must be supported by specialized and significantly more expensive switches and channel adapters. Additionally, the pool of qualified network architects and administrators who can deploy and manage InfiniBand is considerably smaller than those for Ethernet-based solutions.
Which Should You Choose?
Here are the reasons why you should choose between InfiniBand or RoCE:
Choose InfiniBand
If you need the lowest possible latency, or you are setting up an enterprise-level High Performance Computing (HPC) or AI training cluster. You can invest heavily in the most powerful equipment to get a specialized system.
Choose RoCE
If you want to use standard Ethernet switches to minimize costs and have the option to patch performance issues in software if needed. This solution is great for cloud service providers and enterprises that want to build AI clusters without designing a custom network from the ground up.
Many large-scale AI infrastructure providers are expected to adopt RoCE over InfiniBand as the performance of standard switches and Ethernet cables improves.
Conclusion
The choice between RoCE and InfiniBand depends on priorities. The former protocol offers better value in terms of price-to-performance, and it is more flexible in terms of ecosystem. InfiniBand, on its part, provides much better performance, so it is best suited for high-performance computing and AI applications that require the lowest possible latency.
FAQs
Is RoCE as good as InfiniBand?
RoCEv2 provides similar performance to what InfiniBand offers when properly configured with PFC and ECN, but the latency of InfiniBand is generally lower and more predictable.
Does RoCE require any specific hardware?
RoCE can run on standard Ethernet hardware but requires support for RDMA, PFC, and ECN.
Why do AI training clusters use InfiniBand?
The latency of InfiniBand is lower and more predictable, and it has advanced congestion control capabilities, which are critical for distributed AI training.
Is RoCE cheaper than InfiniBand?
Yes, because it can run on standard Ethernet switches and NICs.
Can one use both RoCE and InfiniBand in a data center?
Some companies have adopted the approach of using RoCE-based Ethernet for general-purpose data center networks while using InfiniBand for AI training clusters.
