FAQ

InfiniBand Frequently Asked Questions

Mellanox Technologies Inc. Document Number 2078GI

2900 Stender Way, Santa Clara, CA 95054

Tel: 408-970-3400

Fax: 408-970-3403

www.mellanox.com

1 Rev 1.10

Mellanox Technologies Inc

InfiniBand Frequently Asked Questions

Frequently Asked Questions

Q: Is InfiniBand designed strictly as an I/O fabric? A: No, InfiniBand offers much more. At the lowest layer, InfiniBand provides a high performance, low latency, reliable switch fabric to serve as a scalable I/O interconnect. However, InfiniBand offers higher layers of functionality that enable applications such as application clustering, fast IPC (Inter Process Communication), and SANs (Storage Area Networks). Q: What makes an InfiniBand I/O fabric? A: An InfiniBand fabric is built up from HCAs, TCAs, and Switches. An HCA is a Host Channel Adapter which provides the mechanism to connect InfiniBand devices to pro-cessors and memory. In addition, an HCA provides a common set of required features to which software may interface through a defined verb layer. A TCA is a Target Channel Adapter and is an endpoint of an InfiniBand fabric. The TCA typically pro-vides additional I/O functionality. A switch is used to connect several InfiniBand chan-nel adapters (HCA or TCA), and also to switch devices together, sometimes redundantly. Switch is the mechanism used to create an InfiniBand fabric. Q: Does InfiniBand support RAS? A: Yes. RAS stands for Reliability, Availability, and Serviceability. InfiniBand supports reliability with guaranteed reliable services which deliver in-order packet with multiple CRCs (Cyclical Redundancy Checks) to detect errors. InfiniBand supports availability as it enables redundancy and supports failover by switching to an alternative path, should a link or device fail. InfiniBand supports serviceability through hot swappability and special management functions, which can be either in-band and out-of-band. Q: What basic bandwidths are supported by InfiniBand? A: InfiniBand supports three links (1X, 4X, and 12X) which are multiples of the basic 2.5Gb/s rate. Thus, the rates supported are 2.5Gb/s, 10Gb/s, and 30Gb/s. InfiniBand is actually a full duplex bidirectional. Q: What is the actual data rate of an InfiniBand 1X link? A: The actual data rate of InfiniBand 1X link is 2.0Gb/s. The basic InfiniBand rate is 2.5e9 transfers per second. The raw transfer rate of 2.5Gb/s actually means 2.5G transfers per second where the transfer is a 10 bit symbol. InfiniBand uses the 802.3.z 8B/10B coding to enable clock recovery and avoid a DC bias. Thus, the actual bit rate is 80% of the line rate or 2.0e9 bits per second. In addition, there is some protocol overhead such that the actual payload data rate is somewhat less. Q: What is the InfiniBand physical layer? A: The physical layer is based on the IEEE 802.3.z specification operating at a speed of 2.5Gb/s. This is the same standard used by the 10G Ethernet (3.125Gb/s) and Fibre Channel physical layers (with the exception of the actual bit rate). Q: What is the interface between the physical layer device and the digital device?
Mellanox Technologies Inc 2 Rev 1.10

InfiniBand Frequently Asked Questions

A: The interface between an ASIC and the Serializer-Deserializer (SerDes) is not speci-fied in the InfiniBand standard, however, the de facto standard is defined by the Hari interface. This interface is identical to that used by 10Gb/s Ethernet and Fibre channel. The interface uses a 10 bit, source synchronous, Double Data Rate (DDR) connection with SSTL2 electrical signaling. Q: What is a Virtual Lane (VL)? A: InfiniBand offers link layer Virtual Lanes (VLs) to support multiple logical channels on the same physical link. VLs provide a mechanism to avoid Head of Line (HoL) block-ing and the ability to support Quality of Service (QoS). Q: What is the difference between a Virtual Lane and a Service Level (SL)? A: Virtual Lane is the actual logical lane used on a given point-to-point link. The Service Level stays constant as a packet traverses the fabric, and specifies the desired service level within a subnet. The SL is included in the link header, and each switch maps the SL to a VL supported by the destination link. A switch supporting a limited number of virtual lanes will map the SL field to a VL it supports. Without preserving the SL, the desired SL would be lost in this mapping, and later in the path, a switch supporting more VLs would be unable to recover finer granularity of SLs between two packets mapped to the same VL. Q: Is InfiniBand a layered protocol adopting the OSI 7 layer model? A: Yes. The InfiniBand specification defines the protocol in modular layers loosely based on the OSI model. The specification defines interfaces between a given layer and the one immediately above and below. Thus the lowest physical layer interfaces only to the link layer above. The InfiniBand link layer defines an interface to the physical layer below and another interface to the network layer above. Q: Is InfiniBand a replacement for TCP/IP? A: No. InfiniBand is designed to operate in different environments than the internet proto-col, and complements TCP/IP protocols. In fact, InfiniBand uses IPV6 at the network layer to support inter subnet routing. Q: How does the InfiniBand link layer protocol differ from other link layer proto-cols? A: The InfiniBand link layer protocol is more powerful than typical link layer protocols such as Ethernet or ATM. The InfiniBand link layer offers features such as credit based flow control and VLs, which allow simplification of some of the higher level protocols. For example, the InfiniBand link layer protocol guarantees strong ordering within a vir-tual lane along a given path. This simplifies the higher level transport protocol. Q: How does InfiniBand link layer flow control differ from other link layer flow con-trol protocols? A: The InfiniBand link layer uses an absolute credit based flow control to avoid conges-tion, which is more powerful than an XON/XOFF or CSMA/CD protocol (used by Ethernet). Q: How does the InfiniBand transport service differ from other Internet transport services such as TCP?
3 Rev 1.10

Mellanox Technologies Inc

InfiniBand Frequently Asked Questions

A: The InfiniBand transport offers a richer set of services than TCP, but by leveraging a more powerful link layer protocol is, nonetheless, a lighter weight protocol. InfiniBand transport offers connection and connectionless (datagram) operation, both of which can be reliable or unreliable. Thus, InfiniBand transport can provide a reliable service like TCP or an unreliable datagram service such as provided by UDP. Q: In what way is InfiniBand transport a lighter weight protocol than TCP? A: InfiniBand transport does not need to re-order packets as the lower level link layer pro-vides in order packet delivery. The transport layer is only required to check the packet sequence and deliver packets in order. Further, because InfiniBand offers credit based flow control, the transport layer does not require a drop packet mechanism like the TCP windowing algorithm to calculate the path delay bandwidth product to determine the optimal number of in flight packets. Q: Is the InfiniBand Virtual Interface Architecture (VI) faster than TCP for commu-nication? A: Yes. InfiniBand provides hardware to support the Virtual Interface Architecture. Infini-Band is a channel based architecture and implements in hardware many of the func-tions which TCP requires be done in software. The InfiniBand channel architecture supports data de-multiplexing in hardware, and thus significantly less software is required to enable process to process communication. Further, because the InfiniBand channel architecture supports memory protection, user-space runtime code may access remote memory directly without requiring protected kernel services. This eliminates the high software overhead associated with the transition from user to kernel mode and back. Q: How does InfiniBand layer 2 switching differ from Ethernet layer 2 switching? A: InfiniBand layer 2 switching is based on a 16 bit local address (LID), and thus limits the number of nodes to 2^16 on a subnet. Further, unlike a 48-bit Ethernet MAC address (which is fixed by the vendor at manufacturing time) an InfiniBand LID address is assigned to each device by the fabric manager, and thus can be guaranteed to be contiguous and sequential within a given subnet. Q: Does InfiniBand support congestion management? A: Yes. InfiniBand supports static rate control, which enables a fabric manager to config-ure the network to avoid oversubscribed links in the fabric. For example, a path with a 4X link feeding a 1X link can be constrained to a 1X rate. The 1.0 version of Infini-Band does not address dynamic rate control for congestion management, however it will be addressed in post 1.0 versions. Nonetheless, though it is not explicitly addressed by the spec, it is possible to build devices compliant with the 1.0 specification which support dynamic congestion management. Q: Does InfiniBand replace Fibre Channel for Storage Area Networks (SAN)? A: Over time, InfiniBand will supersede Fibre Channel in Storage Area Network applica-tions. InfiniBand is architected to support the requirements of Storage Area Networks and complements existing FC Storage Area Networks. Native InfiniBand-enabled serv-ers provide optimized service connections to Storage Area Networks without the requirement for an expensive HBA (Host Bus Adapter) card. An InfiniBand based SAN may support a super-set of features
Mellanox Technologies Inc 4 Rev 1.10

InfiniBand Frequently Asked Questions

offered by legacy FC without compromising the operation of the network. During the transition period servers, routers, or hybrid switches can facilitate communication between InfiniBand and FC SANs.

Mellanox Technologies Inc

5 Rev 1.10