InfiniBand Architecture:

Illuminata Bridge Over Troubled Waters

That is a bad bridge which is shorter than the stream.

German proverb
Research Note The PCI peripheral expansion bus has had a long and illustrious history. Since its
inception in 1991, system vendors and users have embraced it like few technical stan-
dards before or since. PCI provides a substantial volume of the I/O bandwidth and
peripheral connectivity across the range of RISC to CISC; PC to enterprise server;
proprietary to commodity. User requirements at the advent of the 21st century,
David Pendery however, have rapidly evolved. Not only has computer performance advanced enor-
Jonathan Eunice mously, the very landscape of IT use and connectivity has changed. The PCI standard
27 April 2000 we have converged and relied upon for close to a decade is being rapidly outstripped
by the demands of ever larger databases, transaction loads, and network user bases.
The bridge is beginning to look shorter than the stream.

Fortunately, help is on the way. The InfiniBand Architecture is the industrys answer
to the growing I/O problem. InfiniBand replaces the bus-based PCI with a high-band-
width (multiple gigabytes per second) switched network topology, and shifts I/O
control responsibility from processors to intelligent I/O engines commonly known as
channels. These approaches have long enabled the worlds largest servers, now
brought down to address virtually every server. InfiniBand is not yet a product, nor
even really a standard. The first full specification wont be available until this summer,
with the first products appearing in 2001. Initial indications, however, are greatly
encouraging. InfiniBand is the right technological advance, emerging at the right time
and for the right reasons. To employ a bit of adolescent patois, InfiniBand rocks.

O N!
O P E NING SO

Copyright 2000 Illuminata, Inc.

Illuminata, Inc. 187 Main Street Nashua, NH 03060
Licensed to InfiniBand Trade Ass'n 603.598.0099 603.598.0199 f www.illuminata.com
Web Use Only - Do Not Reproduce
2
Presto, Change-O
When PCI established itself in the early 1990s, 66 MHz
processors and 10 Mbps networks were fast. 0.8 micron
CMOS semiconductor fabrication was state of the art.
Early transaction processing benchmarks churned out a
whopping 54 transactions per minute.1 Data ware-
housing had just been invented. Client-server applica-
tions and deployments were increasing, but only the
digerati had email, and the Internet as we know it was
still years distant.
What a difference a decade makes! Today, multi-
terabyte databases running on clustered servers, if not
exactly commonplace, are a reality in many shops.
Storage has been decoupled from the server, and often
extended over a storage-optimized network (SAN).
Intels Pentium III Xeons, now the workhorse of
servers not just PCs, are fabbed at 0.18 micron and run
at 800 MHz; 0.13 micron, 1 GHz chips are on the way.
The top TPC-C server does 135,815 transactions per
minute, and the Internet is now the workshop of IT.
These are the new reality, driving ever-higher user
expectations. Fast, unencumbered I/O is the lifeblood
of this evolving corpus. Never has such variety of I/O
been required to link such scale of hardware and soft-
ware in such transparent and accelerated ways. And
yet, never before have the incumbent I/O technologies
been so outstripped by processor capabilities.
PCI = Problematic Computing Interface?
Introduced in response to a morass of incompatible
peripheral connectivity and I/O options of a decade
ago, PCI has been a blessing. Over time it expunged the
alphabet-soup that was AT/ISA, EISA, HP-PB, MCA,
VME, NuBus, SBus, and TurboChannel, among others.
It ushered in a long period of wide industry acceptance
of a single standard, and thus a stability and predict-
ability that made both product development and selec-
tion pleasingly straightforward.
PCI not only standardized I/O attributes, it enabled
high bandwidth. Its initial 133 MBps2 may seem
modest today, but it greatly outpaced then-standard
1. The first TPC-C result, published in 1992.
2. Megabytes per second. Bandwidth figures are nominal,
not typical. Such naive peak rates dont consider practical
slowdowns such as contention and protocol overhead.
Licensed to InfiniBand Trade Ass'n
Web Use Only - Do Not Reproduce
ISAs maximum 10 MBps and EISAs 33 MBps data
transfer rates. As a commodity standard, it minimizes
cost to achieve high shipment volumes. Even so, PCI
has neatly outperformed virtually all alternatives,
including those quite proprietary and specialized.
Then, in a classic case of volume sales providing the
investment dollars needed to move a heretofore
commodity product upmarket, PCI has dramatically
extended its reach. Enhanced versions have doubled
both clock speed and bus width, making 264 MBps
easily achieved today, with 500+ MBps options avail-
able. The HotPlug PCI extension made PCI suitable for
high availability servers, and its CompactPCI deriva-
tive has driven into embedded systems and telco gear.
Although its attributes promised it a long life, PCIs
very architecture is ultimately limiting.
PCI is built upon that simple connectivity structure,
the parallel bus. The simple, economical bus structure
has been at the base of so many electronic products for
so long that its virtually taken for granted. Yet busses
have inherent drawbacks:
Disorderly contention for resources by periph-
erals, memory, and CPUs. Disorder breeds ineffi-
ciency and suboptimal performance.
Vexing failure modes. Not only is the bus a poten-
tial single point of system failure, failure isolation
is difficult or impossible. If one attached card fails,
it can cause the entire system to fail. Worse,
discovering which card caused the failure is at best
a hit-or-miss propositiona misery in a world
needing high availability.
Severe physical stipulations and limitations. As
bus length increases to accommodate more, or
more widely dispersed, expansion devices,
signaling properties become less stable. The same
thing is true for clock rates. The faster the modu-
lation, the shorter the feasible bus, and the fewer
peripheral interconnects are possible. In the
extreme case, the 133 MHz defined for PCI-X,
there can only be a single connector per bus!
PCIs shared structure cannot keep up on a perfor-
mance basis, nor are its manageability and availability
attributes acceptable. As next-generation computing
platforms are being planned and implemented, PCI will
gradually be left behind, as antiquated as 66 MHz
microprocessors and 40 MB disk drives.

Incremental Upgrades
One could continue to improve PCI a bit, or work
around its limitations. Servers needing both high band-
width and large numbers of expansion slots, for
example, are often outfitted with multiple, independent
PCI buses. This comes at a cost, of course, but averts an
immediate capacity crisis.
The latest PCI-X revision goes further, cleaning up the
electrical signal definitions to drive towards 1 GBps
(133 MHz x 64 bits).3 Its a significant and promising
extension that will extend PCIs life by several years.
Even improvements as extensive as PCI-X, however,
have ever diminishing returns. The writing is on the
wall. Despite PCIs notable run of success, and the fact
that it will remain with us for years to come, its ulti-
mate headroom is limited. Bus architectures are funda-
mentally outpaced by our users and applications
voracious need for data, and thus high rates of I/O.
Rather than more patches, what we now need is a jump
as dramatic as PCI was when it was first introduced. As
Mitch Shults, Intels point man on I/O strategies says,
the industry has got to move to some fundamentally
new architecture. Enter InfiniBand.
On the Way to IBTA
The road to a future I/O standard has been rocky. Even
for PCI, vendors were reluctant to give up their favored
proprietary options. Sun for example, while it has
supported PCI, to this day favors its own SBus design
in its premium servers. But the vastly better economics
of a single standard, both for IT producers and
consumers, has won the day.
The once-divergent groups such as NGIO (Next
Generation I/O, led by Intel) and Future I/O (led by
IBM, Compaq, and HP) cast their fates together in
August 1999, a move that led to the foundation of the
InfiniBand Trade Association (IBTA). The IBTA is
largely based on the successful PCI SIG model. Even
after formation there have been some disagreements
about how quickly InfiniBand should appear, and how
encompassing it should be when it does appear. There
have also been tensions between IBTA members and
external constituencies such as the embedded systems
3. A speed that was aggressive for even the best system
busses just five years back.
Licensed to InfiniBand Trade Ass'n
Web Use Only - Do Not Reproduce
3
community. But this is to be expected. The free market
is contentious by nature. And, as they say, you cant
make an omelette without breaking a few eggs. At the
end of the day, these participants know that the
customer uptake rate for their next-generation servers
depends on solving I/O bottlenecks, and on not
creating a divisive standards war. Thus while disagree-
ments they may have, they are all highly motivated to
find a common and standard solution.
IBTA leaders (officially, Steering Members) IBM,
Intel, Compaq, Hewlett-Packard, Dell, Microsoft, and
Sun Microsystems reason that its better to have a
smaller group get something practical and effective out
the door than to hear everyones wishlist. In addition to
the Steering Members, Sponsoring Members include
3Com, Adaptec, Cisco, Fujitsu-Siemens, Hitachi,
Lucent, NEC and Nortel Networks. Its a potent brain
trust, among them the owners of the best I/O technol-
ogies and intellectual property in the industry.
The Goods
InfiniBand is the cavalry to the rescue, the I/O standard
and workhorse emerging for the new generation. So
what exactly is it?
InfiniBand is a network approach to I/O. A system
connects to the I/O fabric with one or more Host
Channel Adapters (HCAs). Devices, such as storage and
network controllers, would attach to the fabric with a
Target Channel Adapter (TCA). InfiniBand adapters
(generically, CAs) are addressed by IPv6 addresses, just
as any other network node might be.
The fabric concept may seem abstract to someone
whos used to fitting a card in a slot, but its exactly
what happens on any other network, whether of the
traditional LAN/WAN/Internet variety, or the storage
area networks (SANs) now rapidly entering data
centers. The physical fabric combines connectors,
cables, and switches. Current specifications call for one,
four, and twelve-wide link options, corresponding to
500 MBps, 2 GBps, and 6 GBps bandwidths.4,5 Whereas
PCI distances can be easily measured in inches or centi-
4. Serial links are conventionally described in bits/sec, not
the bytes/sec of parallel links. Each InfiniBand width
drives 2.5Gbps (250 MBps) in each direction. Doing the
math, 4-wide = 10 Gbps (1 GBps/direction), 12-wide = 30
Gbps (3 GBps/direction).
4
meters, InfiniBand
links are designed to HOST
~17 meters (data
center distances) using
copper cabling, or to Channel Adapter
100 meters (intra
building or small-
Physical Wire
campus distances)
Differential Pair
with fibre optic links.
The connectors will
resemble todays
Ethernet (RJ45) and
Fibre Channel ports. Four Wire
Extenders, protocol InfiniBand Link
switchers, and fibre
cabling may increase this a bit, say to 1km (with
multimode fibre) or a few km (with single mode fibre).
The 1,000km common to WAN links are impractical
given the need to minimize end-to-end latency.
Regardless of the number of wires (i.e., bandwidth
grade) or physical dispersion, InfiniBand uses a single
set of logical structures for how nodes are addressed
(IPv6), what protocols and APIs are used, and how the
components are pieced together.
Time-to-market issues will make early TCA imple-
mentations equivalent to HCAs, but later refined
implementations will rapidly cost- and space-mini-
mize TCAs to enable high-volume sales and inclusion
into denser and more embedded configurations. Card-
sized CAs will give way to multi-chip semiconductor
implementations, then single-chip, and finally will be
modules that can be optionally included in CPUs and
ASICs. As with most I/O options, high-end servers,
storage arrays, and peripherals will be first to imple-
ment and deploy InfiniBand. These are the units that
most need the added performance, and for which the
higher initial costs will be most easily absorbed.
InfiniBand Everywhere
Ultimately, InfiniBand everywhere will be the
rallying cry, just as PCI expanded its purview to both
larger and smaller systems. Within a few years, we
predict that InfiniBand will be the default way of
5. When implemented in copper links, each width unit
uses two wire pairs for differential signalling, resulting
in 4, 16, and 48-wire copper connections.
Licensed to InfiniBand Trade Ass'n
Web Use Only - Do Not Reproduce
Virtual Lanes
Message
Packets
TARGET
Channel Adapter
connecting servers to other servers (in clusters and
MPP systems), to storage (somewhat displacing Fibre
Channel SANs, especially in rack- and room-area
fabrics), and to network adapters and infrastructure
(including directly into Internet routers and switches).
Despite these high-end ambitions, as the successor to
PCI, InfiniBand is still about in-chassis I/O, shipped in
high volume and low price points. This deployment,
which makes a switched network a rack- and mother-
board-level feature, will remake system form factors.
2U and 1U rack-and-stack servers may seem like dense
computing today, but InfiniBands small connectors,
flexible cabling, and network approach will fundamen-
tally compress computing complexes. Within a few
years, expect todays 0.51U per CPU densities to fall
well under 0.5U/processor, perhaps beneath 0.2U per.
Density isnt everythingcost and high-availability
are also keybut ISPs, ASPs, and other service
providers will be particularly glad to further minimize
IT footprints.
Though further out than server and workstation
deployments, embedded computing is another area of
InfiniBand opportunity. Network switches, telco gear,
wireless hubs, industrial automation, and telemetry
units are all eventual targets.6
Part and parcel of the InfiniBand transformation will
be the leveraging of switch design skills and invest-
ments at system, network, and storage OEMs to
6. Albeit in competition with the still-viable CompactPCI
and Motorolas emerging RapidIO initiative.

support generalized InfiniBand fabrics. In great
measure, InfiniBand will work because it brings so
many strong players to the table.
Changing the Guard
Saying that InfiniBand is a networked I/O standard is
true, but hardly scratches the surface of the design. It
is, for example, also a channel-based approach.
Instead of the memory-mapped load/store para-
digm of PCI, InfiniBand uses a message-passing
send/receive model. This, in concert with the
endpoint addressabilty, is essential in ensuring utterly
robust, reliable operations. Transmissions are demar-
cated into distinct work queue pairs, with packets
distributed and disseminated throughout the Infini-
Band network. Adapters take on the responsibility for
handling transmission protocols, and InfiniBand
switches take on responsibility for making sure
packets get where theyre supposed to be. This distri-
bution of work is common, for example, in S/390
mainframes.7
Tom Bradicich, IBMs Intel server technologist and a
prime mover behind InfiniBand, is fond of using a
mailman metaphor. CPUs and hosts pass data into
memory for use by targets, and then move along to
other tasks, just as a mailman drops your messages
and moves along to the next house. This functioning
is fundamental to the specification. Wrenching I/O
off the PCI bus, imbuing it with higher-order organi-
zation schema, and pressing it into better-managed
and more tightly controlled service inside and outside
the box is InfiniBands raison detre.
The controlling mechanisms are quite sophisticated.
Addressing nodes with IPv6, for example, will allow
easy and direct linkage with Internet routers and gate-
ways. And while physical layer implementations are
organized around a given number of wires, the
logical structure is very general. The links are bi-
directional and composed of up to 16 virtual lanes,
any of which a given packet may travel.
7. Perhaps there really is nothing new under the sun!
Licensed to InfiniBand Trade Ass'n
Web Use Only - Do Not Reproduce
5
The idea is getting server I/O onto the network, and
ultimately the Internet. This goes well beyond the
remote I/O, for example, found in a few of todays
newest high-end servers. There are rich possibilities
with this flexible methodology. The technologys
switched design, message/packet basis, fat pipes, and
extensive controlling mechanisms will underpin
architectures and network schema for the next decade.
Thinking Outside The Box
Future IT will be largely dictated by Internet-style
networked computing. In some ways, the Internet
mindset is simply an extension of trends that had
been developing for three decades. Over time,
compute functions have become steadily more atom-
ized and distributed, devices have become more intel-
ligent, client-server has been integrated into the Web,
and the local network has extended into a global
network. In short, data have steadily been cast farther
away from their home bases. I/Oby definition the
movement of datahas of necessity had to be inte-
grated across wider spans.
This dilative phenomenon is the externalization of
I/O. External I/O requires common protocols to link
the traffic between the computing devices and control-
ling mechanisms referred to above.
The application is king. But oftentimes, applications
and databases are starved for data. There are bottle-
necks, latencies, and congestion that simply arrest
performance. Here, InfiniBand will make a difference.
Very large databases are now in the terabyte and
above range, with some spanning to 50 TB. Very large,
distributed engines are needed to process such infor-
mation. These massively parallel systems or compute
farms are essentially clustered systems. Whether the
DBMS provider is IBM or Oracle, NCR or Compaqs
Tandem division, these clusters require high-band-
width, low-latency interconnects. Specialized propri-
etary designs are the common result. IBMs SP
switch, NCRs BYNET, and Compaqs ServerNet and
Memory Channel are commonly used in their largest
distributed engines.
6

HOST

System Interconnect

Compression Engine
CPU
CPU Memory Target
Controller
Memory DMA HCA InfiniBand Switch TCA DMA Function
CPU
CPU

To HCAs, TCAs, by way

InfiniBand Target of IB switches or, directly to
TCA DMA
1, 4, or 12-wire Links Controller(s) devise functions which are
being controlled (disks, etc.)

HOST Network Connection

(InfiniBand or other
network protocols)
System Interconnect

CPU DMA
CPU Memory
Memory DMA HCA InfiniBand Switch CA DMA
CPU Controller CA
CPU

Network Router

As InfiniBand moves into its more refined switched Fibre

generation (in 2002-2003), it will provide exactly
the sort of high-speed, low-latency packet switching
needed by these clusters. Its support for cascaded
switches, fabric partitions (also called zones), and
inherent multicasting are well-suited for large clus-
ters. This prowess combines with its industry stan-
dardization to drive what have been high end cluster
technologies into the mainstream.
Once fully formed, InfiniBand will enable massive
horizontal scalability and transparent I/O sharing
among cluster nodes. Indeed, just as InfiniBand brings
the intelligent channel idea down from the S/390, it
will enable the intelligent-everything (CPU module,
disk, network controller) model of Compaqs Hima-
layas. This is the key to not only huge performance
scalability, but the ability to do so in a highly avail-
ableeven fault tolerantway.
Three Precious Words
Reliability, availability, and serviceability may not
compare to I love you, but CIOs cant say I love
you to any server technology that doesnt have RAS
at the heart of its design. InfiniBand again connects
with the horsehide. Not only does InfiniBand directly
support the RAS attributes inherent in multi-system
clustering, its physical, electronic, and logical design
are RAS-friendly. Also remember that the NCR,
Licensed to InfiniBand Trade Ass'n
Web Use Only - Do Not Reproduce
Tandem, and IBM
TCA DMA Channel Fibre Channel parallel/cluster
Controller
systems discussed
above comprise some of the
highest-quality hardware and software technologies
in IT. InfiniBand will take its place as the backbone of
these clusters.
Further, the InfiniBand protocols support channel-to-
channel I/O failover in InfiniBand links, should a Host
detect a Target failure. Redundant Infiniband links will
of course be required for this functionality.
Note also that clustering is based on a distributed-
memory model that improves availability by diffusing
points of failure. In a similar vein, InfiniBands
concept of creating myriad I/O controllers, most of
them located outside the server chassis, enables
component separation and redundancy, eliminating
the PCI buss single domain of failure. Finally, Infini-
Bands message-passing paradigm and protocols incor-
porate layers of error management. The technology is
also being designed for device hot-addability,
including device look-up and registration, which will
aid IT professionals in dynamically managing, modi-
fying, and augmenting their networks.
Clusters by definition increase scaling: Thats how
vendors get those 64, 128, 256, and 1,024-way
systems. InfiniBands cascadeable switching will

stretch clustering in a big way, dramatically accentu-
ating horizontal scalability. Using InfiniBand
switching, partitioning and fabric management,
combined with memory-management and control,
servers will be configured into first- and second-order
networks with many hosts and I/O end nodes.
Memory and other resources will be shared in these
overlapping subnets of physical and virtual servers and
their supporting components, all playing roles in clus-
tered environments. Additionally, these subnets can be
separated for functional isolation, increasing manage-
ment control, availability and performance.
You Can Take I/O Out of the Network,
But You Cant Take the Network Out of I/O
We have referred to the Internet and its impact on
enterprise computing. The clustered servers and
storage that control inter- and intra-application
communications will be at the eye of the Internet
computing whirlwind for years to come. InfiniBands
physical and logical attributes will extend server I/O
far and wideout of the box, out of the data center,
out of the network, and onto the Internet. The IBTA is
basing the addressabilty of the InfiniBand Architecture
in large part on IPv6, enabling not only efficient local
manageability, but also prepping InfiniBand for its
ventures out into the Internet. Source (HCA) and
destination (TCA) IPv6 addresses are embodied in the
InfiniBand Global Route Header, which is used to route
packets between HCAs and TCAs, across linked
subnets, and out into the world at large. Its a very
well-though out design, simultaneously corresponding
to local-, wide-, and global-area fabric management.
Licensed to InfiniBand Trade Ass'n
Web Use Only - Do Not Reproduce
7
Conclusion
The old order changeth, yielding place to new, wrote
Tennyson. IT professionals live these words. For
system designers, InfiniBand starts this cycle anew
with a generational change in computing architectures.
Bus-based I/O is giving way to serial communication
links, and processor-driven I/O by intelligent I/O
engines, or channels. InfiniBand both enables this
change and provides a standard for it that unifies inter-
connectivity across servers, storage, and networking as
few technologies have done before.
Five IBTA working groups are busily designing Infini-
Bandworking out its protocols, electrical signaling,
register models, data structures, verbs, memory and
semantic operations, software, management, and phys-
ical/mechanical specifications. Version 0.9 runs some
900 detailed pages. Version 1 is due in the summer. We
can hardly wait for the technologys improved band-
width and grand-scale architectural possibilities!
That InfiniBand will eventually meet its own maturity
and demise hardly quells our enthusiasm that
endgame is another ten years distant. We like Infini-
Band because we like the idea of server and worksta-
tion data flowing along communication lines across
lattices of clustered, process-sharing hardware and
software. InfiniBand is the right technology at the
right time for the right reasons to realize this dream.
InfiniBand rocks!
8
InfiniBand Features
Host and Target
Channel Adapters
Switches/Routers
Links
Messages/Packets
Licensed to InfiniBand Trade Ass'n
Web Use Only - Do Not Reproduce
Components and Features of InfiniBand
Attributes
Multi-port; tool-free, single-axis
insertion; commodity form factors;
IPv6 addressable
IPv6 addressable and routeable;
cascadeable switches; partitionable
traffic zoning; QoS enablement
One, four and 12-wide bandwidths;
copper (differential signalling) and
fibre physical links; 1-16 bi-direc-
tional, independently assigned chan-
nels/lanes per link (one reserved for
fabric management, one for applica-
tion usage); credit-based flow
control; static rate control; auto-
negotiation/mapping algorithm
Routing at the packet level; message
segmentation and re-assembly;
Cyclical Redundancy Checking;
interleaved packets across channels;
IPv6 addressing headers; memory
protection; remote DMA
Advantages and Uses
Allows direct access to memory by applica-
tions and other I/O; reduces CPU, OS kernel,
and peripheral traffic to memory; allows for
remote direct memory access; translates and
validates messages; enables load-balancing
and redundancy qualities;
Relieves bus contention; enables partitioning
and subnet creation for clustering and
manageability; enables multicasting; allows
for scalable cascading of multiple switches;
routers dispatch data across switched subnets
Multiplexing and logically connected address
spaces allow for refined manageability and
arbitration; connects hosts and targets of
different speeds and widths; multiple speed
grades match varying price points
Allows for granularity in identifying and
controlling IPC and I/O processes; enables
error detection and correction; allows for
refined application and I/O management and
control; allows for IP compatible fabric
management; enables processor-independent
and serverless I/O