DS8K Architecture

Hardware components and

architecture
This chapter describes the hardware components of the IBM System Storage DS8800. It
provides readers with more insight into individual components and the architecture that holds
them together.
The following topics are covered in this chapter:
_ Frames
_ DS8800 architecture
_ Storage facility processor complex (CEC)
_ Disk subsystem
_ Host adapters
_ Power and cooling
_ Management console network
_ System Storage Productivity Center
_ Isolated Tivoli Key Lifecycle Manager (TKLM) server

Frames:
The DS8800 is designed for modular expansion, there appear to be three types of frames available for
the DS8800. the frames themselves are almost identical. The only variations are the combinations of
processors, I/O enclosures, storage enclosures, batteries, and disks that the frames contain.the frame
variations that are possible with the DS8800. The left frame is a base frame that contains the processors.
In this example, it has two 4-way IBM system p POWER6+ servers, because only 4-way systems
can have expansion frames. The center frame is an expansion frame that contains additional I/O
enclosures but no additional processors. The right frame is an expansion frame that contains simply disks
and no processors, I/O enclosures, or batteries. Each frame contains a frame power area with power
supplies and other power-related hardware. A DS8800 can consist of up to three frames.

Base frame:
The left side of the base frame, viewed from the front of the machine, is the frame power area.
Only the base frame contains rack power control cards (RPCs) to control power sequencing

for the storage unit. The base frame contains two primary power supplies (PPSs) to convert
input AC into DC power. The power area also contains two or three battery backup units
(BBUs). A base frame with a two-way processor contains two BBUs and a base frame with a
four-way processor contains three BBUs
The base frame can contain up to ten disk enclosures, each of which can contain up to 24
disk drives. In a maximum configuration, the base frame can hold 240 disk drives. Disk drives
are either hard disk drives (HDD) with real spinning disks or Solid State Drives (SSD), which
have no moving parts and enable a significant increase in random transactional processing.
A disk enclosure contains either HDDs or SSDs. With SSDs in a disk enclosure, it contains
either 16 drives or a partially populated disk enclosure with eight SSD drives. A disk
Chapter 3. Hardware components and architecture 31

enclosure populated with HDDs contains 16 or 24 drives. Note that only up to 48 SSDs can
be configured per Device Adapter (DA) pair. It is inadvisable to configure more than 16 SSDs
per DA pair.
Standard cabling supports either two-way processors with one I/O enclosure pair or four-way
processors with two I/O enclosure pairs. Standard cabling with one I/O enclosure pair
supports up to two DA pairs and six storage enclosures (144 DDMs). Standard cabling with
two I/O enclosure pairs supports up to four DA pairs and ten storage enclosures (240 DDMs).
Business class cabling utilizes two-way processors and one I/O enclosure pair. Business
class cabling supports two DA pairs and up to ten storage enclosures (240 DDMs).

Note: DDMs(disk drive modules

Inside the disk enclosures are cooling fans located in the storage enclosure power supply
units. These fans blow exhaust to the rear of the frame.
Between the disk enclosures and the processor complexes are two Ethernet switches and a
Storage Hardware Management Console (HMC).
The base frame contains two processor complexes (CPCs). These System p POWER6+
servers contain the processor and memory that drive all functions within the DS8800.
Finally, the base frame contains two or four I/O enclosures. These I/O enclosures provide
connectivity between the adapters and the processors. Each I/O enclosure can contain up to
two device adapters and up to two host adapters.
The communication path used for adapter-to-processor complex communication in the
DS8800 consists of four-lane (x4) PCI Express Generation 2 connections, providing a
bandwidth of 2 GBps for each connection.

The interprocessor complex communication still utilizes the RIO-G loop as in previous models
of the DS8000 family. However, this RIO-G loop no longer has to handle data traffic, which
greatly improves performance.

Expansion frame
The left side of each expansion frame, viewed from the front of the machine, is the frame
power area. The expansion frames do not contain rack power control cards; these cards are
only present in the base frame. Each expansion frame contains two primary power supplies
(PPSs) to convert the AC input into DC power. Finally, the power area can contain zero or two
battery backup units (BBUs), depending on the model and configuration. A first expansion
rack requires two BBUs. A second expansion rack requires two BBUs if the Extended Power
Line Disturbance (EPLD) feature is installed and no BBUs without EPLD. If the EPLD feature
is installed, the modules must be installed on the base frame and all expansion frames. The
EPLD feature is highly recommended as an additional safeguard against environmental
power fluctuations.

The first expansion frame can hold up to 14 storage enclosures, which contain the disk drives.
They are described as 24-packs, because each enclosure can hold 24 small form factor (SFF)
disks. In a maximum configuration, the first expansion frame can hold 336 disk drives.
Disk drives are either hard disk drives (HDDs) with real spinning disks or Solid State Drives
(SSDs), which have no moving parts and enable a significant increase in random
transactional processing. A disk enclosure contains either HDDs or SSDs but not both.
With SSDs in a disk drive enclosure pair, it is either 16 drives or a half-populated disk
enclosure with eight SSD drives. A disk enclosure populated with HDDs contains always 16
or 24 drives. Note that only up to 48 SSDs can be configured per Device Adapter (DA) pair,
and it is not advisable to configure more than 16 SSDs per DA pair. Example 3-1 shows the
configuration required to get to the maximum SSD configuration (384 DDMs).

The second expansion frame can hold up to 20 storage enclosures. In the maximum
configuration, the second expansion frame can hold 480 disk drives. The maximum
configuration with the base frame and two expansion frames is 1056 drives.
An expansion frame contains I/O enclosures and adapters if it is the first expansion frame that
is attached to a DS8800 4-way system. Note that you cannot add any expansion frame to a
DS8800 2-way system
The second expansion frame cannot have I/O enclosures and adapters. If the expansion
frame contains I/O enclosures, the enclosures provide connectivity between the adapters and
the processors. The adapters contained in the I/O enclosures can be either device adapters
or host adapters, or both. The expansion frame model is called 95E. You cannot use
expansion frames from previous DS8000 models as expansion frames for a DS8800 storage
system.
Note: Adding an expansion frame to a business class cabling configuration requires an
upgrade from a 2-way to 4-way system as a prerequisite.
Note: The business class cabling configuration is limited to one expansion frame and a
maximum of 576 DDMs. This restriction can be removed by converting to standard cabling,
which is disruptive. This conversion is only available via RPQ.
In effect, the DS8800 consists of two processor complexes. Each processor complex has
access to multiple host adapters to connect to Fibre Channel or FICON hosts. A DS8800 can
have up to 16 host adapters with 4 or 8 I/O ports on each adapter.

Fibre Channel adapters are also used to connect to internal fabrics, which are Fibre Channel
switches to which the disk drives are connected.

POWER6+ processor

The DS8800 is based on POWER6 p570 server technology. The 64-bit POWER6+
processors in the p570 server are integrated into a dual-core single chip module or a
dual-core dual chip module, with 32 MB of L3 cache, 8 MB of L2 cache, and 12 DDR2
memory DIMM slots. This enables operating at a high data rate for large memory
configurations. Each new processor card can support up to 12 DDR2 DIMMs running at
speeds of up to 667 MHz.
The Symmetric Multi-Processing (SMP) system features 2-way or 4-way, copper-based,
Silicon-on Insulator-based (SOI-based) POWER6+ microprocessors running at 5.0 GHz.
36 IBM System Storage DS8800: Architecture and Implementation
Each POWER6+ processor provides a GX+ bus that is used to connect to an I/O subsystem
or fabric interface card. GX+ is a Host Channel Adapter used in POWER6 systems.

Peripheral Component Interconnect Express (PCI Express)

The DS8800 processor complex utilizes a PCI Express infrastructure to access the I/O
subsystem, which provides a great improvement in performance.
PCI Express was designed to replace the general-purpose PCI expansion bus, the high-end
PCI-X bus, and the Accelerated Graphics Port (AGP) graphics card interface.
PCI Express is a serial I/O interconnect. Transfers are bidirectional, which means data can
flow to and from a device simultaneously. The PCI Express infrastructure involves a switch so
that more than one device can transfer data at the same time
There are two generations of PCI Express in use today:
_ PCI Express 1.1 (Gen 1) = 250 MBps per lane (current P6 processor I/O)
_ PCI Express 2.0 (Gen 2) = 500 MBps per lane (used in the DS8800 I/O drawer)