DS8K Architecture

The DS8K architecture is discussed in the Enterprise Storage Selection page.

Internally, the system consists of device adaptors that connect to the outside
world, disk adaptors and enclosures for the data storage and a large memory
cache to speed up access. The whole lot needs to be connected together and
needs to processing power to run it.
Processing power is supplied by a POWER6+ p570 copper-based symmetric
multiprocessor (SMP) system unit. The processor complex is either 2 way or
4 way and runs at 5.0 GHz. All DS8000 machines have two processor
complexes running as a redundant pair, so if one complex fails, the DS8000
will continue to run on the other complex.
Cache size depends on the CPU configuration, with a 4 way processor using a
significantly larger cache than a 2 way. Cache is used to speed up IO
operations, Sophisticated algorithms are used to detect patterns within read
IOs, so required data can be predicted and loaded up into cache memory
from disk before it is requested. If read IOs are sequential, then the DS8K will
use an Adaptive Multi-stream Prefetching (AMP) caching algorithm, which
basicallty streams the data into the cache.
Disks are a combination of (relatively) slow 'nearline' disks, 'day to day
workload' SAS 2.0 disk drives and optionally very fast solid state drives.
Older DS8K models used Fibre Channel disk drives. All disks come in 16 drive
install groups, except that the 600GB solid state drives can be installed in 8
drive half groups.
'Easy Tier' software is used to dynamically move data within the three
storage classes, depending on how active it is at the present time. Data can
also be placed manually.
A Storage Tier Advisor Tool exists to give guidance on how much solid state
you need to optimise your performance.
All the components are connected together with PCI Express Generation 2 I/O
enclosures. The older DS8K models were connected using shared loops, but
the current models use point-to-point PCI Express cables. Inter-processor
communication is isolated from I/O traffic, and used RIO-G ports.
Data is organised within the DS8K as Extent Pools, How you organise your
extent pools will depend on how many different types of physical disk you
are using, and whether or not you are mixing CKD and FB data on a box. CKD
and FB data must be in separate extent pools, and it pretty much makes
sense to separate slow disks and fast disks into different pools.

Extent pools contain storage extents of either 1GB for FB or the size of a
3390-1 for CKD. To understand where the extents come from we need to
switch to a physical view of the disks.
Enclosures, Arrays and Ranks
Physical disks are installed in groups of 16, called disk groups or enclosures.
Each enclosure contains two fibre channel switches, and every disk has two
fibre connections, one to each switch. These switches are then connected to
two device adaptors over two independent RIO-G loops. Disk enclosures are
installed into the front and rear of the DMX, and two enclosures are then
paired up to form four Array Sites. An Array site contain 8 disks, 4 from a
front enclosure and 4 from a rear enclosure, so an array site will also span
two RIO-G loops. Array Sites are numbered S1, S2 etc. n.b. there is no S0!
This is illustrated in the GIF below

When you format an Array site up as RAID, it becomes an Array. RAID options
are either RAID5, 6 or 10. Typically, a CKD formatted RAID5 array based on
146GB disks will contain 826 GB of data, as one disk is reserved for a spare,
and the equivalent of one disk is needed for RAID5 parity. For the benefit of
you people reaching for your calculators, the missing space is the CKD
formatting overhead. The FB formatting overhead is lower, so an FB RAID 5
array with a dynamic spare will contain 836 GB. The rules for calculating how

many spares are needed are complicated, but in general the DS8K will
require one spare for every Array, until it has four spares per Device Adaptor
pair, but it may reserve more if different disk sizes or speeds are combined in
an Array Site. The Arrays are numbered A0,A1, etc.
At present, a Rank consists of one RAID array. Each Rank is split into extents
of 1GB each for FB format and 0.98GB (the size of a 3390-1) for CKD format.
These are the extents that make up an Extent Pool. Ranks names are
allocated by the system and are called R0, R1, R2 etc.
Ranks should ideally be allocated to Extent Pools so that the capacity is
equally shared between Server0 and Server1. Ranks, Array Sites and Arrays
are independent until the Rank is placed into an Extent Pool. It is then bound
to the server associated with that pool.
As mentioned above, you need an absolute minimum of two Extent Pools per
DS8K, and in theory you can have as many Extent Pools as you have Ranks.
In practice, if you place more than one Rank into an Extent Pool, then you
can stripe the data over the Ranks to improve performance. The IBM
recommendation is to place between 4 and 8 Ranks in an Extent Pool. If you
allocate all the space in an Extent Pool to volumes, then add an extra rank
for more capacity, you will not get the benefit of striping as there is only one
rank to strip across. Always try to add at least 4 ranks to a full pool, or to a
new pool, so striping can be effective.
If you want to use space efficient volumes, then you must define a repository
for them within an Extent Pool. Only one repository is allowed per Extent Pool
and once defined it cannot be resized, except by deleting all the space
efficient volumes, then deleting and reallocating the repository. Space
Efficient volumes is a chargeable option.
Volumes, LCUs and Disk Groups
Ok, so now we have a few Extent Pools all containing lots of 1GB extents.
How do you present those extents to a server? A logical volume is made up
of a set of extents from one Extent Pool. You can define a volume to be
almost any size you want, but the space is added internally is fixed
increments. This means you should make the total size of the logical volume
to be a multiple of 1GB for FB or 3390-1 for CKD to prevent space wastage.
Up to 64K logical volumes can be defined
In a mainframe environment, all the extents in a CKD volume must be
contained in one Extent Pool, but can be striped over several Ranks. A CKD
volume can be any size between a 3390-1 and a 3390-A (1113 to 262,668
cylinders) provided your z/OS release supports this.
The point behind this apparently complicated setup is that it makes the
storage environment more flexible, especially for future maintenance. With

as ESS, if you wanted to resize volumes, you had to empty out an entire
array, delete it then redefine the whole array. The concept of breaking a CKD
volume space into fixed size extents makes it possible to delete one volume
from within a range, then add it back in again as a different size. You can also
convert a 3390-9 to a 3390-27 in place by simply adding more extents to it
and running and IDCAMS REFORMAT . REFVTOC
In an Open Systems environment, a logical subsystem looks like a SCSI
controller with up to 256 associated LUNs. Up to 256 Logical Subsystems can
be defined with even addresses associated with Server0 and odds with
Server1.
The DS8K supports several Open Operating Systems, including Windows,
Linux, AIX, Sun Solaris, HP-UX, VMWARE and Open VMS. Each of these
operating systems has its own quirks, so the best thing is to look up the
latest data for your own OS.