Barry Whyte IBM Master Inventor Virtual Storage Performance Architect June 2013

A Brief History of SVC & Storwize Family

What, How and Why?

2013 IBM Corporation

A Brief History of SVC and Storwize Family

Session goals and welcome

This session presents a brief history of IBM SAN Volume Controller (SVC) and IBM Storwize family. Starting back in 1999, it will explain why IBM decided to develop SVC, and the research projects that lead to the clustering architecture and the I/O processing architectures. The session goes on to describe the benefits of storage virtualization and how the software and hardware have evolved over time to where we are today, with the same software running across twelve different hardware platforms in as many years. From a field-hardened code base to a modern storage appliance/controller architecture that is the fastest thing in anyone's SAN -- even with today's SSD devices you'll learn why these IBM offerings lead the way in storage virtualization.

2013 IBM Corporation

A Brief History of SVC and Storwize Family

A Brief History ...

Professor Steven Hawking's seminal work... By the end of this session, I'll show how we : We are 14+ years ahead of the recent software defined storage trend Set several trends in the storage industry First to use and prove commodity hardware as a storage platform First to implement integrated Flash technology Recently added a time-machine to the Storwize family Without the need for a Flux-capacitor...

2013 IBM Corporation

A Brief History of SVC and Storwize Family

A Brief History of SVC and Stowize Family

From Direct Attached to expanding SANs

2013 IBM Corporation

A Brief History of SVC and Storwize Family

Storage Industry IBM circa 1995 - Direct Attached Storage

Before the mid 1990's most storage systems were direct attached IBM Serial Storage Architecture Developed by the IBM UK Hursley Storage team (who develop the Storwize Family) Dual serial disk loops per adapter 96 drives per loop - 160MB/s Spatial re-use only seen now in todays SAS topologies 8 way server attachment 2 way RAID architecture Over 95% attachment rate to IBM RS6000 server platform (Power Systems) Original storage disk adapters in IBM Enterprise Storage System (ESS) Evolved into IBM DS8000 with FC-AL interfaces using same adapter firmware Returned over $2 billion revenue between 1996 and 2003 SANs started to make direct attach obsolete, when many hosts could attach to one or more disk system

2013 IBM Corporation

A Brief History of SVC and Storwize Family

'Ye canny change the laws of physics, Captain!

Gbit / square inch

25% CGR 60-100% CGR

Moores Law

25% CGR

60-100% CGR

2013 IBM Corporation

A Brief History of SVC and Storwize Family

State of the Storage Industry circa 1999

Back in 1999 we could see the second slow down was coming SAN complexity had grown out the box Storage itself was becoming a commodity Stoage islands wrong place wasted capacity? At the same time : Custom storage function silicon development was too slow and costly 3yrs etc Commodity (x86) silicon was proving to be enterprise ready Questions we asked : How can we tame heterogenous vendor SANs? How can we consolidate the storage functions? How can we make better use of what customers already have? How can we ride the x86 technology wave for years to come?

2013 IBM Corporation

A Brief History of SVC and Storwize Family

More questions... can something help answer them all?

In addition to the technology questions, several business and operational questions were also being asked : The continual growth in data center costs Inability of IT organization to respond quickly to business demands Poor availability or service levels Lack of skilled staff for storage admin tasks Poor asset utilisation So while SANs had opened the potential for more flexibility In reality, they opened up another set of problems Could we radically change the way we think about storage?

2013 IBM Corporation

A Brief History of SVC and Storwize Family

Storage Virtualization an often over-abused term

Server virtualization, wasn't really around in 1999, other than on mainframe systems But what if we could virtualize all those disk systems Consolidate the disparate capacity Make better use of whats already there Help slow down the continual need to buy more disk Provide mechanism to migrate around downtime Implement a common set of storage functions But also make sure we answered the technology questions Use a commodity hardware base Ride Intels technology wave Write the intelligence into the software We'd need a flexible future proof software architecture

+ Software Function

2013 IBM Corporation

A Brief History of SVC and Storwize Family

Storage Virtualization The Approach War

In-band Appliance
Caching and scale-out nodes enhances performance Consolidates Copy Services Grows with business needs

Array Based
No additional HW required Not scalable - limited by box performance Consolidates Copy Services

Out-of-band Switch Based

Potentially scales to large networks Array-based or Switch-based Copy Services

Host Zone

Host Zone

Host Zone

Storage Zone Storage Zone

Storage Zone

10

2013 IBM Corporation

A Brief History of SVC and Storwize Family

IBM Research 1999 Storage Software Architecture

1999 IBM Systems Journal
The software architecture of a SAN storage control system

Cluster of processing nodes

I/O Component

Peer Communications

Abstraction layers Interface layer

Abstract the I/O from the protocol Common platform I/O (plio) definition

Interface Layer

Configuration

I/O Component I/O Component I/O Component

Clustering

Peer to peer comms

Abstract inter-node comms

Common Configuration API

Object manipulation

Common cluster API model

Cluster state interfaces

Stack of I/O components Each implements a specific, encapsulated storage function Upper/Lower common plio interface
2013 IBM Corporation

User Interfaces

A Brief History of SVC and Storwize Family

SCSI Target
Forwarding

Replication Cache A Brief History of SVC and Storwize Family FlashCopy Mirroring

The Birth of SVC

Thin Provisioning Virtualization
Forwarding

Compression Easy Tier

RAID
Forwarding

SCSI Initiator

12

2013 IBM Corporation

A Brief History of SVC and Storwize Family

IBM SVC Inband Storage Virtualization Concept ~2000

SVC nodes deployed as active/active caching pairs Virtual volumes provisioned from striped capacity from a given storage pool Associated with a caching node pair

Control Enclosure

Storage pools of similar performing / RAIDed LUNs All storage pools visible to all SVC nodes

All virtualized storage common across entire cluster (visible to all nodes) C Controller volumes (LUNs) grouped into common attribute storage pools
2013 IBM Corporation

A Brief History of SVC and Storwize Family

IBM SVC v1.1.0 2003

SCSI Target 2000 Hursley development begins work on turning the Almaden research work into an in-band virtualization appliance. By July 2003 the first release was ready for GA 4-way node clustering FlashCopy and MetroMirror functionality Online storage migration Initial controller support IBM DS and FAStT product ranges EMC DMX and Clariion Latest Hardware (4F2) X335 Server Base 2x 1core 2.8Ghz Xeon (Prestonia) 533MHz FSB 4 GB Cache 4x 2Gbit Fibre Channel
2013 IBM Corporation

Supported Hardware SVC 4F2 Node

Replication (MM) Cache FlashCopy

Virtualization

Legend

SCSI Initiator

Virtual Mdisk

A Brief History of SVC and Storwize Family

IBM SVC v2.1.0 and v3.1.0 - 2004/2005

SCSI Target Support for up to 8-way node clustering OS and controller support ++ 2004 Latest Hardware (8F2) X336 Server Base 2x 1core 3.0Ghz Xeon (Irwindale) 800MHz FSB 8 GB Cache 4x 2Gbit Fibre Channel ~ 25% IOPs boost (FSB) 2005 Same as above except: (8F4) 4x 4Gbit Fibre Channel 2x GB/s boost Internal code changes DV (DA)-V Political debates
2013 IBM Corporation

Supported Hardware SVC 4F2 Node 8F2 Node 8F4 Node

Replication (MM) Cache FlashCopy

Virtualization

Legend

SCSI Initiator

Virtual Mdisk

A Brief History of SVC and Storwize Family

IBM SVC v4.1.0 - 2006

SCSI Target Continued OS and controller support ++ First major new set of functional enhancements Multiple-target FlashCopy Up to 256 targets from single volume No overhead (only really 1 copy unless split again) GlobalMirror Added asynchronous replication functions Using same replication layer Sync or Async at create time Legend Virtual Mdisk Replication Cache FlashCopy
Supported Hardware SVC 4F2 Node 8F2 Node 8F4 Node

Virtualization

SCSI Initiator

2013 IBM Corporation

A Brief History of SVC and Storwize Family

IBM SVC 2007 Multi-core CPUs

Late 2006 saw the evolution away from the GHz race More cores per chip Early samples of Intel Woodcrest dual core CPU packages Fundamental change to they way we had previously allocated resources Cores, Threads and Spinlocks SVC up till now had run two threads, one pre CPU Spinlocks relatively low impact While mutli-threaded, base platform code With four threads, spinlocks, or rather spinlock contention wasting ~15% CPU cycles idle cores... Without giving too much IP away Binding of threads, ports and cores Increased threading, batching while locked...
2013 IBM Corporation

A Brief History of SVC and Storwize Family

SVC and Storwize Family Architecture

Since 2007 each node has at least 4 cores Piggy-back on Intel development Latest and greatest PCIe bus/memory Generally run a single fibre thread per core Queues of lightweight fibres No interrupts or kernel code Thread polls hardware from user mode One or more driver polling functions per thread Each port has its own polling function Some management functions like PCIe driver Some kernel functions like iSCSI Object ownership maybe distributed across cores Attempt to avoid context switch between threads if possible

Fibre Thread
(bound to a single core)

FC Port polling fn SAS Port polling fn

...
iSCSI Port polling fn

Fibre Queue
Fibre Fibre Fibre Fibre Fibre Fibre

2013 IBM Corporation

A Brief History of SVC and Storwize Family

IBM SVC v4.2.0 - 2007

SCSI Target Continued OS and controller support ++ First release of new multi-thread changes 2x IOPs boost 33% boost to existing nodes Enhanced FlashCopy Incremental Cascaded multi-target Latest Hardware (8G4) X3350 Server Base 2x 2core 2.33Ghz Xeon (Woodcrest) 1333MHz FSB 8 GB Cache 4x 4Gbit Fibre Channel Replication Cache FlashCopy
Supported Hardware SVC 4F2 Node 8F2 Node 8F4 Node 8G4 Node

Virtualization

Legend

SCSI Initiator

Virtual Mdisk

2013 IBM Corporation

A Brief History of SVC and Storwize Family

IBM SVC v4.3.0 - 2008

SCSI Target Continued OS and controller support ++ Forwarding for asymmetric access Volume Mirroring Two copies of a single volume stored on potentially disparate storage controllers Like LVM mirroring in the SAN Migration using split-mirror Thin Provisioning Fine grained thin provisioning (32KB - 256KB) Use volume mirroring for thick to thin migrate Mix within pools of thick and thin Legend Volume and pool usage warnings Virtual New hardware 8A4 Entry Edition
2013 IBM Corporation

Supported Hardware SVC 4F2 Node 8F2 Node 8F4 Node 8G4 Node 8A4 Node

Forwarding

Replication Cache FlashCopy Mirroring Thin Provisioning Virtualization

Forwarding

SCSI Initiator

Mdisk

A Brief History of SVC and Storwize Family

SCSI Target
Forwarding

A Brief History of SVC and Storwize Family

Replication Cache FlashCopy

The 1,000,000 IOPs Era

(but we were first!) Mirroring
Virtualization
Forwarding

Thin Provisioning

Compression Easy Tier

RAID
Forwarding

SCSI Initiator

21

2013 IBM Corporation

A Brief History of SVC and Storwize Family

2008 - Quicksilver - Scale-out Flash with SVC

Quicksilver was an SVC and Flash technology demonstration in 2008 Proved the capability of a scale-out clustered storage system Proved SVC stack was Flash ready... already Achieved 1.2 million IOPS (70/30 4KB), at under 1 ms response time Technology demo used 14 current generation SVC nodes (8G4) 32 System X servers with Flash PCIe cards
Each of these running a cutdown SVC virtualizing local block Linux /sd devices Presenting the /sd devices as mdisks to the 14 node SVC cluster
22 2013 IBM Corporation

A Brief History of SVC and Storwize Family

IBM SVC v5.1.0 - 2009

SCSI Target Continued OS and controller support ++ First integrated Flash support in virtualizer Integrated SAS HBA / driver
What could we use that for in the future...
Forwarding
Supported Hardware SVC 4F2 Node 8F2 Node 8F4 Node 8G4 Node 8A4 Node CF8 Node

Replication Cache FlashCopy Mirroring Thin Provisioning Virtualization

Forwarding

First single system capable of more than 1M IOPS Low latency key factor 50us overhead Up to four Cluster Partnerships for Replication iSCSI 1Gbit SVC software upgrade strategy All users, all hardware now iSCSI capable Stretched Cluster (Split Cluster) HA Zero detect on new CF8 Hardware Latest Hardware (CF8) Legend X3350M2 Server Base 1x 4core 2.4Ghz Xeon (Nehelan) Virtual QPI replaces FSB Mdisk 24 GB Cache 4x 8Gbit Fibre Channel
2013 IBM Corporation

SCSI Initiator

A Brief History of SVC and Storwize Family

SCSI Target
Forwarding

A Brief History of SVC and Storwize Family

Replication Cache FlashCopy Mirroring

Introducing the Family

Thin Provisioning Virtualization
Forwarding

Compression Easy Tier

RAID
Forwarding

SCSI Initiator

24

2013 IBM Corporation

A Brief History of SVC and Storwize Family

What we need is an Enterprise stack for all...

If you built it.. they will come... Software function everyone wants it What if we could take what we've done for the last ten years, and make it available to all.. SVC is just software running on a server But what about drives, and RAID Remember that SAS driver Remember our SSA history Build a dual-controller integrated disk system Time to market... Just a year away... Lessons learnt... One code base, one build, one binary...
2013 IBM Corporation

A Brief History of SVC and Storwize Family

IBM SVC, Storwize v6.1.0 - 2010

SCSI Target Continued OS and controller support ++ RAID Taken from SSA RAID firmware SAS already added for CF8 Flash Drives New SAS network topologies Enclosure Services SES integration, RAS EasyTier Optimal use of 5% Flash capacity PCIe Driver Inter-node communications inside enclosure Latest Hardware V7000 Dual node SVC inside 2U Enclosure with disks
2013 IBM Corporation

Supported Hardware SVC 4F2 Node 8F2 Node 8F4 Node 8G4 Node 8A4 Node CF8 Node

Forwarding

Replication Cache FlashCopy Mirroring Thin Provisioning Virtualization

Forwarding

Easy Tier
Supported Hardware Storwize V7000

RAID
Forwarding

Legend

SCSI Initiator

Virtual Mdisk Drive

A Brief History of SVC and Storwize Family

IBM SVC, Storwize v6.2.0, v6.3.0 - 2011

SCSI Target Continued OS and controller support ++ 10Gbit Ethernet iSCSI 10Gbit support on SVC and V7000 Storwize V7000 480 drive clusters Storwize V7000 Unified NFS, CIFS, SMB, HTTP, SCP file modules Latest Hardware (SVC CG8) X3350M3 Server Base 1x 6core 2.4Ghz Xeon (Westmere) QPI 6.4GT 24 GB Cache 4x 8Gbit Fibre Channel
Forwarding
Supported Hardware SVC 8F2 Node 8F4 Node 8G4 Node CF8 Node CG8 Node

Replication Cache FlashCopy Mirroring Thin Provisioning Virtualization

Forwarding

Easy Tier
Supported Hardware Storwize V7000 V7000 Unified

RAID
Forwarding
Legend

SCSI Initiator

Virtual Mdisk Drive

2013 IBM Corporation

A Brief History of SVC and Storwize Family

IBM SVC, Storwize v6.4.0, v6.4.1 - 2012

SCSI Target Continued OS and controller support ++ FCoE Support Using existing 10Gbit HBAs Volume Migration NDVM Volume move between IO/Groups without disruption 960 Drive V7000 Clusters Real-time Compression Temporal Locality (I promised you a time machine) Storwize V3500, V3700 Entry level controller
2013 IBM Corporation

Supported Hardware SVC 8F2 Node 8F4 Node 8G4 Node CF8 Node CG8 Node

Forwarding

Replication Cache FlashCopy Mirroring Thin Provisioning Virtualization

Forwarding

Compression Easy Tier

Supported Hardware Storwize V3500 V3700 V7000 V7000 Unified

RAID
Forwarding
Legend

SCSI Initiator

Virtual Mdisk Drive

A Brief History of SVC and Storwize Family

SCSI Target
Forwarding

A Brief Future of SVC and Storwize Family

Replication Cache FlashCopy Mirroring

The Future...
Thin Provisioning Virtualization
Forwarding

Compression Easy Tier

RAID
Forwarding

SCSI Initiator

29

2013 IBM Corporation

A Brief History of SVC and Storwize Family

2003 We were Here

SCSI Target 1st Hardware model 3 Features : Replication (MM) Cache

Peer Communications

Metro Mirror, FlashCopy and Migration (Virtualization)

Interface Layer

Configuration

1 Block protocol :
Fibre Channel

Clustering

FlashCopy

Potential to be: Extensible Flexible Augmentable Was the architecture right?

Virtualization

SCSI Initiator
Fibre Channel

2013 IBM Corporation

A Brief History of SVC and Storwize Family

2013 We are Here

SCSI Target 12 New hardware models running the same software* 8 new major features :
Global Mirror, Volume Mirrroing, Thin Provisioning, Stretched Cluster, EasyTier, Compression, RAID, Global Mirror with Change Volumes Forwarding

Replication Cache

Peer Communications

Interface Layer

Configuration

Clustering

FlashCopy Mirroring Thin Provisioning Virtualization

Forwarding

6 Major and 17 Minor Software releases :

(1.1, 1.2, 2.1, 2.1.1, 3.1, 4.1, 4.1.1, 4.2, 4.3, 4.3.1, 5.1, 6.1, 6.2, 6.3, 6.4, 6.4.1 7.1)

Compression Easy Tier

4 new block protocols:

iSCSI, FCoE, PCIe, SAS

RAID
Forwarding

Extensible, flexible, augmentable - YES! Get the platform and software architecture and philosophy right on day one, and there are no limits or boundaries
2013 IBM Corporation

SCSI Initiator
Fibre Channel iSCSI FCoE SAS PCIe
* Only 4F2 hardware limited to running no later than 5.1 Software due to 32bit CPU

A Brief History of SVC and Storwize Family

SVC and Storwize Family Into the second decade

CG8 Hardware Enhancements - 1H13 2x FC ports (16x 8Gbit per node pair) 12 core CPU per node 3x Compression throughput FlashSystem + SVC Continues the evolution of SVC as the ultimate performance and function platform Greater than 1M Production IOPs in half a rack More information this week : SV-1038A Optimizing Virtual Storage Performance SV-1218A IBM Storage Virtualization Product Directions PU-1670A IBM SAN Volume Controller and Storwize Family: What's New

2013 IBM Corporation