v

© 2011 IBM
1

Exploiting Virtualisation
on IBM Power Systems
with PowerVM v

Nigel Griffiths
IBM Power Systems
Advanced Technology Support
EMEA

© 2011 IBM
2

Processor Virtualisation (CPU Sharing)

Old Style
Separate
Systems

Pre-2000
 v
© 2011 IBM
3

Processor Virtualisation (CPU Sharing)

Old Style LPAR

Separate Server
Systems Consolidation
LPAR size
via start time
boundaries

Pre-2000 ~2001

© 2011 IBM
4

Processor Virtualisation (CPU Sharing)

Old Style LPAR DLPAR

Separate Server Dynamic
Systems Consolidation live boundary
LPAR size changes
via start time Manual or
boundaries scripts

Pre-2000 ~2001 ~2002

 v
© 2011 IBM
5

Processor Virtualisation (CPU Sharing)

Old Style LPAR DLPAR SPLPAR

Separate Server Dynamic Shared Processor
Systems Consolidation live boundary automatic adjusts
LPAR size changes at millisecond level
via start time Manual or by Hypervisor
boundaries scripts

Pre-2000 ~2001 ~2002 ~2005

© 2011 IBM
6

Processor Virtualisation (CPU Sharing)

SPLPAR
Shared Processor
automatic adjusts
at millisecond level
by Hypervisor

~2005
 v
© 2011 IBM
7

Processor Virtualisation (CPU Sharing)

SPLPAR
Shared Processor
automatic adjusts
at millisecond level
by Hypervisor

~2005

© 2011 IBM
8

Processor Virtualisation (CPU Sharing)

SPLPAR Harvesting
Shared Processor “Spare” capacity
automatic adjusts ready for adding
at millisecond level more workloads at
by Hypervisor zero hardware cost

~2005 ~2006
 v
© 2011 IBM
9

Processor Virtualisation (CPU Sharing)

SPLPAR Harvesting Partition Mobility

Shared Processor “Spare” capacity Make a cluster of
automatic adjusts ready for adding your machines &
at millisecond level more workloads at flow your workload
by Hypervisor zero hardware cost between them

~2005 ~2006 ~2008

© 2011 IBM
10

Virtualisation - Value Proposition

 v
© 2011 IBM
11

Virtualisation - Value Proposition

}
10%
10% 40%
10% 40%
10%
10%
10%
10%
10%

Economical I/O Model

© 2011 IBM
12

Virtualisation - Value Proposition

}
10%
10% 40%
10% 40%
10%
10%
10%
10%
10%

Economical I/O Model

Virtual

SAN + Net
Cables
Switches
Ports
+ Power

Reduced Infrastructure
 v
© 2011 IBM
13

Virtualisation - Value Proposition

“Your new system will be ready in …”

}
“20 Minutes” or “20 Days”
10%
10% 40%
10% 40%
10%
10%
10%
10%
10% Quick Deployment
Economical I/O Model

Virtual

SAN + Net
Cables
Switches
Ports
+ Power
Reduced Infrastructure

© 2011 IBM
14

Virtualisation - Value Proposition

“Your new system will be ready in …”

}
“20 Minutes” or “20 Days”
10%
10% 40%
10% 40%
10%
10%
10%
10%
10% Quick Deployment
Economical I/O Model

Virtual

SAN + Net
Cables
Switches
Ports
+ Power

Reduced Infrastructure Server Consolidation

 v
© 2011 IBM
15

Virtual Adapters

The I/O centric view of the world:

CPU used to “modify & feed data”
to the networks & disks

Network Disk

© 2011 IBM
16

Virtual I/O Server (Adapter Sharing)

Network

Disk

Pure Direct
 v
© 2011 IBM
17

Virtual I/O Server (Adapter Sharing)

Production Direct
Network

the rest Virtual

Disk

Network

Disk
Pure Direct

VIOS

© 2011 IBM
18

Virtual I/O Server (Adapter Sharing)

Production Direct
Network

the rest Virtual

Disk

Network

Virtual except
Disk

Pure Direct
high I/O LPARs
Network

VIOS
Disk

VIOS

VIOS
 v
© 2011 IBM
19

Virtual I/O Server (Adapter Sharing)

Production Direct
Network

the rest Virtual

Disk

Network

Virtual except

Disk
Pure Direct
high I/O LPARs

Network
VIOS

Disk
VIOS
Pure Virtual

VIOS

Network
VIOS

Disk
VIOS

Reduced Cost

© 2011 IBM
Where 20

are You?
A B C D E F

4
 v
© 2011 IBM
Where 21

are You?
A B C D E F

© 2011 IBM
Where 22

are You?
A B C D E F

4
 v
© 2011 IBM
Where 23

are You?
A B C D E F

© 2011 IBM
24

Storage Virtualisation since 2005

VIO client

Generic SCSI disk

Virtual SCSI
Adapters
SCSI
SAS vSCSI VIOS

Storage
FC Adapters Virtualiser

SAN

EMC LUN IBM LUN

 v
© 2011 IBM
25

Virtual CD/DVD & Virtual Tape

VIO client

Generic SCSI disk

Virtual SCSI
Adapters
SCSI
SAS vSCSI VIOS
Virtual Tape: Easy to
Storage
FC Adapters Virtualiser move between LPARs

.iso library
SAN Virtual:
CD & DVD Read-only
DVD-RAM Read-Write
EMC LUN IBM LUN

© 2011 IBM
26

Storage Virtualisation with N-Port ID Virtualisation

VIO client VIO client

Generic SCSI disk Note EMC LUN IBM LUN

Virtual SCSI Virtual FC

Adapters Adapters
SCSI
SAS vSCSI VIOS VIOS

Storage Pass Through

FC Adapters Virtualiser FC Adapters mode

VIOS Admin
in charge

SAN NPIV SAN SAN Admin

Back in charge

EMC LUN IBM LUN EMC LUN IBM LUN

 v
© 2011 IBM
27

Dual VIOS since 2001 allows concurrent updates

supporting 2 to 200+ Logical Partitions

SCSI SCSI
SAS SAS

VIOS VIOS

SAN

Noticed: CPUs too fast & memory bloat

© 2011 IBM
28

Dual VIOS since 2001 allows concurrent updates

supporting 2 to 200+ Logical Partitions

VIO client VIO client VIO client VIO client

SCSI SCSI
SAS SAS

VIOS VIOS

SAN

Noticed: CPUs too fast & memory bloat

 v
© 2011 IBM

What about Virtual Memory!

29

Virtual Memory available in UNIX to 30+ years

Modern Software need extreme amounts of memory

Power has two mechanisms to increase memory use

- Active Memory Expansion (AIX only)
- Active Memory Sharing

- AIX Workload Partition can also save memory

© 2011 IBM

AME Conceptual Model

30

Active Memory Expansion Compressed

Memory Pages Memory pages

Dynamically adjusted depending on

compression ratio & target
 v
© 2011 IBM

AME - What is your Plan?

31

Memory Shrinking Memory Growing

to release RAM for other uses for RAM optimisation & performance

10GB 10GB

But want 14GB to

improve performance

Looks like 10GB but

is actually 8GB,
thanks to AME

2GB released Actually still using 10 GB but

for another LPAR looks like 14 GB, thanks to AME

© 2011 IBM
32

Classic Virtual Memory (LPAR)

Virtual Memory

Physical Memory

Not Really Here

Ac
tua
ysll
tor
ed

Tiny 4 KB pages
o
nd
is k
 v
© 2011 IBM
33

Active Shared Virtual Memory (LPAR)

Virtual Memory

Logical Memory

Not Really Here

Ac
tua
ysll
Hypervisor

tor
ed
Blue Physical Loaned Memory

o
nd
Memory Pages To Hypervisor

is k
VIOS

© 2011 IBM
34

Active Memory Sharing (Virtual Virtual Memory!)

VIO client VIO client VIO client VIO client

SCSI SCSI
SAS SAS

VIOS VIOS

SAN
 v
© 2011 IBM
35

Active Memory Sharing (Virtual Virtual Memory!)

VIO client VIO client VIO client VIO client

SCSI SCSI
SAS SAS

VIOS VIOS

SAN
Loaning Mode

© 2011 IBM
36

Active Memory Sharing (Virtual Virtual Memory!)

VIO client VIO client VIO client VIO client

SCSI SCSI
SAS SAS

VIOS VIOS

SAN
 v
© 2011 IBM
37

Active Memory Sharing (Virtual Virtual Memory!)

VIO client VIO client VIO client VIO client

SCSI SCSI
SAS SAS

VIOS VIOS

SAN
Stealing Mode

AME & AMS Comparisons © 2011 IBM

38

Active Memory Expansion Active Memory Sharing

• Jan 2010
• AIX6 TL4+ on POWER7
• Not Linux nor IBM I
• Machine Activation (LPP)
• 60 day trial
• Pure Virtual LPAR
• Internal to single LPARs
• Assume “spare” CPU cycles
for compression
• Simple to setup in LPAR
• Use amepat to predetermine
the compression factor
• Use topas/nmon to monitor
• May 2009
• POWER6
• AIX6 TL3+, Linux & IBM i 6.1
• PowerVM Enterprise
• Pure Virtual LPAR
• Cooperating group of LPARs
• Assumes loanable RAM
• Pages flow between LPARs at
a few MB/s
• More complex to setup on
VIOS & LPARs
• Use topas –C to monitor
 v
© 2011 IBM
39

Summary so far ….

Dedicated World: still available Virtual World

Dedicated CPU
Dedicated Virtual Memory
Dedicated Disk adapters - SCSI/SAS/SAN
Dedicated Network adapters
NFS or NIM !
CPU load balancing in minutes/hours
Memory load balancing in minutes/hours
Virtual CPUs
Shared Virtual Memory (AMS) + AME
Virtual Disks (vSCSI & NPIV)
Virtual Networks
Virtual CD/DVD & Tape & NFS & NIM
CPU load balancing in milli-seconds
Active Memory Sharing between
LPARs in seconds

Live Partition Mobility

© 2011 IBM
40
ho =

Summary so far ….
”
sc ive

Dedicated World: still available Virtual World

ib &
ol

Dedicated CPU Virtual CPUs

l d ns

Dedicated Virtual Memory Shared Virtual Memory (AMS) +AME

Fl ap
le

Dedicated Disk adapters - SCSI/SAS/SAN Virtual Disks (vSCSI & NPIV)

Dedicated Network adapters Virtual Networks
“ O pe

he

NFS or NIM ! Virtual CD/DVD & Tape & NFS & NIM
CPU load balancing in minutes/hours CPU load balancing in milli-seconds
ex
Ex

Memory load balancing in minutes/hours Active Memory Sharing between

C

LPARs in seconds

Live Partition Mobility

 v
Live Partition Mobility (LPM) © 2011 IBM
41

High levels of efficient Virtualisation means

• Hypervisor support (system firmware)
• Virtual I/O Server (VIOS)
• HMC for management
• Operating System support
For new features & fixes … updates are required
LPM allows for zero downtime updates

Also allows
1. Workload balancing across machine room
2. New machine use at day 1
3. Repair actions & upgrades

Live Partition Mobility (1 of 6)

hdisk0
Server Logical Partition Server
A B
vscsi0 en0

Hypervisor

vhost0 hdisk1 en1 en1 hdisk1

vtscsi0 en2 en2
Virtual I/O Server en2
SEA en2 SEA
fcs0 en0 en0 fcs0
HMC

Ethernet Network

Storage Area Network

LUN
Live Partition Mobility (2 of 6)

hdisk0 hdisk0
Server Server
A B
vscsi0 en0 en0 vscsi0

AR
LP
w
en1 ne en1
vhost0 hdisk1 e hdisk1
e at
vtscsi0 en2 en2 Cr en2
SEA en2 SEA
fcs0 en0 en0 fcs0
HMC

Ethernet Network

Storage Area Network

LUN

Live Partition Mobility (3 of 6)

hdisk0 hdisk0
Server Server
A B
vscsi0 en0 en0 vscsi0

vhost0 hdisk1 en1 en1 hdisk1 vhost0

vtscsi0 en2 en2 en2 vtscsi0
SEA en2 SEA
l Ada pter
Virtu a
fcs0 en0
HMC ing en0 fcs0
mapp

Ethernet Network

Storage Area Network

LUN
Live Partition Mobility (4 of 6)

hdisk0 hdisk0
Server Server
A Logical Memory Copy B
vscsi0 en0 en0 vscsi0

vhost0 hdisk1 en1 en1 hdisk1 vhost0

vtscsi0 en2 en2 en2 vtscsi0
SEA en2 SEA
fcs0 en0 en0 fcs0
HMC

Memory Copy over network

Storage Area Network

LUN

Live Partition Mobility (5 of 6)

hdisk0 hdisk0
Server Server
A B
vscsi0 en0 Move program execution to the target en0 vscsi0

vhost0 hdisk1 en1 en1 hdisk1 vhost0

vtscsi0 en2 en2 en2 vtscsi0
SEA en2 SEA
fcs0 en0 en0 fcs0
HMC

Storage Area Network

LUN
Live Partition Mobility (6 of 6)

hdisk0
Server Server
A B
en0 vscsi0

hdisk1 en1 LPAR en1 hdisk1 vhost0

rem
en2 ova en2
en2 l en2 vtscsi0
SEA SEA
fcs0 en0 en0 fcs0
HMC

Ethernet Network

Storage Area Network

LUN

Suspend and Resume © 2011 IBM

48

Resource balancing
- suspend low-priority for more urgent processes

Simplified maintenance
- suspend the whole machine system updates

Debug/forensics
- a workload can be temporarily suspended
- a copy made for offline analysis for security
or performance purposes
Suspend 1) Freeze the LPAR

hdisk0
Server Server
A B
3) Remove the LPAR en0 vscsi0

2) Copy memory to the VIOS disk

en1 en1 hdisk1 vhost0
en2 en2 en2 vtscsi0
SEA en2 SEA
en0 en0 fcs0
HMC

Ethernet Network

Storage Area Network

LUN

Resume
4) Recreate the LPAR
hdisk0
Server Server
A B
vscsi0 en0 6) Unfreeze the LPAR

5)
Co
p ym
vhost0 hdisk1 en1 em en1
or
vtscsi0 en2 y fr o en2
en2 en2
SEA m SEA
the
fcs0 en0 V IO en0
HMC
S
dis
k

Ethernet Network

Storage Area Network

LUN
IBM Power Systems

PowerVM Editions are tailored to client needs

PowerVM Editions PowerVM Editions Express Standard Enterprise
offer a unified VIOS + 10 per core 10 per core
Concurrent VMs
virtualization 2 per VMs (up to 1000) (up to 1000)
solution for all Virtualization Management IVM IVM, HMC IVM, HMC
Power workloads
Virtual I/O Server

PowerVM Express Edition PowerVM Lx86

– Evaluations, pilots, PoCs Suspend/Resume
– Single-server projects
Shared Processor Pools
PowerVM Standard Edition
Shared Storage Pools
– Production deployments
– Server consolidation Thin Provisioning

Live Partition Mobility

PowerVM Enterprise Edition
– Multi-server deployments Active Memory Sharing
– Advanced Functions
*

•IBM i supports shared storage and Suspend &

Resume, but does not support LPM. © 2011 IBM Corporation

© 2011 IBM
52

Virtual Disks 5 type - Reminder

Local Disks
SCSI/SAS/SSD
3 SAN LUN
1 Logical Volume
FibreChannel
from VG Pool
Virtual
I/O Server
2 Whole Local Disk NPIV
over vFC
5 NPIV LUN

4 File-Backed from a vSCSI

file system (pool)
based on local or
Client
SAN disk(s) VM

Client
VM
 v
© 2011 IBM
Shared
Shared 53
Storage
Now one more Storage
Pool
Phase 1 Single VIOS
Pool Thin Provisioning
6 SAN LUN Disks
Local Disks
SCSI/SAS/SSD
3 SAN LUN
1 Logical Volume
FibreChannel
from VG Pool
Virtual
I/O Server
2 Whole Local Disk NPIV
over vFC
5 NPIV LUN

4 File-Backed from a vSCSI

file system (pool)
based on local or
Client
SAN disk(s) VM

Client
VM

Shared Storage Pool © 2011 IBM

distributed cluster Cluster
54

co-operating VIOS’s Pools Disks Repository

FibreChannel

Network
Virtual Virtual
Virtual
I/O Server I/O Server
I/O Server

vSCSI vSCSI
vSCSI

Client Client
VM Client
VM VM
Phase 2
 v
Shared Storage Pool © 2011 IBM
distributed cluster Cluster
55

co-operating VIOS’s Pools Disks Repository

1. Thin & Thick Provisioning FibreChannel

2. Disk online to VIOS only
3. Very simple LPM setup
4. Data Migration
Network
5. Snapshot and Cloning disks
Virtual Virtual
Virtual
6. Linked Clones
I/O Server I/O Server
I/O Server
7. GUI via Systems Director

vSCSI vSCSI
vSCSI

Client Client
VM Client
VM VM
Phase 2

Power Systems
v

© 2011 IBM
56
Power 795
Since September 2010
Power 780

Power 770

Power 750 256 core

Power 740 64 core
Power 720
32 core

4 - 16 core
700/1/2 Blades Power 730
Power 710

4 - 16 core
8 or 16 core