IBM SAP customer experiences August 2006

SAP systems in a customer datacenter on a virtualized IBM POWER5 environment

Irene Hopf International SAP IBM Competence Center Walldorf, Germany Version 1.2 01st August 2006

Operations of SAP systems virtualized on POWER5 Page 2

1. Preface
Edition Notice (August 2006) This is the second published edition of this document. Preface & Scope The objective of this paper is to report from productive customer environments, which run their SAP operations on a virtualized server and storage environment. It should provide best practices information to IBM pre-sales personnel. Special Notices Copyright IBM Corporation, 2006. All Rights Reserved. All trademarks or registered trademarks mentioned herein are the property of their respective holders. The author of this document is: Irene Hopf, Senior IT Architect, Int. SAP IBM Competence Center, Walldorf, Germany Acknowledgements With special thanks for their considerable contributions to the customer and the Andreas Lautensack from c.a.r.u.s. (IBM business partner) involved. Contact for the realization of the utilization analysis: Andreas Lautensack Senior IT Consultant c.a.r.u.s. Hannover Phone +49 511 626261 14 Mobile +49 176 - 1 62626 14 eMail Andreas.Lautensack@carus.de With special thanks to Nicola Krieger, eServer Solutions, IBM Germany who prepared the opportunity to write this document. Feedback We are interested in your comments and feedback. Please send these to isicc@de.ibm.com. Disclaimer See copyrights and trademarks. Edition History Version 1.0: Original Version Version 1.1: Chapter 4.1: Correction regarding CPU entitlement and high level LPAR design for productive SAP systems. Version 1.2: Public Version

Operations of SAP systems virtualized on POWER5 Page 3 Table of Contents

1. 2.
2.1 2.2 2.3

Preface ____________________________________________________________________ 2 Introduction _________________________________________________________________ 4

Objective__________________________________________________________________ 4 Background _______________________________________________________________ 4 Executive Summary _________________________________________________________ 4

3.
3.1 3.2 3.3 3.4 3.5

Infrastructure Architecture _____________________________________________________ 5

Hardware:_________________________________________________________________ 5 Disk Subsystem:____________________________________________________________ 6 Software release combination information ________________________________________ 6 Business Applications _______________________________________________________ 7 LPAR Configuration:_________________________________________________________ 9

4.
4.1 4.2 4.3 4.4 4.5 4.6

Operation Considerations _____________________________________________________ 11

Load Balancing / Performance / Scalability_______________________________________ 11 Availability (Disaster Recovery) _______________________________________________ 12 Data Protection (Backup / Recovery) ___________________________________________ 12 Security _________________________________________________________________ 12 Monitoring _______________________________________________________________ 13 Billing ___________________________________________________________________ 19

Operations of SAP systems virtualized on POWER5 Page 4

2. Introduction
2.1 Objective
IBM Customers and service providers externally and internally frequently ask questions like Can I run my productive systems without significant risks on a virtualized server environment using the virtual I/O server without using dedicated I/O adapters? or What is the savings potential of using shared processor LPAR (Micro-Partitioning) technology for all my SAP systems? This document gives an idea how to judge these questions and describes real life examples considering best practices in IT operations.

2.2 Background
For roughly 2 years the IBM POWER5 technology is available on the market and customers are still not leveraging the available flexibility and virtualization features of this platform. To encourage the presales colleagues to recommend the benefits of p5 to their customers, this document describes how customers manage the technology successfully. The IBM customer does not want to be mentioned and will be called IT Shop in this document. The data basis in this document was gathered in December of 2005. In the daily operations of the IT Shop roughly 90 to 100 SAP systems in the banking industry are managed. IT services are delivered internally, to subsidiaries and customers via service level agreements.

2.3 Executive Summary

The way the customer IT Shop organized the IT business provides extreme flexibility and potential to react to business changes within the magnitude of 30 minutes or less. The SAP systems (and a few non-SAP systems), which are distributed over 4 p570 systems run smoothly and with constant good performance. The SAP systems see their peak workloads at different points in time and therefore the capacity of the system is assigned to the needs of the applications with a minimum of administration effort. Additional buffer capacity is available by providing 4 CPUs per system as capacity upgrade on demand potential to the active 12 CPUs in each machine. The risk of capacity shortage is set to a minimum. The service level agreements can be easily kept and the customers of the IT Shop get an appropriate billing of the consumed IT capacity in terms of CPU consumption (managed SAPS), disk and tape space. The system setup provides the potential for synergy for the IT Shop and their customers. It is not necessary to plan always new hardware if the initiative for a new application arises. A test- or sandboxLPAR can easily and quickly be defined and the new system is available for prototyping for the application departments. This works very well for CPU sharing and virtualization. Memory still needs to be assigned from the start and adjusted manually. Integrated monitoring features in the central CCMS, which is widely used, help in the daily business and planning of the growth in the datacenter management.

Operations of SAP systems virtualized on POWER5 Page 5

3. Infrastructure Architecture
3.1 Hardware:
4 x p570 each with 16 CPUs (1.65GHz) Each system (in the sense of one pSeries machine) is equipped with: 12 active CPUs (4 CPUs are in stand by mode for capacity upgrade on demand) 256 GB Main Memory, 168 GB active 16 x 2GBit FC Adapter SAN-Connectivity (8 active) 8 x 2-Port GB-Ethernet RZ-Backbone (redundant) administration network 2 data centres (DC1 and DC2) in 700 m distance are both equipped with 2 p570 systems installed in one rack. Each location has 2 p570 systems, the Hardware Management Console (HMC) and the Cluster Systems Management (CSM) server in the same rack.
Datacenter 1
server H C R U6 IB M

Datacenter 2
server H C R U6 I BM

System 1

server

pSer ies IBM server pSeries I BM

System 3

server H C R U6 server pSer ies

IB M

server H C R U6 server

I BM

IBM

pSeries I BM

server H C R U6 server pSer ies

IB M

server H C R U6

I BM

I BM

GB Ethernet

server

pSeries I BM

server server H C R U6 server server pSer ies I BM pSeries IB M H C R U6

IBM

I BM

FC SAN

server H C R U6 server pSer ies

IB M server H C R U6 IBM server pSeries IBM IBM

FC SAN

server H C R U6 server pSer ies

IB M server H C R U6 IBM server pSeries IBM IBM

server H C R U6 server pSer ies

IB M server H C R U6 I BM server pSeries IBM IBM

server H C R U6 server pSer ies

IB M

server H C R U6 server pSeries

I BM

I BM

IBM

HMC System 2 System 4

HMC

Datacenter 1: System 1 and System 2

Datacenter 2: System 3 and System 4

High level LPAR Design (85 LPARs) All LPARs are configured as Shared Processor LPARs in uncapped mode Each frame has 2 x virtual I/O servers with 4 x FC and 2 x GB Ethernet Adapters each SAN connectivity of all LPARs is done via Virtual I/O (VIO) LAN connectivity of all LPARs is done via Virtual Ethernet

Operations of SAP systems virtualized on POWER5 Page 6

serv er H C serv er R U6

pSe ries

I B M IB M

serv er H C serv er R U6

pSe ries

I B M IB M

Datacenter 1 p570 system 1 and 2

Datacenter 2 p570 system 3 and 4

serv er H servC er R U6

pSe ries

I B M IB M

serv er H servC er R U6

pSe ries

I B M IB M

serv er H servC er R U6

pSe ries

I B M IB M

server H servC er R U6

pSe ries

I B M IB M

serv er H C serv er R U6

pSe ries

I B M IB M

serv er H servC er R U6

pSe ries

I B M IB M

serv er H C serv er R U6

pSe ries

I B M IB M

serv er H servC er R U6

pSe ries

I B M IB M

serv er H C serv er R U6

pSe ries

I B M IB M

serv er H servC er R U6

pSe ries

I B M IB M

serv er H C serv er R U6

pSe ries

I B M IB M

serv er H servC er R U6

pSe ries

I B M IB M

serv er H C serv er R U6

pSe ries

I B M IB M

serv er H servC er R U6

pSe ries

I B M IB M

Switch 1 Regular

Switch 2 Backup

Switch 3 Regular

Switch 4 Backup

Core Switch 1 incl. loopback

Core Switch 2 incl. loopback

Core Backbone

Fault tolerant SAN and LAN connectivity is setup through redundant switches and cross cabling. The picture shows the schematic setup of LAN connectivity CSM: There is one CSM server per location setup. It manages the following tasks: Central management of the hardware (e.g. power on, power off, etc.) Operating System Boot of all LPARs is done from the SAN and managed via Network Installation Manager (NIM). The OS images are identical on all LPARs running Repository of the SAP software

3.2 Disk Subsystem:

All data is stored on a SAN from Hitachi Data Systems. This is being managed by a different organizational unit in the IT Shop. This is setup in a redundant fashion across the 2 data center locations. For fault tolerance, host based logical volume mirroring with AIX LVM is used.

3.3 Software release combination information

Operating system: AIX 5.3 ML 03 VIO Server 1.20 Database: Oracle 9.2.0.6 SAP products: R/3 Systems (4.6C, 4.7 and WebAS 6.40 based), SRM (Enterprise Buyer Professional) Bank Analyzer and Business Intelligence (BW 3.1 and 3.5), see list below.

Operations of SAP systems virtualized on POWER5 Page 7

3.4 Business Applications

The IT Shop runs in general a classical SAP R/3 system landscape setup: development system (DEV), quality assurance (QAS) and production (PRD). Roughly 90 to 100 SAP systems are being managed in total: Application Group FI/CO/MM IT Shop internal Type PRD QAS DEV PRD QAS DEV PRD QAS DEV Education PRD Revision QAS DEV Sandbox PRD QAS DEV PRD QAS DEV DEV2 PRD QAS DEV DEV2 PRD QAS DEV PRD QAS DEV PRD DEV PRD QAS DEV PRD QAS DEV PRD QAS DEV QAS2 PRD QAS DEV PRD System Location System 2 System 4 System 2 System 4 System 2 System 4 POWER4 hardware SAP System SID P07 C03 C11 P33 C19 C17 P0H Q0H C0H S0H P41 A41 C53 C37 C52 P43 C54 C42 P56 Q56 C56 P91 Q91 C91 P11 C09 C05 P23 C23 C21 P31 C08 P60 C35 C33 P62 C71 C39 P72 C61 C07 Q72 P04 Q04 C04 P40 SAP Basis 4.6C

EBP IT Shop internal

6.20

HR IT Shop internal

4.6C

FI/CO Subsidiaries of IT Shop

BW IT Shop internal

IFRS*) BW IT Shop internal Reporting

System 3 System 4 System 1 System 2 System 3 System 1 System 3 System 1 System 3 System 2 System 2 System 3 System 2 System 4 POWER4 hardware System 4 System 2 System 4 System 2 System 2 POWER4 hardware System 1

6.20

6.20

6.40

IFRS*) Bank Analyzer Implementation IT Shop internal Cost Center Planning Regional Customer Real Estate

6.40

4.6C

FI/CO Customer North

4.6C

EBP Customer North HR Customer North

6.10 4.6C

IFRS*) BW Customer North Reporting IFRS*) Bank Analyzer Implementation Customer North Cost Center Planning CMS Customer North

6.40

System 1 System 4 System 4 System 2 System 4 System 4 System 1

6.40

6.40

BW Customer West

6.40

Operations of SAP systems virtualized on POWER5 Page 8 QAS DEV PRD QAS DEV PRD QAS DEV PRD QAS DEV PRD QAS DEV Sandbox 1 Sandbox 2 Sandbox 3 Sandbox3 System 3 System 1 System 1 System 3 System 1 System 1 System 3 System 1 System 3 System 1 System 3 System 3 C51 C50 P12 Q12 C12 P10 Q10 C10 P14 Q14 C14 T20 T99 T97 C27 T98

IFRS*) BW Customer South Reporting IFRS*) Bank Analyzer Implementation Customer South Cost Center Planning CMS Customer South

6.40

6.40

6.40

BW PoC new

6.40

Test Systems for Basis Purposes

XI Testsystem

System 2 System 2 System 4

4.6C

* IFRS (International Financing Reporting Standard formerly known as International Accounting Standard (IAS) mandatory for companies listed at stock exchange

Operations of SAP systems virtualized on POWER5 Page 9

3.5 LPAR Configuration:

Datacenter 1, System 1
Memory (in GB) CPU-Entitlement vCPU SAP-System LPAR ID Minimum Desired Maximum Desired Minimum Maximum desired min VIO-Server1 2 1 2 4 0,4 0,2 2 2 1 VIO-Server2 3 1 2 4 0,4 0,2 2 2 1 sapp43 13 4 8 16 0,6 0,2 4 4 1 sapp40 14 4 8 16 0,5 0,2 4 2 1 sapc50 15 4 4 8 0,2 0,1 2 1 1 sapc42 16 4 8 16 0,2 0,2 2 1 1 sapp72 17 8 20 32 1,6 0,4 4 8 2 sapp62 18 8 12 32 0,4 0,4 4 2 2 sapc53 19 2 4 8 0,2 0,1 2 1 1 P41_idle 20 8 20 20 0,8 0,4 4 4 2 P91_idle 21 8 16 16 0,8 0,4 8 8 4 P56_idle 22 8 16 16 0,8 0,4 8 8 4 sapa56 23 8 16 20 0,8 0,4 8 8 4 sapp10 24 4 8 8 0,2 0,1 2 1 1 sapp12 25 4 4 8 0,2 0,1 2 1 1 sapc10 26 4 8 8 0,1 0,1 2 1 1 sapc12 27 4 4 8 0,1 0,1 2 1 1 sapq14 28 4 4 8 0,1 0,1 2 1 1 P14_idle 29 4 4 8 0,1 0,1 2 1 1 max 4 4 8 4 2 4 8 8 2 4 12 12 12 4 4 2 2 2 2 Weighting uncapped 128 128 64 64 32 32 64 64 32 64 64 64 32 64 64 32 32 32 32

ME RZ1

168

256

8,5

4,2

66

57

Production Qualityassurance DR-Backup (idle LPAR) Virtual-I/O Server

Development

Datacenter 1, System 2
Memory (in GB) CPU-Entitlement vCPU SAP-System LPAR ID Minimum Desired Maximum Desired Minimum Maximum desired min VIO-Server1 2 1 2 4 0,4 0,2 2 2 1 VIO-Server2 3 1 2 4 0,4 0,2 2 2 1 P07 4 4 16 32 0,8 0,4 4 4 1 P31 5 4 6 12 0,4 0,2 1 2 1 S54 6 4 8 12 0,4 0,2 2 4 1 C23 7 4 8 16 0,4 0,2 0,8 2 1 C19 8 1 3 6 0,2 0,1 0,5 1 1 C11 9 2 4 8 0,2 0,1 0,5 1 1 C29 10 1 2 4 0,2 0,1 0,5 1 1 C37 11 2 3 6 0,2 0,1 0,5 1 1 P33_idle 12 1/4 6 12 0,1 0,1 1,2 1 1 P23_idle 13 1/4 1/2 18 0,1 0,1 2 1 1 T97 14 1 2 4 0,1 0,1 0,5 1 1 C56 15 2 6 6 0,1 0,1 0,5 1 1 C27 16 1 2 4 0,1 0,1 0,5 1 1 C08 17 1 2 4 0,1 0,1 0,5 1 1 IDMSXP2_idle 18 1 4 8 0,1 0,1 0,5 1 1 Q56 19 2 8 12 0,2 0,1 0,5 1 1 Q91 20 2 16 24 0,8 0,2 2 8 1 sa101cafm2 21 1 2 4 0,1 0,1 0,5 1 1 P04 22 4 8 16 0,2 0,1 0,5 2 1 K07_idle 23 1 4 4 0,1 0,1 0,2 1 1 P0H_APP 24 4 12 16 0,1 0,8 1 8 1 ME RZ1 110,5 216 5,6 3,0 23 48 max 4 4 32 10 8 8 5 5 5 5 4 6 5 5 3 5 5 5 12 5 5 2 10 Weighting uncapped 128 128 64 64 32 32 32 32 32 32 64 64 16 32 32 32 64 32 32 64 64 16 48

Production Qualityassurance DR-Backup (idle LPAR) Virtual-I/O Server

Development

Operations of SAP systems virtualized on POWER5 Page 10 Datacenter 2, System 3

SAP-System VIO-Server1 VIO-Server2 sapp41 sapc52 sapc51 P43_idle sapc54 sapp91 sapp56 P40_idle P72_idle P62_idle sapa91 sapp14 sapc14 sapq10 sapq12 P10_idle P12_idle sapsol ME RZ2 Memory (in GB) CPU-Entitlement vCPU LPAR ID Minimum Desired Maximum Desired Minimum Maximum desired min 2 1 2 4 0,4 0,2 2 2 1 3 1 2 4 0,4 0,2 2 2 1 13 16 20 32 0,8 0,4 4 4 2 14 4 8 16 0,2 0,2 2 2 1 15 4 4 8 0,2 0,1 2 1 1 16 4 8 8 0,2 0,1 2 4 1 17 4 8 16 0,4 0,2 2 4 1 18 8 20 32 0,8 0,4 8 8 4 19 8 16 32 0,8 0,4 8 8 4 20 4 8 8 0,2 0,2 2 2 1 21 8 20 20 0,2 0,2 2 2 1 22 8 12 12 0,2 0,2 2 2 1 23 8 16 20 0,8 0,4 8 8 4 24 4 4 8 0,2 0,1 2 1 1 25 4 4 8 0,1 0,1 2 1 1 26 4 4 8 0,1 0,1 2 1 1 27 4 4 8 0,1 0,1 2 1 1 28 4 4 8 0,1 0,1 2 1 1 29 4 4 8 0,1 0,1 2 1 1 30 4 8 12 0,2 0,1 2 1 1 176 272 6,5 3,9 60 56 max 4 4 8 4 2 4 8 12 12 2 2 2 8 4 2 2 2 2 2 2 Weighting uncapped 128 128 64 32 32 64 32 64 64 64 64 64 32 64 32 32 32 32 32 32

Production Qualityassurance DR-Backup (idle LPAR) Virtual-I/O Server

Development

Datacenter 2, System 4
Memory (in G) CPU-Entitlement vCPU SAP-System LPAR ID Minimum Desired Maximum Desired Minimum Maximum desired min VIO-Server1 2 1 2 4 0,4 0,2 2 2 1 VIO-Server2 3 1 2 4 0,4 0,2 2 2 1 P33 4 4 6 12 0,4 0,2 1,2 2 1 P23 5 4 9 18 0,5 0,3 2 3 1 CS-Test 6 1 2 4 0,1 0,1 0,4 1 1 C03 7 2 6 12 0,3 0,2 0,8 2 1 C17 8 1 2 4 0,2 0,2 0,5 2 1 C21 9 2 3 6 0,2 0,2 0,5 2 1 C07 10 2 6 12 0,4 0,2 0,9 2 1 P07_idle 11 1/4 16 16 0,4 0,1 4 4 1 P31_idle 12 4 6 12 0,4 0,2 0,8 2 1 I03 13 1 2 4 0,1 0,1 0,3 1 1 C04 14 3 8 16 0,1 0,1 0,5 1 1 T98 15 2 8 16 0,1 0,1 0,5 1 1 C91 16 2 6 12 0,1 0,1 0,5 1 1 IDMS-X 17 1 4 8 0,1 0,1 0,5 1 1 A41 18 1 4 8 0,1 0,1 0,5 1 1 K07 19 1 4 8 0,1 0,1 0,5 1 1 IDMSXP2 20 1 4 8 0,2 0,1 0,5 1 1 sideprod 21 4 5 6 0,1 0,1 0,5 1 1 Q04 22 4 8 16 0,1 0,1 0,5 1 1 P04_idle 23 1/4 8 16 0,2 0,1 0,5 2 1 Q72 24 4 20 20 1,6 0,4 2 8 2 ME RZ2 141 242 4,7 3,0 19,9 44 max 4 4 2 8 4 2 2 2 4 8 8 3 5 5 5 5 5 5 5 5 5 5 8 Weighting uncapped 128 128 64 64 32 32 32 32 32 64 64 32 32 16 32 32 32 16 64 64 32 64 32

Production Qualityassurance K-Backup (idle LPAR) Virtual-I/O Server

Development

Operations of SAP systems virtualized on POWER5 Page 11

4. Operation Considerations
4.1 Load Balancing / Performance / Scalability
The planning for a new system with 3 landscapes is done in the first step with a sizing exercise in the SAP Quicksizer. The result is a required SAPS value. Memory is also taken from the SAP Quicksizer or the regular rule of thumb with 4 to 8 GB per virtual CPU is applied. The SAPS capacity of one CPU is derived from the published and certified SAP SD standard application benchmark data (e.g. 1 POWER5 CPU 1.65 MHz ~ 1400 SAPS). According to the sum of CPUs in the 4 p570 shared LPARs 205 virtual CPUs are defined and there is room for more. In fact only 4 x 12 = 48 physical CPUs are available. One system provides 12 physical CPUs. This resource is being shared across all defined LPARs on the machine. Also the 2 virtual I/O servers (VIO) run on shared LPARs. All LPARs run in an uncapped mode. Even very small LPARs will get an uncapped configuration. Therefore they are ready to perform the nightly backup very quickly. In general a new defined LPAR gets at least 0.1 physical CPU entitlement (desired) and 1 virtual CPU. The productive LPARs get at least 0.2 CPU entitlement per 1 vCPU. This ensures that the productive LPARs get more guaranteed capacity than the other (non-productive) LPARs. The desired CPU entitlement is selected as low as possible in order to provide maximum flexibility for the hypervisor to assign granular portions of each physical CPUs to the various LPARs according to workload. The memory assignment happens once while basic configuration. After that, operations need to monitor and the memory consumption and manually adjust the dLPAR configuration. The flexibility which is given for the assignment of CPU resources within the system still has room for improvement for the memory allocation for the individual LPAR. For a completely new SAP system landscape to be used for production purposes the production and development LPAR are put into one datacenter and the quality assurance and the idle standby LPAR are put into the other datacenter. Another example in terms of flexibility is the HR system P0H which runs still on POWER4 hardware (p650 6way). This system experience capacity bottlenecks when the payroll run was performed. To solve this bottleneck specifically for the timeframe, these requirements occur, application servers on shared processor LPARs on the p570 systems are provided to the SAP system. 4.1.1 Virtual I/O Server (VIO) The VIO servers get the highest priority, because if those will not have enough resources, all other LPARs suffer from this. The VIO server is configured with 2 virtual CPUs (vCPU) where roughly one is being consumed by the customer network interface (Gigabit Ethernet) and the other one covers the SAN traffic doing I/O and the load on the administration network. Each VIO has 4 FC adapters to access the SAN. The client LPARs get access to disk always via both VIOs. Locally the path priority is set for all even hdisks (0,2,4,6,) to VIO1 and all odd hdisks (1,3,5,7,) to VIO2. Hence the IT shop achieves a load balancing half and half over each VIO or in other words: for each LPAR the I/O traffic is striped over 8 FC adapters. The I/O traffic from all the LPARs on a system adds up in the VIOs. Nevertheless the FC adapters have not seen maximum throughputs so far. The load balancing and the potential peak loads at different times create no bottlenecks on I/O.

Operations of SAP systems virtualized on POWER5 Page 12

4.2 Availability (Disaster Recovery)

Disaster recovery backup is realized through 2 data centres and hot standby LPARs for the production LPARs only. The service level agreements allow a manual take over after problem analysis. That is sufficient for the production systems, which are running. The idle LPAR is up and running, ready prepared with AIX and the according maintenance level. In case of a disaster a few tasks need to be executed, which are well documented and prepared by shell scripts which are readily available on the idle LPAR. The basic tasks are: Make sure that the normal environment is completely down and cleaned up (deactivate the network interface, unmount of filesystems, varyoffvg Assign the virtual hostname (and IP address) to the new (backup LPAR) physical hostname, Import the volume group (optionally forced) and varyonvg Mount of all necessary filesystems Start of the system on the idle (backup) LPAR Post-processing steps Only the basis administration systems are clustered via HACMP. These systems manage the central CCMS, saprouter and serve as central file server, e.g. for /usr/sap/trans.

4.3 Data Protection (Backup / Recovery)

For Backup IBM Tivoli Storage Manager (TSM) is used. The TSM server was recently moved to 2 separate p570 systems. One is located in datacenter 1 and the other is located in datacenter 2. Each TSM server occupies 6 CPUs when heavy backup load is performed. It is also possible to run the TSM server in an LPAR on the regular systems, which can be shared for CPU and memory but does the I/O via dedicated adapters to provide sufficient bandwidth. The procedures for the backup of the SAP relevant data and generic OS (makesysb, filesystems, joblogs etc.) did not change when moving from a dedicated to a virtualized environment. It is planned to use LAN free data backup in the near future. Some posting data is being archived via IBM Commonstore.

4.4 Security
Customers buy a service level agreement from the IT Shop in terms of availability (e.g. 99,5 %) and resources providing capacity in terms of SAPS. They do not care any longer on what type of hardware and whether dedicated or virtualized resources are provided. The security for the end users of the customers is provided via saprouter and the regular authentication with the SAPGUI access. The customers do not have access via telnet or similar tools on OS level. This approach has been considered sufficiently secure by the customers of the IT Shop. The system administrators at the IT Shop access the machines via ssh. All other ports are protected via a firewall.

Operations of SAP systems virtualized on POWER5 Page 13

4.5 Monitoring
Monitoring of the system utilization happens right now with AIX tools (sar) on an hourly average basis. sar collects data in 5 minute buckets and these are averaged over 1 hour. This was integrated into central CCMS via a custom written report. The motivation to use sar was that it works similarly on POWER4 and POWER5 technology and it utilizes already POWER5 virtualization information with physc. The analysis from central CCMS shows that the systems still provide buffer in capacity even considering the variations in workload coming from the different applications. The following charts show on the y-axis the number of physical CPUs used on one p5 system (physc). The maximum is 12 physical CPUs in each system. The x-axis shows the date and time from 28.11.2005, 00:00 until 04.12.2005, 24:00. This includes month-end and a weekend (December 3rd and 4th)Each bar reflects the average of 1 hour (physc in sar). We see 2 different machines and 2 individual LPARs on each of the machines. The first chart is the overview over the whole system and the second and third chart are 2 exemplary LPARs on the machines. Datacenter 2 System 4

This is the overall usage of the whole system. The utilization reaches roughly 11.5 physical CPUs in the evening of November 30th. The system runs 21 LPARs plus 2 VIO LPARs.

Operations of SAP systems virtualized on POWER5 Page 14 System Q72 running in DC2 on system 4: SID Q72 CPU entitlement desired min 1.6 0.4 max 2 vCPU desired min 8 2 max 8

The system Q72 is rather heavily used for BW reporting tests within the Bank Analyzer and reaches 7.8 physical CPUs in the evening of November 30th. High utilization is also obvious over the weekend when a stress test was performed. The application uses heavy parallelization techniques.

Operations of SAP systems virtualized on POWER5 Page 15 System P23 running in DC2 on system 4: SID P23 CPU entitlement desired min 0.5 0.3 max 2 vCPU desired min 3 1 Max 8

The system P23 is a relatively small production system which shows a peak usage of 1.3 CPUs. Every day early in the morning a regular load appears and during the day the usual camel hump curve (peak in the morning peak in the afternoon) can be observed. The system serves remote locations which use financial and controlling applications.

Operations of SAP systems virtualized on POWER5 Page 16 Datacenter 2 System 3

The system very occasionally reaches a high utilization of 11.7 physical CPUs. Over the weekend (December 3rd and 4th) the system is roughly utilized only with 0,5 CPUs on the whole machine. There is lots of capacity available for other LPARs for future applications on this machine.

Operations of SAP systems virtualized on POWER5 Page 17 System P41 running in DC2 on system 3: SID P41 CPU entitlement desired min 0.8 0.4 max 4 vCPU desired min 4 2 max 8

The system P41 shows a relatively constant usage of 2 physical CPUs for roughly 2 days. Other than that the system only shows occasional usage which look like regular batch jobs. The system P41 is the production system which serves the subsidiaries with FI, CO and MM applications. Right now this system runs roughly 30 SAP clients (= Mandanten) which is planned to increase to 180 in the future.

Operations of SAP systems virtualized on POWER5 Page 18 System C14 running in DC2 on system 3: SID C14 CPU entitlement desired min 0.1 0.1 max 2 vCPU desired min 1 1 max 2

C14 is a small test system of the customer south of the IT Shop for collateral handling and risk mitigation. It shows very little workload in the given timeframe. The entitlement of 0.1 CPU is hardly used. The hypervisor reduces the consumed capacity down to 0.02 or 0.03 CPUs frequently.

Operations of SAP systems virtualized on POWER5 Page 19

4.6 Billing
The billing is being done centrally based on the monitoring data. CPU consumption, disk space and tape space consumption is also calculated. The customers of the IT Shop will get a regular constant bill (e.g. monthly) and then a balance at certain points in time (e.g. quarterly or yearly). This is also based on the central CCMS data and is being created automatically. Using information from CCMS and AIX it is possible to perform accounting in a virtualized environment.

Operations of SAP systems virtualized on POWER5 Page 20

Legal Information
IBM Corporation 2006 IBM Corporation Westchester Avenue White Plains, NY 10604 USA ibm.com Produced in Germany September 2006 All Rights Reserved IBM, the IBM logo, eServer, Micro-Partitioning, POWER, POWER4, POWER5, pSeries, AIX and Tivoli are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. For a list of additional IBM trademarks, please see http://ibm.com/legal/copytrade.shtml. SAP, the SAP logo and all other SAP products and services mentioned herein are trademarks or registered trademarks of SAP AG in Germany and several other countries. Hitachi Data Systems is a registered trademark of Hitachi Ltd. in the United States and other countries. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other company, product or service names may be trademarks or service marks of others.

IBM reserves the right to change specifications or other product information without notice. References in this publication to IBM products or services do not imply that IBM intends to make them available in all countries in which IBM operates. IBM PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions; therefore, this statement may not apply to you. This publication may contain links to third party sites that are not under the control of or maintained by IBM. Access to any such third party site is at the user's own risk and IBM is not responsible for the accuracy or reliability of any information, data, opinions, advice or statements made on these sites. IBM provides these links merely as a convenience and the inclusion of such links does not imply an endorsement. Information in this presentation concerning non-IBM products was obtained from the suppliers of these products, published announcement material or other publicly available sources. IBM has not tested these products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. MB, GB and TB = 1,000,000, 1,000,000,000 and 1,000,000,000,000 bytes, respectively, when referring to storage capacity. Accessible capacity is less; up to 3GB is used in service partition. Actual storage capacity will vary based upon many factors and may be less than stated. Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will depend on considerations such as the amount of multiprogramming in the users job stream, the I/O configuration, the storage configuration and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. Maximum internal hard disk and memory capacities may require the replacement of any standard hard drives and/or memory and the population of all hard disk bays and memory slots with the largest currently supported drives available. When referring to variable speed CD-ROMs, CD-Rs, CD-RWs and DVDs, actual playback speed will vary and is often less than the maximum possible. SPW00109-DEEN-00