Professional Documents
Culture Documents
Contents
CHAPTER 1: DEPLOYING ORACLE 11G ON AIX 6.1 WITH IBM DS3400, DS4000 SERIES,
OR DS5000 SERIES STORAGE SUBSYSTEMS
Benefits of Oracle RAC 11g Release 1 Database....................................................................................1-2
Example Configuration ..................................................................................................................................1-3
Configuring the Hosts.....................................................................................................................................1-5
Configuring the IBM Storage Subsystems...............................................................................................1-9
Installing Oracle ............................................................................................................................................. 1-11
Creating Directories for Oracle........................................................................................................... 1-12
Setting Up the Oracle User's Profile ................................................................................................. 1-12
Installing Oracle Clusterware.............................................................................................................. 1-12
Installing and Configuring ASM......................................................................................................... 1-12
Installing an Oracle Database............................................................................................................. 1-13
Configuring Listener Names ............................................................................................................... 1-13
Creating and Configuring a Database............................................................................................. 1-13
Conclusion ....................................................................................................................................................... 1-14
Contact Information..................................................................................................................................... 1-14
APPENDIX A: REFERENCES
AIX......................................................................................................................................................................... A-1
IBM Storage Subsystems .............................................................................................................................. A-1
Oracle................................................................................................................................................................... A-1
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
iii
iv
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
Chapter
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
1-1
Before you install Oracle Database 11g, you must know how to best configure the
storage subsystem for enhanced performance. For this information, refer to Best
Practices for Running Oracle RAC Databases with ASM on IBM DS4800 Storage Subsystems.
Go to:
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101294
The IBM DS3400, DS4000, and DS5000 storage subsystems offer these benefits,
which go hand-in-hand with those of Oracle RAC:
Scalability You can add additional storage capacity to the storage subsystem
while the storage subsystem continues operation.
Lower costs Shared storage within a storage area network (SAN) provides
reduced cost compared to storage that is local to individual hosts and reduces
management expenses associated with administering the storage.
1.Solution also works with DS3400, DS5000, and other DS4000 models.
2.DS3400 has a Fibre Channel host interface with a SAS disk drive interface.
1-2
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example Configuration
The DS4000 Storage Manager software package configures and manages the fully
redundant hardware and high performance architecture with ease. This application
provides management in-band from servers to attached storage, or remotely through
the ethernet. The storage subsystems offer several premium features that add
functionality when used in a database environment. Storage Partitioning feature and
the Enhanced Remote Mirroring feature are two of the important features available.
The Storage Partitioning Premium Feature lets you distribute logical drives on the
storage subsystem to different hosts in your SAN, associating each logical drive with
one particular host or a group of hosts. The logical drives are presented to each host
according to the host type for the different operating systems that are running on
those hosts. The Storage Partitioning premium feature provides great flexibility in the
use of the storage subsystem for storage consolidation.
The Enhanced Remote Mirroring premium feature provides the ability to mirror logical
drives between two IBM storage subsystems. This premium feature provides an
additional redundant copy of the data that you can use for rapid disaster recovery. You
also can use this premium feature to create a copy of live data for test purposes.
Although these premium features are not used in the simple example configuration
shown in this document, many opportunities exist to use these premium features in a
real-world Oracle implementation. For more information, discuss these premium
features with IBM.
Example Configuration
The example configuration contains the following equipment:
The systems are configured as a two-node RAC cluster. ASM is employed to simplify
storage management. The AIX MPIO multi-pathing driver was used to provide failover
support.
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
1-3
Example Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DS4800 Storage
Subsystem
1-4
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
bos.adt.base
bos.adt.lib
bos.adt.libm
bos.perf.libperfstat
bos.perf.perfstat
bos.perf.proctools
rsct.basic.rte
rsct.compat.clients.rte
3 Download and install the updates to the AIX operating system from IBM's website:
http://www-933.ibm.com/support/fixcentral/
The example configuration uses AIX 6.1 Level 1.
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
1-5
Oracle recommends that you put the private interface on a separate network
isolated from all other traffic. Use a separate gigabit ethernet switch for the private
network.
Consider using additional interfaces for most deployments to provide more
flexibility in configuration and higher performance. In the example configuration,
the VIP network shares the interface used for the public network. This network is
configured later in the installation process using the Vipca utility. Entries for the
internet protocol (IP) address, and the public and private networks, are inserted
into the /etc/hosts file.
Make sure that the user ID for the Oracle user and the group numbers for
the dba group and the oinstall group are consistent on all nodes.
6 Set the shell soft limits for the Oracle and root users using the smit chuser
command.
7 Use the smit chgsys command to set the maximum number of processes for
each user to at least 2048.
8 Use the /usr/sbin/no -a command to examine the minimum values for the
following network tuning parameters.
ipqmaxlen = 512
rfc1323 = 1
sb_max = 1310720 (2*655360)
tcp_recvspace = 65536
tcp_sendspace = 65536
udp_recvspace = 655360 (10 * udp_sendspace, must be less than
sb_max)
udp_sendspace = 65536
1-6
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
NOTE
9 Download and install the Open Secure Shell (SSH) package from the IBM web site.
http://www.ibm.com/developerworks/eserver/articles/openssh_updated.html
NOTE
OpenSSH also requires OpenSSL, which you can download from the IBM
website. This software is cryptographic software and the website requires
that you log on to download this type of package.
The example configuration uses OpenSSH. The SSH utility is not required for
Oracle to function, however this utility is recommended because the SSH utility
provides a more secure method of communication between nodes than does rsh.
If SSH is available, Oracle uses SSH. Otherwise, Oracle uses rsh. Configuration of
rsh user equivalence is not described in this document.
f Test the SSH function from each system to itself and to each of the other
systems by issuing the ssh host1 date command. Make sure that the
function completes without using a password.
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
1-7
g Log on as the oracle user and repeat step 11so that Oracle also can perform
remote commands without passwords.
NOTE
The permissions on both the base user directory and the .ssh
directory must restrict write access. If the permissions are not
correctly set, OpenSSH refuses to provide access without a password,
even if the appropriate public key is in the authorized_keys file.
12 Change the .profile file in the Oracle users home directory to set the Oracle
user's parameters.
Table 11 Oracle Users Parameters
Parameter to Add
Description
umask 022
AIXTHREAD_SCOPE=S An environment variable that directs the system to place user threads
within a system-wide scope rather than a process-wide contention
scope. This placement provides better performance and more efficient
memory use.
CRS_HOME
ASM_HOME
Use this parameter if you want to install separate Oracle binaries for
Oracle ASM. This method is preferred if more than one database
instance is supported by Oracle ASM.
ORACLE_HOME
LD_LIBRARY_PATH
ORACLE_BASE
ORACLE_SID
PATH
Include $ORACLE_HOME/bin.
CLASSPATH
Include $ORACLE_HOME/jdbc/lib/
<classes files>.
13 Change the .profile file in the root users home directory to set the Oracle
user's parameters.
Add this entry to the .profile file:
AIXTHREAD_SCOPE=S
14 Synchronize the system time between the hosts in the cluster using any effective
method. If an Network Time Protocol (NTP) server is available, use this method to
set up xntpd.
1-8
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
15 Because some Oracle utilities expect lsattr in the /etc directory and cannot
function correctly without a link to lsattr, use the ln -s /usr/sbin/
lsattr /etc/lsattr command to add a soft link.
1 Use the DS4000 Storage Manager utility to create logical drives on the storage
subsystem that you can share for CRS communication and database storage.
NOTE
You also might use additional logical drives for Oracle binaries and similar
purposes. However, using logical drives local to one particular system
requires the Storage Partitioning premium feature.
One Oracle Cluster Registry (OCR) logical drive (300 MB, RAID 1)
One logical drive for Oracle ASM's SPFILE (100 MB, RAID 1)
Two logical drives for Oracle ASM data storage (300 GB, RAID 5)
The number and the size of the logical drives created for Oracle ASM storage varies
based on the individual implementation, but the sizes used for the OCR disk drives,
voting disk drives, and the Oracle ASM SPFILE disk drives in the example
configuration are sufficient for any configuration.
In the reference configuration, the Oracle binaries are installed on internal disk
drives within each individual server. This configuration uses the JFS2 file system
and is mounted as /u01.
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
1-9
After you have created the logical drives on the storage subsystem, and
mapped to the servers, the cfgmgr command rescans the AIX system for
changes to the physical devices.
The lspv command shows the physical devices that AIX has configured.
3 If the hdisk names for each logical drive on the storage subsystem that the system
can share in the cluster are not identical on all cluster nodes, create new device
nodes for each of the logical drives to give them a consistent name across all nodes.
For example, if the logical drive for the first OCR disk drive was hdisk2 on node1
and hdisk3 on node2, you can create a new devnode called /dev/ORA_OCR1
on each host. The new devnode uses the major numbers and the minor numbers
that correspond with the same logical drive on each system. Show the major
numbers and the minor numbers with the mknod /dev/ORA_OCR1 c
<major# of OCR> <minor# of OCR> command to create the new devnode.
Even when the disk drive names correlate on all of the nodes, still create these
named device nodes for easier location and administration. Named device nodes
were created on the reference configuration for all shared logical drives using the
convention /dev/ORA_<use>.
4 Set the reserve_policy attribute for each logical drive that you want to use for
shared access for OCR logical drives, voting logical drives, and Oracle ASM logical
drives.
For an IBM storage subsystem, reserve_policy=no_reserve is the correct
value.
This value lets the nodes share the logical drives instead of permitting the first
system that accesses the logical drive to reserve the logical drive for exclusive use.
Oracle coordinates the access to the shared logical drives.
NOTE
You must perform this operation on the hdisk* device name rather than
on the named devnode created in step 4.
1-10
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing Oracle
If you created new devnodes for the shared logical drives, set the
ownership and permissions on the new one that you created, not the
original hdisk* devnode.
a Use these commands to set the ownership for each OCR device file:
chown root:oinstall /dev/<devnode>
chmod 640 /dev/<devnode>
b Use these commands to set the ownership and the permission on the voting
disk drive and database device files:
chown oracle:dba /dev/<devnode>
chmod 660 /dev/<devnode>
Do not assign a physical volume identifier (PVID) to hdisks for use with
Oracle ASM.
Installing Oracle
Here are the basic steps for installing Oracle:
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
1-11
Installing Oracle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
You must separate the Oracle Clusterware home directory from the database home
directories.
If more than one database is serviced by Oracle ASM, create a separate home
directory for Oracle ASM. An Oracle ASM home directory lets you modify the Oracle
release for Oracle ASM while not changing the release of any of the databases on
the same server. The directory tree must be owned by oracle.
Because of permission requirements for Oracle CRS, do not make the CRS_HOME
path a subdirectory of the Oracle base directory. For more information, refer to
Appendix E of the Oracle Clusterware Installation Guide.
At the end of the installation process, run the root.sh script as the root user and
run this script on only one node at a time, not concurrently.
1-12
If the Oracle ASM instance was not correctly created by the Oracle Universal
Installer Wizard during installation, use the dbca utility to add the instance and to
create the disk groups.
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing Oracle
If you want additional disk groups, create them using the dbca utility.
NOTE
Using DBCA provides you with the capability to create the Oracle ASM disk groups
if you choose to use Oracle ASM.
Using the Flash Recovery Area (FRA) option is highly recommended. A Flash
Recovery Area permits the setup of a separate area on the disk drive for flashback
capability, an area for the archive logs, a second copy of the multiplexed redo log
files, and an area for RMAN backups. Make sure that the FRA is correctly setup to
handle the performance of the database and that the FRA contains enough space.
Space to hold all of the data for the data retention periods that are set by the
database
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
1-13
Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
For more information about using the Flash Recovery Area, refer to the Oracle
documentation and the following white paper:
Best Practices for Running Oracle RAC Database with ASM on an IBM DS4800 (Models 82,
84, 88) Storage Subsystem. Go to:
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101294
Conclusion
As you have followed the example of setting up a two-node cluster described in this
document, you have seen the simplicity and effectiveness of a deployment of Oracle
RAC 11g Release 1. When you are performing your own installation, make sure to use
the applicable installation information as the basis for your work. This document
describes only tips and techniques. The person who installs Oracle 11g should have the
expertise to know how to and when to modify instructions as necessary. If your
environment is complex and you require outside help, contact your IBM business
partner or sales representative.
Contact Information
For more information and sales office locations, visit the IBM web site at:
http://www.ibm.com
1-14
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
Appendix
References
AIX
For AIX operating system downloads, go to Fix Central:
http://www-933.ibm.com/support/fixcentral/main/System+p/AIX
For AIX documentation, go to AIX Information:
http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.doc/
doc/base/aixparent.htm
IBM Corporation, 2006. Get the Latest Version of OpenSSH for AIX.
http://www.ibm.com/developerworks/eserver/articles/openssh_updated.html
Oracle
For Oracle documentation, go to:
http://www.oracle.com/technology/documentation/index.html
IBM Corporation, 2008. Tuning IBM AIX 5L for an Oracle Database.
https://www-304.ibm.com/jct09002c/partnerworld/wps/servlet/ContentHandler/
VPAA-6V2C4E
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation
A-1
Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A-2
Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems
Copyright 2008, IBM Corporation