Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.

1

Osama Mustafa Page 1


Step By Step to Install Oracle Grid Infra-Structure 11.2.0.3 on Solaris 11.1


Introduction

Oracle Cluster ware is portable cluster software that allows clustering of independent servers so that they cooperate as a single system. Oracle
Cluster ware was first released with Oracle Database 10g Release 1 as the required cluster technology for Oracle Real Application Clusters
(RAC). Oracle Cluster ware is an independent cluster infrastructure, which is fully integrated with Oracle RAC, capable of protecting any kind of
application in a failover cluster.

Oracle Grid Infrastructure introduces a new server pool concept allowing the partitioning of the grid into groups of servers. Role-separated
Management can be used by organizations, in which cluster, storage, and database management functions are strictly separated. Cluster-aware
commands and an Enterprise Manager based cluster and resource management simplify grid management regardless of size. Further
enhancements in Oracle ASM, like the new ASM cluster file system or the new dynamic
Volume manager, complete Oracles new Grid Infrastructure solution.




















Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 2





About the Author

Osama Mustafa Oracle ACE, a database specialist, Certified Oracle Professional (10g, 11g), Certified Ethical hacker (Penetration testing),
and Sun System Administrator, author of book Oracle Penetration Testing. Publishes many articles, including Oracle database articles in his blog.

Twitter: @OsamaOracle.
G+: Osama Mustafa.
SlidShare: Osama Mustafa.
LinkedIn: http://www.linkedin.com/in/osamamustafa.
Blog: http://osamamustafa.blogpsot.com.



The Final Architecture:
The Below Diagram Describe what is my setup, Note theres no IP , Hostname will be mentioned in this document and if you notice any It will
not be the real one, This Part of the document only Describe the Grid and Database installation for Fusion middlware part it will be Published
Separately. Also if you follow this document included with other document you can build full Environment Configured and ready to use with
Single-Sign-On.

Rac Node Number: 2
Database Service Name: TEST
Database Location (Home): /u01/app/oracle/product/11.2.0/dbhome_1
Grid Location (Home): /u01/app/11.2.0/grid
Operating System (OS): Oracle SOLARIS SPARC 11.1


Note: All Software was downloaded from oracle OTN Here and e-delivery here.
Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 3










Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 4

Now after you already know what the setup will look like, Every Information about Locations, and Finally Operating system, during installation
for I faced lot of bugs Since Solaris 11.1 New But it was amazing experience and learn something new.

Lets Start:
Step #1:
You need to know how the ect/hosts will look like after adding IPs:
#########NODES#########
180.111.20.21 Test-db1
180.111.20.22 Test-db2
########################
#########NODE-One-IP###########
180.111.20.28 Test-db1-vip
10.0.0.1 Test-db1-priv
################################
#########NODE-Two-ip############
180.111.20.29 Test-db2-vip
10.0.0.2 Test-db2-priv
################################
######SCAN-IP##################
180.111.20.30 Test-db-scan
###############################


Step #2:
Check OS Version Using the below command:
/usr/platform/uname I/sbin/prtdiag

Step #3 (Optional):
Because I was working remotely not directly from Data Center, I configure Vncserver to enable access to server GUI and Run the Installer from
there.
a. Install Required Package using
pkg info SUNWxvnc
b. Add the below line to /etc/services or use step d as command line
vnc-server 5900/tcp # Xvnc
c. Configure /etc/X11/gdm/custom/conf
[xdmcp]
Enable=true
[security]
DisallowTCP=false
AllowRoot=true
AllowRemoteRoot=true

d. Instead of step b you can do the below , I just want to mention both :

svccfg -s x11-server setprop options/tcp_listen=true
svccfg -s xvnc-inetd setprop inetd/wait=true
svccfg -s xvnc-inetd
setprop inetd_start/exec=astring:"/usr/bin/Xvnc -geometry 1280x720 -inetd -query
localhost -once securitytypes=none"
e. Finally Enable Services
svcadm restart gdm xvnc-inetd
svcadm disable gdm xvnc-inetd;
svcadm enable gdm xvnc-inetd
Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 5


f. Just to make sure my work was Right I restart the Server and check vncsever again.



Step #4 (Optional):
1- I was thinking should I copy oracle software 4 times and its almost 35 GB so 35*4 you are talking about huge miss of time so
what I did here copy the files Once on one server and Configure NFS to share it between all Nodes, more easy and Save your
time.

a. First you have to enable NFS on All Server using the below command
svcadm enable nfs/server
svcadm restart nfs/server

b. If you copied files on server one then this command should be done on server one, if you copied the files on server 2
this command should be done on server 2. Depend where you copied software.

share -F nfs -o rw /base
c. On All Server you can run The Mount command to Share all the files and start setup, More Easy Save time.
mount -F nfs Base-Server-IP:software-location-on-remote-server mount-point-on-all-servers
Step #5: Prerequisite
As any Linux/Unix There is Prerequisite for Operating system follow the below
1. Users And Groups
Oracle Solaris 11 provide you with Command Called ZFS (amazing command to manage File system), I Create Oracle Home Using
this command, also create mount point /u01
This is For Oracle User
zfs create -o mountpoint=/u01 rpool/u01

Create Groups

groupadd -g 1001 oinstall
groupadd -g 1002 dba
groupadd -g 1003 oper

Create Oracle User

zfs create -o mountpoint=/export/home/oracle rpool/export/home/oracle
useradd -g oinstall -G dba oracle
passwd oracle

Create necessary Directory for Our Installation

chown -R oracle:oinstall /export/home/oracle
mkdir -p /u01/app/oracle
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/
chown -R oracle:oinstall /u01




Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 6

This is For Grid User
Create Group for Grid User
groupadd -g 1020 asmadmin
groupadd -g 1022 asmoper
groupadd -g 1021 asmdba

Create Grid User

zfs create -o mountpoint=/export/home/grid rpool/export/home/grid
useradd -g oinstall -G dba grid
usermod -g oinstall -G dba,asmdba grid
passwd grid
chown -R grid:oinstall /export/home/grid

Create necessary Directory for Our Installation

mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/grid
chown grid:oinstall /u01/app/11.2.0/grid
chown grid:oinstall /u01/app/grid
#!important!# chown -R grid:oinstall /u01


Step #6:

Configure .Profile which is located in /export/home/oracle and /export/home/grid

export ORACLE_BASE=/u01/app/oracle/
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export GRID_HOME=/u01/app/11.2.0/grid/
export ORACLE_SID=TEST1
export
PATH=$PATH:/usr/sbin:/usr/X11/bin:/usr/dt/bin:/usr/openwin/bin:/usr/sfw/bin:/usr/sfw/sbin: /usr/ccs/bin: /usr/local/bin:/usr/local/sbin:
$ORACLE_HOME /bin:$GRID_HOME/bin:.


Note: copy .profile to all Database Servers using scp command

cd /export/home/oracle
scp .profile oracle@Server-ip:/export/home/oracle/

Step #7:
Oracle Grid and Networking Notes:
1. The broadcast must work across any configured VLANs as used by the public or private interfaces.
2. Across the broadcast domain as defined for the private interconnect
3. On the IP address subnet ranges 224.0.0.0/24 and 230.0.1.0/24

Regarding to oracle to you need to check udp time using the below command
ndd /dev/udp udp_xmit_hiwat
ndd /dev/udp udp_recv_hiwat

To avoid reboot you can set it on memory and reboot time using the below
On Memory
ndd -set /dev/udp udp_xmit_hiwat 65536
ndd -set /dev/udp udp_recv_hiwat 65536

Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 7

On Reboot
ndd -set /dev/udp udp_xmit_hiwat 65536
ndd -set /dev/udp udp_recv_hiwat 65536


Step #8:

In this step you need to make sure of disks on both nodes, in my case I am using EMC storage.

List Disk Using
/usr/sbin/format
fdisk the raw disk using fdisk command.
Change owner to grid using
Chown grid:asmadmin /dev/rdsk/Disk-name
Chmod 660 /dev/rdsk/disk-name
You can list your available using
ls ltr /dev/rdsk/emc*
Change owner for these disk
Chown 660 emc*
Chown grid:asmadmin emc*


Note: In Unix Each Disk have slide from 0-6 like the below :

crw-rw---- 1 grid asmdba 302, 0 Apr 23 19:02 emcp@0:a,raw
crw-rw---- 1 grid asmdba 302, 8 Apr 23 20:38 emcp@1:a,raw
crw-rw---- 1 grid asmdba 302, 16 Apr 22 14:05 emcp@2:a,raw
crw-rw---- 1 grid asmdba 302, 24 Apr 23 21:00 emcp@3:a,raw
crw-rw---- 1 grid asmdba 302, 32 Apr 23 21:00 emcp@4:a,raw
crw-rw---- 1 grid asmdba 302, 40 Apr 23 21:00 emcp@5:a,raw
crw-rw---- 1 grid asmdba 302, 48 Apr 22 14:05 emcp@6:a,raw



Step #9:

Disable ntp using
svcadm disable ntp


Step #10:

By Default Oracle Solaris SPARC prevent root access, for example we created oracle and grid user but we cannot access to root using su
command to enable root access for oracle and grid do below

Edit file /etc/user_attr

Add the below lines

oracle::::defaultpriv=basic,net_privaddr;roles=root
grid::::defaultpriv=basic,net_privaddr;roles=root


Step #11:

Now we need to configure memory Parameter for Oracle and Grid User and make permanent

To check memory Current Value for Both user

prctl -n project.max-shm-memory -i process $$

Modify memory values using
projmod -a -K "project.max-shm-memory=(privileged,32G,deny)" -U oracle default
projmod -a -K "project.max-shm-memory=(privileged,32G,deny)" -U grid default
projmod -s -K "project.max-shm-memory=(privileged,32G,deny)" default
Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 8

To make sure new memory value effect applied on oracle and grid user open new terminal and run prctl command again.


Step #12:
During installation Oracle will check swap memory so you need to increase swap memory depend on your setup for sure I will use ZFS
command.
Check swap value

bash-3.00# swap -lh

swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 256,1 16 4194288 4194288

Remove Swap using root user

bash-3.00# swap -d /dev/zvol/dsk/rpool/swap

Configure New Swap

bash-3.00# zfs set volsize=20G rpool/swap
bash-3.00# swap -a /dev/zvol/dsk/rpool/swap

Check New Value

bash-3.00# swap -lh


Step #13:
One more step Configure SSH Between Nodes You can do this Step During installation or Using the the Below command, Oracle Provide you
with new way, you can now configure SSH using sshsetup its already exists within media
For Oracle User
# ./sshUserSetup.sh -hosts " node1 node2 node3 node4" -user oracle -advanced noPromptPassphrase

For Grid User

# ./sshUserSetup.sh -hosts " node1 node2 node3 node4" -user grid -advanced noPromptPassphrase


Congratulations you can start your setup now!!



















Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 9

Step #14:

Start Install Grid Infrastructure, in my case I choose to install Software only then Configure ASM in this way if error appeared I will know where
to start troubleshooting. Follow the screens ( this steps should be Done As Grid User )


Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 10



Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 11



Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 12



Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 13









Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 14

Step #15:
After success installation now I need to Configure ASM ( This Step should be Done as Grid User) , From One Node Only.
export ORACLE_HOME=/u01/app/11.2.0/grid/
cd $ORACLE_HOME/bin
Run ./asmca

The Below Screen Should be Open

Three ASM Disks should be created:
1- DATA : DataFiles and Parameter files ( should be big )
2- FRA : FLASH RECOVERY AREA ( should be big )
3- CRS : this Disk for OCR and VOTING ( 4-5G will be enough )
After installation done you have to see MOUNTED on Both Node.
REDUNDANCY:
NORMAL REDUNDANCY - Two-way mirroring, requiring two failure groups.
HIGH REDUNDANCY - Three-way mirroring, requiring three failure groups.
EXTERNAL REDUNDANCY - No mirroring for disks that are already protected using hardware mirroring or RAID.
If you have hardware RAID it should be used in preference to ASM redundancy, so this will be the standard option for most installations.


Step #16:
Finally You Are done with Grid Infrastructure and now you should configure Database on RAC (this step should be done as Oracle User),
Usually I am installing Software only then Called dbca to configure Instance Install Software.
Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 15



Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 16



Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 17



Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 18



Step By Step Install Oracle Grid 11.2.0.3 on Solaris 11.1

Osama Mustafa Page 19



Done!!

You can reboot node to test your Setup.
Reference:
1- Oracle Documentation Here.
2- Oracle White Paper Here