Building A Virtual Oracle RAC - VIRTUAL Box - OLE5

Building a Virtual Oracle RAC – Project Introduction
Filed under: Networking in Virtual Environment, Oracle Database, Virtualization — Tags:

bridged, configuration, host-only, installation, iSCSI server, iSCSI target, Linux, network, oel,
Openfiler, oracle, rac, vbox, virtual box — oraclelabs @ 17:18
19 Votes
Introduction: This learning experiment is about building a totally virtual Oracle RAC
environment at home using only tools and software available to wide public, preferably open
source (when possible).
My project is generally based on an excellent article by Jeffrey Hunter (and other authors) –
Build Your Own Oracle RAC Cluster on Oracle Enterprise Linux and iSCSI
Jeffrey’s article gives instructions on how to build a physical setup. I do recommend reading this
article or at least giving it some 10 minutes scan before you attempt to build a RAC. In that
article Jeffrey estimates a cost to build a production RAC in physical environment between $20K
and $30K. He shows that for the purposes of learning the technology a “mostly physical” RAC
with some parts virtualized can be built for under $2700. I am going to take the virtualization to
the extreme and build a RAC inside a single PC. Yes, I do not care now about performance, and
yes, I am willing to buy some additional memory for the PC. You can not really expect to run
two Linux nodes with databases and a storage (SAN) simulation on a PC with 1GB of memory.
My estimated cost is, therefore, somewhere under $100, if all we need is a memory upgrade.
Sounds good? It is still a lot of work and it will probably take you many hours to do it right. The
benefit is a hands on knowledge, which is priceless.
Now to the choice of hardware and software.
 PC or server with some 200GB space and 3.5GB memory (XP Professional can handle
up to 4GB).
 The host OS: Windows XP SP3 professional (although SP2 will probably do as well)
 Virtualization software: Sun’s Virtual Box 3.1.2 (or higher). Works with no issues on
both Intel and AMD processors.
 Database nodes: Oracle Enterprise Linux 5 update 3 as a guest OS, running Oracle
Database 10.2.0.1
 SAN/NAS emulation software: Openfiler 2.3
My reasons for choosing particular software:

Host OS does not really matter, virtualization software can run on many different platforms. If
you desire so, you can run this whole project using Linux as a host. Virtual Box is selected
because it is an open source software produced by Sun Microsystems (which belongs to Oracle
now).
Why Sun Virtual Box and not VMware? VMware is a popular product but I do not see a
consistency in their support of free versions of the software. In my experiments with VMware I
have encountered some obstacles while running Linux guests (particularly in networking and
CPU clock synchronization). Virtual Box is an open source product and quite a mature one, well
documented and supported. So far I was able to run OEL4 and OEL5 with no issues (even with
no guest additions installed). I like the product’s built-in networking, it seems to be more flexible
and stable. These are all my personal preferences, of course.
Why Oracle’s Linux? This particular flavor of Linux is a modified Red Hat Enterprise Linux 5.
Oracle has made a few adjustments that allow Oracle 10g R2 database (32 bit) to install with no
issues on it (usually, no additional RPM’s required). Since I am aiming to install Oracle in the
guest(s) thus the choice of this Linux flavor. Besides, this Linux comes with OCFS2 and most of
the ASMLib software packages installed. If you have a licensing question about Oracle
Enterprise Linux (OEL) I can answer it. You only need a license if you want Oracle’s support.
For learning purposes you do not need a license. If you will need to download some rpm libraries
and updates you can get those for free, the YUM configuration can be pointed to a free
repository (not discussed here).
Now the database itself. Which edition of 10.2.0.1 are we going to use? There is no clarity in that
article on this matter, for now I assume the Standard Edition will do. There are some additional
pieces of software from Oracle (or used by Oracle) to be installed, those are: OCFS2, ASMLib
2.0 and TAF.
Now that we know what we need to get started let’s split the work into more manageable mini
projects, follow them just in this order, and hopefully all pieces will fit:
 Task 1. Installing and configuring Sun Virtual Box.
 Task 2. Installing Openfiler in Sun Virtual Box.
 Task 3. Configuring Openfiler to simulate iSCSI targets.
 Task 4. Creating a Linux guest, configuring hardware and Installing Linux.
 Task 5. Patching Linux guest.
 Task 6. Configuring iSCSI targets in Linux and partitioning the volumes.
 Task 7. Preparing Linux guest for Oracle installation – part1
 Task 8. Cloning a Linux guest.

 Task 9. Preparing Linux guest for Oracle installation – part2
 Task 10. Oracle installation in RAC nodes – part 1.
 Task 11. Oracle installation in RAC nodes – part 2.
 Task 12. Creating Oracle database in RAC nodes.
 Task 13. Having some fun with RAC.
Virtual Oracle RAC. Task 1: Install and configure Sun Virtual Box
3 Votes
Link back: This guide is a part of the Virtual Oracle RAC project, the index to the whole project
is here.
Here is an overview of network setup that we will do in Virtual Box environment. This picture
below shows three sets of network adapters in guests.
eth0 is for public network
eth1 is for private network

Download and install Sun Virtual Box (follow Sun’s instructions, the installation is quite
straightforward).
Run Virtual Box, access Virtual Box preferences
Make sure Host-Only network is installed (it must be there by default) and it is the only network
present. Create it, if it is not there. The notion of "network" in this particular part of Virtual Box
may be likened to creating a router with a static IP. This "router" will be accessible from host’s
network but not from outside of it (not from other hosts on your network).
Notice, that during the installation of networks there may be pup-ups asking you if you want to
proceed with unsigned driver installation. Do proceed, of course.
Configure Host-Only adapter as below (yes, it is placed on a totally separate network from your
host’s LAN adapter):
Now create another (second) Host-Only adapter:
Click on "Ok", close all settings dialogs and wait till another adapter is created.
Save your changes and enter "ipconfig" command from host command prompt. At this point the
host networks will look like this:
As you can see, Host-Only networks is here but not connected to your gateway (i.e. to your
router).
From the network diagram you will notice that there is a "Bridged" network "router", this one
does not have to be created, it appears automatically when "Bridged" adapters are created in
guests.
Disclaimer: in my network setup I have used a domain name that can be taken or belong to
someone. If such is the case, it is by pure coincidence and I apologize for any inconvenience it
may have caused. Presently it appears that all possible and impossible domain names are taken
but I have no claims on the domain name. I used it just for illustration only.
Virtual Oracle RAC. Task 2: Openfiler – installation in Sun Virtual Box
4 Votes
is here.
If you are planning to experiment with Oracle RAC you will need a SAN (NAS). It all stems
down to having iSCSI targets (server) somewhere on your network. These iSCSI targets can be
created in a variety of ways. Reminder: we are in a virtualized environment so we do not need to
run to a store and buy real hard drives. We will use software to simulate hard drives and disk
volumes. Some projects I looked at use Linux guests (in virtual machine) to simulate iSCSI using
iSCSI server features built into some Linux flavours. Such is, for instance, SUSE Linux
Enterprise Server 10. I’d rather go with open source software for this purpose. My choice is
Openfiler, which is based on a small Linux distro (well, of course) and gives you nice web based
interface. Openfiler is essentially free, however if you want to get the Admin Guide book it will
cost you 40 GBP.
If you want to know more about Openfiler (other than what you can find on their web site) you
can refer to www.rpath.org
Get Openfiler installation media from http://www.openfiler.com/ , it comes in a small image file
of some 300MB size (best things come in a small package, remember?).
I am going to install Openfiler 2.3 (x86) in Sun Virtual Box.
Let’s create a new guest that will become our Openfiler server.
I named the guest Openfiler2 just in case to avoid any possible trouble later when we come to the
networking part. "Red Hat" is the default and I did not bother to change it. Seems to work fine.
For memory, 256MB RAM should suffice for now. Remember, this will take a chunk off your
real memory so do not be too generous.
Now this. The virtual hard drive in this case is the "internal hard drive" in the Openfiler2
machine, i.e. the hard drive that will hold the OS. We will create a new hard disk. This one is not
the hard drive that will be used for iSCSI simulation (be patient).
This screen (can not even call it a dialog) is kind of redundant but we like to be welcomed
anyway, right?
We accept the default of dynamically expanding storage.
Set the size to 800MB initially
Click on Finish and … your new guest with its hard drive are created.
Now go back to the guest machine settings, there is some work to do here. Navigate to the
System and then to Processor tab. This is very important! Enable PAE/NX support as shown
below (if it not enabled by default in your version). If it is not enabled, your installation will fail
at the very end (as it did for me). Look up this part at http://en.wikipedia.org/wiki/NX_bit if you
really really want to.
Almost ready to run the installer, but wait a second. We need to plug some disks into the
Openfiler to get our iSCSI targets, right? Where are those disks coming from? Here is what we
need to do. Close the settings screen for the guest and bring up Virtual Media Manager (that’s a
very cool thing, you have seen a part of it already when we created hard dist for Openfiler2):
Use "New" dialogue to create a virtual hard drive in a file named "openfiler_rac" and sized at
40GB (just like if you were creating a file holding a guest VM). The result will look like this:
The "openfiler_rac" storage will be used in our RAC project to hold iSCSI drives. Now see what
happens next. Bring up the Openfiler2 guest settings and go to "Hard Disks" section. Add the
hard disk to the vacant slot in the IDE controller (the capacity of one controller seems to be three
IDE devices).
When you run installation of your guest these disks will be detected as normal IDE drives.
I must point out that I have seen an article on the internet suggesting the use of physical external
USB hard drive as a source disk for Openfiler (in a Virtual Box environment), another article
suggested using an additional Linux guest machine for such a purpose (to simulate hard drives). I
am wondering, why to go to such extent of complexity when Virtual Box provides much simpler
means?
So, next in our setup is the network, we define two host-only adapters and one bridged adapter.
The bridged adapter will later be used during NTP setup.
Adapter 1:
Adapter 2:
And bridged Adapter 3 (do not copy down my MAC addresses, they are not important to you):
Why this particular adapter type (Intel PRO)? Why not to go with the default? It worked for us
well in other Linux installation. The default type will work for you as well, that is, until first
reboot of the guest. Then your adapters will simply vanish. This has to do with a driver supplied
by VMware to Openfiler to support virtual network adapters (can be a problem in this particular
version of software). To make the story short, only this Intel PRO type seems to survive after
restarting the guest. This problem may not present itself in future releases of Virtual Box and
Openfiler (hopefully).
Now all we need to do is to insert the installation media and push the reset button. But how?
Easy enough. Bring up the Virtual Media Manager again and go to CD/DVD images tab.
Register the installation media (the image file) and notice that the grayed line at the bottom reads
"Not Attached", which means that this file, although registered with the Media Manager, is not
used by any component at this time.
Now bring back the Openfiler2 settings and go to Storage tab. There should be a DVD drive in
our IDE controller, if it is not there it can be added easily. Click on "CD/DVD Device":
Select the appropriate media (so here is the connection between registered media files and DVD
drive in our virtual guest!)
Now we see the the media file (image) appeared in the CD drive:
Finally, we are ready to start installation. Just start your guest, it will detect installation media
and boot into installer.
Follow the generic installation guide for Openfiler here http://www.openfiler.com/learn/how-

to/graphical-installation.
When network cards detected during installation, assign IP addresses shown below in Openfiler
section at the bottom:
# Notice that loopback address do not have a real host name in it

127.0.0.1 localhost.localdomain localhost
# eth0
# Database nodes – Public Network
10.10.2.11 odbn1.harzion.org odbn1
10.10.2.12 odbn2.harzion.org odbn2
# Database nodes – Public Virtual IP (VIP) addresses

10.10.2.200 odbn1-vip.harzion.org odbn1-vip
10.10.2.201 odbn2-vip.harzion.org odbn2-vip
# eth1
# Database nodes – Private Interconnect
10.10.1.11 odbn1-priv.harzion.org odbn1-priv
10.10.1.12 odbn2-priv.harzion.org odbn2-priv
# Openfiler eth0 – public

10.10.2.20 openfiler.harzion.org openfiler
# Openfiler eth1 – private
10.10.1.20 openfiler-priv.harzion.org openfiler-priv
The gateway, of course, will be at 10.10.1.1 and hostname is "openfiler". Notice that naming of
the guest in Virtual Box has no effect on hostnames so do not confuse these two kind of names.
The /etc/hosts file shown above we will place in our Openfiler once installation is done, same
file will be present on all other hosts involved in the project.
All future interactions with the running Openfiler are via a web interface. When the guest boots
it displays a reminder on where the web GUI sits:
Default web user is: "openfiler" and the password is "password". Who could have guessed… I
was going to try "1234", like in "Spaceballs" movie, one of my favourites.
I used a very good practical guide on how to create virtual volumes in the Openfiler that can be
found here: http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_iscsi.html#9 (notice
that it deals with physical hardware rather than virtual but it makes no difference to us).
Virtual Oracle RAC. Task 3: Openfiler – iSCSI targets setup

1 Votes
is here.
Now that we have Openfiler and OEL guests in Virtual Box, we can start configuring our Oracle
RAC. As a general guideline I am going to use this (excellent) article –
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_iscsi.html (the article gives
instructions on how to build physical setup, while we are trying here to do the same in virtual
environment). Roll up your sleeves.
If you have followed instructions in my other articles on installing OEL your network will differ
a bit and will need some adjustments, we will cover that, of course.
The starting point is assigning names and network addresses to all our components, all hosts in
our network will have exactly same /etc/hosts file, as shown below (we have placed it in our
Openfiler already):
# Notice that loopback address do not have a real host name in it

127.0.0.1 localhost.localdomain localhost# Database nodes – Private
Interconnect – (eth0)
10.10.1.11 odbn1-priv
10.10.1.12 odbn2-priv
# Database nodes – Public Network – (eth1)

10.10.2.11 odbn1-pub
10.10.2.12 odbn2-pub
# Database nodes – Public Virtual IP (VIP) addresses – (eth1:1)

10.10.2.200 odbn1-vip
10.10.2.201 odbn2-vip
# Openfiler – private and public interfaces

10.10.1.20 openfiler-priv
10.10.2.20 openfiler
Now we do following steps:
 Enable iSCSI services from [Services] / [Manage Services] page.

 Verify that “iSCSI target server” service is running:
 Network Access Configuration
Navigate to [System] / [Network Setup], notice our three network interfaces. Two of them have
static IP and are active already, one needs a setup (correction – eth0 and eth1 show reversed.
eth0 should be on 10.10.2.20 and eth1 on 10.10.1.20):
Click on eth2 interface and select DHCP protocol:
Confirm default MTU on next screen:
The result will look like this (actual IP not shown for eth2):
Now enable two hosts (two RAC nodes) to communicate to Openfiler as below (notice the
slightly unusual netmask).
Now click on [ Clock Setup ] as shown below:

At this moment you probably ask “why don’t we use the host PC’s clock or install Guest
Additions to keep guest clock synchronized with the host PC?”. Well, it is a legitimate question
and here is the answer, at least what I can come up with. I have not been successful so far with
Guest Additions so far, the clock has drifted wildly and some other problems were introduced
(such as an occasional keyboard mapping mess up). As for synchronizing with host PC’s
Windows Time service (w32time), there was not much success either. It seems that UNIX or
Linux world does not understand Windows SNTP in Windows XP. So here is a sort of
compromise that may work for most of the cases.
If you are setting up this project at home chances are you have access to internet and so you can
use of the “pool” servers, such as 0.pool.ntp.org or similar. If you do not have access to internet
but instead have a Windows Domain Server (so you are in a corporate environment) you may be
lucky and our Linux guest will be able to get time from the domain controller.
Put a name or an IP of the NTP server you are going to use as a time source. Use 10.10.1.1 to
get time from domain controller (somehow Windows XP will relay this to DC). Use internet
NTP server if you have access to internet.
Open an SSH window and connect to openfiler2 as root.
View /var/log/messages file, it may have these lines added after you click on “Setup
synchronization”:
By this time openfiler2 will have its NTP receiving time but it will not distribute it to others. In
order to allow RAC nodes to query this NTP server we have to edit /etc/ntp.conf file on
openfiler2 and add two more lines to it:
restrict 10.10.1.11 nomodify notrap
restrict 10.10.1.12 nomodify notrap
After that you will need to restart the NTP service:
# service ntpd restart

If you have trouble with this NTP setup you should examine /var/log/messages and /etc/ntp.conf
files. Use this source of information on NTP servers and clients to fix your problem. Notice, that
RAC nodes will be able to sync their clocks from openfiler2 even if openfiler2 itself won’t be
able to get precise time from other sources (as long as it permits to query itself), so this resolves
a potential problem with date/time and Oracle installer later on.
 Physical Storage
Navigate to [Volumes]. Most probably you will see a picture like this:
Navigate to [Block Devices] section (see below):
Click on the link for “hdb”, scroll to “Create a partition in /dev/hdb” and click on “Create” thus
accepting defaults:
The result of this action presented below:
Now navigate to [Volumes] / [Volume Groups] and create a new volume group as below:
Not tired yet? Navigate to [Volumes] / [Add Volume] and scroll down to “Create a volume” in
“rac1″ and use the table below (borrowed from original article mentioned in the beginning,
numbers are slightly changed though).
iSCSI / Logical Volumes

Volume Name Volume Description Required Space (MB) Filesystem Type
racdb-crs racdb – Oracle Clusterware 2,048 iSCSI
racdb-asm1 racdb – ASM Volume 1 8,092 iSCSI
Here is the picture of first volume being created (after this one you will have to click on [Add
Volume] again):
In the end you will get a setup like this:

I am getting tired a bit but hold on a little longer, we are getting there. It’s actually getting quite
exciting as we are going to create iSCSI targets now.
Navigate to [Volumes] / [iSCSI Targets] and proceed to add five iSCSI targets with following
names (system will suggest default names that you will change, it is only a matter of aesthetics).
The names will be as below:
We are not going to change any settings for our targets (leave all defaults).
Select first target and click “Change” (this is going to affect sub-tabs on this page). Click on
“LUN Mapping” sub-tab. You will see this (look at it for a while and it will make sense):
If you did not guess yet, you will have to click on “Map” button thus linking LUN
“/dev/rac1/racdb-crs” with iSCSI target “iqn.2006-01.com.openfiler:racdb.crs”.
After that proceed to “Network ACL” sub-tab and grant our RAC nodes the access to this target
as below:
The last sub-tab – “CHAP” is not relevant.
Now go to first sub-tab and select next target, then link it with appropriate LUN and grant
access. Repeat this process until you linked them all. At this point we are done with Openfiler.
I am sending special thanks to Jeffrey Hunter, whose article was guiding me through this setup
so far.
Virtual Oracle RAC. Task 4: Installing Oracle Enterprise Linux 5 in Sun

Virtual Box
Rate This
is here.
This part of the project provides instructions on installing Oracle Enterprise Linux 5 Update 3
(OEL5 U3) in Virtual Box 3.1.2 (r56127)
Download "Oracle Enterprise Linux 5 Update 3" from Oracle’s eDelivery site. The media comes
in five ISO CD images or one DVD image, get the DVD image. I am installing a 32-bit variant
here. Why this Linux is treated as Red Hat was explained in the project index.
Create the first node (new guest in Virtual Box):
Give this machine 512MB of memory for now. Later, when we start installing Oracle we may
need to increase the allocation (or maybe add more base memory to the PC):
Choose to create a new hard disk (which will result in a file that holds an image of this machine):
You get to the next window where where you just click Next:
I recommend going with dynamically expanding storage option:

Choose an appropriate location for the file and give it a max size of 40GB (it will be initially
created quite small):
Verify the settings for the storage and click Finish (there will be another screen with
confirmation):
Once empty and inactive machine is created we need to go over its settings and do some
changes.
Update boot order so CD/DVD is first, then comes Hard Disk and floppy is not selected:
You will need to create three network adapters, first one for private interface:
Second adapter is for public interface:
And the third adapter is for connecting to Internet (since we do not want to bridge either private
or public interfaces to host’s Internet):
Notice MAC addresses of the cards (they will be different in your case but you may assign them
manually and make them just as in this guide). When these adapters are discovered by hardware
installation wizard you need to know which is which.
Now we are ready to proceed with installing the Linux onto our guest machine.
Add installation media image in Virtual Media Manager (do yourself a favour, download a single
DVD image, it is slower but saves you a lot of time later on). Guests can only see media (regular
files, image files, drives, directories) that are added to Media Manager:
Mount (it will become "attached" in Media Manager) the DVD image in guest (your guest has to
have an IDE device that emulates CD/DVD drive, like in the picture below):
Start installation by powering up the guest (I had to pause the guest to take a snapshot):
If boot order is correct you will be presented with a few dialogues where you mostly accept
defaults until you arrive at storage decisions.
OEL5 differs from OEL4 in its management of partitioning. Volume Groups is a new thing here
but do not be afraid, it is a relatively simple setup, just one more layer of abstraction, like having
a software RAID built on top of your (single) hard drive:
Now configuring network (watch out for MAC addresses, the order of adapters here will not
necessarily be the same as you see them in guest settings). So, first adapter will get a static
address on private network (please use 10.10.2.11 instead of what is shown in the picture below):
Second adapter will also get a static IP, it is on public network (please use 10.10.1.11 instead of
what is shown in the picture below):
Third adapter will get DHCP configuration. Finally, make all three cards active on boot.
Give this machine a hostname. Domain name is mandatory for some services (they will fail on
startup if there is no domain) but it does not have to be a real one, make up yours:
Next screens finalize the initial setup and let you choose software you want to be installed.
When you finally get to the software configuration section choose manual mode and make sure
the following is selected:
 Desktop Environments

o GNOME Desktop Environment
 Applications

o Editors
o Graphical Internet
o Text-based Internet
 Development

o Development Libraries
o Development Tools
o Legacy Software Development
 Servers

o Server Configuration Tools
 Base System

o Administration Tools
o Base
o Java
o Legacy Software Support
o System Tools
o X Window System
Few more things: when you get to the Firewall dialog – disable the Firewall, when you are asked
about SELinux – disable that one too. You may also create non-root user now or delay that till
later. When done with all that the installation will proceed and the machine will reboot into your
new OS. Hopefully. Watch for any error messages during reboot.
Virtual Oracle RAC. Task 5: Patching Oracle Enterprise Linux 5

for Oracle Installation
2 Votes
is here.
This part of the project provides instructions on preparing Oracle software installation in Oracle
Enterprise Linux 5 (which itself is in Virtual Box).
In previous guides we have installed OEL5 U3 as a guest machine in Virtual Box. This is our
first node of the RAC. Once we finished with all software installations on that node we will
simply clone it, thus saving a lot of time and effort.
Network verification and configuration:
Please refer to the /etc/hosts file that was introduced in Openfiler installation and setup, it will be
placed in all of our Linux machines.
Edit /etc/hosts on “odbn1″ by copy and paste it or use ftp.
Notice, that node name(s) are not present in the loopback address. That’s right, they should not
be there for the RAC to work properly.
Some adjustments must be done in the kernel for the network (those commands do not make the
changes permanent):
sysctl -w net.core.rmem_default=1048576
sysctl -w net.core.rmem_max=1048576
sysctl -w net.core.wmem_default=262144
sysctl -w net.core.wmem_max=262144
Same changes have to be made permanently in /etc/sysctl.conf file, add following lines to the
config file:
net.core.rmem_default=1048576
net.core.rmem_max=1048576
net.core.wmem_default=262144
net.core.wmem_max=262144
Verify that firewall is indeed stopped by running this command:
/etc/rc.d/init.d/iptables status
Use the command below if the firewall is active:
/etc/rc.d/init.d/iptables stop
Now is the time to install Linux packages needed by Oracle.

Attach the installation DVD to the “odbn1″ machine (right click on CD image in the running
guest window, select “more CD/DVD Images”):
Select the proper image (if it is not displayed you will have to launch Media Manager first and
register the image file):
Use SSH terminal (PuTTY for instance) to log into “odbn1″ with root credentials. It does not
matter which interface you use now for the session, public or private.
Execute these commands:

mkdir -p /media/cdrom
mount -r /dev/cdrom /media/cdrom
cd /media/cdrom/Server
Now let’s run installation for all packages that needed by Oracle, if some of them are already
here the particular command will be skipped.
rpm -Uvh binutils-2.*

rpm -Uvh elfutils-libelf-0.*
rpm -Uvh glibc-2.*
rpm -Uvh glibc-common-2.*
rpm -Uvh libaio-0.*
rpm -Uvh libgcc-4.*
rpm -Uvh libstdc++-4.*
rpm -Uvh make-3.*
rpm -Uvh elfutils-libelf-devel-0.*
rpm -Uvh glibc-devel-2.*
rpm -Uvh glibc-headers-2.*
rpm -Uvh gcc-4.*
rpm -Uvh gcc-c++-4.*
rpm -Uvh libstdc++-devel-4.*
rpm -Uvh unixODBC-2.*
rpm -Uvh compat-libstdc++-296*
rpm -Uvh compat-libstdc++-33*
rpm -Uvh libaio-devel-0.*
rpm -Uvh libXp-1.*
rpm -Uvh openmotif-2.*
rpm -Uvh sysstat-7.*
rpm -Uvh unixODBC-devel-2.*
We can see that some of the packages were there and some were installed. Good.
Now we will install iSCSI package (supplied but not installed by default):
rpm -Uvh iscsi-initiator-utils-6.2.0.868-0.18.el5.i386.rpm

Virtual Oracle RAC. Task 6: Configuring iSCSI targets in Linux and
partitioning the volumes.
5 Votes
is here.
This part of the project provides instructions on configuring iSCSI initiators in Linux.
Configuring iSCSI services.
Start the “iscsid” serivce:
# service iscsid start
Run the following commands to make sure the services will run next time after reboot:
# chkconfig iscsid on
# chkconfig iscsi on
Now, we check if our iSCSI service can communicate with Openfiler. Run this command (shows
available iSCSI targets):
# iscsiadm -m discovery -t sendtargets -p openfiler-priv

Is your output any different from what is shown above? Got less targets or none at all? And you
say that your openfiler is up and running and you followed all instructions to the letter. Well,
let’s go back and recall how we configured our iSCSI targets. There was something called
“Network ACL” setup, that may be likened to a firewall in the openfiler. Remember now? It is
set to “Deny” by default for each target. Go back and set it all to “Allow” for all targets. The
change will take effect immediately, so you can run discovery again.
As it turns out, when the iSCSI initiator discovers the targets it configures the services to start up
automatically on reboot and log into the targets. We can test this now by rebooting our linux
machine. This is what you should see during restart if iSCSI setup was done properly:
If login to the targets did not happen on reboot you will need to execute the commands below:
# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p 10.10.1.20 -l
# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p 10.10.1.20 –op update -n node.startup -v

automatic
The “-l” option means – log into the target (“a node”, do not confuse this “node” with RAC nodes)
The “–op” option means update configuration property specified by “-n” option, it is “node.startup” in this case.
Do these two commands for each of the targets appropriately, I am showing only first of them.
The command below (no operation specified) can be used to query configuration of a target:
# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p 10.10.1.20
Making device names persistent.
Linux “talks” to iSCSI targets using local device names. The mapping of our iSCSI targets to
local SCSI device names is random and may change after reboot. It is a problem that needs
fixing. The mapping of targets to the local devices is illustrated here:
Since we want to have a permanent and consistent mapping across all RAC nodes, we are going
to create persistent local SCSI device names. This is done using “udev”, which is a Dynamic
Device Management tool.
# cd /etc/udev/rules.d/
Create a file called “55-openiscsi.rules” with the following content:
# /etc/udev/rules.d/55-openiscsi.rules
KERNEL==”sd*”, BUS==”scsi”, PROGRAM=”/etc/udev/scripts/iscsidev.sh
%b”,SYMLINK+=”iscsi/%c/part%n”
Navigate to another directory:
# cd /etc/udev/scripts
We create here a new shell script called “iscsidev.sh” with the following content:
#!/bin/sh
# FILE: /etc/udev/scripts/iscsidev.sh
BUS=${1}
HOST=${BUS%%:*}
[ -e /sys/class/iscsi_host ] || exit 1
file=”/sys/class/iscsi_host/host${HOST}/device/session*/iscsi_session*/targetname”
target_name=$(cat ${file})
# This is not an open-scsi drive

if [ -z "${target_name}" ]; then
exit 1
fi
echo “${target_name##*.}”
Make the new script executable:
# chmod 755 /etc/udev/scripts/iscsidev.sh
Let’s restart the iSCSI initiator service:
# service iscsi stop
# service iscsi start
Here is the outcome of all that (the image below may be too wide but I did not want the lines to
wrap):
Well, how do we know if what we’ve done actually worked? We look at the names in /dev/iscsi
(this directory has just been created after we ran these commands, it did not exist before) and
compare them to mapping in /dev/disk/by-path:
And …
Take for instance “/dev/iscsi/asm1/part”, it corresponds to the “asm1″ target (through /dev/sda).
Now we have persistent local names for our targets. We can reboot the odbn1 machine and see
that iscsi devices are still there and properly mapped.
Mapping of iSCSI Target Name to Local Device Name

iSCSI Target Name Local Device Name
iqn.2006-01.com.openfiler:racdb.asm1 /dev/iscsi/asm1/part
iqn.2006-01.com.openfiler:racdb.crs /dev/iscsi/crs/part
Next, we will have to create partitions in our SCSI volumes.
Creating partitions on iSCSI Volumes.
Before we start creating partitions it makes sense to showdown our virtual machines and take a
snapshot of them. This provides an option of reverting to a known state of machine if something
goes wrong.
Assuming that snapshot is taken and our machines are back online.
Notice: some of the material for this article was taken from Oracle author’s article, I recommend
reading that article if you need more detailed information.
The following table lists the five iSCSI volumes and what file systems they will support:
Oracle Shared Drive Configuration

File System iSCSI Target (short) Mount
Size ASM Diskgroup Name File Types
Type Name Point
Oracle Cluster Registry (OCR) File –
OCFS2 crs 2GB /u02 (~250 MB)
Voting Disk – (~20MB)
ASM asm1 8GB ORCL:VOL1 +RACDB_DATA1 Oracle Database Files
ASM asm2 8GB ORCL:VOL2 +RACDB_DATA1 Oracle Database Files
ASM asm3 8GB ORCL:VOL3 +FLASH_RECOVERY_AREA Oracle Flash Recovery Area
ASM asm4 8GB ORCL:VOL4 +FLASH_RECOVERY_AREA Oracle Flash Recovery Area
Total 36GB
In the picture below shown the fdisk dialog that creates a primary partition of a maximum
available size. Red arrows mark your input, where you either type values or accept defaults.
# fdisk /dev/iscsi/asm1/part
Repeat the command sequence for volumes asm2 through asm4 and then for crs, which is
shown below. Remember to always create a primary partition, number 1, of max size):
Verify new partitions
Keep in mind that the mapping of iSCSI target names and local SCSI device names will be
different on each of our RAC nodes (it may even change on each particular node after a reboot).
This does not present a problem as we are using local device names presented to us by “udev”.
So, if you have not restarted your node after partitioning, run following command as root:
# partprobe
And now we will query the partitions with fdisk command:
# fdisk -l
Here is the results:

This is all for the volumes and the partitions at this stage of our project. When we clone our
Linux guest (a node) at a later time, the clone will have all these settings and configurations
already done. Notice that partitioning is only done once, since the storage is shared between all
nodes.
Virtual Oracle RAC. Task 7: Preparing Linux guest for Oracle installation –
part 1
5 Votes
is here.
Before we run installation process for Oracle software there is some preparation needed. If not
specified otherwise, the commands in this chapter are to be executed in SSH session as a user
"root". If you are not familiar with UNIX and Linux, the "#" in the beginning of the command
line denotes that you are logged in as user "root", the "#" is not a part of the command.
We will try to do as much work on a single node of the RAC before we clone it. After that some
changes will have to be done on both nodes (or more, if you are going for more than two nodes).
Optional: Freeing up some system resources
This Linux distribution comes with many services activated by default. Some of them we are not
going to use and so they can be disabled. Following commands are taking care of just that (run
them as root). The change will take place after next reboot.
# chkconfig sendmail off

# chkconfig bluetooth off
# chkconfig ip6tables off
# chkconfig yum-updatesd off
# chkconfig cups off
Use this command to verify the effect of the deactivation:
# chkconfig –list
Another important change concerns default run level of the system. Under normal circumstances
runlevel 3 is sufficient, while by default the system will proceed to level 5 (where it starts GUI
desktop – X Windows). As we are going to use "telnet", i.e. SSH tools to communicate with our
Linux systems we do not need to run our guests at level 5. Here is how to change the default run
level.
# cd /boot/grub
# vi menu.lst
Make changes in the menu.lst as shown in this picture:
Again, changes will take effect after reboot.
If you ever need to get to the X Windows Desktop again all you have to do is to issue "init 5"
command as root. When you are done you can issue "init 3" command to bring the system back
to level 3, no shutdown or reboot is necessary.
Special considerations for user "oracle" and UNIX groups
RAC technology requires that all nodes of the cluster have a shared access to the Oracle
Clusterware software. This is achieved by using Oracle Cluster File System (OCFS2 in our case)
. When using OCFS (and, if you think of this, any other shared storage solution) all nodes will
authenticate themselves to the OCSF with their user IDs. In order to provide consistent access to
files these IDs must match on all nodes. In other words, all users and groups that we create must
have exactly same numerical IDs on different nodes. This part is easy for us, because we create
our IDs only once and all our other nodes will be produced by cloning (that does not alter IDs).
Create Groups
The following O/S groups will be created:
Oracle Oracle Group UNIX Group

Description
Privilege Name name
Oracle Inventory and Software
. . oinstall
Owner
Database Administrator SYSDBA OSDBA dba
Database Operator SYSOPER OSOPER oper
# groupadd -g 501 oinstall
# groupadd -g 502 dba
#groupadd -g 503 oper
Create user "oracle"
#useradd -m -u 501 -g oinstall -G dba,oper -d /home/oracle -s /bin/bash -c "Oracle Software

Owner" oracle
Verify the created user:
# id oracle
Set the password for the oracle account:
# passwd oracle
Verify That the User "nobody" exists (If you have installed the suggested Linux version you
may skip this check)
1. To determine if the user exists, enter the following command:
# id nobody
2. If the user nobody does not exist (English is an interesting language if you think of that), then
enter the following command to create it:
# useradd nobody
Create the Oracle Base Directory
Let’s take a look at our internal disk on odbn1 machine, there are two ways to produce the report
(not all UNIX flavors support -H option):
So we have 33GB of space to work with, 2.7GB is occupied by Linux itself and its swap.
The next step is to create a new directory that will be used to store the Oracle Database software.
This location is later referred to by $ORACLE_BASE environment variable.
# mkdir -p /u01/app/oracle
The "-p" option tell the mkdir command to create the whole chain of directories at once.
After the directory is created, we specify the correct owner and group (why? because we work as
"root" user and newly created objects will have root’s properties):
# chown oracle:oinstall /u01/app/oracle
And now permissions:
# chmod 775 /u01/app/oracle
Create the Oracle Clusterware Home Directory
This location is later referred to by $ORA_CRS_HOME environment variable.
# mkdir -p /u01/app/crs
# chown oracle:oinstall /u01/app/crs
# chmod 775 /u01/app/crs
Create Mount Point for OCFS2 / Clusterware
# mkdir /u02
# chown oracle:oinstall /u02
# chmod 775 /u02
Create Login Script for oracle User Account
Login as user "oracle" and create the following .bash_profile:

Note: After we clone our first node into other nodes we will have to change a number of things
in each of them. One of such changes is to make sure that each RAC node has a unique Oracle
SID (defined in the user’s profile).
# su – oracle
Use some editor to create (or rather edit, since in OEL5 it is already there) the following profile:
.bash_profile for "oracle" User Account

# .bash_profile
# Get the aliases and functions

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
export JAVA_HOME=/usr/local/java
# User specific environment and startup programs

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=/u01/app/crs
export ORACLE_PATH=$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/admin
export CV_JDKHOME=/usr/local/java
# Each RAC node must have a unique ORACLE_SID. (i.e. racdb1, racdb2,...)
export ORACLE_SID=racdb1
export PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS10=$ORACLE_HOME/nls/data
export NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
Swap Space Considerations
Exit your "oracle" user session and now you are "root" again.
We will need at least 1GB of total memory (real + swap) to install Oracle Database 10g Release
2.
Here is how to check how much we’ve got (another way is to run the "top" command):
# cat /proc/meminfo | egrep ‘MemTotal|SwapTotal’
Configure kernel parameters and shell limits for Oracle
Changes to /etc/sysctl.conf file.
You can make changes to your kernel configuration as follows (copy and paste text from "cat" to
"EOF", do not include "#"):
# cat >> /etc/sysctl.conf <<EOF
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
EOF
Just in case I am listing the contents of /etc/sysctl.conf file after the changes:
kernel.shmmax = 4294967295
kernel.shmall = 268435456
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
Activate your changes with following command (or reboot the machine):
# sysctl -p
You can verify the new settings with this command:
#/sbin/sysctl -a
Setting Shell Limits for the oracle User
Run following command to change limits for user "oracle" (while you are still logged in as
"root"):
# cat >> /etc/security/limits.conf <<EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF
# cat >> /etc/pam.d/login <<EOF
session required /lib/security/pam_limits.so
EOF
Shell settings
C shell and its variants use /etc/csh.login profile (this one is loaded before local profile). Add
following lines to it:
# cat >> /etc/csh.login <<EOF
if ( \$USER == "oracle" ) then
limit maxproc 16384
limit descriptors 65536
endif
EOF
All other shells (such as Bourne, Korne and Bash) load /etc/profile, so it has to be amended like
this:
# cat >> /etc/profile <<EOF
if [ \$USER = "oracle" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF
Configure the hangcheck-timer Kernel Module
The module "hangcheck-timer.ko" is already installed in OEL5 and this can be verified by this
command:
# find /lib/modules -name "hangcheck-timer.ko"
Set hangcheck timer parameter to values recommended by Oracle:
# echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >>

/etc/modprobe.conf
To test the module and its new setting run following commands:
# modprobe hangcheck-timer
# tail -10 /var/log/messages

Set up NTP client for synchronized time on RAC nodes
Oracle installer will have a trouble if RAC nodes have different date and time showing by their
clocks. The easy way to fix this problem is to set up NTP client on our machine(s). I chose to use
Openfiler machine as a source of time, rather than pointing to external NTP server.
Use the "tail" command in one SSH window to verify that NTP setup is progressing properly
(leave this window and come back to it later).
# tail -f /var/log/messages
In another SSH window make following changes:
# cd /etc
Make a backup of NTP settings
# cp ntp.conf ntp.conf.bak
Edit the NTP settings file to include these lines (pointing to openfiler-priv network interface):
server 10.10.1.20
restrict 10.10.1.20 mask 255.255.255.255 nomodify notrap noquery
No other NTP servers should be specified in this file, except for the default line with 127.127.1.0
address (which is optional).
Restart ntp daemon:
# service ntpd stop
# service ntpd start
If succeeded, the response will be "OK" and look like this:
The output of the "tail" command in first window may look like this by now:
We can also use "ntpq" to verify the settings (or "ntpdc" utility with "peers" command):
# ntpq -p
If you look carefully at the IP addresses you will notice that "step time server" shows IP of the
"router", rather than that of the Openfiler. It is clarified when you ran "ntpq" command.
Although we used Openfiler’s IP in the configuration file the command shows proper server
name in the output.
As long as "st" column (Stratum) shows value other than 16 and "jitter" is reasonably small, all is
working properly. The result of this setup is that odbn1 has its clock synchronized with that of
the Openfiler.
RAC related changes to the system configuration files
Have this changes done as root.
Add following lines to the /etc/sysctl.conf file
If you are using Linux other than OEL5 U3 you may also need to add these lines:
kernel.shmmax = 4294967295
kernel.shmall = 268435456
kernel.sem=250 32000 100 128
fs.file-max=65536
net.ipv4.ip_local_port_range=1024 65000
Have kernel load the new parameters:
# sysctl -p
Install the OCFS2
The OCFS2 software packages is supplied on CD #3 of the OEL5 U3 (but not installed by
default). Have disk #3 mounted (mount the image through the Virtual Box as you did before) and
issue following commands:
# mount -r /dev/cdrom /media/cdrom
# cd /media/cdrom/Server
# rpm -Uvh ocfs2-tools-1.2.7-1.el5.i386.rpm
# rpm -Uvh ocfs2-2.6.18-128.el5-1.2.9-1.el5.i686.rpm
# rpm -Uvh ocfs2console-1.2.7-1.el5.i386.rpm
Verify the installation of packages:
# rpm -qa | grep ocfs2 | sort
Install ASMLib 2.0
This library is also supplied on CD #3. Install and verify it by issuing the following commands:
# rpm -Uvh oracleasm-support-2.1.2-1.el5.i386.rpm
# rpm -Uvh oracleasm-2.6.18-128.el5-2.0.5-1.el5.i686.rpm

# rpm -qa | grep oracleasm | sort
# cd /
# eject
Install ASMLib Userspace Library
This one is not shipped along with CDs and will have to be downloaded. Get the oracleasmlib-
2.0.4-1.el5.i386.rpm from Oracle.
Have this rpm copied onto the guest by means of sftp into a new directory /root/rpms
Install the rpm by running these commands:
# cd /root/rpms
# rpm -Uvh oracleasmlib-2.0.4-1.el5.i386.rpm
Looks like we are done with most of the preparations that we can do on a single node before
cloning it.
Virtual Oracle RAC. Task 8: Cloning a Linux guest.
3 Votes
is here.
Shut down your virtual machines. You may keep Virtual Box GUI open. Open the command line
window. Type in the following commands (with a correction to your locations, use “oracle”
instead of “sun” if you have latest version of product):
set PATH=%PATH%;C:\Program Files\Sun\VirtualBox\
cd /d G:\VBmachine
mkdir odbn2
VBoxManage clonehd G:\VBmachine\odbn1\odbn1.vdi G:\VBmachine\odbn2\odbn2.vdi
This is how it looks in the process.
A few important moments here. The VBoxManage utility will report errors if source machine is
running, it will also report an error if you use relative path to the vdi file, only absolute path
works (at least that’s the case for release 3.1.2 of Virtual Box. Also, if you are tempted to use the
setvdiuuid tool I would advise against that as it may save you cloning time (5 min) but prove to
give some trouble later.
Next step is to add (or rather register) our newly cloned machine to the Virtual Box.
Bring up the Virtual Box GUI and click on “New” machine, fill out the details like this:
The memory is 512MB for now:

Now, the disk is “existing”, but it is not registered yet so it is not on the list.
Click on the folder button and Media Manager interface appears.
Click on “Add”, navigate to our new file (odbn2.vdi), add it and then “select” it:
Now we are back to the new machine dialog with cloned file available to be used. Click “Next”.
And “Finish”. Do not start this machine or you will ruin it.
What will happen if we run this new machine right now? The Virtual Box knows nothing about
this machine’s hardware at the moment. All these settings are not cloned or copied over when we
clone vdi image. It may be quite different with VMWare but here the image is not self contained.
Hardware setup is separate and contained in an xml file in the same folder as the vdi file. If we
run the machine now with all the default settings from that file the Linux will run new hardware
setup process and overwrite our careful network setup. We need to edit the xml file to give our
new machine proper setup.
In a text editor (Textpad will do, any plain editor) open odbn1.xml and odbn2.xml files. Copy
and paste (overwrite) the whole network section from first to the second. This configuration
section is contained between tags <Network> and </Network>. Save the modified odbn2.xml file
but do not start the machine yet. Do not start odbn1 as well until instructed to do so.
Close the Virtual Box GUI (if it was open) and start it up again, then click on “Settings” for
“odbn2″ machine. Only now our new settings will be read and accepted. Navigate to Storage
section and remove Floppy controller as we have no need for it.
Notice: in newer versions of Virtual Box the default type of disk controller is SATA rather than
IDE. In such case ou will need to delete the SATA controller, attach an IDE controller and then
attach the image of the disk (the vdi file) to it.
Then go to System section and uncheck Floppy from the boot order:
Start the openfiler machine now. When we start odbn2 we would like to see if it can
communicate to openfiler.
Now we can start the odbn2 machine (it is just a precise clone of odbn1 for now). Notice that all
network interfaces are started properly, that means MAC addresses in the hardware (network)
section matched those in the Linux itself, otherwise they would not start. The iSCSI seems to be
working as well.
Now let’s convince odbn1 clone that it is odbn2 (plastic surgery of sorts).
Bring up SSH terminal as if you were connecting to odbn1, log in as root.
# cd /etc/sysconfig/network-scripts/
# vi ifcfg-eth0
Edit the file replacing 11 with 12 in the IP address.
Do the same for the next adapter:
# vi ifcfg-eth1
Now change host name:
# cd /etc/sysconfig
# vi network
Change odbn1 to odbn2 in the HOSTNAME:
Save the file and reboot the machine:
# reboot
Watch the machine messages in the GUI. On the startup the cloned machine will notice its new
name:
Bring up an SSH window and connect to the IP address of odbn2.
Use “ifconfig” command to verify that IP addresses indeed have changed:
We also run “hostname” command to see if host name is correct:
Let’s see if openfiler is visible to the new server and can properly map the iSCSI targets.
While we are at it, let’s also change the ORACLE_SID in the oracle’s profile to “racdb2″.
# su – oracle
# vi .bash_profile
I would now take a backup of the whole farm (i.e. three virtual machines) by shutting them down
and copying all vdi and xml files someplace else. Include also an xml file from
C:\Documents and Settings\%USER%\.VirtualBox\VirtualBox.xml
That file is essential for the backup and restore of Virtual Box setup.
Virtual Oracle RAC. Task 9: Preparing Linux guest for Oracle installation –
part 2
2 Votes
is here.
Configure SSH on RAC Nodes
Oracle Clusterware and the Oracle Database software are installed from one node in a RAC
cluster. Files are copied by Oracle Installer (OI) to other nodes using ssh and scp commands and
OI expects "user equivalence" to be setup on all nodes. This means UNIX id used for installation
("oracle" user in our case) is trusted on all nodes and ssh/scp commands won’t ask for password.
We are going to set up this equivalency.
Log on as the oracle UNIX user account (on both nodes):
# su – oracle
Create the .ssh directory in the oracle user’s home directory (on both nodes):
$ rm -R ~/.ssh
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
Generate an RSA key pair (public and private key) for the SSH protocol. When prompted: accept
the default location for the key files, enter and confirm a pass phrase (make up something, it may
or may not be same on all nodes, write it down, you will need it later). Do it on both nodes.
$ /usr/bin/ssh-keygen -t rsa
The command above creates:
public key in ~/.ssh/id_rsa.pub
private key in ~/.ssh/id_rsa
The results so far will look like this (first node), the arrows indicating your input:
and second node:
Create "authorized_keys" file on first node only. This "authorized_keys" file is a collection of
RSA public keys from all nodes in cluster. When all keys are collected this file will be copied to
other nodes.
The following commands are run from one node (odnb1):
$ ssh odbn1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Notice: yes, you are connecting with SSH from odbn1 to odbn1, this is done in order to produce
additional SSH configuration file "known_hosts" and properly add contents of "id_rsa.pub" to
"authorized_keys". Keep the RSA passphrase and oracle’s password handy, red arrows denote
your input:
Still from the same host odbn1 issue this command:
$ ssh odbn2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
What has been done just now? We have connected to odbn2 and appended the contents of its
"id_rsa.pub" file to the "authorized_keys" file on odbn1. This looks like that so far:
Examine your "authorized_keys" file on odbn1, you will see it includes keys from both nodes.
Now copy this file to the other node (you are still being asked for oracle’s password):
$ scp ~/.ssh/authorized_keys odbn2:.ssh/authorized_keys
Change permissions on the "authorized_keys" file on both nodes:
$ chmod 600 ~/.ssh/authorized_keys
Test connection from odbn1 (first to odbn1 itself, then to odbn2):
$ ssh odbn1 uname -a
$ ssh odbn2 uname -a

Notice: we are not prompted for the oracle’s password, only for the passphrase of the key file.
Now we are going to get rid of that prompt either.
Enter following commands on odbn1:
$ exec /usr/bin/ssh-agent $SHELL

$ /usr/bin/ssh-add
The ssh-agent is a process that will supply the passphrase to new SSH processes in this session.
When we restart the nodes later (or leave this session) the two commands (ssh-agent and ssh-
add) are to be re-run.
Try running same ssh commands as before and you will see no prompts whatsoever (when
running from from odbn1, that is):
Test connection on private and public interfaces as well (this is needed later by Oracle Installer):
$ ssh odbn1-priv
$ ssh odbn1-pub
Well, at this point we have configured the SSH to satisfy the Oracle Installer (we hope).
Virtual Oracle RAC. Task 10: Oracle installation in RAC nodes – part 1
1 Votes
is here.
Taking care of memory
As mentioned before, now comes the time to increase the virtual machines memory to 1024MB.
Take this chance to run a backup of the whole project while guests are shut down, then change
Base Memory settings on both nodes. Start all the machines again.
Getting an X Server running on the PC
There are few small things to take care of before we launch Oracle Installer. Namely, we need an
X Server on our PC. This can be any server of your choosing, such as Cygwin/X, Xming or
other. I will be using Cygwin/X here. Make sure your X Server includes xhost utility, sometimes
it is not installed by default. This is how to locate the xhost utility during instalation:
Have your X Server installed and started on your PC. If using Cygwin, run in CMD window (or
via an icon):
C:\bin\cygwin2\bin\run.exe /usr/bin/bash.exe -l -c /usr/bin/startxwin.exe
This will usually start X server on display 0.0 and bring up an xterm window. If xterm window
did not appear have it started manually. In this xterm window give an access control command:
$ xhost +10.10.1.11
$ xhost +10.10.1.12
This xhost command has enabled the obdn1 host to use your X Server to display x clients. I think
I have just saved you a few days of frustration. The access control matters of X Server are not
covered clearly in documentation and articles would suggest using X11 forwarding in SSH. The
X11 forwarding in SSH will indeed take care of all the neccessary settings automaticatlly,
however this setup won’t be usable by Oracle Installer. Therefore you should check that the SSH
sessions you run from your PC to the RAC nodes are not using the X11 forwarding.
So, if you are using PuTTY make sure the X11 forwarding is not selected for the RAC node
sessions:
As user "oracle", from the odbn1 node, from that same SSH session you have enabled SSH no-
password connections (go back to refresh your memory). Ff you have restarted servers or logged
in again in SSH you will need to repeat steps with ssh-agent and ssh-add as described before (on
odbn1 only).
$ export DISPLAY=10.10.1.1:0
Test that your session can run some X windows application and show it on your PC’s X Server:
$ xclock &
This should spawn an xclock application and show it on your PC via X Server. If this did not
work you will have to troubleshoot your setup and proceed with Oracle installation only when
resolved.
Notice: The IP address we supplied in DISPLAY variable is the virtual "router" on our host PC,
since X Server expect connection from 10.10.1.11, we should "talk" to it on the same network.
Having endured so far we are now going to do more exciting things.
Configure OCFS2
Notice: OCFS2 package (RPM) installation instructions are in Task 7 , go back if you missed
them. If you already cloned the node then do this step on both nodes.
OCFS2 will be configured to use private network i.e. hostnames as in etc/hosts:
10.10.1.11 odbn1-priv
10.10.1.12 odbn2-priv
Notice, that although we will be using the private network IP addresses we will still use
hostnames as they appear on public network. I know, it is confusing. There would not be such a
confusion with just one network (either private or public).
As user "root", on the odbn1 node, start a new SSH session and set the DISPLAY varaible:
# export DISPLAY=10.10.1.1:0
Now, bring up OCFS2 configuration tool to edit /etc/ocfs2/cluster.conf file:
# ocfs2console &
Select [ Cluster ] – [ Configure Nodes ]. This will start the OCFS2 Cluster Stack and bring up
next dialog ("Node Configuration")
There will be a pop-up message:
Dismiss it and proceed to "Node Configuration". Add two RAC nodes as below:
When you click "Apply" the screen changes to node status and node numbers:
Exit the console and repeat the same process on the other node of the cluster supplying same
exact data in the same order.
The /etc/ocfs2/cluster.conf file will look like this after configuration is done (do not just copy
this file over to other node):
O2CB Cluster Service
O2CB is the OCFS2′s cluster stack of services that are listed below:
 NM: Node Manager that keep track of all the nodes in the cluster.conf
 HB: Heart beat service that issues up/down notifications when nodes join or leave the
cluster
 TCP: Handles communication between the nodes
 DLM: Distributed lock manager that keeps track of all locks, its owners and status
 CONFIGFS: User space driven configuration file system mounted at /config
 DLMFS: User space interface to the kernel space DLM
All of the listed services are included in the single service o2cb (/etc/init.d/o2cb).
Now issue following commands to have O2CB start on boot up and have other needed attributes
(on both nodes):
# /etc/init.d/o2cb offline ocfs2

# /etc/init.d/o2cb unload
# /etc/init.d/o2cb configure
Below shown non-default responses to the configuration questions (marked by arrows):

Repeat the same steps on the other node.
Issue this command to review the service status so far:
# /etc/init.d/o2cb status
Make sure that o2cb serivce will be started at proper run level:
# chkconfig –list o2cb
Levels 3 through 5 should be "on":
Format and mount the OCFS2 Filesystem
This task is to be done from one node only
# mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracrsfiles /dev/iscsi/crs/part1
The following output is produced:
Now mount the newly formatted partition (that was labeled "oracrsfiles") under directory /u02:
# mount -t ocfs2 -o datavolume,nointr -L "oracrsfiles" /u02
Repeat the mount command on the other node. Verify that the file system is mounted correctly:
# mount
Repeat verification on the other node. Your output should be similar to the one below (newly
mounted file system is highlighted):
Make these changes permanent by adding the following line to the /etc/fstab file on both RAC
nodes:
LABEL=oracrsfiles /u02 ocfs2 _netdev,datavolume,nointr 0 0
This is how it will look like:
Adjust ownership and permissions on the new filesystem (from one node only):
# chown oracle:oinstall /u02

# chmod 775 /u02
Create directories to contain Oracle Clusterware shared files (also from one node only):
# mkdir -p /u02/oradata/racdb
# chown -R oracle:oinstall /u02/oradata
# chmod -R 775 /u02/oradata
Reboot both nodes. Check if our setup can survive the reboot. Use "mount" command to see if
the new file system is still there and is mounted in the proper place. You can also use o2cb status
check to make sure the serive is functioning properly (from both nodes):
# /etc/init.d/o2cb status
Configure ASMLib 2.0
Notice: ASMLib package (RPM) was added in Task 7 , go back if you missed this step. If you
have already cloned the node then do this step on both nodes.
The configuration task is to be done on both nodes:
# /etc/init.d/oracleasm configure
The following picture shows your input marked by red arrows:

Create ASM disks
We are creating the ASM disks from one node only. Run following commands:
# /etc/init.d/oracleasm createdisk VOL1 /dev/iscsi/asm1/part1

The commands above have created ASM disks in iSCSI targets that are mapped to local devices:
The newly created disks can be listed by using this command:
# /etc/init.d/oracleasm listdisks
On the other node we only need to do a scandisk to recognize the new volumes and then list
them:
# /etc/init.d/oracleasm scandisks
# /etc/init.d/oracleasm listdisks
Prepare Oracle software media
Download the following from http://www.oracle.com/technetwork/database/10201linuxsoft-

097986.html:
Software Title File Name

Oracle Clusterware Release 2 (10.2.0.1.0) 10201_clusterware_linux32.zip
Oracle Database 10g Release 2 (10.2.0.1.0) 10201_database_linux32.zip
Oracle Database 10g Companion CD Release 2 (10.2.0.1.0) 10201_companion_linux32.zip
Now there is two ways to proceed. You can either:
 copy the archives (with sftp) onto odbn1 node as user "oracle", into some directory
 unzip the archives and re-pack them into an ISO image, then mount the image in VB
guest
I will be using the second option as not to inflate the file system of the virtual machine (after all,
it is a one time install).
So, have the archives unzipped into separate directories on your PC, then use a utility such as
Infra Recorder to pack the software into a DVD ISO image, just like in the picture below:
Name this image "rac32media.iso":
Add newly created image to Virtual Media Manager and then mount it in both VB machines.
Then mount it in the nodes like this:

# cd /media/cdrom
Install cvuqdisk package
This package is needed later by Cluster Verification Utility (CVU). Issue the following
commands (do it on both nodes):
# cd /media/cdrom/10201_clusterware_linux32/clusterware/rpm
# export CVUQDISK_GRP=oinstall
# rpm -iv cvuqdisk-1.0.1-1.rpm
Verify installation of the package:
# ls -l /usr/sbin/cvuqdisk
Install and run Cluster Verification Utility (CVU)
This utility will be run from odbn1 node as user "oracle". The SSH agent will have to be initiated
(again) for this purpose.
Open another SSH session (we may want to use our "root" session later) and login as "oracle",
then execute following commands:
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Install the CVU utility:
Verifying Oracle Clusterware Requirements with CVU
Once all prerequisites for running the CVU utility have been met, we can now check that all pre-
installation tasks for Oracle Clusterware are completed by executing the following command as
the "oracle" UNIX user account from linux1:
$ mkdir -p /home/oracle/bin/cluvfy
$ cd /home/oracle/bin/cluvfy
$ cp /media/cdrom/10201_clusterware_linux32/clusterware/cluvfy/* .
$ mkdir -p jdk14
$ unzip jrepack.zip -d jdk14
$ export CV_HOME=/home/oracle/bin/cluvfy
$ export CV_JDKHOME=/home/oracle/bin/cluvfy/jdk14
Run pre-requisites check report
$ ./runcluvfy.sh stage -pre crsinst -n odbn1,odbn2 -verbose | tee cvu_report_1.txt
Review the CVU report (cvu_report_pre.txt). There are several errors like "Check: Package
existence" that can be ignored.
There is a messages in connectivity section reading "Could not find a suitable set of interfaces
for VIPs". This one can be ignored too, as it is caused by a known bug documented in Metalink
Note 338924.1 (has to do with non routable ranges of IP addresses).
If you get any errors other than those in my report there might be a problem with your setup.
Connectivity errors must be resolved. For reference and troublesooting see Metalink ID
338924.1 and Forum
Run the Hardware and OS Setup check report
$ ./runcluvfy.sh stage -post hwos -n odbn1,odbn2 -verbose | tee cvu_report_2.txt

Review the report, it should say "Post-check for hardware and operating system setup was
successful."
The utility checks connectivity again and then checks storage, proper report will look like this:
When reports are ok and all serious problems have been resolved we can move on.
Install Oracle Clusterware
The two shared files (file groups) used by Oracle Clusterware will be stored on the Oracle
Cluster File System (OFCS2) we created earlier. The two shared Oracle Clusterware file groups
are:
 Oracle Cluster Registry (OCR)
File 1 : /u02/oradata/racdb/OCRFile
File 2 : /u02/oradata/racdb/OCRFile_mirror
Size : (2 * 100MB) = 200M
 CRS Voting Disk
File 1 : /u02/oradata/racdb/CSSFile
File 2 : /u02/oradata/racdb/CSSFile_mirror1
File 3 : /u02/oradata/racdb/CSSFile_mirror2
Size : (3 * 20MB) = 60MB
Issue these commands as user "oracle":
$ cd ~
$ /media/cdrom/10201_clusterware_linux32/clusterware/runInstaller -ignoreSysPrereqs &
The Oracle Installer will appear, click "Next" as in the picture below:
Have Name and Path filled as shown, click Next:

In the next screen the installer runs various checks (the checks must be all right if our manual
check reports were fine) and shows the final result, click Next:
The Cluster Configuration screen appears with one node filled out already, click on Add and
supply information for the other node as shown so it looks like this:
Click "Next". The Network Interface configuration screen appears (it shows subnets and network
adapter names). As you can see, both interfaces are shown as "Private". We need a public
interface in this configuration. Select "eth0" and click "Edit".
Designate "eth0" as public interface. Click "Ok"
Our interfaces and their purpose look like this now:
Click "Next" and OCR location dialogue appears.
Supply following OCR locations in the next screen (leave Normal Redundancy selected):
Specify OCR Location: /u02/oradata/racdb/OCRFile

Specify OCR Mirror Location: /u02/oradata/racdb/OCRFile_mirror
Click Next:
Supply following locations of voting disks in the next screen (again, leave Normal Redundancy
selected):
Voting Disk Location: /u02/oradata/racdb/CSSFile

Additional Voting Disk 1 Location: /u02/oradata/racdb/CSSFile_mirror1
Additional Voting Disk 2 Location: /u02/oradata/racdb/CSSFile_mirror2
Click Next:
Summary screen appears. Click Install:
Notice how, as the install progresses, the software is being copied over to the other node in the
cluster
Notice: Sometimes one of the virtual machines may crash on memory related errors (running out
of usable memory on the host PC).
If this happens on odbn2 it is safe to reboot it and wait till installer continues (say 5 minutes after
reboot).
If this happens on odbn1 it is more complicated. You will need to:
 cancel the installer

 restart machines if they are down
 delete subdirectories under of the /u01/app/crs and /u01/app/oracle
 shut down all VM’s
 free up host PC’s memory or reboot it
 restart VM’s
 restart the Installer
 good luck
Upon completion of the remote copy operations (ssh and tar processes) the next screen shows up:
And pop up screen shows:
Now pay attention closely.
Important: Do not close this Oracle Installer dialog until instructed to do so.
Important: run orainstRoot.sh script on both nodes, one at a time.
Important: run /u01/app/crs/root.sh script on odbn1 and wait until it completes, then proceed
with the same script on the odbn2.
Output of the root.sh script on odbn1:

The /var/log/messages shows this on the first node at the completion if the script:
Now proceed with the root.sh on odbn2 node (read a note further on). The output is here:
The error at the end (libpthread.so.0) is expected, here is the manual fix to this trouble.
Edit "/u01/app/crs/bin/vipca" file on both nodes adding the following statement after line 124:
unset LD_ASSUME_KERNEL
The snapshot below shows edited text with added "unset" command:
Next, edit "/u01/app/crs/bin/srvctl" file on both nodes adding same statement after line 167 so it
will look like this:
To take care of this problem in future, when more software is installed into new Oracle Homes or
patches applied to existsing software, the following files have to be re-edited:
<ORA_CRS_HOME>/bin/vipca
<ORA_CRS_HOME>/bin/srvctl
<RDBMS_HOME>/bin/srvctl
<ASM_HOME>/bin/srvctl
Since "vipca" utility failed, we will re-run it manually. Run vipca from odbn2 (still as root):
# export DISPLAY=10.10.1.1:0
# /u01/app/crs/bin/vipca
The vipca fails again with this cryptic message:
" [Error 0(Native: listNetInterfaces:[3])]"

The message has to do (again) with the range of IP addresses we selected for the interfaces.
Notice: If you have received another cryptic message [Exception in thread "main"] this simply
means you forgot to set the DISPLAY variable.
This trouble has to be manually fixed.
Run following commands as root from odbn2 (notice that the IP addresses here are of the
networks, rather than specific adapters):
# /u01/app/crs/bin/oifcfg setif -global eth0/10.10.2.0:public

# /u01/app/crs/bin/oifcfg setif -global eth1/10.10.1.0:cluster_interconnect
Let’s see now the resulting configuration of VIP network:
# /u01/app/crs/bin/oifcfg getif
So, let’s run vipca again (still on odbn2)
# /u01/app/crs/bin/vipca
The following screen pops up (click on Next):

Agree wih the next screen (eth0 is indeed the public interface):
As soon as we supply first IP alias name as "odbn1-vip" the rest of fields is magically filled out.
Remove the domain name from the second alias. Click "Next".
Summary screen appears. Click "Finish".
The progress screen is shown:
When this is finished and you uncross your fingers, this screen shows. Click "Ok":
The summary appears:

Exit the vipca.
Now you can back to Clusterware installer dialog and dismiss the "Execute Configuration
scripts" dialogue.
The installer continues

And … fails on Cluster Verification Utility:
We ignore this message and exit the installer.
Now let’s run a few verification commands on both nodes:
# /u01/app/crs/bin/olsnodes -n
# /u01/app/crs/bin/crs_stat -t -v
# /u01/app/crs/bin/crsctl check crs
# ls -l /etc/init.d/init.*
Here are the results (one node shown, the other must show identical results):
Now we are ready to install Oracle database software.
Virtual Oracle RAC. Task 11: Oracle installation in RAC nodes – part 2
Rate This
is here.
Verifications before installation
Now we are ready to install Oracle database software but before that we need to run the
verification utility again (database pre-installation requirements). Run these commands from
odbn1 as user oracle:
$ cd /home/oracle/bin/cluvfy
$ export CV_HOME=/home/oracle/bin/cluvfy
$ export CV_JDKHOME=/home/oracle/bin/cluvfy/jdk14
$ ./runcluvfy.sh stage -pre dbinst -n odbn1,odbn2 -r 10gR2 -verbose | tee cvu_report_3.txt
Review the report. There are some errors that are expected and can be ignored, in particular:
"Could not find a suitable set of interfaces for VIPs" and two errors with "Package existence
check failed for …"
The connectivity section looks like this (see below), it says "failed" but we can safely continue
with installation.
Oracle Database software installation
As "root" execute this command from odbn1:
As "oracle" execute these commands from odbn1:
$ cd ~
$ /media/cdrom/10201_database_linux32/database/runInstaller &
The installer window appears, click "Next":
In the next screen choose "Custom" and click on "Next":

Agree with defaults in the next screen and click "Next":
In the next screen make sure "Cluster Installation" and all nodes are selected. Click "Next":
Acknowledge the summary screen and click "Next":
The verifications screen appears. Hopefully you will get a clean bill of health. Click "Next":
In the next screen correct the name of group for OSOPER as shown. Click "Next":
In this screen select "Install Database Software Only". Click "Next":
Summary screen appears. Click "Install":
Installation proceeds.
Make sure the Virtual Box machines (GUI) are somewhere visible. You can log into them and
run "top" to make sure the machines are alive. It may happen that during the installation one of
the machines will stop responding complaining about low memory (it will be automatically
paused and Oracle installer will appear stuck). In such case you can un-pause (resume) the
machine and/or reboot it. In some cases the machine (well, the clusterware) will reboot itself and
then the installer will proceed. I have observed this behaviour in the phase of "Remote
Operations In Progress".
If all goes well, this configuration pop-up appears:

Run the "root.sh" script on odbn1 and then on odbn2 (one at a time only).
The output from odbn1 is show here:

Exit the OUI.
A little cleanup needs to be done. This bug fix was mentioned before. Edit following file (on
both nodes):
$ vi $ORACLE_HOME/bin/srvctl
Add the following text at line 168:
Save the file and exit.
This concludes the software installation phase.
Adding TNS Listener
This is to be done as user "oracle" from odbn1 only:
$ netca &
The NETCA screen appears, proceed with "Cluster configuration":
Make sure both nodes selected and click "Next":

Select first option:
Agree to "Add":
Accept default LISTENER name:

Accept default protocol:
And default port:
Leave "No" for "configure another listener", proceed (this may take some time, screen appears
frozen):
Then confirmation screen appears, click "Next":
Hey, we’ve been here, right? Right, now click on "Finish":
We can verify the results on both nodes as shown here:
and…
We are done with listeners.
Virtual Oracle RAC. Task 12: Creating Oracle database in RAC nodes
Rate This
is here.
Start Database Configuration Assistant from odbn1 node:
$ dbca &
Leave fisrt option selected:
Leave "Create a Databse" selected:

Click "Select All" nodes:
Select "Custom Database":
Fill out the Global Database Name. ,We have discussed the importance of the domain name in
the networking of the RAC in the very beginning of this article. Although the domain name has
to be there, it is not a real internet domain. Still, it has to match what you have in your "hosts"
file.
I am configuring a lightweight RAC configuration so I am not installing OEM or other advanced
options (it is up to you if want OEM, Java etc):
Specify password for all admin accounts (and write it down)
Select ASM as a storage option:
Specify SYS password for ASM instance and fill out location of SPFILE as shown (SPFILE has
to be on a shared disk):
Acknowledge the confirmation:
Promising signs…
When next screen appears click "Create New" disk group:
Fill out details as shown, click "Ok":

Click "Create New" again:
Fill out details as shown:
In the next screen select RACDB_DATA1 as a storage option for the database (unselect first
group):
Agree to the "Oracle Managed Files" option:
Fill out details for Flash Recovery as shown (total size of the disk group minus some 10%):
As mentioned before, I am installing barebone database, but you can add more options:
I am even removing standard components as I am not going to use them in this configuration:
In the next screen click "Add":
Fill out "Service Name" as racdb_taf (this is for Transparent Application Failover):
Leave both nodes as "Preferred" and select "Basic" as TAF Policy:

I am accepting default settings (you can change them as you wish):
Same with storage options, I am accepting all defaults:
Leave "Create Database" option selected and select "Generate Database Creation Scripts" as well
(for later review). Click "Finish":
Summary screen pops up. Cross fingers and click "Ok"
Database installation progresses and in the end this screen appers, click "Exit":
A few moments later the RAC starts (this phase may take a while):
Then the OUI will close by itself.
A little cleanup needs to be done (again). Edit the following file (on both nodes):
$ vi $ORACLE_HOME/bin/srvctl
Add the following text at line 168:
Save the file and exit.
Now we should have a fully functional Oracle RAC.
Let’s run a few checks.
To make sure all registered RAC applications are running (as user "oracle" from any node):
$ $ORA_CRS_HOME/bin/crs_stat -t
To verify the racdb_taf service, start a sqlplus (on any node):
$ sqlplus ‘/as sysdba’
Give following command:
SQL> show parameter service
See that there are two services displayed. If there is only one or none they can be added
manually:
SQL> alter system set service_names = ‘racdb.idevelopment.info, racdb_taf’ scope=both;
Just for reference
Stopping the Oracle RAC 10g Environment on a single node
$ export ORACLE_SID=racdb1
$ srvctl stop instance -d racdb -i racdb1
$ srvctl stop asm -n odbn1
$ srvctl stop nodeapps -n odbn1
Starting the Oracle RAC 10g environment on a single node
$ export ORACLE_SID=racdb1
$ srvctl start nodeapps -n odbn1
$ srvctl start asm -n odbn1
$ srvctl start instance -d racdb -i racdb1
Start or stop all Instances of the RAC at once
$ srvctl start database -d racdb

$ srvctl stop database -d racdb
Virtual Oracle RAC. Task 13. Having some fun with RAC
1 Votes
is here.
We could do a few more things to make the good work perfect, for instance we can enable
archive log mode in our RAC and set up backup with RMAN but this is already outside of the
scope of this little (I am kidding, it feels quite big to me) project.
What I really want to do now is to connect to the RAC database, run a long query and then kill
one of the nodes. Wondering if RAC will stand to its reputation.
Configure client side TNS (tnsnames.ora)
Add following lines:
RACDB_TAF =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.10.2.200)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.10.2.201)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = racdb_taf.harzion.org)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 180)
(DELAY = 5)
)
)
)
In some cases you will need to add a following line to sqlnet.ora (to overcome possible ORA-
12170 error):
sqlnet.inbound_connect_timeout = 30
Connecting with sqlplus
Bring up a sqlplus in your host PC and connect to the cluster:
SQL> connect system/password@RACDB_TAF
Run this "long" query that will keep the RAC occupied for a while:
SQL:> select * from help t1, help t2;

While the query’s output is running (it may run like this for hours), we will reboot odbn1 node (I
did the "shutdown" but you can do "reboot"):
Next picture shows that OCFS was stopped properly:
The other node is running happily if quite busy all the meanwhile:
And the sqlplus is still producing the output…
The odbn1 node is coming back, OCFS loaded, then ASM follows
If you log in to the console and run the "top" command filtering user "oracle" (press "u" to
specify user), you will notice that for a minute or two there will be no processes for "oracle",
then they will start appearing.
and finally you will have plenty of them, just as on the other node:
If noting happens within 5 minutes though, then most probably you have skipped the step where
we edited "svrctl" script. Go back, edit and reboot again.
Now, just for fun, I will reboot the second node, odbn2. I do not have much hope that my query
will survive this too. It would be too good.
Sure enough, just as OCFS is going down, the query dies as well, although the first node is
running again:
Well, I consider this project concluded, thanks for your patience and comments.

Building A Virtual Oracle RAC - VIRTUAL Box - OLE5

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Building A Virtual Oracle RAC - VIRTUAL Box - OLE5

Uploaded by

Copyright:

Available Formats

Building a Virtual Oracle RAC – Project Introduction

Filed under: Networking in Virtual Environment, Oracle Database, Virtualization — Tags:

Now to the choice of hardware and software.

My reasons for choosing particular software:

 Task 1. Installing and configuring Sun Virtual Box.

 Task 2. Installing Openfiler in Sun Virtual Box.

 Task 3. Configuring Openfiler to simulate iSCSI targets.

 Task 4. Creating a Linux guest, configuring hardware and Installing Linux.

 Task 5. Patching Linux guest.

 Task 6. Configuring iSCSI targets in Linux and partitioning the volumes.

 Task 7. Preparing Linux guest for Oracle installation – part1

 Task 8. Cloning a Linux guest.

 Task 10. Oracle installation in RAC nodes – part 1.

 Task 11. Oracle installation in RAC nodes – part 2.

 Task 12. Creating Oracle database in RAC nodes.

 Task 13. Having some fun with RAC.

eth0 is for public network

eth1 is for private network

Run Virtual Box, access Virtual Box preferences

Now create another (second) Host-Only adapter:

Virtual Oracle RAC. Task 2: Openfiler – installation in Sun Virtual Box

I am going to install Openfiler 2.3 (x86) in Sun Virtual Box.

Follow the generic installation guide for Openfiler here http://www.openfiler.com/learn/how-

# Notice that loopback address do not have a real host name in it

# Database nodes – Public Virtual IP (VIP) addresses

# Openfiler eth0 – public

Virtual Oracle RAC. Task 3: Openfiler – iSCSI targets setup

# Notice that loopback address do not have a real host name in it

# Database nodes – Public Network – (eth1)

# Database nodes – Public Virtual IP (VIP) addresses – (eth1:1)

# Openfiler – private and public interfaces

Now we do following steps:

 Enable iSCSI services from [Services] / [Manage Services] page.

Confirm default MTU on next screen:

Now click on [ Clock Setup ] as shown below:

restrict 10.10.1.11 nomodify notrap

restrict 10.10.1.12 nomodify notrap

After that you will need to restart the NTP service:

# service ntpd restart

Navigate to [Block Devices] section (see below):

iSCSI / Logical Volumes

In the end you will get a setup like this:

The last sub-tab – “CHAP” is not relevant.

Virtual Oracle RAC. Task 4: Installing Oracle Enterprise Linux 5 in Sun

I recommend going with dynamically expanding storage option:

Virtual Oracle RAC. Task 5: Patching Oracle Enterprise Linux 5

Network verification and configuration:

Edit /etc/hosts on “odbn1″ by copy and paste it or use ftp.

Verify that firewall is indeed stopped by running this command:

Use the command below if the firewall is active:

Now is the time to install Linux packages needed by Oracle.

Execute these commands:

rpm -Uvh binutils-2.*

rpm -Uvh iscsi-initiator-utils-6.2.0.868-0.18.el5.i386.rpm

Configuring iSCSI services.

Start the “iscsid” serivce:

# service iscsid start

# iscsiadm -m discovery -t sendtargets -p openfiler-priv

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p 10.10.1.20 –op update -n node.startup -v

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p 10.10.1.20

Making device names persistent.

Create a file called “55-openiscsi.rules” with the following content:

Navigate to another directory: