Symantec

Using Veritas Storage Foundation
Cluster File System HA from Symantec to

provide Oracle Fast Failover
Discussion and Demo
Who should read this paper Who should read this paper
The contents of this document should be read by anyone responsible for
providing high availability Oracle DBMS solutions DBAs, Architects,
System Administrators, etc.
Intended audience
The contents of this document are intended for the Symantec Sales staff
and Symantec customers.
W
H
I
T
E

P
A
P
E
R
:
V
E
R
I
T
A
S

S
T
O
R
A
G
E

F
O
U
N
D
A
T
I
O
N

C
L
U
S
T
E
R
F
I
L
E

S
Y
S
T
E
M

H
A

F
R
O
M

S
Y
M
A
N
T
E
C
.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.
Content
Value proposition of Storage Foundation Cluster File System High Availability Oracle fast failover. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Oracle database environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Cost savings for deploying Storage Foundation Cluster File System HA compared to Oracle RAC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Key Features Storage Foundation Cluster File System HA for achieving fast failover. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Oracle parameters to consider for tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Supporting documents for demo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Link to Demo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Using Veritas Storage Foundation Cluster File System HA from Symantec to provide Oracle Fast
Failover Discussion and Demo
Value proposition of Storage Foundation Cluster File System High Availability Oracle fast failover
Highly available Oracle Database Management System (DBMS) environments are almost always a requirement for the Oracle customer base.
Oracle positions Oracle Real Application Clusters (RAC) as the best solution to provide high availability that can eliminate any outages
associated with a database instance failure. However, Oracle RAC is operationally complex, expensive, and typically requires substantial
efforts to ensure that the application running on top of the DBMS will perform well and take advantage of seamless client failovers when a
database instance fails.
This is not to say that there is no place for Oracle RAC. There are applications with very stringent Service Level Agreements (SLAs) where
Oracle RAC will be the best fit, and in those cases the Veritas Storage Foundation for Oracle RAC will be the best solution. However, there
are many applications where short outages (seconds to a few minutes) can be tolerated that can be served very well using Single Instance
Oracle in a 'fast failover' environment. In these cases, deploying Veritas Storage Foundation Cluster File System High Availability from
Symantec for the Oracle environment can be an excellent solution. We note here that Storage Foundation Cluster File System HA is the
integrated product bundle that includes Storage Foundation and Veritas Cluster Server.
Oracle database environments
Refer to Figure 1 for a comparison of Oracle 'single instance failover' vs. Oracle RAC architectures. In the case of failure, Oracle single
instance database will take some additional amount of time for the failover to complete, when compared with Oracle RAC, because the Oracle
instance has to be started anew on the failover server. With Oracle RAC, the instance is already running on the other server(s). Thus, Oracle
DB client sessions needing to migrate their connections from the failed server to the surviving server will be able to resume 'operations' more
quickly with Oracle RAC compared to Oracle single instance because they will be connecting to an instance that is already running. Keep in
mind, that, in both cases, the database clients that were connected to the failed server will have to establish new database connections with
a different server.
Oracle database environment architectures
Figure 1
1
Cost savings for deploying Storage Foundation Cluster File System HA compared to Oracle RAC
The cost savings of deploying with Storage Foundation Cluster File System HA/single instance Oracle versus deploying with Oracle RAC are
considerable. To do a cost comparison analysis between the two options, let's first note several Oracle RAC financial facts:
Oracle RAC Software License cost is a 48 percent uplift over single instance Oracle (See http://www.oracle.com/us/corporate/pricing/
technology-price-list-070617.pdf )
Annual maintenance cost is always 22 percent of list license cost which means maintenance costs for RAC will also be at least 48 percent
higher than single instance Oracle. (See http://www.oracle.com/us/corporate/pricing/technology-price-list-070617.pdf )
Oracle cost is calculated as follows:
Cores x Oracle Processor Core Factor x License cost
The CPU chip architecture (e.g. Intel or Sun Scalable Process Architecture (SPARC) determines how many cores per CPU-a quad-core
CPU has 4 cores/CPU. There is a 'Oracle Processor Core Factor' for each chip arcthitecture e.g 0.25 for Intel and 0.25 for SPARC (see
http://www.oracle.com/us/corporate/contracts/processor-core-factor-table-070634.pdf)
Let's compare costs for the Storage Foundation Cluster File System HA failover solution vs the RAC solution. This comparison is based on
providing 32 cores of database compute power for each option.
For the failover environment, we have an active/passive 2 node cluster, with each node having 8 quad-core CPUs. Note that the passive node
requires no Oracle license cost.
For the RAC configuration, we have a 2 node cluster active/active of course, which each node having 4 quad-core CPUs. The Oracle DB
license plus the RAC uplift license cost is incurred on both nodes.
2
So for each of the two configurations we have 32 cores of active database compute power. The RAC configuration requires just 50 percent of
the hardware cost since both nodes of the cluster are active.
Referring to Figure 2, let's look at the respective costs for these two options based on Sun SPARC architecture.
RAC will require $179,000 (+25 percent) more capital investment than our failover solution. This may seem counter intuitive as the failover
solution requires double the compute power of the RAC solution. But, when you factor in the added RAC license costs, it makes the RAC
solution more expensive. For the recurring annual maintenance costs, it is a similar story. Maintenance is $97,000 (+35 percent) more
expensive for the RAC solution, as maintenance costs are pegged to license costs.
A similar analysis for the Intel/AMD architecture shows thatcapital costs for RAC are 23 percent higher and RAC maintenance costs are 38
percent higher.
The assumptions we in this analysis are:
50 percent discount for licenses
15 percent discount for Hardware
Support not discounted
Minimal Oracle database options selected (just Oracle Database Enterprise Edition and RAC)
Figure 2
3
If one were to deploy Oracle DB server clusters with N+1 configurations, instead of the active/passive clusters discussed here, the cost
savings would be even more compelling, because instead of configuring 100 percent of additional hardware as we do for this 2-node active/
passive configuration (1 to 1 ratio of active to passive), we would configure much less e.g. for a 4-node cluster, the additional hardware
would be a 3 to 1 ratio (3 active and one passive). So, for example, if and enterprise needs 100 active servers to support its workload, it would
need a total of 200 servers for an 1:1 active/passive cluster environment, whereas it would need only 133 total servers for a 3:1 active/
passive cluster approach.
A proof point regarding a customer saving money and simplifying operations by moving from RAC to Storage Foundation Cluster File System
HA is noted here:
Implementation of Storage Foundation HA CFS: Case Study Large Pharmaceutical
Deployed widely with Oracle RAC (Storage Foundation RAC)
Deployed across Web and Application tiers with Storage Foundation, in addition to Oracle tier
Determined that they over-deployed with RAC
Added cost
Added complexity
Added performance challenges
Converted 80 percent of Storage Foundation RAC environments to Storage Foundation Cluster File System HA
Reduced cost
Reduced complexity
Improved performance
Continue to have a single stack (Storage Foundation) to support all Oracle environments
So, to summarize regarding the value proposition of Storage Foundation Cluster File System HA, using the appropriate Storage
Foundation stack (Storage Foundation RAC for RAC and Storage Foundation Cluster File System HA for single instance Oracle failover), an
enterprise can optimize for Availability and Cost across its diverse Oracle environments. For "supercritical" workloads where downtime has to
4
be as close to zero as possible, Storage Foundation RAC is the appropriate solution. For all other workloads, Storage Foundation Cluster File
System HA is the right choice.
Key Features Storage Foundation Cluster File System HA for achieving fast failover
In order to achieve the fastest Oracle DB instance failover times, two key components of the Storage Foundation stack come into play
Cluster File System and Intelligent Monitoring Framework (IMF). It is certainly possible to deploy Storage Foundation HA using the VxFS (non-
clustered file system) and without using IMF for the the various Veritas Cluster Server agents. But to meet stringent SLAs, these two
components will come into play.
Let's discuss the 'value add' for these two features:
Cluster File SystemThe significant benefit of using CFS versus the non-clustered file system (VxFS) is that in the event of server failover,
all of the time associated with importing disk groups, starting volumes, and mounting file systems is eliminated.
The steps required for failover with SF HA for Oracle (single instance Oracle) using a local Veritas file system are:
1. Import disk group
2. Start Volumes
3. Mount file system-VxFS
4. Start database
5. Resolve transactions (replay logs)
6. Update IP
7. Clients Reconnect
The steps required for failover with SFCFSHA for Oracle (single instance Oracle) using a Cluster Veritas file system are:
1. Start database
2. Resolve transactions (replay logs)
3. Update IP
4. Clients Reconnect
The positive impact of this benefit is directly proportional to the number of disk groups, volumes, and file systems on which the Oracle DB
resides. The larger the DB, greater is the benefit.
And one need not be concerned that the file systems are persistently mounted on the 'standby' server, because Oracle places a lock on its
'control file' which resides on CFS, thereby preventing inadvertent or intentional starting up of the database instance on the standby server
when the database instance is running on the 'active' server.
Intelligent Monitoring Framework (IMF)Prior to the introduction of IMF with the 5.1SP1 release of the Storage Foundation HA product, all
Veritas Cluster Server agents did poll based monitoring, which continues to be supported.
Poll based monitoring works as follows:
Checks to see if the application is online during an interval of time
Attributes for Cluster Server monitoring which are controlled per resource type:
5
MonitorInterval when the application is online (default 60 sec)
OfflineMonitorInterval when the application is offline (default 300 sec)
MonitorTimeout is the amount of time given to a monitor process before giving up (default 60 sec)
Resources are monitored on all systems they are configured to run on
If an Oracle database is configured to run on node-1, node-2 and node-3 then each of those three systems will validate the
state of the resource based on the current resource stateonline/offline
Each instance of a resource is monitored
If there are 20 mount resources in a service group, then 20 monitors will be run per system in the cluster based on the
current resource stateonline/offline
If there are many resources to monitor (e.g. disk groups, volumes, file systems, Oracle instances, etc.), then poll based monitoring can
become fairly resource intensive. Additionally, the amount of time it takes to detect a resource fault is directly related to the polling interval
typically 60 seconds.
With IMF, monitoring is much less resource intensive because of zero polling overhead, and fault detection is instantaneous because
monitoring is being done asynchronously within the OS kernel.
Oracle related Cluster Server agents that are IMF enabled for the 5.1SP1:
Process based agents
Physical environments, containers
IMF is enabled for Process agents running within a container
Oracle agent-starts, stops, and monitors Oracle DB instance
Netlsnr agent-starts, stops, and monitors Oracle listener -
CVMvxconfigd-starts and monitors the vxconfigd daemon which maintains disk and disk group configurations, communicates
configuration changes to the kernel, etc.
Mount based agents
Mount, CFSMount
Additional Cluster Server agents that are IMF enabled for the 6.0 release are:
Virtualization based agents
Solaris Zones
AIX WPAR
IMF updates
Agent Framework update for Custom Agent support
Support for IMF-PCV
Prevention of Concurrency Violation with this feature, inadvertent startup of an application on one cluster node, when the
application is already running on another cluster node, is prevented.
For Oracle environments, the Oracle, Netlsnr, Cluster Volume Manager, and Mount agents will all benefit from the IMF enablement and
contribute to fast failover of Single Instance Oracle.
6
Oracle parameters to consider for tuning
We noted that when database failover occurs, one of the tasks that needs to take place is 'Resolve transactions (replay logs)', and we also
noted that log replay is likely to account for the majority of the recovery time for a large active DB environment with many concurrent users.
Thus it is prudent to examine several Oracle init.ora parameters for possible tuning efficiencies:
Log_checkpoint_interval specifies the frequency of checkpoints in terms of the number of redo log file blocks that can exist between an
incremental checkpoint and the last block written to the redo log. This number refers to physical operating system blocks, not database
blocks.
Fast_start_mttr_target enables you to specify the number of seconds the database takes to perform crash recovery of a single instance.
When specified, Fast_start_mttr_target is overridden by Log_checkpoint_interval. (This means if you use log_checkpoint_interval, then
you will not use fast_start_mttr_target, and visa versa.)
Fast_start_parallel_rollback determines the maximum number of processes that can exist for performing parallel rollback. This
parameter is useful on systems in which some or all of the transactions are long running.
Optimized tuning of these parameters can further contribute to reducing failover times, in addition to leveraging the Cluster File System and
IMF features of Storage Foundation HA. Please refer to the Oracle 11gR2 Performance Tuning Guide at http://docs.oracle.com/cd/
E11882_01/server.112/e16638.pdf for guidance on tuning of these parameters.
7
Supporting documents for demo
Demo Screen Shots
This demo consists of a single client doing a looping series of inserts and updates to the 'orders' table. The demo begins with the client
connecting to Node 1, and then seamlessly failing over to Node 2 when a fault occurs on Node 1.
We have 3 telnet sessions one for the client DB connection, and 1 each for the active and passive oracle DBMS servers.
8
Denoting the 'Order Entry Client'
Denoting the Node 1 Active Production DB Node
9
Denoting the Node 2 Standby DB Node
List Oracle DB instance processes running on Node 1
10
Starting the Order Entry Workload from the Client telnet window which puts active workload on the Node 1 DB server
Showing that Order Entry Workload from the Client is active
11
Output of hastatus command shows that database file system is mounted on both nodes, as CFS is in use
hastatus also shows the status of resources for Node 1 and Node 2. Note that oragrp is online on Node 1 and offline on Node 2.
12
Introducing DB Fault on Node 1 by killing Oracle pmon process
IMF immediately detects fault
13
Failover from Node 1 to Node 2 in progress
Failover to Node 2 completed listing Oracle processes now running on Node 2.
14
Client workload seamlessly failed over to Node 2
15
VCS configuration file (main.cf)
============================
include "types.cf"
include "CFSTypes.cf"
include "CVMTypes.cf"
include "OracleTypes.cf"
cluster sfcfs_clus (
UserNames = { admin = dKLdKFkHLgLLjTLfKI }
Administrators = { admin }
HacliUserLevel = COMMANDROOT
)
system node1 (
)
system node2 (
)
group cvm (
SystemList = { node1 = 0, node2 = 1 }
AutoFailOver = 0
Parallel = 1
AutoStartList = { node1, node2 }
)
CFSMount oradata_mnt (
Critical = 0
MountPoint = "/oradata"
BlockDevice = "/dev/vx/dsk/oradb_dg/oradb_vol"
)
CFSfsckd vxfsckd (
)
CVMCluster cvm_clus (
CVMClustName = sfcfs_clus
CVMNodeId = { node1 = 0, node2 = 1 }
CVMTransport = gab
CVMTimeout = 200
)
16
CVMVolDg cvmvoldg (
Critical = 0
CVMDiskGroup = oradb_dg
CVMVolume = { oradb_vol }
CVMActivation @node1 = sw
CVMActivation @node2 = sw
)
CVMVxconfigd cvm_vxconfigd (
Critical = 0
CVMVxconfigdArgs = { syslog }
)
cvm_clus requires cvm_vxconfigd
oradata_mnt requires cvmvoldg
oradata_mnt requires vxfsckd
vxfsckd requires cvm_clus
// resource dependency tree
//
// group cvm
// {
// CFSMount oradata_mnt
// {
// CVMVolDg cvmvoldg
// CFSfsckd vxfsckd
// {
// CVMCluster cvm_clus
// {
// CVMVxconfigd cvm_vxconfigd
// }
// }
// }
// }
group oragrp (
SystemList = { node1 = 0, node2 = 1 }
AutoStartList = { node1 }
)
IP IP_oraprod (
Device = hme0
17
Address = "10.212.102.13"
)
NIC NIC_oraprod (
Device = hme0
)
Netlsnr LSNR_oraprod_lsnr (
Owner = oraprod
Home = "/prod/u01/oracle/product/11.2.0.2"
TnsAdmin = "/prod/u01/oracle/network/admin"
Listener = LISTENER_PROD
MonScript = "./bin/Netlsnr/LsnrTest.pl"
)
Oracle ora_db (
Sid = orademo
Owner = oracle
Home = "/local/oracle/dbbase/dbhome"
)
IP_oraprod requires NIC_oraprod
LSNR_oraprod_lsnr requires IP_oraprod
LSNR_oraprod_lsnr requires ora_db
requires group cvm online local firm
// resource dependency tree
//
// group oragrp
// {
// Oracle ora_db
// }
==================================
18
VCS dependency graphs
Service groups 'oragrp' and 'cvm' for 2 node cluster
Resources for 'oragrp' group of 2 node cluster (node1 and node2)
19
Resources for 'cvm' group of 2 node cluster (node 1 and node 2)
20
21
Link to Demo
http://www.symantec.com/connect/videos/cfsha-oraclefailover
Summary
Most applications running on top of Oracle DBMS have high availability (HA) requirements. These requirements will vary depending on the
criticality of the application. For the most stringent SLAs, Oracle RAC may be required, in which case, the Veritas Storage Foundation RAC
product will be the best enterprise class solution. But for the majority of cases, HA can be provided with the Storage Foundation Cluster File
System HA product that can deliver very short application outages, or no outage at all if the application is intelligently architected to deal
with DB server failover. By leveraging the full capabilities of Storage Foundation Cluster File Systems HA, the cost and complexity of Oracle
RAC can be mostly or entirely avoided.
22
About Symantec
Symantec protects the worlds information and is
the global leader in security, backup, and
availability solutions. Our innovative products and
services protect people and information in any
environmentfrom the smallest mobile device to
the enterprise data center to cloud-based systems.
Our industry-leading expertise in protecting data,
identities, and interactions gives our customers
confidence in a connected world. More information
is available at www.symantec.comor by connecting
with Symantec at go.symantec.com/socialmedia.
For specific country offices
and contact numbers, please
visit our website.
Symantec World Headquarters
350 Ellis St.
Mountain View, CA 94043 USA
+1 (650) 527 8000
1 (800) 721 3934
www.symantec.com
Copyright 2012 Symantec Corporation. All rights
reserved. Symantec, the Symantec Logo, and the
Checkmark Logo are trademarks or registered
trademarks of Symantec Corporation or its affiliates in
the U.S. and other countries. Other names may be
trademarks of their respective owners.
8/2012 21264688

Symantec

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Symantec

Uploaded by

Copyright:

Available Formats

Using Veritas Storage Foundation

Cluster File System HA from Symantec to

You might also like