Configuring Snapmirroring

Configuring Snapmirroring
Eircomnet

Version 1.0

Owner
EircomNet

Prepared by
Chander Sangra
DBA Team

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 1 of 20

 Configuring Snapmirroring

1 Document Control
1.1 Revision History
Revision Previous Summary of Changes Author
version revision date
1.0 Chander Sangra

1.2 Distribution
This document has been distributed to

Name Title Date of Issue Version

Dervla Monaghan Operations 21-April-2009 1.0

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 2 of 20

 Configuring Snapmirroring

2 Table of Contents
1 Document Control......................................................................... 2
1.1 Revision History................................................................................... ................2
1.2 Distribution............................................................................................ ..............2
2 Table of Contents.........................................................................3
3 Overview...................................................................................... 5
3.1 Preface................................................................................................................ .5
3.2 Purpose and Scope....................................................................... .......................5
3.3 Prerequisites and Assumptions.......................................................... ..................5
4 Snapmirroring..............................................................................6
4.1 Type of Snapmirroring ................................................................................. ........6
4.1.1 Asynchronous........................................................................ ................6
4.1.2 Synchronous................................................................................. .........6
4.1.3 Semi-Synchronous..................................................................... ...........6
5 Snapmirroring Requirements........................................................7
5.1 Assumptions.................................................................................... ....................7
5.2 Requirements ....................................................................... ..............................7
5.2.1 Network..................................................................................... ............7
5.2.2 Root Volume................................................................... .......................8
5.2.3 Rsh access......................................................................................... ....8
5.2.4 Permissions............................................................. ..............................8
5.2.5 Mount................................................................................ ....................8
5.2.6 Ownership..................................................................... ........................8
5.2.7 Others...................................................................... .............................8
6 Architecture.................................................................................9
7 Configuring Snapmirroring..........................................................10
7.1 Configuring Database................................................................... .....................10
7.2 Configuring Snapmirror.................................................................................. ....11
7.2.1 Identify Source & Target filer...................................... .........................11
7.2.2 Identify Volumes...................................................................... ............11
7.2.3 Identify Schedule........................................................................ .........11
7.2.4 Enable SnapMirror........................................................ .......................12
7.2.5 Create Baseline......................................................... ..........................12
7.2.6 Update SnapMirror............................................................................. ..12
7.2.7 Check the Status......................................................... ........................12
8 Database Recovery Using SnapMirror Async and Sync..................13
8.1 Generate Data............................................................................... ....................13
8.2 Snapshot Backup........................................................................... ....................13
8.3 Simulate Disaster.............................................................. ................................13
8.4 Perform Recovery................................................................ ..............................14
8.4.1 Recovery from SnapMirror Update Point of Data Volumes...................15
8.4.2 Recovery from Snapshot Backup of the Data volumes........................15
9 Special Consideration.................................................................17
9.1 Initialisation Phase................................................................................... ..........17
9.2 Throttle Rate.............................................................................................. ........17
9.3 Incremental Phase......................................................................... ....................17
9.4 Stopping & Restarting Snapmirroring................................................................ .17
9.5 Breaking the Mirror........................................................................... .................17
9.6 Re-synching the Mirror.............................................................................. .........17

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 3 of 20

 Configuring Snapmirroring
10 Appendices...............................................................................19
10.1 Appendix A (Scripts)................................................................. .......................19
10.2 Appendix B .......................................................................... ...........................19
11 References...............................................................................20

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 4 of 20

 Configuring Snapmirroring

3 Overview

3.1 Preface

The base of this document is NetApp’s document on “Disaster Recovery With

SnapMirror Technology” (Ref. TR 3057). The document is prepared to fit the existing
database environment in eircomnet. The document is prepared based on the
theoretical knowledge of the DBA without any practical experience. The next version of
this document would be produce once the testing is performed in the test environment.

3.2 Purpose and Scope

The purpose of this document is to presents an overview of SnapMirror
technology, its importance in the database environment, evaluating the pros
& cons of its implementation and finally the recommendations to assist the
DBA/Ops team in designing an optimal SnapMirror solution.

3.3 Prerequisites and Assumptions

This document is useful to the DBA and the Operations team members, the
following assumptions are made that the reader has:
- Minimal knowledge of NetApp platforms and products, particularly in the
area of data protection
- General knowledge of disaster recovery (DR) solutions
- Working knowledge of the NetApp Snapshots and Snaprestore

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 5 of 20

 Configuring Snapmirroring

4 Snapmirroring
SnapMirror software provides automated file system replication from a source to a
destination system. The NetApp storage system from which data is transferred is
referred to as the SnapMirror source, and the NetApp storage system to which the
data is transferred is referred to as the SnapMirror destination. The SnapMirror
source and destination can be miles apart, provided that both NetApp storage
systems can communicate with each other across a network.

The SnapMirror configuration details are maintained in a configuration file called

snapmirror.conf, which resides on the destination NetApp storage system. This
file, along with information entered via the snapmirror.access option or the
snapmirror.allow file, is used to establish relationships between specified source
volumes or qtrees and the destination volume or qtree where the mirrored data is
kept.

More details on Snapmirroring can be found at a separate document “Proposal on

Snapmirroring” at the following location:
http://obsscs/cgi-
bin/display.pl?file=configuringsnapmirror_v1.0.doc&type=download

4.1 Type of Snapmirroring

4.1.1 Asynchronous
In Asynchronous mode, SnapMirror performs incremental, block-based
replication as frequently as once per minute. Performance impact on the
source FAS system is minimal, as long as the system is configured with
sufficient CPU and disk I/O resources.

4.1.2 Synchronous
SnapMirror in synchronous mode is a mode of replication that sends
updates from the source to the destination as they occur, rather than
according to a predetermined schedule. This guarantees that data written
on the source system is protected on the destination even if the entire
source system fails. In synchronous mode, SnapMirror immediately
replicates all data written to the source file system. This guarantees zero
data loss in the event of a failure, but can have a significant performance
impact. It is not necessary or appropriate for all applications.

4.1.3 Semi-Synchronous
A semi-synchronous mode is also provided which minimizes data loss in a
disaster while also minimizing the extent to which replication impacts the
performance of the source system. Unlike asynchronous mode, which can
replicate either volumes or quota trees, synchronous and semi-
synchronous modes work only with volumes.

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 6 of 20

 Configuring Snapmirroring

5 Snapmirroring Requirements

5.1 Assumptions
This document covers Disaster Recovery Solutions offered by NetApp for
Oracle 10gR2 single database instance using SnapMirror Async as well as
Sync technology.

The following assumptions are made in this document:

- Source:
o Server: testmsstaging00
o NetApp filer: "bfiler03"
o Oracle Admin User: "SNAPDBA"
o NetApp Volume: “/vol/dbasnaptestdata” for Oracle data
“/vol/dbasnaptestlog” for Oracle logs
o NetApp Mount Points): “/snaptestdata” for Oracle data
“/snaptestlog” for Oracle data
o Aggregate on the NetApp storage system used for database
storage: “dbaggr”
(The flexible volumes snaptestdata & snaptestlog reside in
aggregate dbaggr)

- Target:
o Server: testdba00
o NetApp filer: "bfiler03"
o Oracle Admin User: "SNAPDBA"
o NetApp Volume: “/vol/testdba/testdba00” for Oracle data
“/vol/ora_data_testdba00” for Oracle logs
o NetApp Mount Points: “testdba” for Oracle data
“oradata” for Oracle logs
o Aggregate on the NetApp storage system used for database
storage: “dbaggr”
(The flexible volumes testdba & oradata reside in aggregate
dbaggr)

- Both Target & Source:

It is also assumed that the database host used for accessing the
database at the Disaster Recovery site has a similar setup to the
database host on the primary site and has all required privileges to
access the volumes on the SnapMirror destination NetApp storage
system.

5.2 Requirements

5.2.1 Network

Ensure good connectivity between the following:

- Source Database Host and Source NetApp filers
o Make entries to the “/etc/hosts” files
- Target Database Host and Target NetApp filers
o Make entries to the “/etc/hosts” files
- Others

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 7 of 20

 Configuring Snapmirroring
5.2.2 Root Volume

The root on the database server must have access to the root volume
/vol/vol0.
For example, to grant access on a root volume named /vol/vol0 on a
NetApp storage system to the user root on the database host system
<servername>, execute the following command on the storage systems:
exportfs –p rw=<servername>,root=<servername>,anon=0 /vol/vol0
exportfs -a

5.2.3 Rsh access

Enable ‘rsh’ access from the database server if it is going to be used. Add
the IP address of the database host to the “/etc/hosts.equiv” file.

5.2.4 Permissions

Ensure the user has permissions on all the volumes to be used for the
database

exportfs –p rw=<ip address>,root=<ipaddress>,anon=0 /vol/<data

volume>
exportfs -a

5.2.5 Mount

Mount the volumes to be used for the database on database host

5.2.6 Ownership

Ensure the file system on the mount points is owned by the user ‘oracle’
on the database server.
Chown –R oracle:dba <mount point>

5.2.7 Others

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 8 of 20

 Configuring Snapmirroring

6 Architecture
The following network & storage model is to be followed in this exercise:

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 9 of 20

 Configuring Snapmirroring

7 Configuring Snapmirroring

7.1 Configuring Database

Prepare database according to the following setup:

Database Configuration:

- Database: Oracle 10g R2

- Instance: Single Instance (Instance Name)
- Oracle Home: Local on database host
(/home/oracle/ora10g/product/10.2.0/db_1)
- Data volume for database data: <data vol>
- Log volume for database logs: <log vol>
- Softlinks to follow the OFA directory structure
o Ln –s /mnt/<data vol> /home/oracle/ora10g/<data vol>
o Ln –s /mnt/<log vol> /home/oracle/ora10g/<log vol>

Database Layout:

Control files:
/<data vol>/control01.ctl
/<data vol>/control02.ctl
/<log vol>/control03.ctl

Redo logs:
/<data vol>/redo_01.log
/<log vol>/redo_02.log

Datafiles:
/<data vol>/*.dbf

Database Setup:

DB Structures:
Execute ‘create_tables_demo.sql’ to create the tables

DB Data:
Execute ‘create_data_demo.sql’ to generate data into the demo
tables

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 10 of 20

 Configuring Snapmirroring

7.2 Configuring Snapmirror

Configure the SnapMirror using the following steps:

7.2.1 Identify Source & Target filer

Identify the SnapMirror Source and the Destination NetApp storage

systems. The source NetApp storage system should already have an Oracle
10g R2 database created on it.

We are using “testmsstaging00.bfiler03” as SnapMirror Source that has Oracle

10g R2 database on it and “testdba00.bfiler03” as SnapMirror destination.

7.2.2 Identify Volumes

Identify the volumes on the source NetApp storage system that is to be

replicated across the Target NetApp storage systems using SnapMirror

We are using “/vol/dbasnaptestdata” and “/vol/dbasnaptestlog” as the two

volumes to be mirrored across Target system.

7.2.3 Identify Schedule

Define a schedule for each SnapMirror relationship in the

/etc/snapmirror.conf file, which resides on the SnapMirror destination’s
NetApp storage system.

The /etc/snapmirror.conf file controls where data is replicated from and

how often the mirror is updated. The entry in the /etc/snapmirror.conf file
should be in the following format:
<Source Filer>:<Volume/Qtree Path> <Destination Filer>:<Volume/Qtree
Path> ‘-‘ <Schedule>

Where:
The <Schedule> is defined by [minute] [hour] [days of month] [days of
week]
A snippet from the /etc/snapmirror.conf file:

bfiler03:/vol/dbasnaptestdata bfiler03:/vol/testdbavol/testdba00 – 0,30 9-

17 * 1-5

bfiler03:/vol/dbasnaptestlog bfiler03: /vol/testdba/testdba00 – 0,30 * * 1-5

In the above snippet,

- The volume named “/vol/dbasnaptestdata” on bfiler03 is replicated to
the volume named “/vol/testdbavol/testdba00” on bfiler03.
- The argument dash ‘-‘ represents that data is replicated at the fastest
rate possible.
- The NetApp storage system updates the volume
“/vol/dbasnaptestdata” at every 30-minute interval between 9 a.m. and
5 p.m., Monday through Friday.
- The asterisk (*) in the example means that the mirror is updated on
every day of the month.

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 11 of 20

 Configuring Snapmirroring

7.2.4 Enable SnapMirror

Enable the SnapMirror feature on the Source and the Target by issuing the
following command:
option snapmirror on

7.2.5 Create Baseline

Create one-time baseline transfer by initialising the SnapMirror relationship

for each volume. Issue the following command from the SnapMirror
destination:

snapmirror initialize –S <Source Filer>:<Volume> <Target

Filer>:<Volume>

for example:
snapmirror initialize –S bfiler03:/vol/dbasnaptestdata
bfiler03:/vol/testdbavol/testdba00

7.2.6 Update SnapMirror

After the baseline transfer, the SnapMirror updates can be triggered based
on a predefined schedule in the /etc/snapmirror.conf file. It is also possible
to trigger a SnapMirror update manually by executing the following
command from the destination:

snapmirror update <volume name>

For example,
execute the following command on the destination to start the manual
SnapMirror update for a volume named /vol/testdbavol/testdba00:

snapmirror update /vol/testdbavol/testdba00

7.2.7 Check the Status

Monitor the SnapMirror status by executing the following command on the

Destination filer:

snapmirror status

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 12 of 20

 Configuring Snapmirroring

8 Database Recovery Using SnapMirror

Async and Sync
To simulate a Disaster Recovery strategy using mixed modes of SnapMirror,
we configured the data volume replication using SnapMirror Async and the
transaction logs volume replication using SnapMirror Sync mode. The
frequency of SnapMirror and Snapshots are summarized as:

Volume SnapMirror Snapshot Backup

vol/dbasnaptestdata 2 hrs 30 min
vol/dbasnaptestlog Sync n/a

The snippet from the /etc/snapmirror.conf file looks somewhat similar to the following:

bfiler03:/vol/dbasnaptestdata bfiler03:/vol/testdbavol/testdba00 – 0 0-22/2

* 1-5

bfiler03:/vol/dbasnaptestlog bfiler03: /vol/testdba/testdba00 – sync

The argument ‘sync’ in the above entries controls whether a volume is

Sync snapmirrored with its source or Async. If the argument is left out of
the conf file, then the volume will be Async snapmirrored.

Perform the following steps to simulate disaster on the primary site

(Source) and perform recovery on the Disaster Recovery site (Target):

8.1 Generate Data

Execute the following database command to run the load:

Execute scott.demobld(25000000);

8.2 Snapshot Backup

Take snapshot backup of the database.

- Move the database into the ‘Backup Mode’
- Take Snapshot of the data & log volumes
- Move the database out of ‘Backup Mode’

8.3 Simulate Disaster

Let us assume disaster strikes at 3:05 p.m. In order to simulate the disaster,
we made
the SnapMirror source NetApp storage system unavailable by executing the
following
command:

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 13 of 20

 Configuring Snapmirroring

After a disaster, the SnapMirror destination volumes corresponding to the

database data will be in SnapMirror Snapshot consistent state. That means
the data volumes at the Target site will have the same data, including
Snapshot Backup copies, as the corresponding source volume had at the
time of successful SnapMirror update. The transaction logs volume is Sync
snapmirrored; therefore, the corresponding volume at the Disaster Recovery
site will have all the changes up to the point the disaster started.

For the Test Scenario,

the data volume named dbasnaptestdata is Async snapmirrored and the
transaction logs volume named vol/dbasnaptestlog is Sync snapmirrored;
therefore, these two volumes will be in the following states:

dbasnaptestdata – consistent with the source volume at the time of

SnapMirror update at 2.00 p.m.
dbasnaptestlog – consistent with the source volume at the time disaster
started at 3.05 p.m.

8.4 Perform Recovery

After the baseline transfer, the destination volumes are available to clients,
but in a read-only state. The status of a destination will show that it is
snapmirrored. A disaster makes the source unavailable; to use the
destination volumes for writing as well as reading, you would need to end
the SnapMirror relationship by executing the following command on the
destination NetApp storage system:

snapmirror break <volume name>

This command changes the destination volume’s status from snapmirrored to broken-off, thus making
it writable.

After a disaster, in this type of configuration, there are two possible options for the database recovery:

1) Recovery from the SnapMirror update point of the data volumes.

2) Recovery from an application-coordinated Snapshot copy of the data volumes.

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 14 of 20

 Configuring Snapmirroring
8.4.1 Recovery from SnapMirror Update Point of Data Volumes

The database’s transaction logs volume is sync snapmirrored; as a result,

all the changed data up to the point the disaster started is replicated to the
destination volume. Therefore, after database recovery, there shouldn’t be
any data loss; the RPO (Recovery Point Objective) will be zero. The RTO
(Recovery Time objective) will vary based on the transaction logs that need
to be applied for recovery.

To perform recovery from the SnapMirror update point of the data volumes,
we need to follow the following steps on the database server:

Mount each volume used for the database to a database host by executing
the
following command :

mount –o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,vers=3,
timeo=600 <bfiler03>:<volume> <mount point>

The control file that resides on the transaction logs volume will have the
latest database information, so copy that control file from the dblogs
volume to the dbdata volume by executing the following commands on the
database server:
<<cp control files >>

Start the database instance and perform database recovery by executing

the following commands on the database server:

Startup mount
Set autorecovery on
Recover database
Alter database open

8.4.2 Recovery from Snapshot Backup of the Data volumes

This type of recovery may become necessary in rare cases, such as when
database recovery is not possible from the SnapMirror update point. In this
kind of situation, the database can be recovered using application-
coordinated Snapshot copies of the data volume.

In this configuration, the database transaction logs volume is sync

snapmirrored; as a result, all the changed data up to the point the disaster
started is replicated to the destination volume. Therefore, after database
recovery, there shouldn’t be any data loss; RPO (Recovery Point Objective)
will be zero. The RTO (Recovery Time objective) will vary based on the
transaction logs that need to be applied for recovery.

In order to perform database recovery using an application-coordinated

Snapshot copy of the volume holding the database’s data, you would need
to complete the following steps:

First you need to restore the volumes that hold data for the database from
the application-coordinated Snapshot copy by executing the following
command on the NetApp storage system:

snap restore -s <snapshot name> <volume name>

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 15 of 20

 Configuring Snapmirroring

Mount the NetApp storage system volumes that hold the database’s data
and transaction logs to a database host by executing the following
command :

mount -o rw,bg,hard,nointr,rsize=32768,wsize=32768,
tcp,vers=3,timeo=600 <bfiler03>:<volume> <mount point>

The control file that resides on the transaction logs volume will have the
latest database information, so copy that control file from the dblogs volume
to the dbdata volume by executing the following commands on the database
server:

<<cp control files >>

Start the database instance and perform database recovery by executing

the following commands on the database server:

Startup mount
Set autorecovery on
Recover database
Alter database open

After completing the above steps, the database recovery will be complete.

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 16 of 20

 Configuring Snapmirroring

9 Special Consideration
Several special scenarios should be considered as:

9.1 Initialisation Phase

If an initialization (level 0) replication event is interrupted for more than 9
minutes (ex. a network outage, filer reboot) it will abort. Partial level 0
events are not recoverable, so the process must be restarted.

9.2 Throttle Rate

The KB per second maximum throttle parameter in /etc/snapmirror.conf can
be changed at any time, and the edited values will take effect within two
minutes. The exception to this is mirror initialization where a bandwidth
throttle is already in effect. In this case, the bandwidth throttle cannot be
modified until the initialization phase is complete or the process is
interrupted.

9.3 Incremental Phase

- Incremental updates will not start until the level 0 initialization is
complete.
- Any incremental updates which are missed are simply skipped.
Subsequent incremental updates transmit any data which was skipped,
so no data is lost.
- Incremental updates in progress will run to completion according to the
configuration settings and available network bandwidth. If a new
incremental update is scheduled to start while an existing update is in
progress, it is considered a schedule miss and skipped.

9.4 Stopping & Restarting Snapmirroring

The SnapMirror process can be stopped and restarted at any time by issuing
the following command sequence on either filer:
vol snapmirror off
[arbitrary time interval]
vol snapmirror on.

As long as the target volume remains read only during the time between
turning the SnapMirror process off and on, the mirror remains intact.

9.5 Breaking the Mirror

To break the mirror, you issue the following command on the target filer:
vol options oracle_mirror snapmirrored off
As a result the target volume is still online but changed from read-only to
read/write mode.

9.6 Re-synching the Mirror

Currently the ability to restart a broken mirror is not available, although this
feature is planned for a future release. Presently, the method for
reestablishing a mirror is to reinitialize the mirror. If the target volume has
been brought online and then written to, this may need to be done twice.

- Once to replicate the data back to the source volume. (In this case the
mirror relationship between the two filers is flipped.)
- Once to reinitialize with the mirror relationship back to normal.

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 17 of 20

 Configuring Snapmirroring
An alternative is to back up the target volume's data to tape, and then
restore that data onto the source filer. This avoids the necessity to
reinitialize the mirror twice.

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 18 of 20

 Configuring Snapmirroring

10Appendices

10.1 Appendix A (Scripts)

Stored procedure script

----------------------------------------------------------------
-- Script Name: create_demobld.sql
-- Purpose: This script generate database load by inserting
specified number of rows in the table.
----------------------------------------------------------------

CREATE OR REPLACE PROCEDURE demobld( rcount IN number DEFAULT 10) IS

--
-- Purpose: Generate database load by inserting specified number of
-- rows in the table

BEGIN
for i in 1..rcount loop
insert into scott.tab1 values (i,
'ABDFADFAFAFAFASFDASDFFFFFFFFFFFFFFFFFFFFFFFFFF');
if mod(i,1000)=0 then
commit;
end if;
end loop;
END;
/

Snapshot Backup
----------------------------------------------------------------
-- Script Name: snapshot_backup.ksh
-- Purpose: This script perform the following tasks:
o Put the database in hot backup mode (backupmodein.sh)
o Create snapshot (create_snapshot.ksh)
o Put the database out of backup mode (backupmideout.sh)
----------------------------------------------------------------

10.2 Appendix B

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 19 of 20

 Configuring Snapmirroring

11References
- SnapMirror Best Practices Guide (TR-3446)

- SnapMirror Async Overview and Best Practices Guide (TR-3446)

- SnapMirror Sync and SnapMirror Semi-Sync Overview and Design

Considerations Guide (TR-3326)

- EircomNet: Backup & Recovery guideline document for the

Mediasurface databases where snapshot based backup & recovery
solution is implemented:
http://obsscs/cgi-
bin/display.pl?file=backuppolicy_v1.0.doc&type=download

- EircomNet: Proposal on Net-App based backup & recovery solution:

http://obsscs/cgi-bin/display.pl?file=b_rproposal.doc&type=download

- EircomNet: Practical demonstration of Net-App based database

recovery:
http://obsscs/cgi-bin/display.pl?file=recoverydemo-
1.doc&type=download

11 July 2008 DBATeam/ NetApp/ Snapmirroring Page 20 of 20