Professional Documents
Culture Documents
Openfiler 2.3 Active/Passive Cluster (Heartbeat, DRBD) With Offsite Replication Node
Introduction
Openfiler is a Linux based NAS/SAN application which can deliver storage over nfs/smb/iscsi and ftp. It has a web interface over that you can control these services. The howto is based on the Howto from Kyle Gililland. A lot of thanks to him for this. The cluster we build will consist of two nodes replicating each other and taking over services and storage in case of emergency. Furthermore we have an Offsite Replication Server, which ideally stands in a physically different position and replicates the configurations/storage from which ever node is active. In case of emergency this Offsite Replication Server can be used to restore the cluster and to deliver the services. I used the Openfiler VMware Images 2.3 x86 and Openfiler 2.3 x86 Installation Media to test out it this howto. Hopefully you will be able to repeat my steps and have a safe and cheap Storage Solution.
Overview
- 1. Installation - 2. Prepaire the partitions for DRBD - 3. Create basic heartbeat configuration - 4. DRBD Configuration - 5. Prepaire the configuration partition - 6. Initiate Data/Storage partition - 7. Create final heartbeat configuration - 8. Enable/Disable sytem services - 9. Test failover on node1 and node2 - 10. Test recover of filer01 and filer02 - 10.1 DRBD Configuration - 10.2 filer01 and filer02 redo configuration
Copyright 2011 All Rights Reserved. HowtoForge Page 1 of 35
http://www.howtoforge.com/
- 10.3 Retake resources and run cluster again - 11. Use Replication Node as Main Node - 11.1 Finished Replication, how to turn Replication Node in standby again - 12. Add another Storage Partition
RequirementsHardware
- 3x boxes that meet the minimum hardware requirements of openfiler - 2x ethernet interfaces at each box
1. Installation
In the Installation process of openfiler leave the portion of the disk you wanna use for Storage Services unused. In this case it was a second harddisk in each system. We create the partitions for Storage Replication after the installation.Networkconfiguration - on filer01: - eth0: 10.10.11.101 - eth1: 10.10.50.101 - on filer02: - eth0: 10.10.11.102 - eth1: 10.10.50.102 - on filer03: - eth0: 10.10.11.103 - eth1: 10.10.50.103 - virtual devices: ( this will be created by heartbeat, dont add these ip's to any interface ) - eth0:0 10.10.11.100 ( iscsi target server ip )
HowtoForge
Page 2 of 35
http://www.howtoforge.com/
- eth1:0 10.10.50.100 ( replication interface ) After finishing the installation build yourself a nice working environment while you update the openfiler systems. It's important to update the systems now as openfiler 2.3 standard installation media dont bring DRBD 8.3.x which is needed for 3-node replication.
conary update conary
While the systems update you add the other filers to the /etc/hosts file and generate some ssh-keys and exchange them.
root@filer01 ~# nano /etc/hosts
10.10.50.102filer02 10.10.50.103filer03
10.10.50.101filer01 10.10.50.103filer03
HowtoForge
Page 3 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node #Installed by rBuilder 127.0.0.1 filer03 localhost.localdomain localhost
http://www.howtoforge.com/
10.10.50.101filer01 10.10.50.102filer02
Then we create ssh-keys without a passphrase which we use for easier file-exchange.
root@filer01 ~# ssh-keygen -t dsa
Generating public/private dsa key pair. Enter file in which to save the key (/root/.ssh/id_dsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_dsa. Your public key has been saved in /root/.ssh/id_dsa.pub. The key fingerprint is: 4c:76:ec:61:10:2a:07:9f:35:8d:9b:16:89:5c:bc:b9 root@filer01
HowtoForge
Page 4 of 35
http://www.howtoforge.com/
HowtoForge
Page 5 of 35
http://www.howtoforge.com/
Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1566, default 1): Using default value 1 Last cylinder, +cylinders or +size{K,M,G} (1-1566, default 1566): 67 Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (68-1566, default 68): Using default value 68 Last cylinder, +cylinders or +size{K,M,G} (68-1566, default 1566): Using default value 1566 Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): 8e Changed system type of partition 2 to 8e (Linux LVM)
You have to exist fdisk with w to write the changed to the disk. You can check the partitions after this:
fdisk -l
Disk /dev/sda: 2355 MB, 2355978240 bytes 16 heads, 63 sectors/track, 4565 cylinders Units = cylinders of 1008 * 512 = 516096 bytes
HowtoForge
Page 6 of 35
http://www.howtoforge.com/
Disk identifier: 0x00000000 Device Boot /dev/sda1 * Start 1 End 4565 Blocks 2300696 Id 83 System Linux
Disk /dev/sdb: 12.9 GB, 12884901888 bytes 255 heads, 63 sectors/track, 1566 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xff9e0345 Device Boot /dev/sdb1 /dev/sdb2 Start 1 68 End 67 1566 Blocks 538146 12040717+ Id 83 8e System Linux Linux LVM
NOTE: Before you can continue the systems should have finished update.
HowtoForge
Page 7 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 bcast eth1 keepalive 5 warntime 10 deadtime 120 initdead 120 udpport 694 auto_failback off node filer01 node filer02
http://www.howtoforge.com/
Go to https://10.10.11.101:446 and start iscsi target service ( this will create /etc/ha.d/haresources on filer01 ). Copy haresource file to filer02 and start heartbeat IP:
root@filer01 ~# scp /etc/ha.d/haresources root@filer02:/etc/ha.d/haresources
HowtoForge
Page 8 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node root@filer02 ~# service heartbeat start
http://www.howtoforge.com/
This should create a failover where filer01 gets the ip address 10.11.50.100, check this with ifconfig on filer01, the output should be like this.
root@filer01 ~# ifconfig
If this is set you can now continue to start creating the drbd devices. If not check the output of /var/log/ha-log and try to figure out what is wrong.
4. DRBD Configuration
This manual implies that you have worked with drbd in the past and know howto tune the drbd common section regarding to your hardware. Using protocol C is save on stacked resources but not the best regarding network performance. Remove the old drbd.conf and create a new one:
HowtoForge
Page 9 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node root@filer01 ~# rm /etc/drbd.conf
http://www.howtoforge.com/
global { usage-count yes; } common { protocol C; disk { on-io-error detach; } net { after-sb-0pri disconnect; after-sb-1pri disconnect; after-sb-2pri disconnect; rr-conflict disconnect; } syncer { al-extents 257; rate 100M; } handlers { pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f"; pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f"; local-io-error "echo O > /proc/sysrq-trigger ; halt -f"; } startup { degr-wfc-timeout 120; # 2 minutes. } }
HowtoForge
Page 10 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node resource meta { on filer01 { device /dev/drbd0; disk /dev/sdb1; address 10.10.50.101:7788; meta-disk internal; } on filer02 { device /dev/drbd0; disk /dev/sdb1; address 10.10.50.102:7788; meta-disk internal; } } resource data { on filer01 { device /dev/drbd1; disk /dev/sdb2; address 10.10.50.101:7789; meta-disk internal; } on filer02 { device /dev/drbd1; disk /dev/sdb2; address 10.10.50.102:7789; meta-disk internal; } } resource meta-U { stacked-on-top-of meta { device /dev/drbd10; address 10.10.50.100:7788; }
http://www.howtoforge.com/
HowtoForge
Page 11 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node on filer03 { device /dev/drbd10; disk /dev/sdb1; address 10.10.50.103:7788; meta-disk internal; } } resource data-U { stacked-on-top-of data { device /dev/drbd11; address 10.10.50.100:7789; } on filer03 { device /dev/drbd11; disk /dev/sdb2; address 10.10.50.103:7789; meta-disk internal; } }
http://www.howtoforge.com/
HowtoForge
Page 12 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node root@filer01 ~# drbdadm create-md data
http://www.howtoforge.com/
HowtoForge
Page 13 of 35
http://www.howtoforge.com/
drbd driver loaded OK; device status: version: 8.3.7 (api:88/proto:86-91) GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by phil@fat-tyre, 2010m:res cs ro ds p mounted fstyp 0:meta Connected Primary/Secondary UpToDate/UpToDate C 1:data Connected Primary/Secondary UpToDate/UpToDate C 10:meta-U^^0 Connected Primary/Secondary UpToDate/UpToDate C 11:data-U^^1 Connected Primary/Secondary UpToDate/UpToDate C
01-13 17:17:27 e
This way it should look when all nodes are synced and filer01 is the primary.
HowtoForge
Page 14 of 35
http://www.howtoforge.com/
Openfiler to Meta-Partition:
root@filer01 ~# mkdir /meta
root@filer01 ~# rm /meta/opt/openfiler/sbin/openfiler
root@filer01 ~# rm /meta/opt/openfiler/etc/rsync.xml
HowtoForge
Page 15 of 35
http://www.howtoforge.com/
HowtoForge
Page 16 of 35
http://www.howtoforge.com/
NOTE: If it doesnt restart check if you have linked the httpd Modules, probably you use lib64 instead of lib!
HowtoForge
Page 17 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node mkdir /meta
http://www.howtoforge.com/
mv /opt/openfiler/ /opt/openfiler.local
ln -s /meta/opt/openfiler /opt/openfiler
umount -a -t rpc-pipefs
rm -rf /etc/samba/
ln -s /meta/etc/samba/ /etc/samba
rm -rf /var/spool/samba/
ln -s /meta/var/spool/samba/ /var/spool/samba
rm -rf /var/lib/nfs/
ln -s /meta/var/lib/nfs/ /var/lib/nfs
rm -rf /etc/exports
ln -s /meta/etc/exports /etc/exports
HowtoForge
Page 18 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node rm /etc/ietd.conf
http://www.howtoforge.com/
ln -s /meta/etc/ietd.conf /etc/ietd.conf
rm /etc/initiators.allow
ln -s /meta/etc/initiators.allow /etc/initiators.allow
rm /etc/initiators.deny
ln -s /meta/etc/initiators.deny /etc/initiators.deny
rm -rf /etc/proftpd
ln -s /meta/etc/proftpd/ /etc/proftpd
to
filter = [ "a|drbd1[0-9]|", "r|.*|" ]
HowtoForge
Page 19 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node root@filer01 ~# vgcreate data /dev/drbd11
http://www.howtoforge.com/
Go to https://10.10.11.101:446 and start/stop iscsi target service ( this will recreate /etc/ha.d/haresources on filer01 ). Copy haresources to filer02:
root@filer01 ~# scp /etc/ha.d/haresources root@filer02:/etc/ha.d/haresources
HowtoForge
Page 20 of 35
http://www.howtoforge.com/
HowtoForge
Page 21 of 35
http://www.howtoforge.com/
Shutdown node 1:
root@filer01 ~# shutdown -h now
Check if you can login on https://10.10.11.100:446. Start filer01 and do a manual failover:
root@filer01 ~# /usr/lib/heartbeat/hb_takeover
HowtoForge
Page 22 of 35
http://www.howtoforge.com/
Now set up two complete new filer01 and filer02 from step 1. to step 3. From there on our recovery will be slightly different to the installation.
HowtoForge
Page 23 of 35
http://www.howtoforge.com/
At this point DRBD will recognize the inconsistent data and start to sync from filer03.
root@filer01 ~# service drbd status
service drbd status drbd driver loaded OK; device status: version: 8.3.7 (api:88/proto:86-91) GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by phil@fat-tyre, 2010-01-13 17:17:27 m:res cs ro ds p mounted fstype ... sync'ed: 0.2% (11740/11756)M ... sync'ed: 1.8% (11560/11756)M
HowtoForge
Page 24 of 35
http://www.howtoforge.com/
C C C C
For the lower resource meta and data filer01 is the SyncSource, while for the upper resource meta-U and data-U it is the SyncTarget. This shows us that the rebuild process has started. Before you finish the synchronisation you can prepare the configuration for Openfiler and its Storage Services.
mv /opt/openfiler/ /opt/openfiler.local
ln -s /meta/opt/openfiler /opt/openfiler
HowtoForge
Page 25 of 35
http://www.howtoforge.com/
umount -a -t rpc-pipefs
rm -rf /etc/samba/
ln -s /meta/etc/samba/ /etc/samba
rm -rf /var/spool/samba/
ln -s /meta/var/spool/samba/ /var/spool/samba
rm -rf /var/lib/nfs/
ln -s /meta/var/lib/nfs/ /var/lib/nfs
rm -rf /etc/exports
ln -s /meta/etc/exports /etc/exports
rm /etc/ietd.conf
ln -s /meta/etc/ietd.conf /etc/ietd.conf
rm /etc/initiators.allow
ln -s /meta/etc/initiators.allow /etc/initiators.allow
rm /etc/initiators.deny
ln -s /meta/etc/initiators.deny /etc/initiators.deny
rm -rf /etc/proftpd
HowtoForge
Page 26 of 35
http://www.howtoforge.com/
ln -s /meta/etc/proftpd/ /etc/proftpd
When the synchronisation process has finished we can prepare the cluster now for rerunning the services on filer01. If you run the services for the cluster on filer03 ( Step 11.) you have to stop these services like described in Step 11.1 before you can continue.
HowtoForge
Page 27 of 35
http://www.howtoforge.com/
Mount the meta Partition and generate a new haresource file with openfiler:
root@filer01 ~# mount -t ext3 /dev/drbd10 /meta
Now login into https://10.10.11.101:446/ and start/stop some service you don't use to regenerate the /etc/ha.d/haresource file. Then we can copy this file to filer02, start the heartbeat services on both machines and do a takeover.
root@filer01 ~# service openfiler stop
root@filer01 ~# /usr/lib/heartbeat/hb_takeover
After the network and filesystem mounts have happened you should see everything running fine again under the cluster IP 10.10.11.100. You can check this by trying to login to https://10.10.11.100:446. Try a manual failover on filer02 now, too.
root@filer02 ~# /usr/lib/heartbeat/hb_takeover
HowtoForge
Page 28 of 35
http://www.howtoforge.com/
At this point we are able to start openfiler and the services we need, but we need the virtual IP which the cluster used to deliver services first. We use the resource.d scripts from heartbeat to do this.
root@filer03 ~# /etc/ha.d/resource.d/IPaddr 10.10.11.100/24/eth0 start
HowtoForge
Page 29 of 35
http://www.howtoforge.com/
Give up the cluster IP by using the resource.d scripts from heartbeat again.
root@filer03 ~# /etc/ha.d/resource.d/IPaddr 10.10.11.100/24/eth0 stop
After this you can retake all services from filer01 like you found in Step 10.3.
HowtoForge
Page 30 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node address 10.10.50.101:7790; meta-disk internal; } on filer02 { device /dev/drbd2; disk /dev/sdc1; address 10.10.50.102:7790; meta-disk internal; } } resource data2-U { stacked-on-top-of data2 { device /dev/drbd12; address 10.10.50.100:7790; } on filer03 { device /dev/drbd12; disk /dev/sdc1; address 10.10.50.103:7790; meta-disk internal; } }
http://www.howtoforge.com/
Create the metadata on the lower resource before we can start the upper resource again.
root@filer01 ~# drbdadm create-md data2
HowtoForge
Page 31 of 35
Openfiler 2.3 Active/Passive Cluster (heartbeat,DRBD) With Offsite Replication Node root@filer02 ~# drbdadm create-md data2
http://www.howtoforge.com/
Make it primary:
root@filer01 ~# drbdsetup /dev/drbd2 primary -o
After this we are ready to add the new device to our existing LVM Device and increase our storage. Note: It's out of scope of this manual to resize the
HowtoForge
Page 32 of 35
http://www.howtoforge.com/
storage that you actually use on it. Now we create a PV on the new stacked resource device and add it to the existing VolumeGroup:
root@filer01 ~# pvcreate /dev/drbd12
Recreate the /etc/ha.d/haresource like we've done before by restarting some unused service over the Openfiler GUI, exchange this new haresource file to filer02. After this you can log into your openfiler cluster IP and use the extended data storage. Instead of increasing you could just create another VolumeGroup. Refer to Step 6 for this.
HowtoForge
Page 33 of 35
http://www.howtoforge.com/
Another problem seems to be in the discovery off LVM Devices with Openfiler, the lvm config i posted is good to use for a system with stacked resources, but probably not right for a drbd only system, the drbd documentations mention the following lvm configurations for drbd and lvm which will only show the drbd or drbd stacked resources to lvm.
filter = [ "a|drbd.*|", "r|.*|" ]
and
filter = [ "a|drbd1[0-9]|", "r|.*|" ]
like you found in this howto. This will allow that the devices /dev/drbd10 - /dev/drbd19 are exposed to lvm. If you need more devices you have to change your lvm configuration regarding to this. You can find the example configurations in the drbd documentation here. Edit the /etc/rc.sysinit file on Line 333-337 and comment out these lines: From:
if [ -x /sbin/lvm.static ]; then if /sbin/lvm.static vgscan --mknodes --ignorelockingfailure > /dev/null 2>&1 ; then action $"Setting up Logical Volume Management:" /sbin/lvm.static vgchange -a y --ignorelockingfailure fi
HowtoForge
Page 34 of 35
http://www.howtoforge.com/
to:
# # # # # fi fi if [ -x /sbin/lvm.static ]; then if /sbin/lvm.static vgscan --mknodes --ignorelockingfailure > /dev/null 2>&1 ; then action $"Setting up Logical Volume Management:" /sbin/lvm.static vgchange -a y --ignorelockingfailure
Restart your filers now to make the changes happen. You should be fine discovering the iSCSI LUNs with your Citrix Xen systems now.
HowtoForge
Page 35 of 35