A Foundation for Understanding

Storage Area Networks

Objectives
This training prepares students to:
Use and understand basic SAN terminology Distinguish SAN from NAS and other storage network technologies Understand where Sans can help users the most Installing HBA on AIX host Installing HBA driver on AIX host Determine WWPN of new HBA for AIX HBA diagnosis on AIX Installing HBA on Solaris host Installing HBA driver on Solaris host Determine WWPN on Solaris HBA diagnosis on Solaris Installing HBA on Windows Powerpath troubleshooting

Agenda

Introduction to Sans Installing HBA on AIX host Installing HBA driver on AIX host Determine WWPN of new HBA for AIX HBA diagnosis on AIX Installing HBA on Solaris host Installing HBA driver on Solaris host Determine WWPN on Solaris HBA diagnosis on Solaris Installing HBA on Windows Powerpath Troubleshooting

What is a Storage Area Network?

A dedicated network for servers and storage systems

LAN

SAN

Why Give Storage Its Own Network?

A good LAN does not make a good SAN !
LANs
LANs

use different protocols, different tools

are physically insecure at the desktop and potentially vulnerable at the server LANs seldom have spare capacity for storage networking LANs are tuned to favor short, bursty user transmissions vs large, continuous data transfers

SAN Storage

RAID: Redundant Array of Inexpensive Disks

Fault-tolerant grouping of disks that server sees as a single disk volume Combination of parity-checking, mirroring, striping Self-contained, manageable unit of storage

JBOD: Just a Bunch of Disks

Drives independently attached to the I/O channel Scaleable, but requires server to manage multiple volumes Do not provide protection in case of drive failure

SAN Components

Servers with host bus adapters Storage systems

RAID JBOD Tape Optical

Hubs and switches Bridges and extenders SAN management software

Promise of SAN

Massively extended scalability Greatly enhanced device connectivity Storage consolidation LAN-free backup Serverless (active-fabric) backup Server clustering Heterogeneous data sharing Disaster recovery - Remote mirroring

Scalability and Performance

Storage expansion

No impact on servers No impact on storage Active parallel paths Robust topology

Server expansion

Load balancing

Bandwidth on demand

High Availability

Multiple levels of redundancy are configurable throughout data path Multiple access paths allow failover cluster configurations De-coupling of storage from application service allows it to be managed independently Data vaulting and disaster recovery configurations can address loss of service due to site failure

F A I L O V E R

What is HBA
host bus adapter (HBA) connects a host system (the computer) to other network and storage devices.

The terms are primarily used to refer to devices for connecting SCSI, Fibre Channel and eSATA devices, but devices for connecting to IDE, Ethernet, FireWire, USB and other systems may also be called host adapters.

Installing HBA on AIX Host

Stop all I/O to the storage systems connected to where the new HBA will be installed and shutdown AIX.

With host system power removed, install the host bus adapters. Label each cable to identify the HBA and the FA, hub, or switch port that it connects.

Reapply power and allow the system to boot normally. By default, the hdisks will be in a failed over state.

Installing HBA driver on AIX Host (1 of 2)

When installing or upgrading drivers, be sure to use the latest versions supported by EMC. On the host, find the installed driver version using the lslpp command. For example: lslpp -al |grep df1000 devices.pci.df1000f7.com 5.1.0.0 COMMITTED Common PCI FC Adapter Device devices.pci.df1000f7.diag 5.1.0.0 COMMITTED PCI FC Adapter Device devices.pci.df1000f7.rte 5.1.0.0 COMMITTED PCI FC Adapter Device Software devices.pci.df1000f9.diag 5.1.0.0 COMMITTED 64-bit PCI FC Adapter Device Get appropriate drivers: Compare the installed driver version with the required version as identified in the ELab Interoperability Navigator. If the correct version is not installed, obtain the diver(s) from the operating system vendor and install them as described in the operating system documentation

Installing HBA driver on AIX Host (2 of 2)

Verify that you have a copy of the EMC config share file. Download the emc_cfgmgr.sh file from the following EMC ftp URL: ftp://ftp.emc.com/pub/elab/powerpath/aix

Install the IBM Fibre Channel Interface Support software (ODM) on the AIX server. The ODM software enables the host operating system to recognize devices as belonging to a Symmetrix storage system. The file can be found in power link website at the following Location : Home > Support > Downloads and Patches > Downloads A-C > AIX ODM Definitions Uncompress and untrar the file to /tmp location. If the SMIT menu interface is preferred, invoke smit installp from the /tmp directory. Select Install and Update from LATEST Available Software

Use the List function to select /tmp as the installation directory. Use the List function to select EMC Symmetrix AIX Support Software, EMC CLARiiON Fibre Channel Support Software, or both depending on your configuration. EMC Symmetrix AIX Support Software; Standard utility support.EMC Symmetrix FCP Support Software; IBM Fibre Channel driver support.EMC Symmetrix FCP MPIO Support Software; IBM default PCM MPIO support.EMC Symmetrix FCP PowerMPIO Support Software; PowerPath custom PCM MPIO support.EMC Symmetrix iSCSI Support Software; IBM iSCSI driver support.

Determine WWPN of new HBA for AIX

For each Fibre Channel HBA that you are connecting to the storage system, identify the World Wide Port Name (WWPN) of each HBA using the following command: lscfg vpl device_name Where device_name is the name of the HBA device.

NOTE: To find the names of devices, enter the lscfg command with no arguments and look for device names beginning with fsc. In the output, the Network Address field identifies the WWPN. Record the WWPN of each HBA as you will need to refer to this information

HBA Diagnosis AIX (1 of 6)

Logon onto Host to determine Operating System, Multipathing software, Host Bus Adapter and Storage Vendor.

Determine the host operating system in bold below: uname a

Determine the type of storage: lsdev Cc disk

HBA Diagnosis AIX (2 of 6)

HBA Diagnosis AIX (3 of 6)

Determine the type of multipathing software and the number of status of the HBAs which are configured. /usr/local/bin/sudo powermt display or /usr/local/bin/pbrun powermt display

HBA Diagnosis AIX (4 of 6)

Special Note: If the displays above indicate an even number of HBAs with a minimum of 2 are in an ACTIVE Mode. Stop here all the HBAs are in place. Otherwise continue on. Identify the number of HBAs recognized by the host and compare to the earlier POWERPATH adapter display.

lsdev Cc adapter

HBA Diagnosis AIX (5 of 6)

Note which one of the adapters is missing from POWERPATH display and validate that both HBAs are running the same firmware lscfg vl fcsX X=fcs0, fcs1 from previous command

 Note which one of the adapters is missing from POWERPATH display and validate that both HBAs are : Same Type Same firmware From the previous displays you know: Host name Multipathing software Storage Array Vendor Status of the HBA. Working HBA and non working HBA are the same type Working HBA and non working HBA are the same firmware microcode. Identified the WWN of the working and non working HBA. If all the above is true continue to Diagnose Switch Section. Otherwise remediate what is wrong or missing before continuing.

HBA Diagnosis AIX (6 of 6)

Installing HBA on Solaris Host

1. Record IEEE and serial numbers. Each HBA is shipped with a unique 64-bit identifier called the IEEE address. The Fibre Channel industry uses a World Wide Name (WWN) derived from the IEEE address. This number is needed for FC connectivity. The serial number is used for communicating with Sun. All numbers are clearly marked on the board. Record these numbers before installation. 2. Shut down, power off, and unplug the system. 3. Remove the system case. 4. Remove the blank panel from an empty PCI or PCI-X slot. 5. Insert the HBA into the empty PCI or PCI-X slot. Press firmly until the adapter is seated. 7. Secure the HBAs mounting bracket to the case with the panel screw or clip. 8. Replace the system case and tighten the case screws. The HBA is now installed in the system and power up

Installing HBA driver on Solaris (1of 2)

Overview: This procedure describes how to install the QLogic driver on a Solaris host. Refer to the
release notes provided with the driver for information that might be unique to new driver revisions.

Note: The naming convention for the drivers, associated files, and messages is QLA; however, the
Solaris SPARC driver supports standard QLogic PCI and SBus boards (QLAxxxx Download the HBA driver To download the latest HBA driver: Create a directory to which you will save the driver. For example: mkdir /qlogic_driver Use a web browser to access the QLogic website at http://www.qlogic.com. And Click Downloads at the left side of the screen. Click the EMC link under OEM Models. On the next page, select EMC SYMMETRIX, CLARiiON, & CELERRA supported software. Select the HBA model in the Product column of the table. On the next table, find the supported driver and click the associated Download link. Extract the downloaded file to the directory created earlier.

Installing HBA driver on Solaris (2 of 2)

Uncompress the driver file. Change to the /qlogic_driver directory you created earlier and uncompress the driver file by using uncompress command. Install the driver. Install the driver into the host by entering the following applicable command pkgadd d <pkg name> A list of available packages appears. Select an HBA driver and press Enter. You are prompted to select the directory into which the driver files will be installed. Press Enter at the next three prompts to accept the defaults. The pkgadd program performs a series of checks, then posts a script warning and asks whether to continue the installation. Type y at the prompt and press Enter to continue the driver installation. The pkgadd program will notify you when the driver installation is complete. If the driver is version 4.13 or earlier, reboot the host. Note: There is no reboot required for driver versions 4.15.03 and later. Verify that the QLogic HBAs FCode is installed. To do this, view the QLogic device path name in the /var/adm/messages file or from the output of the format command: If the device path name contains QLGC, qla, the FCode is already installed. For example: /ssm@0, 0/pci@19, 700000/pci@2/QLGC, qla@5 If the device path name contains fibre-channel or scsi, the FCode has not yet been installed. If this is the case, update the HBA FCode before continuing. For example: /ssm@0, 0/pci@19, 700000/pci@2/fibre-channel@5 /ssm@0, 0/pci@19, 700000/pci@2/QLGC, scsi@5

Determine WWPN of HBA on Solaris

Solaris10:
# prtconf -vp | grep wwn port-wwn: 2100001b.3202f94b node-wwn: 2000001b.3202f94b port-wwn: 210000e0.8b90e795 node-wwn: 200000e0.8b90e795

Solaris 8 and 9:
Run the following script to determine the WWNs of the HBAs that are currently being utilized: #!/bin/sh for i in `cfgadm |grep fc-fabric|awk {print $1}`; do dev=`cfgadm -lv $i|grep devices |awk {print $NF}` wwn= \ `luxadm -e dump_map $dev |grep Host Bus|awk {print $4}` echo $i: $wwn done To Check link status of card bash-2.03# luxadm -e port Found path to 2 HBA ports /devices/ssm@0,0/pci@18,700000/SUNW,qlc@2/fp@0,0:devctl CONNECTE D /devices/ssm@0,0/pci@19,700000/SUNW,qlc@2/fp@0,0:devctl CONNECTE D

HBA Diagnosis on Solaris (1 of 4)

1. Kernel modules loaded?
Check whether the needed kernel modules/drivers are loaded: # modinfo|grep fc 93 ffffffffef17d000 19ac0 94 ffffffffef196000 a2a0 157 ffffffffef3e8000 9620 160 ffffffffef494850 5610 87 1 fcp (SunFC FCP v20090210-1.134) - 1 fctl (SunFC Transport v20090210-1.59) 83 1 fcip (SunFC FCIP v20090210-1.50) 92 1 fcsm (Sun FC SAN Management v20070116)

2. Which FC interfaces do I have?

Use "fcinfo":

# fcinfo hba-port HBA Port WWN: 2100001b321fbd9c OS Device Name: /dev/cfg/c4 Manufacturer: QLogic Corp. Model: 375-3355-01 Firmware Version: 4.04.01 FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8; Serial Number: 0402R00-0725369970 Driver Name: qlc Driver Version: 20081115-2.29 Type: N-port State: online Supported Speeds: 1Gb 2Gb 4Gb Current Speed: 4Gb Node WWN: 2000001b321fbd9c Result: Two ports available, one port per fiberchannel card. The WWPNs are printed in bold. There's also the port type (N_port in this case) and the current data bit rate.

HBA Diagnosis on Solaris (2 of 4)

3. Which WWPNs (storage devices) are seen from my fiberchannel interface ports? Use "fcinfo remote-port -p" with the interface port WWPN found in step 2: # fcinfo remote-port -p 2100001b321fbd9c Remote Port WWN: 210000e08b0d82c6 Active FC4 Types: SCSI,IP SCSI Target: no Node WWN: 200000e08b0d82c6 Remote Port WWN: 2100001b321fb79d Active FC4 Types: SCSI,IP SCSI Target: no Node WWN: 2000001b321fb79d "SCSI" types are storage disks (storage volumes). SCSI,IP" types are devices capable of talking IP and SCSI (storage). In this case this is a Solaris host with IP configured.

HBA Diagnosis on Solaris (3 of 4)

4. Are all disks configured as solaris disks (usable by Solaris)? # cfgadm -al Ap_Id Type Receptacle Occupant Condition c3 scsi-bus connected configured unknown c3::dsk/c3t0d0 disk connected configured unknown c4 fc-fabric connected configured unknown c4::2100001b321fb79d unknown connected unconfigured unknown c4::210000d0231c50ab disk connected configured unknown c4::210000e08b0d82c6 unknown connected unconfigured unknown c4::220000d0232c50be disk connected configured unknown c5 fc-fabric connected configured unknown c5::2100001b321f9eb3 unknown connected unconfigured unknown c5::210000d0231c50be disk connected configured unknown c5::220000d0232c50ab disk connected configured unknown usb0/1 unknown empty unconfigured ok usb0/2 unknown empty unconfigured ok usb0/3 usb-device connected configured ok usb0/4 usb-storage connected configured ok usb0/5 usb-floppy connected configured ok

HBA Diagnosis on Solaris (4 of 4)

5. Is multipathing working?
Use "luxadm display" with the target WWPN of the storage disk device to list the paths: # luxadm display 220000d0232c50be DEVICE PROPERTIES for disk: /dev/rdsk/c6t600D0230006C1C4C0C50BE16FF1FE200d0s2 Vendor: ADVUNI Product ID: OXYGENRAID 416F4 Revision: 347B Serial Num: 0C50BE16FF1FE200 Unformatted capacity: 0.000 MBytes Write Cache: Enabled Read Cache: Enabled Minimum prefetch: 0x0 Maximum prefetch: 0xffff Device Type: Disk device Path(s): /dev/rdsk/c6t600D0230006C1C4C0C50BE16FF1FE200d0s2 /devices/scsi_vhci/disk@g600d0230006c1c4c0c50be16ff1fe200:c,raw Controller /dev/cfg/c4 Device Address 220000d0232c50be,2 Host controller port WWN 2100001b321fbd9c Class primary State ONLINE Controller /dev/cfg/c5 Device Address 210000d0231c50be,2 Host controller port WWN 2100001b321fe69b Class primary State ONLINE

Solaris "FC Device Address" syntax they are defined as wwpn,lun.

Installing HBA driver on Windows

Install the QLogic adapter card in the host system. Connect the cable to the storage unit port. Install and configure the adapter driver. See the installation instructions that are provided by your host adapter vendor for specific adapter and adapter driver instructions. Press Ctrl+Q to get to the FAST!Util menu. Use FAST!Util to set the parameters to the values that are listed below table Restart the server.

Find the WWPN for a Windows host

To locate the WWPN, perform the steps for the adapter you are using. Emulex adapter
Click Start Programs Emulex Configuration Tool. From the Emulex Configuration Tool menu in the Available Adapters window, double-click an adapter entry to display the WWPN information.

QLogic adapter using the SANsurfer toolNote: This tool enables you to locate the WWPN without restarting the server.
Go to the QLogic website at QLogic and click the Downloads tab. On the Downloads page, select your adapter type, model, and operating system, and then click Go. On the adapter page, go the tools list to download the SANsurfer tool.

QLogic adapter using the FAST!Util utility

Restart the server. Press Alt+Q to get the FAST!Util menu.Note: If you have more than one fibre-channel adapter installed, all the fibre-channel adapters display. Scroll down to the adapter you want and press Enter. From the FAST!Util menu: Scroll down and select Select Host Adapter. Scroll up and select Configuration Settings. Press Enter. From the Configuration Settings menu, select Host Adapter Settings. Write down the 16-digit alphanumeric string that is displayed on your screen.

Verifying storage configuration on windows

Complete this task to determine if Windows Server 2000, 2003, or 2008. is configured for storage. About this task To verify that the Windows host is configured for storage, perform the following steps: Partition new drives with Disk Management. From the Windows Server desktop, right-click Start. Click Explore and verify that you can see the fibre-channel drives. Select a large file (for example, 9 MB file), and drag (copy) it to a fibre-channel drive. Verify that the file was copied to the selected drive.

Powerpath Installation
The powerpath software can be downloaded from the following URL. https://powerlink.emc.com

1. On Powerlink, select Support > Downloads. 2. Download the tar file to /usr. 3. Unzip the installation package. Type: gunzip EMCpower.AIX.5.3.0.GA.tar.gz 4. Extract EMCpower_install. Type: tar -xvf EMCpower.AIX.5.3.0.GA.tar.gz 5. Perform one of the following: For an existing PowerPath installation, run rm.toc to delete the .toc file. For new PowerPath installations, skip to step 6 . 6. Install the software. Type: installp -agXd . EMCpower 7. Run powermt config.

Upgrading powerpath (1 of 3)
PowerPath can be upgraded from versions 4.5.x or later to 5.3 SP1. The upgrading process requires knowledge of the PowerPath environment and its unique upgrading process. Based on your environment and requirements, you can upgrade PowerPath by using one of the following 2 methods: 1) Full package installation: You can directly upgrade PowerPath from a previous version with the later version of PowerPath package. 2) Uninstall the previous version and install the later version Before upgrading PowerPath: Download the latest version of EMC Reports, available on Powerlink, and then run the PowerPath Configuration Checker (PPCC). Check the Powerlink website, for the most current release notes and service packs. Determine which service packs (if any) you want to install. You do not need to remove the previous version of PowerPath before upgrading to PowerPath 5.3.

Upgrading powerpath (2 of 3)
During an upgrade: During an upgrade, do not add devices to your PowerPath configuration.

After an upgrade: You do not need to restart the host after the upgrade, provided you close all applications that use PowerPath devices before you install PowerPath 5.3.
You do not need to re-enter license information for PowerPath multipathing. PowerPath will use your existing license key, if necessary, converting the old 12character license key to a 24-character key.

Upgrading powerpath (3 of 3)
Upgrade procedure:
To upgrade from PowerPath 4.5.x or later to PowerPath 5.3 SP1: 1. Close all applications that use PowerPath devices, and vary off all volume groups except the root volume group (rootvg). If EMC Solutions Enabler daemons, such as storwatchd and storapid, are running, stop the daemons. Type: stordaemon shutdown <daemon> If EMC ControlCenter is running on the host, stop the ControlCenter agents. In a VNX OE and CLARiiON environment, if the Navisphere Host Agent is running, type: /etc/rc.agent stop 2. Optional. Run powermt save in PowerPath 4.x to save the changes made in the configuration file 3. Install PowerPath as described in Powerpath Installation on slide 25. 4. Run powermt config. 5. Optional. Run powermt load to load the previously saved configuration file. When upgrading from PowerPath 4.x to PowerPath 5.3, an error message is displayed after running powermt load, due to differences in the PowerPath architecture. This is an expected result and the error message can be ignored.

Powerpath troubleshooting (1 of 6)
Most problems reported as PowerPath problems are host configuration problems. For example, the host does not have the correct HBAs drivers installed or the storage system settings are incorrect The host must be able to see its external storage devices before these devices can be placed under PowerPath control. Specifically, if the host cannot see the storage devices, power Path will be unable to view the storage devices. PowerPath does not make storage devices visible to the host.

Powerpath troubleshooting (2 of 6)
Use the AIX lsdev command to confirm that the host can see its external storage devices. Use cfgmgr if any devices are missing. If devices are still missing after this command, there is a connectivity problem between the host and the array. You must fix the connectivity problem before continuing with the troubleshooting procedure In the following example, the host can see eight CLARiiON devices: lsdev -Cc disk hdisk0 Available 1S-08-00-8,0 16 Bit LVD SCSI Disk Drive hdisk1 Available 1S-08-00-9,0 16 Bit LVD SCSI Disk Drive hdisk2 Available 1S-08-00-10,0 16 Bit LVD SCSI Disk Drive hdisk3 Available 11-08-01 EMC CLARiiON FCP RAID 5 Disk hdisk4 Available 11-08-01 EMC CLARiiON FCP RAID 5 Disk hdisk5 Available 11-08-01 EMC CLARiiON FCP RAID 5 Disk hdisk6 Available 11-08-01 EMC CLARiiON FCP RAID 5 Disk hdisk7 Available 11-08-01 EMC CLARiiON FCP RAID 5 Disk hdisk8 Available 11-08-01 EMC CLARiiON FCP RAID 5 Disk hdisk9 Available 11-08-01 EMC CLARiiON FCP RAID 5 Disk hdisk10 Available 11-08-01 EMC CLARiiON FCP RAID 5 Disk

Powerpath troubleshooting (3 of 6)
Are PowerPath devices configured on the host?
Type the PowerPath powermt display command to verify the correct number of storage devices are under PowerPath control. The powermt display output shows, for each type of storage system attached to the host, the number of storage devices visible to host under PowerPath control (the logical device count). The following is an example of powermt display output. In the example, 50 CLARiiON devices are under PowerPath control: Symmetrix logical device count=0 CLARiiON logical device count=50 Invista logical device count=0 Ess logical device count=0 Hitachi logical device count=0 HP xp logical device count=0 HP HSx logical device count=0 ============================================================================== ----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats -----### HW Path Summary Total Dead IO/Sec Q-IOs Errors ============================================================================== 1 fscsi1 optimal 200 0 - 0 0 2 fscsi2 optimal 200 0 - 0 0 Ensure that the total number of storage devices under PowerPath control is correct. If any devices are missing, complete the following teps to bring the missing devices under PowerPath control:

Powerpath troubleshooting (4 of 6)
Note: Stop all I/O before performing the following procedure. 1. Type the AIX cfgmgr command to refresh the storage devices. 2. Type the PowerPath emc_cfgmgr command to configure the PowerPath devices. Note: The emc_cfgmgr utility resides in the /lpp/usr/EMC/clariion/bin and /lpp/usr/EMC/symmetrix/bin directories. If there are third-party storage devices that should be under PowerPath control, and the output of the powermt display command does not include these devices, type the powermt manage command to place these devices under PowerPath control. 4. Type powermt config to add the storages devices to the PowerPath configuration. 5. Type powermt save to save the configuration.

Powerpath troubleshooting (5 of 6)
Are the PowerPath devices active and alive?
Type powermt display dev=all to check the state of the PowerPath devices. The following are examples of powermt display dev=all output. The first shows a CLARiiON device; the second a Symmetrix device. You should see output resembling the following for each storage device. Pseudo name=hdiskpower0 CLARiiON ID=APM00052905641 [lcla148_150_sg] Logical device ID=600601603558150092D0414EB73CDA11 [LUN 85] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A ========================================================================= ===== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --### HW Path I/O Paths Interf. Mode State Q-IOs Errors ========================================================================= ===== 0 fscsi0 hdisk1 SP A0 active alive 0 0 0 fscsi0 hdisk21 SP B0 active alive 0 0 1 fscsi1 hdisk41 SP A0 active alive 0 0 1 fscsi1 hdisk61 SP B0 active alive 0 0

Powerpath troubleshooting (6 of 6)
Are the PowerPath devices active and alive? cntd
Failed PowerPath devices; PowerPath devices with dead paths If the output from powermt display dev=all indicates that a PowerPath device has failed or has I/O paths that are dead: 1. Stop all I/O. 2. Type powermt restore to test and mark all paths dead that are missing or point to the wrong logical device. 3. Type powermt check. When prompted to remove a dead path, respond using the -a option to remove all dead paths. Note: If you intentionally keep certain paths in the dead state, do not use The -a option. Instead you will need to remove paths individually. 4. Type powermt config to configure all disks possibly pointing to storage system logical devices different from the devices of which PowerPath is aware.

Questions and Evaluation

Thank you