Professional Documents
Culture Documents
Revision History
1.0 June 14, 2009
Initial Version
Notices:
This paper is intended to provide information regarding IBM BladeCenter Open Fabric Manager. It
discusses findings based on configurations that were created and tested under laboratory
conditions. These findings may not be realized in all customer environments, and implementation
in such environments may require additional steps, configurations, and performance analysis. The
information herein is provided AS IS with no warranties, express or implied. This information
does not constitute a specification or form part of the warranty for any IBM or non-IBM products.
Information in this document was developed in conjunction with the use of the equipment
specified and is limited in application to those specific hardware and software products and levels.
The information contained in this document has not been submitted to any formal IBM test and is
distributed as is. The use of this information or the implementation of these techniques is a
customer responsibility and depends on the customers ability to evaluate and integrate them into
the customers operational environment. While each item may have been reviewed by IBM for
accuracy in a specific situation, there is no guarantee that the same or similar results will be
obtained elsewhere. Customers attempting to adapt these techniques to their own environments
do so at their own risk.
IBM may not officially support techniques mentioned in this document. For questions regarding
officially supported techniques, please refer to the product documentation, announcement letters,
or contact the IBM Support Line at 1-800-IBM-SERV.
This document makes references to vendor-acquired applications or utilities. It is the customer
responsibility to obtain licenses of these utilities prior to their usage.
Copyright International Business Machines Corporation 2009. All rights reserved.
U.S. Government Users Restricted Rights Use, duplication, or disclosure restricted by GSA ADP
Schedule Contract with IBM Corp.
Table of Contents
Table of Contents ........................................................................................................................3
Executive Overview .................................................................................................................4
How does Fibre Channel Over Ethernet Work? .........................................................................5
Logical Connectivity ..............................................................................................................6
IBM BladeCenter and System x FCoE Product Details .............................................................9
10Gb Pass-Thru Module Features........................................................................................9
QLogic 2-port 10 Gb Converged Network Adapter (CFFh) for IBM BladeCenter (QMI8142) 10
QLogic 2-port 10Gb Converged Network Adapter for IBM System x....................................10
Brocade 2-port 10 Gb Converged Network Adapter for IBM System x .................................11
Brocade 8000 CEE Switch ...................................................................................................11
Implementing Blade Boot from SAN Solution with BOFM Enabled........................................14
Useful Links ........................................................................................................................35
Trademarks ...........................................................................................................................36
Executive Overview
This document illustrates the technical implementation details of Fibre
Channel over Ethernet (FCoE) solution with IBM BladeCenter. The objective of
this document is to demonstrate the configuration steps for blade boot from FC
SAN using the BladeCenter 10Gb Ethernet Pass-Thru Module and the Brocade
B32 Converged Enhanced Ethernet (CEE) solution.
The BladeCenter chassis-based FCoE solution will also leverage
BladeCenter Open Fabric Manager (BOFM) for Ethernet MAC address and Fibre
Channel WWPN virtualization solution to enable blade pre-provisioning, simplify
blade server re-deployment and automatic blade failover using the Advanced
Open Fabric Manager. In addition to address virtualization and automatic blade
failover, BOFM simplifies and automates the configuration tasks such as enable
adapter BIOS and selectable boot device settings performed in the HBA firmware
utility. Traditionally, this requires the administrator to interrupt the blade boot
process and change the default HBA settings on one or more blades individually.
This can be painful and time consuming if the configuration is performed
manually on hundreds of blade servers. With BOFM, you can simply modify the
configuration file and apply the profile to one or more blades in multiple chassis
at the same time.
The following sections list the configuration process for blade boot from
SAN using the Qlogic CNA attached to DS4700 and System x rack server booting
locally, and access IBM DS4700 using the Qlogic CNA.
The FCoE test environment setup includes the following hardware:
The following figure shows the final topology implemented for blade boot from
SAN environment.
FC fabric services for FCoE VN_port devices, which provide access to FCoE
VN_port devices similar to those provided by FC F_port to FC N_port
devices
FCoE-to-FC switching and translation services:
- FCoE servers to/from FC targets
- FCoE targets to/from FC servers
NOTE
NPIV is an FC facility allowing multiple N_port IDs to share a single
physical N_port. This allows multiple FC initiators to occupy a single
physical port.
For FCoE ENodes, there is support for only directly-connected links.
Flow isolation
Logical Connectivity
Figure 2 shows a logical view of the Brocade 8000 CEE switch. There are
two independent switches, one for CEE and the other for FC. There are six
embedded FCoE virtual ports bridging the two switches. Each of these port is a
10-Gigabit FCoE port. The Fabric OS has FCoE-specific CLI commands for the
configuration and management of these FCoE ports.
The FCoE ports differ from the usual concept of a port in a Brocade switch
as they are embedded ports and are not directly associated with an external
physical port on the switch. Configuration of the FCoE ports through the regular
Fabric OS FC CLI is disabled; only the Fabric OS FCoE CLI commands can be
used to configure and manage these ports.
The FCoE ports are displayed as ports in some show command displays.
The data displayed for a single FCoE port is the sum of the individual ports
comprising the FCoE port. CEE statistics are not shown for the FCoE ports.
NOTE
In the Fabric OS CLI the FCoE ports can be enabled or disabled; there is no other
configuration required. Each of the embedded FCoE ports is used explicitly for
FCoE VF_port service and provide four MAC addresses and flow isolation for up
to four flows per FCoE port.
The host must have FCoE adapter installed and connected to CEE + FCoE
switch
10Gb Ethernet and 8Gb FC device drivers must be installed on the host
The link has to be established from FCoE to the corresponding 10Gb CEE
port
The Host sees the LUNs from the FCoE HBA firmware if boot from SAN
Low cost solution for Clients to connect to any Top Of Rack 10Gb or CEE
capable Switch
No Configuration necessary
No On-Board Management
Support for SAN boot over CEE, PXE boot, and iSCSI boot
Support for BladeCenter Open Fabric Manager for BIOS, UEFI, and FCode
Two SFP+ cages for either SFP+ Fiber SR or SFP+ Active Copper
Standard PCI Express half length card with low profile form factor
Support for both standard PCI-E slot and low profile PCI-E slot
10
Support for SAN boot over CEE, PXE boot, and iSCSI boot
Support for BladeCenter Open Fabric Manager for BIOS, UEFI, and FCode
Two SFP+ cages for either SFP+ Fiber SR or SFP+ Active Copper
Standard PCI Express half length card with low profile form factor
Support for both a standard PCI-E slot and a low profile PCI-E slot
Support for SAN boot over CEE, PXE boot, and iSCSI boot
Eight Fibre Channel universal (E, F, M, and FL) ports with 1, 2, 4, and 8
Gbps
CEE ports
FCoE features
The FCoE translation entity built into the hardware engine provides:
11
FCoE)
Extraction of Fibre Channel frames from FCoE Ethernet packets (FCoE >
FC)
Performance
ISL Trunking
Classes of service
Port types
12
Media types
USB
CEE services
Spanning Tree Protocol (STP, MSTP, RSTP, VLAN Tagging (802.1q), MAC
address learning and aging; native FCoE switching; IEEE 802.3ad Link
Aggregation (LACP); access control lists based on VLAN, source,
destination address, and port; eight priority levels for QoS and 4k VLANs;
Priority-based Flow Control (PFC); Data Center Bridging eXchange
(DCBX)-Capabilities Exchange; Enhanced Transmission Selection (ETS)
Licensing options
Fabric OS 6.1.2_cee includes the following optional features that can be enabled
13
via license keys and are applicable only to the Fibre Channel ports of the Brocade
8000:
Management software
Management access
One 10/100/ Gigabit Ethernet (RJ-45), in-band over Fibre Channel, one
serial port (RJ-45), and one USB port
Ensure FCoE HBA has the latest firmware that is also compatible with
BOFM
Connect the cables from the 10Gb Pass-thru to 10Gb interface on Brocade
8000
The link status shows UP on the 10Gb Ethernet Interface on Brocade 8000
14
o Trunking
Note: Verify and confirm that BOFM is enabled on the Blade slot and the status is
Normal, from the AMM GUI, select Blade tasks Configuration Open Fabric
Manager and the blade slot. The ports 1 and 2 are on-board 1Gbps Broadcom NICs
and ports 5 & 7 are 10Gbps FCoE ports. Both Ethernet and Fibre Channel ports map to
high-speed switch modules in bays 5 and 7. So, when creating the BOFM configuration,
you must select ports 5 and 7 to enable BOFM on the Qlogic CNA.
The following figure lists the WWPNs of the FCoE HBA ports virtualized by
BOFM and the WWPNs of the storage controller is also applied on the HBA.
Important: Once BOFM is enabled on the blade slot then it will automatically
configures the following tasks:
BOFM eliminates the manual configuration tasks listed above. BOFM also
overwrites the adapter BIOS and Selectable Boot Settings configuration
performed manually from the HBA firmware utility.
15
1. Verify the Link status on 10Gb Ethernet interface on Brocade 8000 CEE.
From the Brocade Web interface, select Port Admin CEE interfaces as
shown below. The blade is connected to Ethernet interfaces 3 and 14 on
the Brocade 8000 switch via the pass-thru module. The link status on
ports 3 & 14 shows Online
16
17
The following commands can be used to verify the configuration from the
Brocade 8000 CLI:
The fcoe loginshow displays the FCoE device WWPNs connected to the
fabric. The WWPNs in blue font belongs to the blade in bay 12 connected to
ports 3 & 14 on the Brocade 8000 switch. The WWPNs listed below should also
be available from the zone configuration menu to add as zone members with FC
target WWPNs.
18
19
20
!
interface TenGigabitEthernet 0/9
no shutdown
lldp fcoe-priority-bits 0x8
!
interface TenGigabitEthernet 0/10
switchport
switchport mode converged
switchport converged allowed vlan add 1002
no shutdown
cee fcoe-test1
!
interface TenGigabitEthernet 0/13
shutdown
!
interface TenGigabitEthernet 0/14
switchport
switchport mode converged
switchport converged allowed vlan add 1002
no shutdown
cee fcoe-test1
!
interface TenGigabitEthernet 0/15
shutdown
!
protocol lldp
advertise dcbx-fcoe-app-tlv
advertise dcbx-fcoe-logical-link-tlv
!
line console 0
login
line vty 0 31
login
!
end
swd77#
21
Effective configuration:
cfg: FCoE_Test
zone: BCH5_BS12p1_DS4700
20:15:00:a0:b8:26:12:10
21:80:00:e0:8b:00:01:1e
zone: BCH5_BS2p1_DS4700
20:14:00:a0:b8:26:12:10
20:15:00:a0:b8:26:12:10
21:00:00:c0:dd:10:10:99
zone: BCH5_BS2p2_DS4700
20:14:00:a0:b8:26:12:10
20:15:00:a0:b8:26:12:10
21:00:00:c0:dd:10:10:9b
zone: X3550p1_DS4700
20:14:00:a0:b8:26:12:10
20:15:00:a0:b8:26:12:10
21:00:00:c0:dd:10:0e:19
zone: X3550p2_DS4700
20:14:00:a0:b8:26:12:10
20:15:00:a0:b8:26:12:10
21:00:00:c0:dd:10:0e:1b
swd77:admin>
Note: For instructions to configure zoning, refer to the Brocade FOS guide.
Verify Storage Configuration
From the DS4000 Storage Manager GUI, verify and confirm that the boot LUN is
mapped to the host as shown in figure below:
Once the above configuration steps are completed, the blade is now setup to
boot from SAN and ready for OS installation.
22
2. Download the latest device driver for the FCoE adapter and extract to a
floppy diskette
3. The FCoE device driver contents should include the files listed in the
following figure:
23
4. Setup the boot sequence for blade so that it boots from CD first. From
AMM GUI select Blade Tasks Configuration Boot Sequence Blade.
5. Insert the OS CD1 and power ON the blade. For this test, we will be
installing Windows 2003 R2 SP2 integrated.
Note: The Service Pack 2 integrated is required for successful install of
Win2003 R2 image. Without SP2 integrated, the OS will not load the FCoE
device driver correctly.
6. Once the Windows 2003 R2 SP2 CD is detected, immediately look for F6
prompt and quickly press F6 key.
24
7. If F6 key was pressed during the initial setup, then the following prompt
shown in figure below will be displayed that allows you to load the FCoE
driver. Ensure that the USB Floppy diskette is inserted and select S to
specify Additional Device.
8. The following menu is displayed after reading the contents of the diskette.
25
9. The following Windows Setup menu shows that the FCoE driver was
successfully read from the diskette. Do not remove the diskette as
Windows Setup needs to copy the driver from the diskette. Press Enter to
continue as no additional driver is needed for the blade to boot from SAN.
26
Figure 13: Windows Setup will load the FCoE driver for Qlogic CNA
10. Press F8 key to agree to Windows Licensing Agreement and proceed with
OS installation.
11. The following screen shows that the Windows Setup can see the 20GB
27
12. The following figure shows that new partition is created on boot LUN ID
=0, bus = 0 on qlfcoe adapter:
28
13. From the following menu, select the File System and Format type and
press Enter key.
14. The following figure shows the Windows Setup is formatting the boot
LUN:
15. The following figure shows the Windows Setup is copying the FCoE device
29
16. The following figure shows that the Windows 2003 R2 SP2 OS installation
process in progress:
From this point, you can follow the prompts to complete the Windows 2003 OS
30
installation. Remember that the OS was installed on the boot LUN via the single
path. Download and install RDAC driver on the host and add additional paths
from the host to the boot LUN.
31
32
3. The following figure shows that the MPIO driver is successfully installed.
4. The following figure shows that there are multiple paths from the host to
the boot LUN with MPIO driver:
33
Figure 24: The Windows Device Manager shows the MPIO driver is installed
5. You can perform the failover test by disabling one path at a time from the
host to storage. This can be done by disabling the switch ports to which
the blade or storage ports are connected.
6. Once the path is disabled on host side or the storage side, the OS should
still be available via the alternate path. The MPIO driver automatically
moves the LUN to the available path.
Conclusion
This concludes the configuration process to deploy blade boot from SAN using
the Qlogic 10 Gbps CNA attached to the Brocade 8000 CEE switch and IBM
DS4700 storage subsystem.
34
Useful Links
1.
42C1830 - QLogic 2-port 10Gb CNA (CFFh) for IBM BladeCenter (FRU P/N
42C1832, Card P/N 42C1831):
http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/Product
_detail_new.aspx?oemid=394&companyid=6
2.
3.
4.
https://www947.ibm.com/systems/support/supportsite.wss/selectproduct?brandind=5
000028&familyind=0&oldbrand=0&oldfamily=0&oldtype=0&taskind=1&ps
id=bm
QLogic 2-port 10Gb Converged Network Adapter (CFFh) for IBM BladeCenter
Tech Note
http://www.redbooks.ibm.com/abstracts/tips0716.html?Open
5.
http://www.redbooks.ibm.com/abstracts/tips0718.html?Open
35
Trademarks
IBM, the IBM Logo, BladeCenter, DS4000, eServer, and System x are trademarks of International
Business Machines Corporation in the United States, other countries, or both.
For a complete list of IBM Trademarks, see http://www.ibm.com/legal/copytrade.shtml.
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other
countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
36