Fundamentals

SAN and Fibre Channel
Troubleshooting
Storage Competency Centre SEA

Troubleshooting
Agenda
Module 1 - What is a SAN ?
Module 2 - Fiber Channel Protocol
Module 3 Fibre Channel Topologies
Module 4 Brocade Switches
Module 5 - Fibre Channel on HP-UX

Troubleshooting
Agenda
Module 6 Troubleshooting Fibre Channel
on HP-UX
Module 7 Troubleshooting Brocade

Switches
Module 8 Further Information and Web

Links.

Troubleshooting
Module 1
What is a SAN?
What is a SAN?
Storage Area Network:

A high availability, high-performance dedicated storage network
that connects servers and storage in a secure, flexible and scalable
architecture.
A pool of storage devices that can be used by any host within the
storage network.
Consists of Servers, Switches, Hubs, Bridges, & Storage Devices.
The current standard is Fibre Channel. Fibre Channel is the

lower level transport used, with SCSI-3 providing the upper
layer protocol.
SANs represent a third type of network, behind LANs and

WANs
What is a SAN?
Benefits of a SAN
Increased Performance
- The
Fibre Channel based SAN provides the ability for the Fibre
Channel HBA to access storage with up to 100 megabytes of
bandwidth. In some cases the duplexing capabilities of Fibre Channel
can provide up to 200 megabytes of potential throughput. 2 Gigabit
GBICs are now available with a new 10 Gigabit standard being
developed.
Increased Availability
- Redundant
data paths to disks (LUNS)
What is a SAN?
Benefits of a SAN
Increased Scalability and Flexibility
-Supports up to 14 million devices
-layered architecture supports three connection topologies (direct, loop
and switched fabric)
Flexible Use of Storage Devices

- Dynamic
reconfiguration with centralized management tools
- Same storage device may be accessed by multiple servers
What is a SAN?
Clients
Network
Attached
Storage
Network
Printers
Traditional
Servers
w/ attached
Storage
LAN
Application
Servers
Database
Servers
File &
Print
Servers
Fibre
Channel
Fibre Channel
Switches, Hubs, etc.
High-end
Storage
Arrays
Mid-range
Arrays &
JBOD
SAN
Secondary
Storage
(DLTs, etc.)
What is a SAN?
Components of a SAN
SAN Management
SAN
Software
Disk/Lun Access
Tape Access
Operating System & Driver

FC Host Bus Adapter
FC SAN
Hardware
SAN Storage
FC Hub & Switch

FC Bridge
Disk Arrays
Tape

Troubleshooting
Module 2
Fibre Channel
Protocol
Fibre Channel Protocol
What is Fibre Channel?

A new transport protocol
Based on an integrated set of standards
Standards are being developed by an ANSI
committee
Ideal for these Applications:
Mass Storage
Real Time
Video
BackBone
Networks
Avionics
Fibre Channel is a comprehensive standard. It is very large
and is undergoing continual development by several ANSI committees.
Fibre Channel Protocol Mass Storage
Fibre Channel for Mass Storage

Provides communication between system units and
storage devices
Transports existing protocols
SCSI
IP
HIPPI
IPI
IEEE 802.2
Mass storage has several device classes: tapes, disks and disk arrays.
Current HP Fibre Channel mass storage products support tapes and disk arrays.
Fibre Channel can transmit a number of existing protocols; however, HP
supports only SCSI-3 over Fibre Channel for its mass storage devices.
Fibre Channel Protocol Advantages
Advantages of Fibre Channel for Mass Storage

Distance- Up to 10km using 9 micron, single mode Fiber
Connectivity- San Loop 126 nodes. SAN Fabric is scalable from 2
to 16 million ports.
Performance- High bandwidth ( 100 Mbytes/s and beyond)
Reliability- Error rate is very low
Heterogeneous support
Fibre Channel may can transmit a number of existing networking
and I/O protocols. (SCSI,HIPPI,IPI), (IP,IEEE 802.2)
Fibre Channel Protocol Advantages
Advantages of Fibre Channel for Mass Storage

Fibre channel greatly exceeds SCSI-3 Interface distance limitations of 25
meters
HP supports configurations up to 10km apart, using 9 micron single mode
Fiber cable
HP currently supports speeds up to 1063 Mbps. Speeds up to 4GB are being
tested
Fibre Channel allows a large number of intermixed devices and topologies
Fibre Channel transports data with a very low error rate
Fibre Channel Protocol Media

Media
Fiber optic cable for long distances and wide distribution
9 um single mode fiber (10km)
50 um multimode fiber (500m)
62.5 um multimode fiber (175m) {used mainly for FDDI}
Copper for short distances and high performance
Video and long video coax
Mini coax
Shielded twister pair (STP)
Twinax
Single mode fibers are optical fibers 9um in diameter. Nine um is so small that only a single frequency
range of light can enter the fiber, hence a single mode. The fiber length can be about 10km at 1Gbps.
HP supports fiber optic cable for Fibre Channel.
Fibre Channel Protocol

The Fibre Channel Protocol suit is based in a Multi-Layered
Architecture.
These layers define standards and services provided by mechanisms built
into the protocol.
Fibre Channel uses 5 layers labeled FC-0 through FC-4
Some that are defined have not been implemented ( currently the FC-3
layer is not used, however we will still look at this layer ).
Fibre Channel Protocol Functional Levels

Fibre
Channel Functional Levels

FC-0 Level -- Physical
FC-1 Level -- Encode / Decode
FC-2 Level -- Framing protocol / Flow control
FC-3 Level -- Common services
FC-4 Level -- Protocol mapping
Fibre Channel Protocol FC Levels
Level FC-0 deals with physical media type, cable length and signaling methods.
Fibre Channel Protocol FC Level Placement
Fibre Channel Protocol FC-0 Components

FC-0 Component Location
Fibre Channel Protocol FC-0 Connections
Fibre Channel Protocol Optic Cable

Main parts of a fiber optic cable
Fiber is immune to interference, so it is relatively secure from eavesdropping.

Fiber is not subject to electromagnetic interference.
Fibre Channel Protocol Fiber Internal Reflection
Fibre Channel Protocol Microbend Loss
Maximum bend radius is 3 cm.

If the cable is bent beyond 3 cm, data loss or corruption is likely to occur.
The cable can crack or break.
Make sure to give some strain relief for connectors.
Fibre Channel Protocol Microbend Loss
Microbend Loss How it shows up in a SAN

Massive Bad TX characters on Links.
PVLink failovers in HP-UX ( POWERFAIL Messages ).
Out Of Sync Storms.
Link Failure/CRC errors.
NT/2000 systems could hang.
Sometimes shows up only when link is loaded with frames.
Fibre Channel Protocol Media Types

FC-0 Media choices for ports
Runs up to 10km require single mode Fiber (9micron)

Runs up to 175 meters require multimode
50 and 62.5 are supported. But 50 micron allows greater speed and distance
Fibre Channel Protocol FC-1

FC-1 Transmission Character
Each 8-bit byte transmitted is encoded into a
10-bit transmission character
- Known as 8B/10B
- 256 symbols are used for data bytes
- Data transparency results because special characters are
used to identify control functions in the bit stream
- 12 special characters can be used for control; only one is
used at present
Allows error checking by detecting unrecognized
codes and other errors
Fibre Channel Protocol FC-2 Framing Protocol

FC-2 Framing protocol
A frame is the basic unit of information transfer
- Smallest indivisible unit of user data sent
- Length can be up to 2148 bytes
A sequence is made up of one or more frames
- A ULP ( Upper Layer Protocol ) object to be transmitted
- The error recovery boundary for Fibre Channel
An exchange is made up of one or more sequences
- SCSI task, an IP connection
- Has an Initiator and a Target
A packet is made up of one or more exchanges
Fibre Channel Protocol FC-3 Common Services

FC-3 Common Services
The
FC-3 level, Common Services, concerns itself with

functions spanning multiple N_ports.
FC-3 is the single point in the architecture layout through

which all traffic must flow in both directions.
A node may have several ports. A node may have several
ULPs and FC-4 mappings. However, there is only one
FC-3 Common Services level per node.
Fibre Channel Protocol FC-3 Common Services

FC-3 Common Services
The FC-3 level can manage a set of tables holding the
login information for other active ports. Each port may
be logged into any number of other ports. The FC-3
level knows which ports are busy and what exchanges
they are busy with.
HP-UX and most current Fibre Drivers do not support
this FC-3 level as of this time. This level is usually
transparent in the driver.
Fibre Channel Protocol FC-4 Mapping Protocol

FC-4 Mapping Protocol
The FC-4 level is the mapping level between the upper level
protocols ( ULP ) and the lower Fibre Channel levels.
Each FC-4 provides rules for transforming Information

Units ( LUs ) into Fibre Channel sequences and
exchanges. Conversely, each FC-4 provides rules
for transforming sequences and exchanges into LUs.
Fibre Channel Protocol FC-4 Mapping Protocol

FC-4 Mapping Protocol
To send data, the FC-4 level takes an IU from the ULP

and transforms it into sequences for FC-3/FC-2.
To receive data, the FC-4 level takes a sequence from
the FC-3/FC-2 level and transforms it into an IO for a
ULP.
FC-4s developed by the ANSI committees are intended
to provide clear mapping instructions between the ULP
and the FC-3/FC-2 levels to enhance interoperability
between applications.
Upper Level Protocols ( ULP )

Upper Level Protocols ( ULP )
ULPs allow two devices to communicate
. Computer and disk array
. Computer and tape library
. Client and server
ULPs preserve legacy protocols (like SCSI)
ULPs are specified as a separate protocol standard
ULP SCSI-3 Architecture

SCSI-3 Overview
CDB
CPD
Command
STATUS
CRD
Logical
Data
CDB Command Descriptor Block

CPD Command Parameter Data
CRD Command Response Data
Sense
Data
Autosense
Preserved
NOT USED
Classes of Service
Classes of Service
Class 1 Dedicated connection service
. Connection-oriented
. Acknowledged delivery
Class 2 Multiplexed service
. Connectionless
. Acknowledged delivery
Class 3 Datagram service ( HP Provides Class 3 Service )
. Connectionless
. Unacknowledged delivery
Class 4 Virtual connections (being defined) Fibre Channel
classes of service are managed by the FC-2 level
Basic Definitions ( N Port )
Basic Definitions ( NL_Port )

Basic Definitions ( NL_Port )
Basic Definitions ( Other Ports )
Hosts & Devices
N_Port = Node
NL_Port = Node Loop
Switches
F_Port = Fabric
FL_Port = Fabric Loop
QL_Port = Private Loop
E_Port = Extension
G_Port = F + E
U_Port = F + FL + QL+ E
Note: switch ports will be discussed more in the Brocade Switch Chapter
Fibre Channel Addressing

Fibre Channel Addressing
A node is dynamically assigned a 24 bit N_Port ID, that is used in
frame routing. This address is embedded in the D_ID and S_ID of
the frame header. This shortened address optimizes performance in a
switched environment. The switch is responsible for assigned the
N_Port ID, and enters the information in a name server within the
switch.
Within a loop, the devices themselves select unique addresses. If no
switch exists, the upper two bytes are zeroed out Private Loop.
If the Loop is attached to a switch and the devices are fabric
aware, the upper 2 bytes are assigned a value by the switch, this is a
Public Loop.
24 bit N_Port ID
Domain
Area
Port

Troubleshooting
Module 3
Fibre Channel
Topologies
Fibre Channel Topologies
Fibre Channel consists of 4 Topologies.

Direct Connect
Arbitrated Private Loop
Arbitrated Public Loop
Fabric Attach ( Point to Point )

100 MB/sec
TX
RX
100 MB/sec
RX
Direct
Connect
TX
2 fibres = 1 cable
Storage
Device
Direct Connect topology:

+ guarantees maximum bandwidth availability
+ freedom from collision or congestion
-extremely limited with only two nodes
-uses Private AL to operate

TX
RX
1 Fibre
TX
Basic 4 node
private Loop
100 MB/sec
per loop
TX
RX
RX
TX
Loop Topology:
+ good connectivity
+ moderate bandwidth
- bandwidth is shared
-latency can be a problem in high node configurations
- LIPs can cause issues

Loop ID and AL_PA
$01
$EF
Loop ID
EF
E8
E4
00
01
02
02
01
00
7C
7D
7E
higher Prio
$02
$00
AL_PA
FCAL internally uses AL_PAs

Devices use Loop IDs
Fibre Channel Topologies - LIP
What is a LIP ?
A Loop Initialization Primitive (LIP) is used to initiate
a procedure that results in a unique addressing for all
nodes, to indicate a loop failure, or to reset a specific
node.
A LIP is generated when a device is unplugged from
a switch or plugged into a switch, powered off or
powered on.
Fibre Channel Topologies AL and

the FL_Port
FL_Port
Connects Private and Public Arbitrated
Loops.
Full Fabric citizenship status for Loop Devices
FAN Support ( AL_PA address Keeping )
Added Value Non-Blocking Queues & Translative
Mode
Automatic registration of Private Loop devices in
the Simple Name Server
Fibre Channel Topologies AL and

the FL_Port
FL_Port
( Fibre Loop Port ) is only avaiable when

there are switches in the loop.
Only one active FL_Port on a Loop

An FL_Port has the highest Priority on the loop
An FL_Port does not have to arbitrate fairly
An FL_Port always wins the job of
Loop Initialization Master
Fibre Channel Topologies Loop

Initialization
Loop Initialization Begins
Loop Initialization Primitive
Loop Initialization
Select Master
LIP
LISM
LIPs sent over the loop, whether

originated by a FL_Port or sent
from other L_Port in Loop
FL_Port sends LISM frame with

D_ID = 0x00, rest of L_ports sends
LISM with D_ID= 0xEF. FL_port
becomes loop master when it
receives the same LISM that was
sent. However, if no FL_Port the
L_port with lowest WWN gets
Loop master
To ARB ( Loop Control )


Initialization
From LISM
Master Selected
ARBx Address x for
Loop control.
Loop Initialization
Fabric Assigned
ARB(F0)
LIFA
FL_Port ( or Master L_Port )

sends ARB(F0) to notify the
L_Ports that a loop mast is
selected.
FL_port send LIFA frame to L_ports

to find what L_ports had logged
into the Fabric previously. Those
that have will seta bit in the bitmap
corresponding to their previous
AL_PA. If this is the first Loop
Initialization after power up, the
Login bit is also set in the LIFA frame.
To LIPA
* ARB Arbitrate Primitive

Initialization
From LIFA
Loop Initialization
Previously
Assigned
LIPA

sends LIPA sequence with received
LIFA bitmap. LIPA is made from the
received LIFA bitmap.
This is where the device marks a bit
in the bitmap if it had not logged in
with Fabric in the previous Loop
Initialization
Loop Initialization
Hard Assigned
LIHA
To LISA
If a device has a Hard Assigned

AL_PA, set a bit in the bitmap that
corresponds to the hard assigned
AL_PA. If the bit is already taken by
other device, let it pass through and
set another unused bit during LISA
sequence.

Initialization
From LIHA
Loop Initialization
Soft Assigned
Loop Initialization
Hard Assigned
The LISA payload is made from the

LIHA Sequence payload received.
This is where an L_Port set the first
available bit in the bit map. Once set
this becomes the devices Soft
Assigned AL_PA. The received LISA
contains the accumulation of all bits
set by the loop devices.
LISA
NO
ID= 0x11050100
YES
Send LIRP Sequence
If the Loop Init.ID of the

received LISA is
0x11050100 LIRP is
supported by the Loop
Devices, then the FL_Port
sends LIRP and LILP. If
not, FL_Port sends CLS
to complete the Loop
Initialization.

Initialization
From LISA
Loop Initialization
Report Position
Loop Initialization
Loop Position
LIRP
LILP
To CLS

sets the Byte 0 of the AL_PA position
map to 0x01( offset ) and byte 1 to
0x00 ( Loop Master ). The frame is
Sent over the loop and each L_Port
Writes its AL_PA in the offset
Location in the bitmap.
The returning LIRP contains the

Accumulation of all the AL_PA
Position map. The FL Port simply
Changes the Loop Init ID to LILP and
Circulates in the loop once more to
Inform L_Ports of their relative AL_PA
position

Initialization
From LILP
Send CLS
Sending CLS transitions all the

L_Ports to the Monitoring State
Receive CLS
Initialization Complete
Fibre Channel Topologies Arbitration

To send data over the loop a NL_Port must win
the loop through Arbitration.
Arbitration observes fairness:
- loop device AL_PA
- access bit
Once a NL_Port has won Arbitration it cannot
attempt to arbitrate until no other devices on the
loop are arbitrating.
Fibre Channel Topologies Arbitration

A NL_Port wanting to win the Loop sends a ARB(x)
Frame around the loop. If this comes back with its
own AL_PA it wins the Loop.
The NL_Port will set its access bit to 0. It now owns
the loop until it sends a ARB(f0) Frame without its
AL_PA.
The NL_Port cannot Arbitrate again until a idle
ARB(x) is received.
Fibre Channel Topologies AL Fault

Isolation
RX
TX
RX
Hub
TX
RX
TX
Causes a LIP to occur.
Is disruptive to Loop communication

Backups ( if running ) will FAIL.
Fibre Channel Topologies - Fabric

100 MB/sec
100 MB/sec
F
100 MB/sec
100 MB/sec
Switch
100 MB/sec
F
100 MB/sec
100 MB/sec
per port
4 node fabric
100 MB/sec
F
100 MB/sec
24 bit Address - 16 million addresses

A rich set of class services...Name Services, fabric login, port login,
registration
Login process queries connection to determine type of connection

A Fabric must contain at least 1 switch.
Switches function as the core of the Fabric they contain

And Provide the following functionality
- SNS ( Simple Name Server )
- Fabric Controller
- RSCN ( Registered State Change Notification )
- Zoning
- Simple Routing ( Shortest Path First )

FLOGI Fabric Login;
A process by which a node establishes a logical
connection to a fabric switch.
PLOGI Port Login;
A port-to-port login process by which initiators
establish sessions with targets.

Addressing Private Loop
Example of a Private Loop Address
42
Domain
Area
Port
42 is decimal = 2A in hex which is the AL_PA which can be converted

into a loop ID.
Private Loop only gets a 8 bit address

Addressing Public Loop
Example of a Public Loop Address
24 bit N_Port ID
1
Domain
22
Area
42
Port
1 = Domain # assigned to switch Note: Do not assign Domain # 8 to a

switch if HP-UX is in the environment.
22 = 22 - 16 = 6 which is the port it is plugged into on the switch.
42 is decimal = 2A in hex which is the AL_PA which can be converted
into a loop ID.

Addressing Fabric Devices
Example of a Fabric Address
24 bit N_Port ID
1
Domain
22
Area
0
Port
1 = Domain # assigned to switch Note: Do not assign Domain # 8 to a

switch if HP-UX is in the environment.
22 = 22 - 16 = 6 which is the port it is plugged into on the switch.
0 = Fabric Point to Point is not Loop therefore it will be zero.
Fibre Channel Topologies Brocade

Native Mode Addressing
Native Mode Addressing on Brocade Switches
XX 1Y ZZ
Where:
XX is a value between 0x1 and 0xEF
The 1 will always be there
Y is the port number ( 0-15 )
ZZ is the AL_PA for loop device or 00 for F_Port
Fibre Channel Topologies WWN

and PWWN
Each Node has a fixed 64-bit Node_Name WWN assigned by the
Manufacturer. The address can be considered analogous to a MAC address.
This guarantee's uniqueness within a large switched networked
Unlike the MAC address, the WWN is not used to transport frames across
the network
There is two WWNs assigned.
N_Port Node WWN
N_Port Port WWN
The N_Port Node WWN is assigned for the whole HBA.
The N_Port Port WWN will have a WWN assigned for each port on the
HBA.
The N_Port Port WWN is the WWN used when setting up Secure Manager
on an XP Storage Array.

Definitions Summary
FC Definitions
Fabric: A switching network that consists of one or more Switch elements. The function of the
Fabric is to receive frames from source Ports and to route them to destination Ports. Switch
elements can have F Ports, FL Ports, E Ports and QL (EPL) Ports.
Fabric Login Protocol: In the Fabric Login Protocol, the N Port interchanges sequences with the
Fabric to determine the service parameters for the operating environment. Parameters such as flow
control buffer credit, class of service and Port Identifiers are exchanged. Both N Ports and Public
NL Ports can participate in the Fabric Login.
Arbitrated Loop: A connection of 2 or more Nodes without the use of a Fabric. Loops can consist
of physical elements called Hubs. Hubs are a way to house the loop in a box and physically
connect to the Nodes in a point to point fashion (this is NOT the Point to Point FC connection).
Loops can be considered Private with connections involving Private NL Ports & QL(EPL) Ports
or Loops can be considered Public with connections involving Public NL ports & FL ports.
Arbitrated Loop Initialization Protocol: In Arbitrated Loop topology, the assignment of the 127
possible loop addresses to the different ports attached to the loop is carried out through the
transmission of a set of sequences around the loop, alternately collecting and broadcasting
mapping of addresses to Nodes.
LIP:. Loop Initialization Primitive. This is an ordered sequence that is sent around the loop to
begin the initialization process of assigning the AL PA for each node.
AL PA: Arbitrated Loop Physical Address. This address is assigned through the loop initialization
process and is what allows nodes to communicate with each other.

Troubleshooting
Module 4
Brocade Switches
Brocade Switches - Hardware

Common Hardware for the 2400/2800 Brocade Switches
i960RP-33Mhz Processor
16 Mbyte SDRAM
8 Mbyte ( 2 X 4 Mbyte ) Flash Memory for up to two banks
of stored Image; expandable to 16 Mbyte NOTE: the 2400
model has only 1 bank; max of 8 Mbyte.
3.3V & 5V power monitor
Integrated LCD display & four push-button module 2800
Model Only
10/100BaseT Ethernet Port
Bridged duel PCI Bus; Primary PCI for ASICS; Secondary
PCI for Ethernet
Duel Redundant Power supplies
Brocade Switches - Hardware

SilkWorm 2X00 Block Diagram
BOOT
PS0
PS1
Local BUS
RTC
Drv
12C BUS
FPGA
Sensors
LCD
I/O
Secondary PCI BUS
Intel
i960RP
CPU
33Mhz
SDRAM
UART
Flash
RS232
Primary PCI BUS
MAC
LAN
Loom
0
Loom
1
NOTE: Primary Bus NOT used for DATA
Loom
2
Loom
3
Only on 2800
Brocade Switches - Administration

There are different methods to Administrate the Brocade
Family of Switches;
2800
- Front LCD Panel
- Web Interface
- Telnet Interface
2400/2050/2100
- Serial Port
- Web Interface
- Telnet Interface
Brocade Switches Port Interfaces

Port Types and Definitions
- Connection Ports
- E_Port
- F_Port
- FL_Port
- Transitional Ports
- G_Port
- U_Port
SilkWorm Port Addressing
- SilkWorm 2000 Native Mode to FC Standard ( ANSI )
- SilkWorm 2000 Compatibility Mode Connection to Older
Switches
- SilkWorm 2000 Exact Match Mode Tachyon chip Mode
Brocade Switches E_Port

Expansion Port ( E_Port ) allows 2 switches to be
connected to create a Fabric

Supports Class F Service switch-to-switch Virtual Channel
Standard
Connections are called ISLs ( Inter-Switch Links )
E_Port Connectivity creates a cascaded or meshed Fabric
Supports eight Virtual Channels per ISL
Multiple E_Ports to one/many switches is supported
Brocade Switches E_Port Virtual

Channels
Eight Virtual Channels to improve performance
VC 0 for Link control Frames
VC 1 For Class 2 Acks and Link Control Frames
VC 2 For Loom 0
VC 3 For Loom 1
VC 4 For Loom 2
VC 5 For Loom 3
VC 6 For Broadcast traffic
VC 7 For Broadcast traffic
Switch
One ISL, 8 Virtual

Channels
Switch
Brocade Switches U_Port and

G_Port
Universal Port ( U_Port )
Can Operate in any of the port Interfaces defined
( G/E/F/FL )
Port mode automatically detected during port Initialization
Will Negotiate up to N_Port from NL_port if attached
device supports both.
General Port ( G_Port )
Transition Port from U_Port to E/F port
Port transition when deciding to go F or E port
Brocade Switches Port

Initialization
Start Mode Detection
One or more NL Ports
responds to LIPs
U_Port attempt
Loop Init
Late LIP or Loop Fail
Mode Known
FL_Port
U_Port attempt
Pt-to-Pt
Loopback detected
ELP Received
Mode Known
E_Port
LIP Timeout
Mode G_Port
Attempt E_Port
Link Init Succeeds
FLOGI Received
Mode Known
F_port
Brocade Switches Simple Name

Server
The Simple Name Server ( SNS ) is:
A directory service to translate various names, addresses and
attributes related to Fibre Channel objects.
- Objects defined in the SNS are:
- Fibre Channel Ports
- Fibre Channel nodes
Uses a simple request and respond model
Based on ANSI Standard FC-GS Simple Name Server
To the client/end user, a multi-switch fabric just looks like one switch!
Simple Name Server has a special reserved 24 bit address in the
Fabric - FFFFFC

Server
Name Server Characteristics
In multi switch environment no single point of failure.
Transparent distribution
- Each Distributed Name Server maintains and owns local
information and retrieves remote information from other
Name Servers
- Server-to-Server communication is transparent to the external
Name Server Client
Caching
- A distributed Name Server may cache remote information
for a period of time ( ~15 Minutes )

Server
Port Attributes
Port Identifier ( Native port 24 bit address ID )
Port Name ( WWN )
Class of Service ( 1,2 or 3 )
FC-4 Types ( IP/SCSI )
Port Type
Symbolic Port Name
Node Attributes
Node Name ( WWN )
IP Address
Initial Process Associator ( Not Used )
Symbolic Node Name

Server
Access Protocol
Based on FC-CT ( Fibre Channel Common Transport )
Request/ Respond Model
Three groups of requests
- Queries
- Registration
Response may be an ACCEPT or REJECT

Server
Registered State Change Notification ( RSCN )
A Initiator must request to receive RSCN frames.
Switch notifies all devices/initiators that something has changed in
the Fabric via a RSCN Frame.
Devices PLOGI into the SNS and reread the entries and compare
it with what they already have.
This is not destructive to traffic flow like a LIP.
Brocade Switches - QuickLoop

What is QuickLoop Mode?
A Hub alternative Allows switches to replace hubs in a
private loop environment.
QL is a Logical PLDA which complies with the FC-AL standard
( A max of 126 devices supported )
Allows hosts with drivers that are not Fabric-aware to communicate
with all devices connected to a QL.
Public devices that support an AL and are attached to QL ports are
treated as Private devices.
Other Public hosts can also communicate with devices attached to
QL via Translative mode.
Brocade Switches - QuickLoop

What Is QuickLoop Mode? ( cont. )
Each Looplet supports transfer rates of 100 MB/sec and
multiple devices can communicate simultaneously in different
looplets.
Maximum of two switches in one QL
Only ONE QL per Fabric
Can activate QL in two modes;
- Full Switch ( qlenable )
- Port by Port ( qlportenable )
Can mix F_Ports, FL_Ports and QL Ports on one switch.
Brocade Switches Port Definition

Summary
Port Definitions
N_Port: A port on a Host/Device that is attachable via a non-loop link to either an N
Port on another Host/device OR an F Port on a switch. N ports participate in Fabric
Login process but not in the Arbitrated Loop process.
Private NL_Port: A port on a Host/Device that is attachable, in an Loop topology, to
Private NL Ports, Public NL Ports, FL Ports or EPL Ports. It cannot connect to N
Ports or F Ports. Private NL Ports participate in the Arbitrated loop process but not
the Fabric Login process. One exception is that a Host Private NL port cannot hook to
an FL port on the Switch, it needs to hook to the EPL port on the Switch.
Public NL_Port: A port on a Host/Device that is attachable, in an Loop topology, to
Private NL Ports, Public NL Ports, FL Ports or EPL Ports. It cannot connect to N
Ports or F Ports. Public NL Ports participate in the Arbitrated loop process as well as
the Fabric Login process (if connected to a Fabric). A Public NL Port can act as a
Private NL Port.
F_Port: A Switch port to which a non-loop, point-to-point N Port can attach.
FL_Port:. A Switch port to which a Public NL Port can attach.
E Port: A Switch port that is able to communicate with other Switch ports. This is
used for cascading Switches into the switched network Fabric.
EPL Port or QL Port:. A Switch port to which a Private NL Port or Public NL port
can attach.
Brocade Switches - Zoning
2 fabric
zones
N
Divides fabric virtually

Zones may overlap
Not on LUN level

Zoning
Allows a finer partitioning of SANs
Used to set up barriers between different operating systems
To deploy logical fabric subsets
Flexibility to manage a SAN to meet specific user needs
Zones may be changed dynamically (may cause a LIP, refer to firmware ??)
Zones vary in size and shape
Zones members see only members in their zones, and may access only
each other
Zones may overlap ...

Why Zone ?
Unify Heterogeneous environments
Create Functional (i.e. test environments)
Security
Simplify Implementation
Does Zoning impact performance ?
Zoning is based on SNS (simple name server)
Zoning is not an ANSI standard ...issues
Zoning does not change the SNS protocol
A requestor does not have the capability to query members in other zones, only its on
Zone
If no zoning is in effect, all responses are based on all Fabric attached devices
An example :
2,12;2,14:10:00:00:60:69:00:00:8a
Switch 2; port 12 14;WWN

Troubleshooting
Module 5
Fibre Channel On
HP-UX
Fibre Channel On HP-UX

The Fibre Channel Environment on HP-UX is made
up of two parts
Mass Storage Driver ( fcms )
Fibre Channel Adapter Driver ( td, fcT1 )
Fibre Channel On HP-UX - Adapters

HSC Adapters
A3404A
PCI Adapters
A3740A *
A3591A/B
A3636A*
A6684A
A5158A
A6795A
A6685A
* Discontinued

HSC/EISA Adapters
Uses old Tachyon Chip V 2.1

Does NOT support Fabric
Private AL only
Uses the fcT1 driver
Supported 10.20, 11.00
Fibre Channel on HP-UX HSC TL

Fibre Channel Adapters
A6684A HSC TL Adapter
Uses new TL Chip V 3.0
Full Fabric Support
Supports most D/R Class
Systems.
Uses the td Driver
Support 10.20, 11.00, 11i
Fibre Channel on HP-UX HSC TL

A6685A HSC TL Adapter
Full Fabric Support
Supports most K Class
Systems.
Uses the td Driver
Support 10.20, 11.00, 11i
Fibre Channel On HP-UX PCI

Uses old Tachyon Chip V 2.1
Does NOT support Fabric
Private AL only
Supported on V class only
Uses the fcT1 driver
Supported 11.00 only
Card is discontinued
Fibre Channel On HP-UX PCI TL

A5158A PCI TL Adapter
Full Fabric Support
Supports V,N, L and A Class
Systems.
Uses the td Driver
Support 11.00, 11i
1 Gigabit Not Upgradable
Complete boot, dump, and swap support
Fibre Channel On HP-UX PCI XL2

A6795A PCI XL2 Adapter
Uses new Tachyon XL2 Chip

Full Fabric Support
Supports Systems.
Support 11.00, 11i ( 4 bit only )
2 Gigabit Auto Sensing.
Supports N, L, A, Superdome
Uses the td driver ( 11.00.09 or greater )
Complete boot, dump, and swap support
Fibre Channel On HP-UX Fibre

Channel Adapters
For More information on installation and details on
HP Fibre Channel Cards see
http://techsolutions.hp.com/fc/A3636-96011.pdf
Fibre Channel On HP-UX Drivers

fcT1
Old fcT1 drivers
used on Tachyon cards prior to the TL models.
fcT1 Fibre Channel Storage Adapter Driver
fcT1_cntl - Fibre Channel Storage Controller Driver
Ioscan(1M) output fc
0 8/8
lan
2 8/8.5
fcT1
CLAIMED
fcT1_cntl CLAIMED
INTERFACE
INTERFACE
HP Fibre Channel Mass Storage Adapter

HP Fibre Channel Mass Storage Cntl
/dev/fcms1
Fibre Channel On HP-UX Drivers

td
TL td driver
- used on Tachyon TL and XL2 Fibre Cards.
td Tachyon TL/TS Fibre Channel Mass Storage Adapter
Ioscan(1M) output ba
2 0/4
fc
0 0/4/0/0
Storage Adapter
lba
CLAIMED
BUS_NEXUS
Local PCI Bus Adapter (782)
td
CLAIMED
INTERFACE
HP Tachyon TL/TS Fibre Channel Mass
/dev/td0
Fibre Channel On HP-UX Fibre

Drivers
General Fibre Drivers
fcms - Fibre Channel Mass Storage Driver *
fcp Fibre Channel ( FCP ) Domain/Protocol Driver
fcp_cdio Fibre Channel Protocol Context Dependant
I/O Module Driver *
Fibre Device Drivers
fcparray Fibre Channel Array Driver
fcpdev Fibre Channel General Device Driver
fcpmux Fibre Channel SCSI MUX Driver
* Driver not seen in ioscan
Fibre Channel On HP-UX Driver

Stacking
Logically the drivers stack on top of each other ( some
are not seen in ioscan ).
At the top is the Card driver, followed by a protocol
driver then a device driver.
fc
1 0/6/2/0
td
Adapter
/dev/td1
fcp
1 0/6/2/0.1
CLAIMED
fcp
ext_bus 12 0/6/2/0.1.20.23.0
Interface
INTERFACE
CLAIMED
fcpmux
target
47 0/6/2/0.1.20.23.0.4
tgt
tape
3 0/6/2/0.1.20.23.0.4.0 stape
HP Tachyon TL/TS Fibre Channel Mass Storage
INTERFACE
CLAIMED
CLAIMED
CLAIMED
FCP Domain
INTERFACE
HPA3308 FCP-SCSI MUX
DEVICE
DEVICE
Quantum DLT4000
Fibre Channel On HP-UX Driver

Stacking
More ioscan Examples:
fc
1 8/12
lan
3 8/12.5
fcT1
CLAIMED
INTERFACE
fcT1_cntl CLAIMED
HP Fibre Channel Mass Storage Adapter
INTERFACE
HP Fibre Channel Mass Storage Cntl
/dev/fcms3
fcp
ext_bus
1 8/12.8
fcp
2 8/12.8.0.255.0
CLAIMED
fcpdev
target
4 8/12.8.0.255.0.0
tgt
disk
3 8/12.8.0.255.0.0.0 sdisk
INTERFACE
CLAIMED
CLAIMED
CLAIMED
FCP Protocol Adapter
INTERFACE
FCP Device Interface
DEVICE
DEVICE
SEAGATE ST39102FC

Addressing Direct Connect
Interpreting the Device Hardware Path
( Direct Connect )
Logical Unit or Volume Set Addressing
8/12.8.0.255.0.1.0
AdapterProtocol_Type
Protocol_TypeArea
AreaPort
PortBus
BusTarget
TargetLun
Lun
Adapter
* Protocol_Type always 8 for Mass Storage

* Area
- always 0 for Direct Connect
* Port
- always 255 for Direct Connect

Addressing Private Loop

Addressing Public Loop

Addressing Fabric Attach
Fibre Channel On HP-UX - fcmsutil

HP-UX supplied tool to display the following information:
- Configuration data
- SAN Configuration from the Hosts Point of View
- Statistics
Good Troubleshooting tool on HP-UX Fibre Issues
Can function with all HP-UX Fibre Channel Adapters
Binary found under /opt/fcms/bin/fcmsutil
Fibre Channel On HP-UX TL Card

Initialization
A5158A uses Auto-Discovery of Topology
Upon completion of loop initialization or link initialization, the Nx_port is aware of its mode
of link operation (loop or point to point)
Upon completion of FLOGI, the Nx_port driver is aware of the presence or absence of
Fabric.
If a Fabric is not present and:
- Current mode of operation is Loop, then topology is Private Loop.
(fcmsutil shows topology as IN_LOOP)
If a Fabric is present and:
- Current mode of operation is Loop, then topology is Public Loop.
(fcmsutil shows topology as PUBLIC_LOOP currently not supported )
- Current mode of operation is Point to Point, then topology is Direct Fabric Attach.
(fcmsutil shows topology as PTTOPT_Fabric)

Addressing examples:
1. 0/4/0/0.1.20.23.0.4.0
2. 0/4/0/0.1.20.255.1.7.0
3. 0/4/0/0.1.25.0.0.2.0
4. 8/4.8.0.255.0.5.0
5. 0/2/0/0.4.18.0.3.0
6. 0/8/0/0.8.0.116.2.1.1

Troubleshooting
Module 6
Troubleshooting
Fibre Channel On
HP-UX
Troubleshooting Fibre Channel On

HP-UX
Discuss tools to help Troubleshoot SAN/FC
What information to collect.

Analyzing information to create action plan.
Patching and Driver Versions.

HP-UX - fcmsutil
The fcmsutil command is a diagnostic tool to be used for
the TACHYON A3740A, A3591A, A3404A, A3636A

and TACHYON TL A5158A/A6795A Fibre Channel
Host Bus Adapters.
This command provides the ability to perform Fibre
Channel Test and Echo functionality, read the card's
registers, etc.
This command requires the use of a device file to
indicate the interface over which the requested command
needs to be performed e.g /dev/fcms1, /dev/td0

HP-UX - fcmsutil
Using fcmsutil with only the device file provides the following information # fcmsutil /dev/fcms3
Local N_Port_ID is = 0x000008

N_Port Node World Wide Name = 0x10000060B03E56F2
N_Port Port World Wide Name = 0x10000060B03E56F2
Topology = IN_LOOP
Speed = 1062500000 (bps)
HPA of card = 0xFBF4C000
EIM of card = 0xFFFA2009
Driver state = READY
Number of EDB's in use = 0
Number of OIB's in use = 0
Number of Active Outbound Exchanges = 1
Number of Active Login Sessions = 2

HP-UX - fcmsutil
# fcmsutil /dev/td0
Vendor ID is = 0x00103c
Device ID is = 0x001028
TL Chip Revision No is = 2.3
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x000006
Topology = PTTOPT_FABRIC
Local N_Port_id is = 0x011a00
N_Port Node World Wide Name = 0x50060b000009f8f5
N_Port Port World Wide Name = 0x50060b000009f8f4
Driver state = ONLINE
Hardware Path is = 0/4/0/0
Number of Assisted IOs = 8168
Number of Active Login Sessions = 0

HP-UX - fcmsutil
The following are the driver states:
DEAD. The driver couldn't come up due to some problem. Check the connectivity with the
loop/loopback hood.
LOOPBACK_STATE . The host bus adapter is in the loop back test phase.
OFFLINE. The host bus adapter card is not participating on the loop.
READY/ONLINE. The driver is up and functional.
RESETTING. The host bus adapter card is being reset.
All other states are only "transient" and should not continue for long. If the "transient" state
persists, there might be a problem in the hardware connectivity or configuration.

HP-UX - fcmsutil
Getting stats from the T1/td driver.
Usage: fcmsutil <device file> stat
Both the T1 and td driver output for this command is
different!
Only certain fields are important for us.

HP-UX - fcmsutil
Important fields to look at in stat output.
# fcmsutil /dev/td0 stat | more
Wed Jan 23 11:57:06 2002
Channel Statistics
Statistics From Link Status Registers ...

Loss of signal
38
Bad Rx Char
Loss of Sync
13
Link Fail
Received EOFa
Discarded Frame
Bad CRC
Protocol Error
242
73
185
0
* Same fields for the T1 driver.


HP-UX STM logtool
Most Fibre Critical Events are logged to the system
Diagnostics Log.
Access logtool from cstm, mstm or xstm
Always try and collect the formatted log to help in
analyzing Fibre connection issues.
Troubleshooting Fibre channel On

HP-UX STM Logtool Example
Event # : 1004
Severity : Critical
Label
: FCMS Adapter is offline
Description of Error:
An ioscan failed because the Fibre Channel

Mass Storage Adapter is OFFLINE.
Probable Cause / Recommended Action:
The Fibre Channel Mass Storage Adapter is not connected in a supported topology.

HP-UX - EMS
EMS ( Event Monitoring System ) On 11.X systems monitor FC
cards and devices.
Logs entries into syslog.log when events occur
Events are also logged to logtool depending on the event type
Example of EMS Message
Jan 22 14:49:42 lisa EMS [2074]: ------ EMS Event Notification ------ Value: "
CRITICAL (5)" for Resource: "/adapters/events/TL_adapter/8_8.8" (Threshold:
>= " 3") Execute the following command to obtain event details: /opt/resmo
n/bin/resdata -R 135921666 -r /adapters/events/TL_adapter/8_8.8 -n 135921665 -a

HP-UX syslog.log/dmesg
Major Sources of Information for Troubleshooting
Fibre Channel
/var/adm/syslog/syslog.log
- EMS entires are logged here
- SCSI Driver logs Errors here
- LVM Driver logs Errors here
- Filesystem Errors are logged here
dmesg(1m) ( Kernel Message Buffer )
- SCSI Driver logs Errors here
- LVM Driver logs Errors here
- Filesystem Errors are logged here

HP-UX syslog.log/dmesg
Examples of syslog.log entries Dec 30 00:20:28 nuh507 vmunix: LVM: vg[6]: pvnum=9 (dev_t=0x1f052500) is POWERFAILED
Dec 30 00:20:28 nuh507 vmunix: LVM: vg[5]: pvnum=7 (dev_t=0x1f051100) is POWERFAILED
Dec 30 00:20:28 nuh507 vmunix: LVM: PV 7 has been returned to vg[5].
Dec 30 00:20:28 nuh507 vmunix: lv_readvgdats: Could not read VGDA 1 header & tra
iler from disk H/W path 8/8.8.0.0.0.2.5 (error = 5)
Dec 30 00:20:28 nuh507 vmunix: LVM: PV 9 has been returned to vg[6].

HP-UX Kernel Debugging
Built in Debugging in T1 and td driver
Outputs to /var/adm/syslog/syslog.log
Very Verbose! Possible performance issues when enabled
Only to be used under instructions from the SCC or WTEC
Do NOT have the customer enable Debugging without supervision
Only needed if all other Troubleshooting Methods have failed.

Enabling SCSI and FC Debugging on the T1 Driver
#echo "scsi_log_mask/W-1;fcp_log_mask/W-1;fcT1_dbg_level/W8\
> | adb w /stand/vmunix /dev/kmem
Error from elf64_getehdr(application core file)
Not an Elf file: No Elf header
scsi_log_mask: 1F238B10
=
-1
fcp_log_mask: 0
=
-1
fcT1_dbg_level: 1
=
8
Disabling SCSI and FC Debugging on the T1 Driver

#echo "scsi_log_mask/W0;fcp_log_mask/W0;fcT1_dbg_level/W1\
scsi_log_mask: -1
=
0
Note: does not survive a Reboot.
fcp_log_mask: -1
=
0
fcT1_dbg_level: 8
=
1

Enabling SCSI and FC Debugging on the td Driver
#echo "scsi_log_mask/W-1;fcp_log_mask/W-1;td_dbg_level/W6\
scsi_log_mask: 1F238B10
=
-1
fcp_log_mask: 0
=
-1
td_dbg_level: 1
=
8
Disabling SCSI and FC Debugging on the td Driver

#echo "scsi_log_mask/W0;fcp_log_mask/W0;td_dbg_level/W1\
scsi_log_mask: -1
=
0
fcp_log_mask: -1
=
0
td_dbg_level: 6
=
1

Log Levels for the td driver
# define LVL_ALERT 1 /* action must be taken immediately */
# define LVL_CRIT 2 /* critical conditions */
# define LVL_ERR 3
/* error conditions */
# define LVL_WARN 4 /* warning conditions */
# define LVL_NOTE 5 /* normal but may need special handling */
# define LVL_INFO 6 /* informational */

HP-UX - PVLinks
PVLinks is LVMs way of using a backup link
PVLinks must be manually configured on HP-UX Hosts
Not a Load balancing product.
Can cause the Flip Flop effect in some Failures
Use pvchange(1m) to change Failover mode
Highly recommended to set pvchange S n on all PVs to
prevent Flip Flop.

HP-UX Load Balancing
Although SANs run a Gigabit speeds there is still a need to Load
Balance access to Disks over the SAN. There is really two ways
to do this:
1. Manual on HP-UX using LVM PVLinks or VxVM
2. Automatically using software such as EMC PowerPath ,
AutoPath XP, or AutoPath VA.
Some Hardware such as the VA7x00 has specific requirements to
get best performance out of them.
Use of Disk stripping can increase Performance over a SAN.

HP-UX SAN Backup
Tape Devices can exist in the same SAN as disk
In SAN requires device locking such as Omniback

Best to run Tape Libraries in Fabric not AL mode.
Can use same FC Adapter for tape and Disk
Recommended SAN Zoning for Tape Libraries.
Keep Firmware Updated on Libraries and MUX devices
See SPOCK for latest SAN Backup Support Matrix:
http://turbo.rose.hp.com/spock/index.shtml

HP-UX Patching 10.20
Latest patches for FCMS and Adapters for 10.20
PHSS_23581 Fibre Channel Mass Storage Driver Patch
- Required for Fabric Login on 10.20
- Required FCMS product installed on System
PHNE_21923 HSC Fibre Channel B.10.20.2[023] cumul. Patch
- Requires FCMS Product installed on System
- Requires the HSC Fibre Channel Adapter Driver installed
Note: PHSS_23581 depends on PHKL_16751. NEVER reinstall PHKL_16751!

* Current as of 01/02/02

HP-UX Patching 11.00
Latest patches for FCMS and Adapters for 11.00
- FCMS product installed by default on 11.00
- Depends on PHKL_18543. Never reinstall PHKL_18543!
PHSS_23996 Tachyon TL Fibre Channel Driver Patch
- Requires the TL/TS Adapter Driver Installed on system
- Driver version is 11.00.08
- Require a Minimum of 11.00.03 for Fabric Login
Patches depend on PHKL_18543. NEVER reinstall PHKL_18543!
* Current as of 01/02/02

HP-UX Patching 11i
Latest patches for FCMS and Adapters for 11i
- FCMS product installed by default on 11i
- No patch dependencies.
PHSS_24121 Tachyon TL Fibre Channel Driver Patch
- Requires the TL/TS Adapter Driver Installed on system
- Driver version is 11.11.06

HP-UX WTEC Tools
WTEC do supply some other Troubleshooting Tools
http://essd.boi.hp.com/fctools/CurrentTools/index.htm
Fct_mon Fibre Channel Monitor
http://essd.boi.hp.com/fctools/CurrentTools/fct_mon-v2.htm
Fct_Util Collecting and Interpreting Fibre Channel Status
http://essd.boi.hp.com/fctools/CurrentTools/fct_util.htm
The best tool to Troubleshoot Fibre Issues is still fcmsutil!

HP-UX AL Issues
When Dealing with AL issues on HP-UX collect the following
information:
- Symptoms the customer is seeing.
- syslog.log and dmesg from the affected system
- STM logtool formatted output.
- fcmsutil output from Adapters in Loop including stat output
- A diagram of the Loop Environment including AL_PAs.
- Patching and Driver versions ( from td/T1 drivers )

HP-UX AL Fault Isolation
Troubleshooting AL with HUB involved.
Server1
Disk1
2
FC HUB
Server2
Disk2
The Loop flow follows the Hub port numbers


HP-UX AL Issues
My Loop wont Initialize, why is this?
- Duplicate Hard AL_PAs in Loop
- Faulty Cabling causing continuous LIPs
- Device adapter not configured for AL
- Non compatible FC Cards ( addressing )

HP-UX AL Issues
What are Sync, Elastic Store and LIP Storms?
- A Sync storm is when a Adapter fails to keep in Sync with
the upstream Interface. A Storm happens when a certain number of
these occur in a minute.
- The Elastic Store is a buffer used by the Tachyon cards to
buffer incoming and outgoing frames from the Loop. If this fills up
and is not emptied a Elastic Store error occurs. A storm is when a
number of these occur withina certain time span.
- A LIP Storm is when the threshold for LIPs per minute is
exceeded on the Loop.

HP-UX Fabric Issues
When Dealing with Fabric issues on HP-UX collect the following
information:
- Symptoms the customer is seeing.
- syslog.log and dmesg from the affected system
- STM logtool formatted output.
- fcmsutil output from Adapters in Loop including stat output
- A diagram of the Fabric Environment.
- Patching and Driver versions ( from td/T1 drivers )

HP-UX Fabric Issues
When Dealing with Fabric issues on HP-UX collect the following
information ( Continued ) :
- SNS details from Switch nsshow *
- Port information from switch portshow *
- Error Logs from FC switch errShow *
These commands will be discuss later in the Brocade Switch
Troubleshooting Section.
* Can be collected by using the supportShow Command


HP-UX Fabric One step away
Like a Network, hosts could be one or more hops from the source
of the problem. So when looking a Fabric issues you must look at the
entire Fabric, not just the host.
Disk1
Server1
FC Switch
Server2
Disk2
How does Server1 and Server2 react to Disk2 problems?

HP-UX Fabric Issues
Using fcmsutil for testing and checking config.
To query the SNS of the local switch:
# fcmsutil /dev/td0 ns_query_ports
The following is the list of nport ids:
1. 11000
2. 11101
3. 11417
4. 11500
Reports
Reports Nport
Nport Ids
Ids of
of all
all hosts/devices
hosts/devices
5. 11602
that
that this
this host
host is
is allowed
allowed to
to see.
see.
6. 11700
7. 11900
8. 11a00
9. 11d00
Note: N_port Ids used with fcmsutil command require
0x at the beginning as these are Hex addresses.

HP-UX Fabric Issues
Using fcmsutil for testing and checking config ( cont ).
To Send a test frame to a device:
Usage fcmsutil <device> test <frame size> <number>
Example send a 64 byte frame once to 011900:
# fcmsutil /dev/td0 test 0x011900 64 1
Sent a Test frame of size 64 bytes to nport_id 0x011900
Useful when testing connectivity issues.

HP-UX Fabric Issues
Using fcmsutil for testing and checking config ( cont ).
Other useful fcmsutil commands:
Get Fabric Login Parameters details from N_Port or
all N_ports in the fabric
Usage fcmsutil <device> get remote {all | <remote N_Portid>}
Replace WWN details of device after WWN Change. N_Port has
not changed.
Usage - fcmsutil device_file replace_dsk remote-N-Port-ID

HP-UX Final Words.
Collect all the information before replacing hardware.
Understanding the problem is the key to solving it!

Understanding SAN issues is understanding all the
environment from the Hosts, switches to the devices.
Its not always a easy task and is very time consuming.
Support and configuration details change all the time!
Dont Blame the Transport!

Troubleshooting
Module 7
Troubleshooting
Brocade Switches
Troubleshooting Brocade Switches

Basic Switch Troubleshooting Tips
Requires switch to be network accessible

The Web interface is not overly good for troubleshooting
Make sure you use a Terminal Emulator that can capture output.
Terminal setup is vt100. You may need to setup the Erase key.

Basic Switch Troubleshooting Tips
Some errors are displayed on the LCD display on 2800 models

Check LED status on switch Orange Light means warning.
Default IP address on switch from factory is 10.77.77.77
Subnet 255.0.0.0.
Supported Firmware versions v2.1.9f and a2.4.1

Telnet Commands
Using Telnet we can collect a lot of useful information
when troubleshooting Switch issues. We will look at the
following commands:
version
switchshow
Nsshow/nsAllshow
Errshow
portShow
supportShow
Troubleshooting Brocade Switches version

This command displays firmware version information and build dates.
The lines of the display show:
Kernel: the version of the switch's Kernel Operating System;
Fabric OS: the version of the switch's Fabric Operating System;
Made on:
the build date of the firmware running in the switch;
Flash: the build date of the firmware stored in flash proms;

BootProm:
the build date of the firmware stored in the boot prom.

brocade1:admin> version
Kernel:
5.3.1
Fabric OS: a2.4.1

Made on:
Flash:
Tue Mar 20 17:45:07 PST 2001

Tue Mar 20 17:46:25 PST 2001
BootProm: Thu Jun 17 15:20:39 PDT 1999

brocade1:admin>
Troubleshooting Brocade Switches switchshow

This command displays switch and port status information. Some information
varies with the switch model, e.g. number of ports, and Domain ID values.
brocade1:admin> switchshow
switchName:
switchType:
brocade1
2.4
switchState: Online
switchRole:
Principal
switchDomain: 1
switchId:
switchWwn:
fffc01
10:00:00:60:69:10:62:ac
switchBeacon: OFF
port 0: sw Online
F-Port 50:06:0b:00:00:09:f9:04
port 1: sw Online
L-Port 1 public
port 3: sw Online
L-Port 1 private, 7 phantom
port 4: sw Online
F-Port 50:06:0b:00:00:09:f9:26

nsshow/nsAllShow
Nsshow - This command displays local Name Server
information, which includes information about devices
connected to this switch, and cached information about devices
connected to other switches in the Fabric.
NsAllShow - This command displays the 24-bit Fibre Channel

addresses of all devices in all switches in the Fabric.

nsshow/nsAllShow
brocade1:admin> nsshow
The Local Name Server has 3 entries {
Type Pid
N
COS
011000;
PortName
NodeName
TTL(sec)
3;50:06:0b:00:00:09:f9:04;50:06:0b:00:00:09:f9:05; na
FC4s: FCP
Fabric Port Name: 20:00:00:60:69:10:62:ac
NL 011101;
3;50:06:0b:00:00:02:27:c2;50:06:0b:00:00:02:27:c3; na
FC4s: FCP
N
011900;
3;50:00:60:e8:01:8a:55:01;50:00:60:e8:01:8a:55:01; na
FC4s: FCP [HP
OPEN-XP256
5248]

}
Troubleshooting Brocade Switches errshow

This command displays the error log, prompting the user to
type return between each log entry. It is identical to errDump,
except that errDump displays all entries without page breaks.
Each entry in the log has the same format:
Error Number
-----------taskId (taskName): Time Stamp (count)
Error Type, Error Level, Error Message
Diag Err#
Troubleshooting Brocade Switches portErrShow

This command displays an error summary for all ports. One output line is displayed per port,
and shows error counters in ones, thousands (the number is followed by 'k'), or millions (the
number is followed by 'm').
brocade1:admin> porterrshow
frames enc crc too too bad enc disc link loss loss frjt fbsy
tx rx in err shrt long eof out c3 fail sync sig
--------------------------------------------------------------------0: 234k 30k 0
0 13
3 0
1: 37k 529
1 0
2: 96k 27k 0
0 0
3: 149k 12k 0
0 16
0 0
4: 177k 1.6k 0
0 0
5: 97k 12k 0
0 0
Troubleshooting Brocade Switches portShow

This command displays port status information.
Syntax portShow <port number>
brocade1:admin> portshow 1
portFlags: 0x23806b
PRESENT ACTIVE F_PORT L_PORT U_PORT LOGIN NOELP LED ACCT
portType: 3.1
portState: 1 Online
portPhys: 6 In_Sync
portScn: 6 F_Port
portRegs: 0x80020000
portData: 0x10f5eab0
portId:
011100
portWwn: 20:01:00:60:69:10:62:ac
Troubleshooting Brocade Switches supportshow

This command prints the switch's information for debugging purposes by
executing the following commands in the order as follows:
1) version
7) errDump
14) portRegShow
21) nsAllShow
2) uptime
8) switchShow
15) portRouteShow
22) cfgShow
3) tempShow
9) portFlagsShow
16) fabricShow
23) configShow
4) psShow
10) portErrShow
17) topologyShow
24) faultShow
5) licenseShow
11) mqShow
18) qlShow
25) traceShow
6) diagShow
12) portSemShow
19) faShow
26) portLogDump
13) portShow
20) nsShow
* Dumps to screen requires to be captured by Terminal Software.

Port Auto Disable/Enable
Ports are disabled on the switch when issues occur that
are greater than set thresholds.

Issues include
- Sync Failures
- To Many LIPs
- To Many FLOGI attempts
- Link Failures
Ports disabled are ported in the errShow output.

Private Devices without QL
A Private Loop device cannot be connected to a Switch
without QuickLoop Mode enabled on the port.
If a Private Device is connected to a switch it will not
complete port Initialization and remain as a G_Port.
Enable QL on the Port and the device will login to the
Fabric.
Remember that with QL Public devices can see Private,
but Private cannot see Public!
Troubleshooting Brocade Switches Zoning

Complex Zoning can be hard to troubleshoot
Draw a Diagram how things are meant to be configured
use fcmsutil to see what the HP-UX hosts can see
from the SNS.
Remember zoning is from the SNS!
If using WWN check for errors!
If all else fails disable Zoning and start again!

Performance Monitoring
To monitor the throughput and performance of the switch/SAN
you can use either the Web or Telnet Interface.
Web Open the Switch Management Application and click
on the Perf Icon. This will Graphically show you throughput
on each port including Transmitted and Received Frames and
CRC errors.
Telnet use the portPerfShow command. For each port it will
show the throughput.By default the interval is 1 second.
But this can be changed.

You can also get some idea of possible performance issues from
each port. Most important is the BB_Credit information. Each
interface is given a certain number of buffers to buffer the
incoming/outgoing frames. If the port is not clearing these
buffers quick enough it could be due to a number of reasons:
- HBA does not have enough Buffers to be able to receive 1GBs Traffic
- Disc may not be able to Handle 1 Gbits/Sec data coming at it
- May be congestion in the fabric if not enough ISLs are used to
accommodate the traffic being sent.
BB_Credit Zero is used by the HBA to tell the switch to stop sending frames.

Example of a possible BB_Credit issue brocade1:admin> portstatsshow 13
stat_wtx
65 811125
4-byte words transmitted
stat_wrx
651517215
4-byte words received
stat_ftx
1976542811
Frames transmitted
stat_frx
1903265007
Frames received
stat_c2_frx
Class 2 frames received
stat_c3_frx
1265007
Class 3 frames received
stat_lc_rx
Link control frames received
stat_mc_rx
Multicast frames received
stat_mc_to
Multicast timeouts
stat_mc_tx
Multicast frames transmitted
tim_rdy_pri
Time R_RDY high priority
tim_txcrd_z 479865431
Time BB_credit zero
Possible Issue

FLOGI/PLOGI Decoding
When troubleshooting switches/hosts login issues it is
useful to understand the FLOGI/PLOGI login frames.

A host will first FLOGI ( Fabric Login to the switch Fabric
Controller
After a FLOGI it will PLOGI ( Port Login )into the SNS
server.
If you are on the telnet session on the switch you will see
these happen.

Example of a FLOGI Sequence
Jan 23 20:54:02.599 port 13: FLOGI 0x011d00 cos=0x8

bb_credit=4 df_size=1024 cf=0x0
Things to note in the above
Port Port the FLOGI was detected on.
FLOGI 0x011d00 24 bit address assigned to the host.
Bb_credit number of credits assigned to this port as per the request
Of the HBA.

Example of a PLOGI Sequence
Jan 23 20:54:02.633 port 13: PLOGI s_id=0x11d00

d_id=0xfffffc cos=0x8 df_size=1024
Things to note in the above
Port Port the FLOGI was detected on.
PLOGI s_id 0x011d00 24 bit address of source.
D_id destination address in this case the SNS

Config Upload/Download
Before doing any major changes or Firmware updates it is highly
recommended that the configuration of the switch is saved in case
of problems.
Requires Network connectivity to switch.
Requires either RSH or FTP access
ConfigUpload Save switch config to a server
brocade1:admin> configupload
Server Name or IP Address [host]: 15.30.130.46
User Name [user]: stuartm
File Name [config.txt]: config2.txt
Protocol (RSHD or FTP) [rshd]: FTP
Password:
upload complete

Config Upload/Download
ConfigDownload Load switch config from a server
brocade1:admin> switchdisable
brocade1:admin> configdownload
File Name [config.txt]: config2.txt
Password:
Committing configuration...done.
Updating flash ...
* Can only be performed on a

Switch that is Offline!
download complete
brocade1:admin> switchenable

Firmware Updates
Switches Firmware can be updated Online.
After uploading new Firmware the Switch will need to be
rebooted to have the new Firmware take effect.
Firmware can be reverted back to Earlier Version.
Beware 2.1.3 Firmware Upgrade instructions postdisable
Require Network connectivity to switch to update Firmware
Require FTP or RSH to access the Firmware for upload.

Firmware Updates
firmwareDownload
Download firmware into switch
brocade1:admin> firmwaredownload
File Name [/usr/switch/firmware]: ./a2.4.1
Password:
60464+2248+128404, csum 6c14
loading to ram .......................
writing flash 0 .......................
writing flash 1 .......................
download complete

Hardware Diags
Most Hardware errors will be reported in the errshow log
There are two types of Hardware Diags on Brocade Switches
- Online Tests
- Offline Tests
Most Offline tests are done in Post Testing when Switch is rebooted
or Power cycled.
Most problems can be solved without using these tests.

Hardware Diags
.
OFFLINE TESTS
ONLINE & OFFLINE TESTS
portRegTest
centralMemoryTest
cmiTest
sramRetentionTest
cmemRetentionTest
camTest
portLoopbackTest
spinSilk
ramTest
crossPortTest

Troubleshooting
Module 8
Further Information
and Web Links
Further Information and Web Links

Books
Designing Storage Area Networks Tom Clark
Addison-Wesley ISBN 0-201-61584-3
Building Storage Area Networks Marc Farley
Osborne ISBN 0-07-212050-9
Fibre Channel connection to the Future
Fibre Channel Association J2389-96001

Web ( Internet )
www.t11.org - Fibre Channel Standards Web Page
www.fibrechannel.com - Fibre Channel Association
http://www.hp.com/products1/storage/san/index.html
- Hewlett Packard SAN Products

Web ( Intranet )
http://hpsu.rose.hp.com/hw_prods/san_infra/san_infra.htm
-Storage Uni. SAN Course notes
http://downy.cup.hp.com/internals/dart48/111classgloss.html
- Glossary of TL/TS and fcmsutil output
http://techcom.cup.hp.com/dir_fcms/supinfo_index.htm
- Fibre Channel Adapter Information

Fundamentals

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Fundamentals

Uploaded by

Copyright:

Available Formats

SAN and Fibre Channel

Storage Competency Centre SEA

SAN and Fibre Channel

Module 2 - Fiber Channel Protocol

Module 3 Fibre Channel Topologies

Module 4 Brocade Switches

Module 5 - Fibre Channel on HP-UX

Storage Competency Centre SEA

SAN and Fibre Channel

Module 7 Troubleshooting Brocade

Module 8 Further Information and Web

Storage Competency Centre SEA

SAN and Fibre Channel

Storage Competency Centre SEA

Storage Area Network:

Consists of Servers, Switches, Hubs, Bridges, & Storage Devices.

The current standard is Fibre Channel. Fibre Channel is the

SANs represent a third type of network, behind LANs and

data paths to disks (LUNS)

Storage Competency Centre SEA

Flexible Use of Storage Devices

reconfiguration with centralized management tools

- Same storage device may be accessed by multiple servers

Storage Competency Centre SEA

Storage Competency Centre SEA

Operating System & Driver

FC Hub & Switch

Storage Competency Centre SEA

SAN and Fibre Channel

Storage Competency Centre SEA

Fibre Channel Protocol

What is Fibre Channel?

Storage Competency Centre SEA

Fibre Channel Protocol Mass Storage

Fibre Channel for Mass Storage

Storage Competency Centre SEA

Fibre Channel Protocol Advantages

Advantages of Fibre Channel for Mass Storage

Storage Competency Centre SEA

Fibre Channel Protocol Advantages

Advantages of Fibre Channel for Mass Storage

Storage Competency Centre SEA

Fibre Channel Protocol Media

Storage Competency Centre SEA

Fibre Channel Protocol

Storage Competency Centre SEA

Fibre Channel Protocol Functional Levels

Channel Functional Levels

Storage Competency Centre SEA

Fibre Channel Protocol FC Levels

Storage Competency Centre SEA

Fibre Channel Protocol FC Level Placement

Storage Competency Centre SEA

Fibre Channel Protocol FC-0 Components

Storage Competency Centre SEA

Fibre Channel Protocol FC-0 Connections

Storage Competency Centre SEA

Fibre Channel Protocol Optic Cable

Fiber is immune to interference, so it is relatively secure from eavesdropping.

Storage Competency Centre SEA

Fibre Channel Protocol Fiber Internal Reflection

Storage Competency Centre SEA

Fibre Channel Protocol Microbend Loss

Maximum bend radius is 3 cm.