01 SAN Overview

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.
The objectives for this module are shown here. Please take a moment to read them.
SAN Overview - 1
The objectives for this lesson are shown here. Please take a moment to read them.
SAN Overview - 2
Fibre Channel (FCP) is a serial data transfer interface that operates over copper wire and/or optical fiber at data rates up to 8 Gbps (gigabits per second) and up to 10 Gbps when used as ISL (E_Ports) on supported switches. Networking and I/O protocols (such as SCSI commands) are mapped to Fibre Channel constructs, and then encapsulated and transported within Fibre Channel frames. This process allows high-speed transfer of multiple protocols over the same physical interface. Fibre Channel systems are assembled from familiar types of components: adapters, hubs, switches and storage devices. Host bus adapters (HBAs) are installed in computers and servers in the same manner as a SCSI host bus adapter or a network interface card (NIC). Hubs link individual elements together to form a shared bandwidth loop. Fibre Channel switches provide full bandwidth connections for highly scalable systems without a practical limit to the number of connections supported (16 million addresses are possible). The word fiber indicates the physical media. The word fibre indicates the Fibre Channel protocol and standards.
SAN Overview - 3
The Fibre Channel standards define a layered protocol: ULP (Upper level Protocol) is not actually part of Fibre Channel FC-4 level of Fibre Channel is designed to hand off to another protocol such as SCSI. Fundamentally, the commands at FC-4 for SCSI allow SCSI initiators and targets to communicate over Fibre Channel FC-3 Defines a set services to support advanced functions FC-2 level serves as the transport mechanism of Fibre Channel. The transported data is transparent to FC-2 and visible to FC-3 and above. Includes data framing, frame sequencing, flow control, and class of service FC-1 defines the transmission protocol including serial encoding and decoding, special characters and error control. Information is encoded 8 bits at a time into a 10 bit transmission character and transmitted over the cable FC-0 is the lowest level and defines the physical link in the system, including the fiber, connectors, optical and electrical components. This level covers a variety of media and the associated drivers and receivers capable of operating at a wide range of speeds
SAN Overview - 4
Frames are the basic building blocks of a Fibre Channel connection. The frames contain the information to transmit, the address of the source and destination ports, and link control information. All information in Fibre Channel is passed in frames. The maximum amount of data carried in a frame is 2112 bytes; the total frame size is 2148 bytes. The header contains the Source and Destination Addresses, which allow the frame to be routed to the correct port. The Type field interpretation is dependent on whether the frame is a link control or Fibre Channel data frame. For example, if the frame is a data frame, a 08 in the Type field indicates SCSI FCP information in the Data field.
SAN Overview - 5
Fibre Channel addresses are used to designate the source and destination of frames in the Fibre Channel network. The Fibre Channel address field is 24 bits/3 bytes in length. Unlike Ethernet, these addresses are not burned in, but are assigned when the node enters the loop or is connected to the switch. The Fibre Channel Address identifiers are three bytes in length. The Frame Header contains two threebyte fields for address identifiers, the D_ID and S_ID fields. Each N_Port has a fabric-unique identifier, the N_Port Identifier, by which it is known. The source and destination N_Port Identifiers and alias address identifiers are used to route frames within the fabric. The Physical Address is switch-specific and dynamically generated during the Fabric Login (FLOGI).
SAN Overview - 6
Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved. 2007
A Fabric is a virtual space in which all storage nodes communicate with each other over distances. It can be created with a single switch or a group of switches connected together. The primary function of the fabric is to receive data frames from a source N_Port and route them to the destination N_Port whose address identifier is specified in the frames. Each N_Port is physically attached through a link to the fabric. Many switches, such as the Cisco MDS, support multiple fabrics within the same switch and also within the same physical switch topology. When a device logs into a fabric, its information is maintained in a database. At the same time, information required for it to access other devices or changes to the topology is provided by another database.
SAN Overview - 7
The following are some of the common services found in a fabric: Login Service: This Service (at address FFFFFE) is used by all nodes when they perform a Fabric Login (FLOGI). For a node to communicate in a fabric, it has to register itself with this service. When it does so, it sends a Source Identifier (S_ID) set to zero, that is, having the following hex value 0x000000. The login service then returns a D_ID to the node with the with the Domain, Area, and Port fields qualified. This gives the node (N_Port) a 24-bit address that identifies it uniquely in the entire fabric. The fabric uses this unique address for routing services. Name Service: This Service (at address FFFFFC) is used to store information about all devices attached to the fabric. When a node performs a PLOGI, it registers itself with the name server. The name server stores all these entries in a locally resident database on each switch. Each switch in the fabric topology exchanges its Name Service information with other switches in the fabric to maintain a synchronized, distributed view of the fabric. Fabric Controller: This Service (at address FFFFFD) provides state change notification to all registered nodes in the fabric. Devices that require a fabric controller are those devices that need to keep track of changes in a fabric, for example scanning storage targets in a fabric. This is requested by submitting a SCN (State Change Notification) frame to the controller which then responds back with a RSCN (Registered State Change Notification) to the frame.
SAN Overview - 8
There are three types of login supported in Fibre Channel; Fabric, Port and Process. All node ports must attempt to log in with the Fabric. This is typically done right after the link or the Loop has been initialized. When a device is physically connected to a fabric switch port, the Fibre Channel protocol establishes a logical connection between the node and the fabric switch. This is called Link Initialization. Ordered Sets are sent between the node and the switch to establish the link.
SAN Overview - 9
Once the physical link is established, the node sends a special frame (FLOGI) to the port to allow it to communicate with the rest of the fabric through the Fabric Login or FLOGI. The FLOGI frame contains the S_ID field filled in with its ALPA value (00 for a switched fabric). This frame is received by the login server, which is located at address FFFFFE. The login server responds back with the D_ID field filled with the domain ID and area location. In other words, the device now gets a 24-bit address by which it is identified in the fabric. All ports except private NL_Ports have to go through this sequence for them to be able to communicate with each other in the fabric. Next, each node performs a Name Server registration. The Name Server is located at address FFFFFC and obtains information from the node through the port login frame (PLOGI) and through subsequent registration frames. Information in the Name Server is stored in the form of database objects. The node may register values for all or some database objects depending on the requirement. The most commonly registered objects are: 24-bit fabric address, 64-bit Port Name (WWPN), 64-bit Node Name (WWNN), ULPs (SCSI, IP etc.), Classes of service supported and Port type. The Node also requests a list of nodes that support the same FC-4 Upper Layer Protocols as itself. This list usually depends on whether there are restrictions placed on which devices the node can talk to i.e. zoning.
SAN Overview - 10
The node then attempts a Port Login (PLOGI) to all nodes from the list it receives from the switchs Name Server. It provides a specific set of operating characteristics associated with the destination N_Port, Classes of Service are supported. It also initializes the destination end-to-end credit. The process is repeated as other nodes are attached to other ports on the switch. Finally, the node then sets up the environment between itself (originating N_Port) and the device its communicating with (responding N_Port) by performing a Process Login. This environment is then used to determine if there is a LUN present. This is the point at which storage connectivity is established. A group of related processes is collectively known as an image pair. The processes involved can be system processes, system images, control unit images or ULP processes. The use of process login (PRLI) is required by a specific upper-level protocol such as SCSI-FCP mapping.
SAN Overview - 11
This table documents the various processes that occur when a Fibre Channel device is connected to an FC-based SAN. When a link light on the switch or on an HBA does not come on, it means that the link initialization component in the FLOGI process is not complete. To troubleshoot this, one usually starts with the hardware and then moves up to the HBA driver level.
SAN Overview - 12
The Internet Small Computer Systems Interface (iSCSI) protocol provides a means of transporting SCSI packets over TCP/IP. iSCSI works by wrapping SCSI commands into TCP and transporting them over an IP network. Since iSCSI is IP based traffic, it can be routed or switched on standard Ethernet equipment.
SAN Overview - 13
Traditional Ethernet adapters (NIC) are designed to transfer packetized file level data among PCs, servers, and storage devices such as NAS appliances. However, NICs do not traditionally transfer block level data, which is handled by a storage host bus adapter, such as Fibre Channel or parallel SCSI. In order for a NIC to process block level data, the data needs to be placed into a TCP/IP packet before being sent over the IP network. Through the use of iSCSI drivers on the host or server, a NIC can transmit packets of block level data over an IP network. When using a NIC, the server handles the packet creation of block level data and performs all of the TCP/IP processing. This is extremely CPU intensive and lowers the overall server performance. The TCP/IP processing performance bottleneck has been the driving force behind the development of TCP/IP offload engines (TOE) on adapter cards. A TOE moves the TCP/IP processing from the host CPU to the TOE card. Thus, a TCP/IP offload storage NIC operates more like a storage HBA rather than a standard NIC. An iSCSI HBA is similar to a Fibre Channel HBA where all operations occur on the card. The iSCSI HBA is required for EMC Boot from SAN environment.
SAN Overview - 14
Because a message is divided into a number of packets, each packet can, if necessary, be sent by a different route across the network. Packets can arrive in a different order than the order they were sent in. The Internet Protocol just delivers them. It's up to another protocol, the Transmission Control Protocol (TCP) to put them back in the right order. An iSCSI packet contains SCSI data and the iSCSI header, which is created by the iSCSI initiator, and is then wrapped in other protocol layers to facilitate its transport. The Ethernet Header is used to provide addressing for the physical layer while the IP Header provides packet routing information used for moving the information across the network, and the TCP Header contains the information needed to guarantee delivery to the target destination. The iSCSI Header defines how to extract SCSI commands and data.
SAN Overview - 15
Native iSCSI allows for all communications using Ethernet. Initiators may be directly attached to iSCSI Targets, or may be connected using standard Ethernet routers and switches. Bridging architectures allow for the Initiators to exist in an Ethernet environment, while the storage remains in a Fibre Channel SAN.
SAN Overview - 16
To ensure that data reaches all the users who need it, organizations are now looking for new ways to transport data throughout the enterprise locally over the SAN as well as over much longer distances. One of the best ways to achieve this goal is to interconnect geographically dispersed SANs through reliable, high-speed links. This approach involves transporting Fibre Channel block data over existing IP infrastructures currently used throughout the enterprise. The new FCIP protocol standard has rapidly gained acceptance as a manageable, cost-effective way to blend the best of both worlds: Fibre Channel block data storage and proven, widely deployed IP infrastructure. As a result, organizations now have an excellent way to protect, store, and move their data while leveraging existing technology investments.
SAN Overview - 17
The top layer of the FCIP protocol stack has SCSI Applications, which include the SCSI driver program that executes read and write commands. These block, stream, and other command types are grouped into a layer which also contains SCSI data and status information. Below the SCSI layer is the Fibre Channel Protocol (FCP) layer, which is simply a fibre channel frame whose payload is SCSI. The FCP layer rides on top of the Fibre Channel transport layer. This layer can then run natively within a SAN fabric environment or be encapsulated into IP at the Fibre Channel over IP (FCIP) layer. TCP and IP are then used to transport the encapsulated information across wired or wireless Ethernet, or another transport that supports TCP/IP traffic. Notice how FCIP creates a logical tunnel between two Fibre Channel protocol stacks. In reality, the information flows through the IP network represented by the cloud.
SAN Overview - 18
FCIP entities are switches or other network adapters used during FCIP. The primary purpose of an FCIP entity is to forward FC frames. Primitive signals, sequences and class 1 FC frames are not transmitted through FCIP because they can't encode using FC Frame encapsulation. An IP network sees the FCIP entities as peers, therefore require TCP/IP communication. FCIP entities contain 1+ TCP endpoints in IP-based network. From a Fibre Channel perspective, the pairs of FCIP entities and their FC entities, forward FC frames between FC fabric elements. The End nodes dont know an IP link exists, therefore the path taken by the FC frames follow the normal routing procedure established by IP network. FCIP doesnt participate in the FC frame routing. FCIP control and service may use QoS features. FCIP entities don't participate in the discovery of FC source and destination identifiers. Before a link is established between two FCIP entities the IP address and TCP port of its pair must be configured with the initiating entity.
SAN Overview - 19
FCIP can transport existing Fibre Channel services across the IP network such that two or more interconnected SANs can appear as a single large SAN and can be managed by traditional SAN management applications. In addition, FCIP enables SAN applications to support additional protocols without modification. These applications might include disk mirroring between buildings in a campus network or remote replication over the WAN. The type of applications utilized are based on the distance the data must travel, the network bandwidth, and the QoS requirements and/or abilities of the network connection. While some implementations of FCIP are point-to-point tunnels, the protocol does not require that the gateways support only point-to-point tunneling. The FCIP standard supports all Fibre Channel services, including FSPF routing algorithms, such that multiple logical links created from a single gateway can route Fibre Channel packets over the IP infrastructure. Not only is FCIP routable, but IP networks do not need to know anything about the packets being routed. The Fibre Channel services handle all routing between logical links, while the TCP protocol handles the delivery of packets to the specific gateway device.
SAN Overview - 20
A Fibre Channel switch, acting as a FCIP Gateway, encapsulates FC frames into IP packets. Once encapsulated into IP packets, they can be sent across the IP network through the FCIP link. The IP network could be made up of Gigabit Ethernet, Fast Ethernet, IP switches and routers, etc. At the other end of the IP network, the original Fibre Channel frames are recovered and sent through the SAN by the receiving Fibre Channel Switch. The FCIP connected FC SANs essentially form into a new unified fabric. The IP network is transparent to the Fibre Channel fabric. Only edge devices need to be aware of FCIP encapsulation. This solution can take advantage of existing Fiber Channel networks and tie them together using existing IP networks.
SAN Overview - 21
SAN Overview - 22
The Port initialization takes place first. In this step the port determines the Port Type. For this discussion, we will assume the Port has configured itself as an E_Port (Expansion Port). Once this port is a recognized E_Port, Exchange Link Parameters (ELP) are sent. The ELP exchanges information about the switch such as its WWN and Class of Service. Next, the Fabric must elect a Principal Switch. This is accomplished by the Exchange Fabric Parameters (EFP) and Principal Switch Selection (DIA) operations. Once the Principal Switch is elected, Domain_IDs are assigned to the other member switches of the Fabric. Switches will attempt to keep their previous Domain_ID. The Final steps in configuring the Fabric are to build the Topology Database using Fibre Shortest Path First (FSPF) and then create the Routing Tables. Although not specified, the switches attempt to combine the existing Zone information into a single Zone Set to be distributed throughout the new Fabric.
SAN Overview - 23
An E_Port is the point at which a Switch is connected to another Switch to create a multi-Switch Fabric. An E_Port is the point at which a Switch is connected to a Bridge device. It normally functions as a conduit between the Switches for frames destined for remote N_Ports and NL_Ports. An E_Port is also used to carry frames between Switches for purposes of configuring and maintaining the Fabric. An E_Port shall support the Class F service. An E_Port shall also be capable of routing one or more of the following classes of service: Class 1 service, Class 2 service, Class 3 service. An E_Port shall not admit to the Fabric any Primitive Sequences, or any Primitive Signals other than Idle, that the E_Port receives on its inbound fibre. An E_Port contains an FC-PH Transport element through which all frames are passed, and Primitives are transferred across the Link to and from the other E_Port. Frames received from the other E_Port are either directed to the Switch Construct via the Switch Transport element, or directed to the Link_Control_Facility. The Link_Control_Facility receives frames related to Switch Fabric Internal Link Services such as ELP, and transmits responses to those Link Service frames. Frames received from the FC-PH Transport element that are destined for other ports are directed by the Switch Transport to the Switch Construct for further routing. Frames received from the Switch Construct by the Switch Transport are directed either to the FC-PH Transport for transmission to the other E_Port, or to the Internal_Control_Facility. The Internal_Control_Facility receives frames related to Switch Fabric Internal Link Services, and transmits responses to those Internal Link Services frames. Information is passed between the Internal_Control_Facility and the Link_Control_Facility to effect the control and configuration of the Transport elements.
SAN Overview - 24
Switches in the fabric communicate through EFP (exchange fabric parameter) frames to determine the principal switch as follows: If one (and only one) director/switch is set to Principal, it becomes the principal switch. This switch remains the principal switch as long as it is on line in the fabric, even if someone changes another switchs Switch Priority to Principal. If more than one director/switch is set to Principal, the director/switch with the lowest WWN number among those set to Principal becomes the Principal switch. The Switch Priority setting for every other switch/director (including any others that were previously set to Principal) automatically changes to Default. If no director/switch is set to Principal but one or more are set to Default, the director/switch with the lowest WWN number among those set to Default becomes the principal switch. The Switch Priority setting for every other switch/director remains unchanged. If all directors/switches are set to Never Principal, no director/switch can be principal. In this case, all of the ISLs will segment, with reason code 05 (no switch is capable of becoming a Principal switch).
SAN Overview - 25
Each switch in a multiswitch fabric is identified by a unique Domain ID, in the range 1 through 239. Domain IDs are used in 24-bit Fibre Channel addresses to identify source and destination ports in a fabric. When adding a new switch, assign it a Preferred Domain ID that is not in use. When the switch comes on line, it requests that it be assigned this ID. If the ID is already in use, the principal switch assigns an unused one. To change a Domain ID, a switch must be taken off line. Some HBAs can handle a Domain ID change without requiring device driver administration changes. In some operating environments, however (currently HP-UX and AIX), drivers require device driver-level administrative changes when the Domain ID is changed. If two operational fabrics join, they determine if any Domain ID conflicts exist between the fabrics. If a conflict exists, the interswitch link (ISL) E_Ports segment themselves from each other, preventing the fabrics from joining. To avoid this problem, assign a unique Preferred Domain ID to each switch in a data center. If there are plans to go data center to data center, apply this uniqueness across both data centers. This will facilitate future fabric mergers, and is particularly important if zoning is implemented through port number (and by default Domain ID) rather than WWN.
SAN Overview - 26
EMC recommends that you always attempt to give each switch a unique preferred Domain ID, whether it is targeted for an existing fabric or a new fabric. (The exception to this rule would be if you were swapping a new switch for a failed switch.) This provides the best opportunity to merge a switch or fabric into another fabric with the least amount of manual intervention. Since Domain IDs become part of an HBA or storage array port's fabric address, and data is transferred to and from devices based on their address, Domain IDs cannot be changed without first bringing the switch to an offline state. Every switch that is currently online will have been assigned a Domain ID from the principal switch. If the fabric contains only one switch, that switch is the principal switch for that fabric. Domain ID negotiation done during a fabric merge will not change the current Domain ID when the switch is connected to the fabric while online. During the exchange of parameters following a merger event, if the switches identify a domain conflict, they will not merge and the ISLs connecting them will remain segmented. Taking a switch that has segmented from the fabric due to a Domain ID conflict offline and then bringing it back online allows the switch to renegotiate for a new unique Domain ID. It should be noted that taking a switch offline terminates communication between the devices that are on that switch. EMC recommends that if a conflict does exist that you change the preferred Domain ID of the switch yourself so that you can ensure that it is unique in your entire environment and that it always attempts to use the same ID, even after future merges and fabric changes
SAN Overview - 27
One switch in the fabric is responsible for the distribution of Domain IDs, and plays a role in the route creation for fabric management traffic. This switch is known as the principal switch. Since this switch must communicate with all other switches and is the basis for fabric traffic routing, this switch should be centrally located. This will assist in the uniform delivery of information in the fabric, and provide a consistent response to fabric build events. Principal switches are selected during both the creation of the fabric and during fabric reconstruction events. Two pieces of information located on the switch will determine the selection of the principal switch in the fabric. The Domain ID negotiation is governed by the entity in the fabric called the principal switch. Principal switch selection is made based on the switch with combination of the highest principal switch priority and then the lowest World Wide Name. Proper placement of the principal switch in the fabric can lead to short negotiation times, resulting in the fabric returning to a normal state in an shorter amount of time. For these reasons, EMC recommends that the switch at the logical center of the fabric be made the principal switch. This would normally mean the switch with both the least amount of hops to the farthest extent of the fabric and/or the switch that has connections to the most other switches in the fabric. These two placement strategies will help to ensure that your principal switch access times are as quick as they can be. The basic rules of Principle Switch selection are: A standalone switch will declare itself the Principal Switch. Each Fabric can have only one Principal Switch. If both switches are ONLINE, a principal switch election will occur. Switch Priority determines who becomes Principal Switch. Domain IDs DO NOT determine who becomes Principal Switch.
SAN Overview - 28
The fabric-capable device drivers on Symmetrix-supported host bus adapters (HBAs) discover devices by using a Name Server, which is a fabric-based function accessible to all nodes on the fabric via a well-known fabric address. When an N_Port logs onto the fabric, it registers its World Wide Port Name and FC-4 type (SCSI) with the Name Server. A driver performing device discovery queries the Name Server for a list of all SCSI devices that have registered with it. The driver uses this list to attempt a login to each of the registered devices. In this manner, a fabric device driver discovers all logged-in Symmetrix ports attached to the switch.
SAN Overview - 29
Connectrix Directors and switches use a number of algorithms to distribute the load over a set of ISLs. These algorithms include: Fibre Shortest Path First Multiple Equal-Cost Paths Load Assignment on World Wide Node Names The ANSI standard for routing in a Fibre Channel fabric is Fibre Channel Shortest Path First (FSPF). This standard measures routing cost in terms of the number of hops a frame must take from its source to its destination. A hop is a direct path between two switches with no other switches in between. A link between an end device and a switch does not count as a hop. The FSPF algorithm creates routes of minimum cost throughout the fabric. Every entry port into a switch has a designated exit for each of the possible destinations. These exit assignments change only if there is a reconfiguration event in the fabric. This single-static exit ensures in-order delivery of the frames within an exchange. If an ISL or a new switch is added or removed, the assignments may change. These assignments are done with logged-in N_Ports. No routing assignments are done for F_Ports that have no logged-in ports. When there are multiple equal-cost paths between a source and a destination, the routing algorithm spreads the load over the multiple paths in a round robin fashion, sequentially assigning a set of entry ports to a set of exit ports so that each exit port has the same number of assigned entry ports (or a difference of only one).
SAN Overview - 30
In this standard, the term path selection indicates the discovery of the best path from source to destination, and the term routing indicates the actual forwarding of frames to a specific destination. FSPF performs hop-by-hop routing which means that a Switch only needs to know the next hop on the best path to the destination. The replicated topology database insures that every Switch in the Fabric has the same view of the Fabric itself, and therefore all Switches will make consistent routing decisions. Typically a Switch needs to know, for each destination domain in the Fabric, which path should be used to route a frame to that domain. A routing table entry minimally consists of a destination Domain_ID, and an E_Port to which frames are forwarded to the destination Switch.
SAN Overview - 31
The fewest hop path is always selected. If a new path with the least cost becomes available, frames will immediately be switched to the newer path. If multiple minimum hop paths between switches exist, switches will do load sharing In this example, the cost for each link is equal: There are two ISLs between switches Green and Yellow Three HBAs login to switch Green with data destined to Switch Yellow HBS 1 uses Path C (ISL 3) HBA 2 uses Path D (ISL 4) HBS 3 uses Path C (ISL 3) Each link has a specific cost associated with it. The higher the throughput, the lower the cost. For example a 1Gbps link has a cost of 1000 while a 2Gbps link costs 500. In this case, the 2Gbps link is always used. You also can configure link costs for switches.
SAN Overview - 32
Zoning is a switch function that allows devices within the fabric to be logically segmented into groups that can communicate with each other. When a device logs into a fabric, it is registered by the name server. When a port logs into the fabric, it goes through a device discovery process with other devices registered as SCSI FCP in the name server. The zoning function controls this process by only letting ports in the same zone establish these link level services. A collection of zones is called a zone set. The zone set can be active or inactive. An active zone set is the collection of zones currently being used by the switched fabric to manage data traffic. Single HBA zoning consists of a single HBA port and one or more storage ports. A port can reside in multiple zones. This provides the ability to map a single storage port to multiple host ports. For example, a Symmetrix FA port or a CLARiiON SP port can be mapped to multiple single HBA zones. This allows multiple hosts to share a single storage port.
SAN Overview - 33
Switch Domain IDs must be unique If two enabled switches are cabled together, they must have unique Domain Ids, or the Fabric is segmented. Ensuring unique Domain ID If a new switch is cabled to the Fabric and then enabled, the master switch assigns a unique Domain ID to the new switch. Domain IDs can be configured by the administrator to ensure that uniqueness. If you power cycle, the switch will receive a unique Domain ID. Other configuration items that may affect cascade Non-SCSI Tachyon Mode Disable Device Probing VC Encoded Address Mode Per-frame Route Priority
SAN Overview - 34
SAN Overview - 35
There are several ways to monitor and manage Fibre Channel switches in a fabric: If the switches in the fabric are contained in a cabinet with a Service Processor (SP), console software loaded on the SP can be used to manage them. Some switches also offer a console port, which is used for serial connection to the switch for initial configuration using a Command Line Interface (CLI). This is typically used to set the management IP address on the switch. Subsequently, all configuration and monitoring can be done via IP. Telnet or ssh may be used to log into the switch over IP, and issue CLI commands to it. The primary purpose of the CLI is to automate management of a large number of switches/directors with the use of scripts, although the CLI may be used interactively, too. In addition, almost all models of switches support a browser-based graphical interface for management. There are vendor-specific tools and management suites that can be used to configure and monitor the entire fabric. They include: B-Series: Web Tools and Connectrix Manager M-Models include Connectrix Manager Basic MDS-Series: Cisco Fabric Manager SAN Manager, an integral part of EMC ControlCenter, provides some management and monitoring capabilities for devices from both vendors. A final option is to deploy a third-party management framework such as Tivoli. Such frameworks can use SNMP (Simple Network Management Protocol) to monitor all fabric elements.
SAN Overview - 36
Web Tools is an easy-to-use, browser-based application for switch management and is included with all Connectrix B-Series products. Web Tools simplifies switch management by enabling administrators to configure, monitor, and manage switch and fabric parameters from a single online access point. Web Tools supports the use of aliases for easy identification of zone members. With Web Tools, firmware upgrade is a one-step process. The Switch View allows you to check the status of a switch in the fabric. The LED icon for the port reporting an issue changes color.
SAN Overview - 37
Connectrix Manager is a licensed software product widely used for the management of Connectrix B switches and directors. It can be run locally on the Connectrix Service Processor, or remotely on any network-attached workstation. Since this application is Java-based, IT administrators can run it from virtually any type of client device.
SAN Overview - 38
Fabric Manager and Device Manager must be installed in a server and can contain several clients. This Java-based tool simplifies management of the MDS Series through an integrated approach to fabric administration, device discovery, topology mapping, and configuration functions for the switch, fabric, and port. Features of Fabric Manager include: Fabric visualization: Automatic discovery, zone and path highlighting Comprehensive configuration across multiple switches Powerful configuration analysis including real-time monitoring, alerts, zone merge analysis, and configuration checking Network diagnostics: Probes network and switch health, enabling administrators to pinpoint connectivity and performance issues Comprehensive security: Storage Media Encryption Protection against unauthorized management access with Simple Network Management Protocol Version 3 (SNMPV3), Secure Shell Protocol (SSH), and role-based access control (RBAC)
SAN Overview - 39
SAN Manager provides a single interface to manage LUN Masking, switch zoning, device monitoring and management. The integration of SAN Manager into ControlCenter provides a distributed infrastructure allowing for remote management of a SAN. It offers reporting and monitoring features such as threshold alarms, state change alerts and component failure notifications for devices in the SAN. SAN Manager has capabilities to automatically discover, map, and display the entire SAN topology at a level of detail desired by the administrator. It can also display specific physical and logical information about each object in the fabric. Administrators can view details on physical components such as host bus adapters, Fibre Channel switches and storage arrays, as well as logical components such as zones and LUN masking policies. SAN Manager offers support for non-EMC arrays such as HDS Lightning, HP StorageWorks, and IBM Shark.
SAN Overview - 40
Command Line Interface is available for any Connectrix Switch. Each switch vendor implements its own syntax for common switch and fabric functions.
SAN Overview - 41
SAN Overview - 42
You can install a switch as a stand-alone unit on a flat surface, in an Electronic Industries Association (EIA) cabinet or the EC-1700 Cabinet using the Slide Rack Mount Kit included with the switch. For this example, we review the configuration of a B-Series switch, The following modules cover the details for each Connectrix series.
SAN Overview - 43
Floor loading is a badly understood issue in Data Centers. Not just in the place where the equipment eventually ends up but also en-route from the loading bay to the final resting place. Most data centers have a raised floor made of specialist tiles held up on struts and a tile mounting framework pushing down on the underlying slab and fabric of the building. Each of these elements must be strong enough to withstand the weight of the equipment to be installed. In addition, the floor must be strong enough to deal with carrying its design weight with a number of the tiles removed for maintenance. Typically a data center floor will have a loading capability measured in Kilo Pascals (kPa) aka Kilograms per square meter (Kg/m2) or Pounds per square foot (lbs/ft2) depending on your preferred units of measure. For most EMC scenarios, the floor tiles must be concrete filled or Heavy-duty tiles. Perforated and vented tiles as well as damaged tiles and tiles with holes for cables (particularly on the edge of the tile rather than in the middle) are significantly weaker than complete tiles. There are three kinds of load that we need to take into account: Static Load - the load a stationary object places on the floor (kg/sq m) Concentrated Load - The point load transferred directly to the floor from a wheel or foot support (kg/sq cm) Rolling or Dynamic Load - The load placed on the floor as the object is pushed through the data center into its final location (kg/Tile) The underlying slab needs to be strong enough to support the gross load of the entire floor, including the struts, tiles, equipment and M&E plant. If in doubt instruct a Structural Engineer or consult the initial design documents for your building. As an example, the installation of the MDS 9513 Director in a rack requires a mechanical lift to place the chassis in the rack. A fully loaded 9513 can weigh about 300 pounds. A full rack with all components will be over 1000 pounds of static load.
SAN Overview - 44
Verify the following requirements are met prior to product and management interface installation. Ensure: A site plan is prepared, configuration planning tasks are complete, planning considerations are evaluated, and related planning checklists are complete. Fibre Channel SAN design is complete. Support is available for one of the following product management methods: A browser-capable PC and Internet connectivity to support the product-resident Web Tools, or A rack-mount service processor or browser-capable PC and LAN segment connectivity to support operation of Connectrix Manager and Element Manager applications. Support equipment and technical personnel are available for the installation. The required number and type of fiber-optic jumper cables are delivered and available. Ensure cables are the correct length and have the required connectors. An EC-1700 cabinet or customer-supplied 19-inch equipment rack and associated hardware are available (optional). Remote workstations or simple network management protocol (SNMP) workstations are available (optional). Workstations are customer-supplied and connected through a public or dedicated LAN segment.
SAN Overview - 45
Take the following measures to ensure a successful installation and operation of the switch. In this example the switch requires specific environmental conditions. Install the switch with the non-port side facing the cool-air aisle to ensure proper cooling. This prevents the fan assemblies from pulling in heated exhaust air. Verify that the supply circuit, line fusing and wire size conform to the electrical rating of the switch. Verify that the ambient air temperature at the non-port side of the switch does not exceed the listed rating of the switch. Verify air flow is available to the nonport vents.
SAN Overview - 46
The exact installation procedures vary among the different cabinet systems. Always review the product documentation prior to installation. Assemble the two side slide assemblies. Install the inner slide rails on the switch chassis: Position one of the inner slide rails with the flat side against the switch, oriented as needed for slide removal. Attach the rail using the three Phillips head 8-32 x 3/16 inch screws provided with the installation kit. Tighten the screws to a torque of 15 inch-pounds. Repeat with the second inner slide rail on the other side of the switch. Verify that two of the slide mount L-brackets are attached to each outer slide rail. These screws should be at a torque of 15 inch-pounds. Install the power cord clips onto the outer slide rails. The clips must be installed on the outer surface of the slide rails so that they do not interfere with rail movement. Insert the power cords into the power cord clips with the pronged end of the power cords toward the cool air aisle for access to the power source. Install the outer slide rails in the cabinet. Install the switch in the cabinet. Warning: Warning Using screws longer than the specified length can damage the switch.
SAN Overview - 47
Installation of the SFP transceivers is the final step: Remove the shipping plugs from the ports to be used. The transceiver is keyed so that it can be inserted into the port only with the correct orientation. If it does not slide in easily, ensure it is correctly oriented. Position a transceiver so that it is oriented correctly and insert into the port until it is firmly seated and the latching mechanism clicks. For instructions specific to the type of transceiver, refer to the transceiver manufacturers documentation. Repeat for the remaining ports as required.
SAN Overview - 48
Connect the power cord to one or both of the power supplies and to a power outlet. Ensure that the power cord has a minimum service loop of 6 inches available at the connection to the switch, and is routed so that it is not exposed to stress. Turn on the power supply by flipping the AC switch to 1. The switch is powered on and begins running the POST. Repeat these steps for the second power supply. For optimal redundancy, ensure that separate circuits are provided for each power supply. When the switch is powered on, it automatically runs the POST to guarantee switch stability. Errors that occur during POST are written to the system error log. Verify that the POST completes successfully.
SAN Overview - 49
Connect the serial cable to the serial port on the switch and to an RS-232 serial port on the workstation. If the serial port on the workstation is RJ-45 instead of RS-232, you can remove the adapter on the end of the serial cable and insert the exposed RJ-45 connector into the RJ-45 serial port on the workstation. The serial cable must be a pass-through serial Cable. Null Modem cables establish a connection but do not transfer data or commands. It is recommended that you use your laptop to configure these switches through HyperTerminal.
SAN Overview - 50
Change the IP address, subnet mask, and gateway address as directed by the customers network administrator. Verify the address was correctly set by entering the ipAddrShow command at the prompt. Record the IP address on the label provided for this purpose on the port side of the switch. Once the IP address is verified as correct, log off of the serial console, remove the serial cable, and replace the shipping plug in the serial port.
SAN Overview - 51
When configuring the switch using the telnet, Web Tools or other out-of-band management systems, you must be aware of LAN configuration. Devices and applications such as routers, firewalls, proxy servers, DNS, DHCP could interfere with network connectivity between the managements station and the switch. In the event of a failure to communicate, some simple tests should be performed. To determine if connectivity is possible, begin with the basic TCP/IP test of PING <switch address>. If this is successful you have an IP connection. For higher level applications to function you may next need to test for Port availability and LAN timeouts. Commands such as TRACERT, NSLOOKUP and NETSTAT are useful to determine the network configuration between station and switch.
SAN Overview - 52
Before you install Web Tools on your workstation, verify that your switches and workstation meet the Web Tools requirements listed in the current product documents. Web Tools requires any browser that conforms to HTML version 4.0, JavaScript version 1.0, and Java Plug-in 1.6.0 or higher. If your fabric includes a switch with a limited switch license and you are opening Web Tools using that switch, if the fabric exceeds the switch limit indicated in the license, Web Tools allows a 30-day grace period in which you can still monitor the switch through Web Tools. However, Web Tools displays warning messages periodically. These messages warn you that your fabric size exceeds the supported switch configuration limit and tells you how long you have before Web Tools will be disabled. After the 30-day grace period, you are no longer able to open Web Tools from the switch with the limited switch license if that switch is still exceeding the switch limit. Web Tools is part of the Fabric OS of a switch. When you open Web Tools on a switch, you can manage other switches in the fabric that have lower or higher firmware versions. It is important to note that when accessing these switches you are opening the switchs version of Web Tools, and the functionality available for those switches might vary.
SAN Overview - 53
To perform the initial login into a switch: Verify that the switch is connected to your IP network through the RJ-45 Ethernet port to enable connection through telnet. Open a telnet connection to the switch. The login prompt is displayed if the telnet connection successfully found the switch in the network. Enter the user ID (usually user or admin) at the login prompt. login: admin Enter the default admin password. The default password is password. You are prompted to change the password. Enter the new password. Verify that the login was successful. A prompt is displayed showing the switch name and user ID to which you are logged.
SAN Overview - 54
It is important to have consistent system configuration settings since inconsistent parameters amongst switches in the same fabric can cause fabric segmentation. To display and check system configuration settings, perform the following procedure. Fabric parameters include all the items listed in the configure command. Fabric parameters must be identical for each switch across a fabric. To save time when configuring the fabric parameters: Configure one switch first (using the configure command. Use the configUpload command to save the configuration information. Use the configdownload command to download it onto each of the remaining switches. The default Domain ID is 1. If the default domain ID is already in use when the switch is connected to the fabric, the Domain ID for the new switch is automatically reset to a unique value. The Domain IDs that are currently in use can be determined using the fabricshow command. Domain IDs are assigned dynamically when a switch is enabled. However, the Domain ID can be set manually in order to control the number, or to resolve a Domain ID conflict when merging fabrics. Valid Domain IDs are 1 through 239.
SAN Overview - 55
Switch Date and Time - All switches maintain current date and time in non-volatile memory. Date and time are used for logging events. Switch operation does not depend on the date and time; a switch with an incorrect date and time value still functions properly. Switch Status Policies - The policy parameter determines the number of failed or non-operational units for each contributor that will trigger a status change in the switch. Each parameter can be adjusted so that a specific threshold must be reached before that parameter changes the overall status of a switch to MARGINAL or DOWN. For example, if the FaultyPorts DOWN parameter is set to 3, the status of the switch will change if 3 ports fail. Tracking Switch Changes - The Track Change feature allows you to keep record of specific changes that may not be considered switch events, but may be useful. The output from the track changes feature is dumped to the error log for the switch. Use the errdump command or errshow command to view the error log. Items in the error log created from the Track changes feature are labeled TRACK. An SNMP-TRAP mode can also be enabled. Routing - In a stable fabric, frames are always delivered in order, even when the traffic between switches is shared among multiple paths. However, when topology changes occur in the fabric (for instance, a link goes down), traffic is rerouted around the failure. When topology changes occur, some frames could be delivered out of order.
SAN Overview - 56
Switch Name - You can customize the switch names for the logical switches. If you chose to change the default switch name, use a switch name that is unique and meaningful. Changing the switch name causes a domain address format RSCN to be issued. Switch names: Can be up to 15-characters long Must begin with an alpha character Can consist of any combination of alphanumeric and underscore characters
SAN Overview - 57
Licensed features such as Fabric Watch and Performance Monitoring are already loaded onto the switch firmware, but must be enabled with a license key. Once you have purchased these features, you are provided with a key to unlock the features in the firmware. You must log on to PowerLink or contact EMC Customer Support to convert the transaction key to a license key. Verifying License Activation To verify that the required licenses are activated on the switch, perform the following steps: Log into the command line interface as the Admin user. Enter the licenseshow command at the command line prompt. A list displays all of the licenses currently activated on the switch.
SAN Overview - 58
Configure the software features (such as Fabric Watch, Zoning, and Secure Fabric OS) for each switch. To save time you may configure the software features on one switch, then save the configuration file, and download it to the each of the remaining switches.
SAN Overview - 59
Power off all devices (to minimize PLOGIs) and connect them to the switch, according to your topology. For devices that cannot be powered off, connect the devices, but use the portdisable command to disable the port on the switch. When powering the devices back on, wait for each device to complete the fabric login before powering on the next one. EMC recommends that you create a blocking zone config and load that to the fabric before connecting devices. More information on this is presented in the B-Series Zoning module. To verify that you have fabric wide device connectivity, display the fabric wide device count. To display the fabric-wide device count from a switch: Log in to the switch as the admin user. Enter the nsallshow command at the command line. This command displays 24-bit fibre channel addresses of all devices in the fabric.
SAN Overview - 60
To verify that your switch is operating correctly, display information about the switch and port status. To display information about the switch and port status: Log in to the switch as the admin user. Enter the switchshow command at the command line. This command displays a switch summary and a port summary. Verify that the device is connected to the switch by entering the nsshow [-r] command. Use the -r option to replace the TTL attribute output with SCR (state change registration) information in the display. SCR is the state change registration of a device. This value indicates what type of RSCN a device registers to receive.
SAN Overview - 61
To verify that you have fabric-wide switch connectivity, display a summary of information about the fabric. To display a summary of information about the fabric: 1. Log in to the switch as the admin user. 2. Enter the fabricshow command at the command line. This command displays a summary of all the switches in the fabric.
SAN Overview - 62
It is recommended that you make a hard copy backup of all key configuration data, including license key information for every switch, and store it in a safe and secure place for emergency reference. Refer to the following procedures. Print out the information from the licenseshow command and store in a secure location. Print out the information from the configUpload command (contains license and configuration information) and store in a secure location. Print out the information from the ipaddrshow command (select option 4 to display all configured addresses) and store in a secure location. Depending on the security procedures of your company, you may want to keep a record of the user levels and passwords for all switches in the fabric. This is sensitive information and access to such information should be limited. Save all key configuration data, including license key information for every switch and upload it to a host for emergency reference. The configuration file is written as three sections, and is broken up as follows: The first section contains the switch boot parameters. It has variables such as the switch's name and IP address. This section corresponds to the first few lines of output of the configshow command. The second section contains general switch configuration variables, such as diagnostic settings, fabric configuration settings, and SNMP settings. This section corresponds to the output of the configshow command (after the first few lines), although there are more lines uploaded than shown by the command. The third section contains zoning configuration parameters.
SAN Overview - 63
These are the key points covered in this module. Please take a moment to review them.
SAN Overview - 64

01 SAN Overview

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

01 SAN Overview

Uploaded by

Copyright:

Available Formats

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2009 EMC Corporation. Do not Copy - All Rights Reserved.