IBM Systems Technology Group

System Networking Technical

Education
October 23, 2012

Delivering on the Promise of Smarter Computing

2011 IBM Corporation

Welcome to the Webinar!

If you are using the
telephone, please make
sure this button is selected
and that you enter your
Audio PIN

If you have questions

or problems, type
them here
We will be starting shortly!

Webinar Instructions

Questions During The Call?

Should be submitted via the Go To Meeting Tool

This Presentation Will Be Recorded

All or a portion of the recording will be placed in the System
Networking course catalogs of IBMs Education SMART Zone and
PartnerWorlds System College when non confidential

Please Remember That Registration Is Required For Each Go To

Webinar Event

Any other questions

Email debrann@us.ibm.com

Todays Session
DATE: October 23, 2012
TIME: 10:00 AM EST
EVENT: System Networking Technical Education Session
TITLE: IBM Flex System Fabric CN4093 FC/FCoE Capabilities Deep
Dive
SPEAKERs: Meenakshi Kaushik and Deepak Kumar
Bios:
Meenakshi Kaushik: PLM (FC/FCoE, Fabric Architecture and
Virtualization)
Deepak Kumar: Architect, Software Development, FC/FCoE

Content in Todays Session is Confidential Until Nov

Upcoming Sessions
Please Remember to REGISTER for the Events Just Click on
Button in the Calendar Notice To Be Safe Register at least ONE
HOUR PRIOR to Event

10.24
10.30

IBM System Networking Monthly Roadmap and Marketing Update

How to Leverage the STG Tech Support Model

11.05
IBM Flex System Fabric EN4093 10Gb Scalable Switch 4Q Update
11.13
(TBC)
11.20
Blocked - for Thanksgiving
11.27
IBM System Networking Monthly Roadmap and Marketing Update
11.29
Overview of System Networking Environments TRILL (Comparison to other
technologies:
FabricPath, etc)

Do you have a topic you would like covered?

Send recommendations to debrann@us.ibm.com
5

System Networking
Meenakshi Kaushik

IBM Flex System Fabric CN4093

FC/FCoE Capabilities Deep Dive

2012 IBM Corporation

System Networking

Agenda
CN4093 FC/FCoE feature overview
Data I/O Convergence overview
FCoE Protocol Building Blocks
FIP and FCoE

Full Fabric FC/FCoE Switch

NPV Gateway
CN4093 FC/FCoE Features and Scaling Limits
CN4093 Fabric Architecture

2012 IBM Corporation

System Networking

Agenda
CN4093 FC/FCoE Configuration
NPV Gateway
-

Configuration CLI
Display CLI
Debugging and Trouble Shooting
Login Redistribution (Load balancing)

Full Fabric FC/FCoE

-

Configuration CLI
Display CLI
Debugging and Trouble Shooting
Ghost FC Ports

Design Tips

IBM Confidential

2012 IBM Corporation

System Networking

Overview
10Gb
to Server

SFP+
Ports

Omni
Ports

QSFP+
Ports

Base System (00D5823)

14

Upgrade #1 (00D5845)

28

Upgrade #2 (00D5847)

28

12

Upgrade #1 & #2

42

12

Total Ports

Leadership
OmniPorts allow flexibility 10GbE or 4/8Gb FC
Proven IBM Networking Operating System
Performance - Low latency, up to 1.28Tbps
Scalable pay-as-you-grow design
VM aware & VM Mobility with VMready
Virtual Fabric carve up virtual NICs and pipes
Seamless interoperability with other vendors
Warranty is 1 year or will match the chassis warranty
(Includes software upgrades)

2012 IBM Corporation

1GbE Mgmt

Pay as you grow

scalability

Optimized for
performance

Designed for
Convergence

Enhanced virtualization
intelligence

Lower TCO

Seamless interoperability

12 x 10GbE OmniPorts (6 in base + 6 #2)

Networking Infrastructure

Networking

2X10Gb SFP+ #1 = 2x40GbE

IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch

System Networking

IBM Flex System Fabric CN4093 Converged Scalable Switch

Industry leadership - price/performance, scalable 10/40Gb Ethernet switches
Key Features

Customer Benefits

Convergence

Ability to reduce adapters and switches

Full Fabric FC/FCoE and NPV Gateway can co-exist at the same time
Full Fabric FC/FCoE including end-to-end FCoE (target to the initiator). Ability to leverage integrated
storage node. All done (storage and Compute) within the Pure Flex chassis.
NPV gateway to connect to any SAN switch. Reduces cost while integrating to the existing SAN.
Support for connectivity to existing FC SAN or FCoE Storage

Easy & Flexible

Reduce IT staff cost to deploy and manage, higher availability/reliability.

Reduces cable clutter (no cables from server to switch).
Flexible and Scalable ports up to 42 node ports (6 per server*), plus up to two SFP+ 10Gb uplinks
(1/10GbE), 12 OmniPorts (10GbE, 4/8Gb FC) and 2x40Gb uplinks
High availability via enterprise class features like failover
Seamless integration with 3rd party switches.

Better end users/customer experience.

Low Latency under 1 microsecond
Industries first 40Gb Ethernet embedded switch

Virtualization

VMready makes the network VM aware for simpler management & VM mobility.
Virtual Fabric allows for vNICs and virtual pipes between adapter and switch
Works with IBM DVS 5000V simple virtual & physical switches management

Cost of Ownership

Better and shorter return of investment, savings could be used elsewhere.

Acquisition cost is less than separate Ethernet and FC adapters and switches

Where to win

14 x 10Gb node port, 2 x SFP+ and 6

OmniPorts
Part Number: 00D5823

Switch upgrade 1:

28 x 10Gb node ports, 2 x SFP+, 6

OmniPorts & 2 x 40Gb uplinks
Part Number: 00D5845

Switch upgrade 2:

Performance

Base Configuration

28 x 10Gb node ports, 2 x SFP+, and 12

OmniPorts
Part Number: 00D5847

Switch with both upgrades:

42 x 10Gb node ports, 2 x SFP+, 12

OmniPorts & 2x40Gb Uplinks

Competes Against:
HP FlexFabric
Dell Force10 MXL
Cisco UCS Fabric Extender and
Fabric Interconnect

Target Customers: Clients wanting native FC connectivity from the chassis

Clients looking to implement convergence to reduce cost and complexity
Clients looking wanting features that help improve performance & available, energy efficiency and more automated virtualization

* 6-port adapter not available at this time

2012 IBM Corporation

System Networking

IBM Capabilities and Competitive advantages

Capabilities:
Ability to provide flexibility of ports as either 10Gb or 4/8Gb Fiber Channel
Ability to provide cost effective FCoE clients connectivity into existing FC SAN (NPV Gateway)
Ability to provide end to end FCoE (Full Fabric) compute node to integrated storage node

Competitive Advantages
1.High Availability (Avoid down-time)
No reboot of switch needed to convert the OmniPorts from Ethernet to Fibre Channel or
Fibre Channel to Ethernet
No reboot of switch needed to make it FC switch or NPV-Gateway
2.Reduce Complexity
On a VLAN basis, it can be configured to be an Full Fabric FC/FCoE switch (only 1 VLAN)
On a VLAN basis, configure the switch to be a FCoE->FC NPV-Gateway switch (up to 12
VLANs)
Simultaneous FC switching and Gateway functionality (on different VLANs )
- Not a Interop Tested Configuration for Nov 2012

2012 IBM Corporation

System Networking

Traditional segmented fabric approach to networking

Separate fabrics for local area networking (LAN) and storage area
networking (SAN)

LAN
LAN
Ethernet

SAN
SAN
Fibre
Channel

Advantages
Traffic isolation
Independent administration
Challenges
Underutilization of resources
Higher infrastructure costs
Complexity of management

2012 IBM Corporation

System Networking

Convergence Options for the Enterprise

Loss-Less
Ethernet

FCoE Storage Array

iSCSI
Storage
Array
Gateway

FCoE
Gateway

Fibre Channel

NFS/CIFS NAS

13

This educational material is intended for your use in selling. It is NOT a deliverable for your clients.

FC Arrays

2012 IBM Corporation

System Networking

Why FCoE convergence is interesting now

1. Data transmission speed - Ethernet is now able to meet or exceed the
speeds available with Fibre Channel.
2. Ethernet capabilities that are designed to make it more similar to Fibre
Channel such as Data Center Bridging protocols, lower latency and
lossless.
3. Improved asset utilization and cost of ownership (procuring, installing,
managing and operating ).

14

This educational material is intended for your use in selling. It is NOT a deliverable for your clients.

2012 IBM Corporation

System Networking

Why convergence versus separate LAN and SAN switching?

Clients can see over a 25% acquisition cost savings through convergence
Clients can see significant operational cost savings
50% fewer switches and less cables
Lower power consumption
Fewer components to manage
Delivers separate LAN & SAN outside the chassis requirement by some clients

Using x240 nodes

Separate LAN & SAN

Converged Solution

10Gb = Std. on x86 nodes

8Gb FC ($935 x 14 = $13,090)

SW upgrade to LOM
CNA ($99 x 14 = $1,386)

10G Fabric

EN4093 ($13,199 x 2 = $26,398)

CN4093 ($20,899* x 2 = $41,798)

FC Fabric

FC3171 ($9,899 x 2 = $19,798)

Included with CN4093

$59,286

$43,184

I/O

Acquisition Costs

* NTE price subject to change and confidential until announce

2012 IBM Corporation

System Networking

FCoE Protocol Components

FCoE Standards

Fibre Channel on Network

Media
www.t11.org
CN4093 supports
FC-BB-5 (FC-BB_E -FC over Ethernet)

DCB (Lossless Ethernet)

IEEE 802.1
PFC, ETS, DCBX

2012 IBM Corporation

System Networking

FCoE Network Layering and FCoE Frame

Based on FC Model

No FC Frame Fragmentation;
Jumbo frames
Source: IBM Redbooks
http://www.redbooks.ibm.com/redbooks/pdfs/sg247986.pdf
2012 IBM Corporation

System Networking

Basic Components of FCoE

CN4093

Full Fabric FC/FCoE Forwarder

Services Login Request FLOGIs
Consumes a Domain ID
Provides FC Function and Services

Zoning, Name Server, RSCN (Registered State Change Notification)

Also FSPF (not used in CN4093 Nov. Release)

Forwarding based on FC Domain IDs

FC/FCoE Encap/Decap (optional)
Source: IBM Redbooks
http://www.redbooks.ibm.com/redbooks/pdfs/sg247986.pdf
2012 IBM Corporation

System Networking

CN4093 in Full Fabric mode (End-to-End FCoE)

Key value:
Directly connects to internal storage node via midplane
in FCoE mode
Integrated Storage and Compute

Black line is FCOE

Green line is Ethernet

Ethernet Network
Ex: G8264/G8316

LOM or CN4054
(pNIC, vNIC
mode)

Storage node

CN4093 Flex System

Fabric Converged
Switch

CN4093 in Full Fabric

2012 IBM Corporation

System Networking

CN4093 in Full Fabric mode (End-to-End FCoE)

Black line is FCOE

Green line is Ethernet

Ethernet Network
Ex G8264/G8316

Ext.
FCoE
Storage

Key value:
Reduce cost and complexity of external SAN network
removes Cisco and Brocade SAN
Ideal for new storage deployments
Removes need for FC cables and network all together
LOM or CN4054
(pNIC, vNIC
mode)

CN4093 Flex System

Fabric Converged
Switch
Adapter

Integrated Switch

Storage Target

OS levels

LOM & CN4054 4-port adapter (BE3)

pNIC and vNIC modes

CN4093 10Gb Switch

Storwise V7K

Win2008, ESX 4/5, RHEL 5/6, SLES 10/11

2012 IBM Corporation

System Networking

FCoE Protocol Control and Data Plane

FIP (FCoE Initialization Protocol)
Control Plane
EtherType -- 0x8914
Discover FC Entities in Ethernet cloud
+ FC Login Services
MAC == Burned In Address of CNA

FCoE
Data Plane
EtherType -- 0x8906
Carries ALL SCSI traffic ; most of the
FC frames
MAC == FPMA
Destination MAC Address
Source MAC Address
(IEEE 802.1Q Tag)
ET = FCoE
Ver
Reserved
Reserved
Reserved
Reserved
SOF
Encapsulated FC Frame (minimum of 28 Bytes)
(including FC-CRC)
EOF
Reserved (Pad Bytes)
Ethernet FCS
2012 IBM Corporation

System Networking

FCoE is Fibre Channel on Ethernet Transport

CNA

FCF

Not in
CN4093

2012 IBM Corporation

System Networking

FCoE is Fibre Channel on Ethernet Transport

SAN
SAN

SAN
SAN

FCoE HBA

FCoE HBA

FCoE HBA

FCoE HBA

This is
harder! Enet Switch

FCoE HBA

FCoE HBA

to here FCoE Switch

is easy!
FCoE HBA

FC HBA

FC HBA

FC HBA

FCoE Switch

FCoE HBA

Going
from
here

FC Switch
FC HBA

SAN
SAN

This is the config we want

Still have a one to one direct connection

between host ports and switch ports
CEE Switch capable of FCoE is known
as a Fibre Channel Forwarder (FCF) in
the standard

No longer a one to one direct

connection
Have to discover FCoE switches (FCFs)
They arent just at the other end of the link any more
MAC Addresses

IBM Internal Use Only

Source: Dan Eisenhauer and Roger Hathorn

23

2012 IBM Corporation

System Networking

FIP and FCoE

Enode
VLAN
Discovery

FCF (FC-BB-5)
VLAN
Discovery

Default VLAN; ALL-FCF-MACs

FCF
Discovery

FIP
FCF
Discovery

Solicitations/ Advertisement;

FLOGI and
FDISC

FLOGI and
FDISC
Accept

FC Command

FC Command
Response

(FCoE
Initialization
Protocol)

FCoE Protocol

2012 IBM Corporation

System Networking

FSB (FIP Snooping Bridge)

FIP Snooping Bridge
Snoops FIP messages (VLAN Disc, FCF Disc and FLOGI)
Secure host-FCF path with dynamic ACL

Server1 FCoCEE
CNA

VNIC
VHBA

SA=MAC2, DA=FCF_MAC, Type=FCoE_Type, permit;

SA = {FCFs}, deny;
DA = ALL_FCF_MACs, Type = FIP_Type, permit;
DA = {FCFs}, Type = FIP_Type, permit;
Type = FIP_Type, deny;
Type = FCoE_Type, deny;
SApre = FC-MAP, deny; for FPMA fabric
Any non-FCoE related ACEs.

MAC1
MAC2

Lossless
Enet Bridge

FCF MAC

FCF

2012 IBM Corporation

System Networking

Full Fabric FC/FCoE Switch (End-to-End FCoE)

CN4093 in Full
Fabric Mode
Black line is FCOE
Green line is Ethernet

Ethernet Network
Ex: G8264/G8316

LOM or CN4054
(pNIC, vNIC mode)

Storage node

Full Fabric FC/FCoE Switch

Services Login Request FLOGIs
Consumes a Domain ID
Provides FC Function and Services

Zoning, Name Server, RSCN (Registered State Change Notification)

Forwarding based on FC Domain IDs

FC/FCoE Encap/Decap (optional)
2012 IBM Corporation

System Networking

NPV Gateway
Ethernet
Ports

Host

Storage

CEE/FSB

CEE/FSB

FC
Ports

FC
Ports
F

52 F
ETH

F
FCisco MDS/
Brocade
F SAN

CN4093 53 NP
NPV
Gateway
Mode 64 NP

Storage

FC

NPV - N_Port Virtualization (Extend port density of the Full Fabric Switch)

Reduces port requirements on expensive SAN switch

SAN switch agnostic; To the FC switch, the gateway NP_Ports appear to be native
FC N_Ports that are capable of performing NPIV

FC/FCoE Gateway

Simple FC/FCoE Frame translation 1:1 frame mapping

Stateless
Low cost
2012 IBM Corporation

System Networking

NPV Gateway
Enode

NPV
FLOGI

FC

FC Switch
FLOGI
Accept

VLAN
Discovery

VLAN
Discovery
Default VLAN; ALL-FCF-MACs

FIP
(FCoE
Initialization
Protocol)

FCF
Discovery

FCF
Discovery
Solicitations/ Advertisement;
FLOGI and
FDISC

Send FDISC
to FCF

FDISC
Accept

Relay FC
DISC via FIP

FC Command
FCoE / FC
Encap/Decap

FC Command
Response

FCoE
Protocol
2012 IBM Corporation

System Networking

CN4093 in NPV Gateway

Key value:
Acquisition costs
Reduce number of director class SAN switch
ports

Ext.
Storage

Cisco or Brocade
SAN

Ethernet Network
Ex. G8264/G8316

Leverage existing external SAN infrastructure

Black line is FCOE
Red line is Fibre Channel
Green line is Ethernet

LOM or CN4054
(pNIC, vNIC
mode)

CN4093 Flex System

Fabric Converged
Switch
Adapter

Integrated Switch

FCoE Top Of Rack

Switch

SAN Switch

Storage Target

OS levels

LOM & CN4054 4-port

adapter (BE3) pNIC
and vNIC modes

CN4093 10Gb Switch

NPIV mode

N/A

Cisco SAN
Brocade SAN

Storwise V7K, FC:

SVC, DS3K/5K, DS8K,
Tape, XIV

Win2008, ESX 4/5,

RHEL 5/6, SLES 10/11

2012 IBM Corporation

System Networking

CN4093 in NPV Gateway

Key value:
More control of the overall solutions and leverage
existing storage
Directly connects to internal storage node via midplane
in FCoE mode
Can connect to external storage via SAN switch also.
Black line is FCOE
Red line is Fibre Channel
Green line is Ethernet

Ext.
Storage

Cisco or Brocade
SAN

Ethernet Network
Ex: G8264/G8316

LOM or CN4054
(pNIC, vNIC
mode)

Storage node

CN4093 Flex System

Fabric Converged
Switch
Adapter

Integrated Switch

FCoE Top Of Rack

Switch

SAN Switch

Storage Target

OS levels

LOM & CN4054 4-port

adapter (BE3) pNIC
and vNIC modes

CN4093 10Gb Switch

NPIV mode

N/A

Cisco SAN
Brocade SAN

FCoE: Storage V7000 node

FC: Storwise V7000, SVC,
DS3K/5K, DS8K, Tape, XIV

Win2008, ESX 4/5,

RHEL 5/6, SLES 10/11

2012 IBM Corporation

System Networking

CN4093 in NPV Gateway (Power Compute Node)

Key value:
Integrated FCoE on compute node
Directly connects to internal storage node via midplane
in FCoE mode
Can connect to external storage via SAN switch also.
Black line is FCOE
Red line is Fibre Channel
Green line is Ethernet

Ext.
Storage

Cisco or Brocade
SAN

Ethernet Network
Ex. G8264/G8316

CN4058 Adapter

Storage node

CN4093 Flex System

Fabric Converged
Switch
Adapter

Integrated Switch

Too Of Rack
Switch

SAN Switch

Storage Target

OS levels

IBM Flex System

CN4058 8-port 10Gb
Converged Adapter

CN4093 10Gb
Switch
NPIV mode

NA

Cisco & Brocade

SAN

FCoE: V7000 node

FC: Storwise V7000,
SVC, DS3K/5K,
Tape, XIV

AIX 5/6/7
VIOS 221/222
IBM i6/i7
2012 IBM Corporation

System Networking

FSB, NPV and Full Fabric FC/FCoE Switch

Features

FSB

NPV Gateway

Full Fabric FC/FCoE

Switch

FCoE End Device

Connectivity

(F)

(F)

FC Connectivity

(NP) and
(NP-F)*

(F)*

FCoE/FC Gateway
(Encap/Decap)

(FIP ACL)

(FIP ACL, Zoning)

Security

(FIP ACL)

Zoning

S_ID/D_ID Routing

* == Not available in Nov. 2012

2012 IBM Corporation

System Networking

FC/FCoE Features and Scaling

NPV Gateway
FC Port Speeds: 4/8 Gb
Bridging to Fibre Channel SANs
- Login load distribution

End-to-end FCoE (initiator to target). FCoE

initiator /target can be attached to any port
that is configured as ethernet
Direct Attachment of FCoE targets.
Manageable via ISCLI / BBI

Scaling
Logins
- # of FCoE Logins/Port = 175
- # of FCoE Logins/Switch= 175

Full Fabric FC/FCoE Switch

FC Port Speeds: 4/8 Gb
FC-BB-5 Compliant Full Fabric FC/FCoE
- Fabric Services: Name Server, Registered State
Change Notification (RSCN), Login services,
Zoning
- WWN, FCID or Alias based Zoning
- Login load distribution

FC classes of service: Class 2 and Class 3

Manageable via ISCLI / BBI

Scaling
Zone
-

MAX ZONESETS
:4
MAX ZONES per ZONESET : 64
MAX MEMBERS per ZONE : 20
MAX ZONES
: 200
MAX ALIASES
: 200
MAX MEMBERS
: 1000

Logins
- # of FCoE Logins/Port = 175
- # of FCoE Logins/Switch= 175

2012 IBM Corporation

System Networking

FC-BB-5 vs. FC-BB-6 ( Not in Nov. 2012)

FC-BB-5

FC-BB-6

2012 IBM Corporation

System Networking

CN4093 Architecture (Not for Customer )

FCoE
Initiator

FCoE
Target

CNA

FCoE
Initiator

FC
Switch

CNA

12x
10GbE / 2/4/8GbFC

10x10GbE

CN4093 Architecture

FC ASIC

Ethernet ASIC

42x10GbE

FCoE
Initiator
CNA

FCoE
Target

Login Load Distribution

Throughput
Performance
2012 IBM Corporation

System Networking

References
T11 FC-BB-5
http://www.t11.org/ftp/t11/pub/fc/bb-5/09-056v5.pdf
IBM Redbook
http://www.redbooks.ibm.com/redbooks/pdfs/sg247986.pdf
Understanding NPIV and NPV Blog
http://blog.scottlowe.org/2009/11/27/understanding-npiv-and-npv/
http://www.juniper.net/techpubs/en_US/junos12.2/topics/concept/fibrechannel-fip-functions-understanding.html

2012 IBM Corporation

System Networking

BACKUP SLIDES

2012 IBM Corporation

System Networking

IBM Supported Interop Solutions

System Storage Interoperation Center

(SSIC)
http://www03.ibm.com/systems/support/storage/ss
ic/interoperability.wss
Provides End to End tested / supported
IBM FCoE and iSCSI solutions
Outlines all testing building blocks
Server Models
Operating System
Adapter (ASIC)
Connection Protocol
Storage
Supported Solution Requirement
All Building Blocks = Supported
One missing = Non Supported
Interop Reference Guides
Networking Interop Guide
BladeCenter Interop Guide

38

2012 IBM Corporation