Cisco Systems - Data Center Networking - Infrastructure Architecture

Data Center Networking: Infrastructure
Architecture
Solutions Reference Network Design
March, 2003
Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 526-4100
Customer Order Number: 956513

Text Part Number: 78-xxxxx-xx
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
CCIP, the Cisco Arrow logo, the Cisco Powered Network mark, the Cisco Systems Verified logo, Cisco Unity, Follow Me Browsing, FormShare, iQ Breakthrough, iQ Expertise,
iQ FastTrack, the iQ Logo, iQ Net Readiness Scorecard, Networking Academy, ScriptShare, SMARTnet, TransPath, and Voice LAN are trademarks of Cisco Systems, Inc.;
Changing the Way We Work, Live, Play, and Learn, Discover All That’s Possible, The Fastest Way to Increase Your Internet Quotient, and iQuick Study are service marks of Cisco
Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, the Cisco IOS logo,
Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherSwitch, Fast Step,
GigaStack, Internet Quotient, IOS, IP/TV, LightStream, MGX, MICA, the Networkers logo, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, RateMUX, Registrar,
SlideCast, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.
All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship
between Cisco and any other company. (0208R)
Data Center Networking: Infrastructure Architecture

Copyright © 2003, Cisco Systems, Inc.
All rights reserved.
C ON T E N T S
Preface vii
Target Audience vii
Document Organization vii
Document Conventions viii
Obtaining Documentation ix
World Wide Web ix
Documentation CD-ROM ix
Ordering Documentation ix
Documentation Feedback ix
Obtaining Technical Assistance x
Cisco.com x
Technical Assistance Center x
Cisco TAC Web Site xi
Cisco TAC Escalation Center xi
CHAPTER 1 Data Center Overview — Infrastructure Architecture 1-1

Benefits of Building Data Centers 1-1
Data Centers in the Enterprise 1-2
Data Center Architecture 1-3
Aggregation Layer 1-5
Front-End Layer 1-6
Application Layer 1-7
Back-End Layer 1-8
Storage Layer 1-8
Metro Transport Layer 1-8
Distributed Data Centers 1-9
Data Center Services 1-9
Infrastructure Services 1-10
Metro Services 1-10
Layer 2 Services 1-10
Layer 3 Services 1-11
Intelligent Network Services 1-11
Application Optimization Services 1-11
Storage Services 1-11

956513 iii
Contents
Security Services 1-12

Management Services 1-12
Summary 1-13
CHAPTER 2 Data Center Infrastructure 2-1

Data Center Infrastructure Topology Recommendations 2-2
Choosing the Spanning-Tree Algorithm 2-3
Routing in the Data Center 2-3
Predictable Convergence Time 2-4
System Components 2-5
Hardware 2-5
Aggregation Switches 2-7
Access Switches 2-8
Software 2-8
Aggregation Switches 2-8
Access Switches 2-8
Design Details 2-9
Layer 1 2-9
Service Appliances 2-9
Redundant Supervisors 2-9
Channel Configuration 2-10
Layer 2 2-11
VLAN Configuration 2-12
Access Switch Port Configuration 2-13
Trunk Configuration 2-14
Logical Ports 2-15
Spanning-Tree: 802.1w 2-16
Spanning-Tree: Rapid PVST+ 2-17
Protection From Loops 2-20
Layer 3 2-22
OSPF 2-24
EIGRP 2-27
VLAN Interfaces 2-28
HSRP 2-28
IBM Mainframes 2-29
Attachment Options 2-30
IP Addressing 2-31
OSPF Routing on the Mainframe 2-32
Sysplex 2-32

iv 956513
Contents
Summary 2-35
CHAPTER 3 Enhancing Server to Server Communications 3-1

Benefits 3-3
Restrictions 3-4
System Components 3-4
Hardware Requirements 3-4
Software Requirements 3-4
Features 3-5
Policy Based Routing 3-5
Default Next-Hop 3-5
Static Routes 3-6
Network Topology 3-6
Topology Description 3-6
Traffic Paths for Client-to-Server Traffic 3-8
Server-to-Client Traffic 3-8
Server-to-Server Traffic (Servers Directly Connected) 3-8
Server-to-Server Traffic (Remote Servers) 3-9
Configuration Description 3-9
Configuration Sequence and Tasks 3-9
Configuring the Server VLANs on the MSFC 3-9
Configuring the “MSFC-to-CSM” VLAN 3-10
Sample Configurations 3-10
Infrastructure Details 3-10
PBR Details 3-12
CSM1 Configuration 3-12
MSFC-AGG1 Configuration 3-13
MSFC-AGG2 Configuration 3-15
CHAPTER 4 Addressing Spanning-Tree Limitations 4-1

Spanning Tree Overview 4-1
Protocols Overview 4-2
Root 4-2
Designated Bridge 4-2
Root Port and Designated Ports 4-2
Forwarding and Blocking 4-2
BPDU Format 4-2
Hello Time and MaxAge 4-2
Port States 4-3

956513 v
Contents
Convergence Time 4-3

Aggressive Spanning-Tree Timers 4-4
Spanning Tree Limitations 4-4
A Simple Spanning-Tree Failure 4-4
Failure 1 4-4
Possible Reasons for Spanning Tree Instability 4-5
Possible Spanning-Tree Failures in the Aggregation and Access Layers 4-5
Failure 2 4-5
Failure 3 4-6
Failure 4 4-7
Design Guidelines 4-7
Loopguard 4-8
Examples 4-8
Automatic Recovery 4-9
Key Benefits 4-9
Interoperability with Other Features 4-9
PVST+ - Uplinkfast 4-9
PVST+ BackboneFast 4-9
Rapid PVST+ - 802.1w 4-9
Rootguard 4-9
PagP and LACP 4-10
UDLD 4-10
Implementation Details 4-10
Where to Enable Loopguard 4-10
How to Configure Loopguard 4-11
How to Test Loopguard 4-11
Summary and Recommendations 4-12
APPENDIX A Configurations 1
Aggregation1 - Rapid PVST+ - OSPF 2
Aggregation2 - Rapid PVST+ - OSPF 5
Access - Rapid PVST+ 8
Aggregation1 - MST - OSPF 9
Aggregation2 - MST - OSPF 12
Access - MST 15
INDEX

vi 956513
Preface
The convergence of voice and video in today’s enterprise networks has placed additional requirements
on the infrastructure of enterprise data centers. These data centers:
• House enterprise-wide servers
• Support critical application services
• Support traditional data services
• Require 24X7 support
These requirements are based on the applications supported rather than the size of the data center. The
process of selecting the proper data center hardware and software versions that meet the necessary Layer
2, Layer 3, QoS, and Multicast requirements can be a daunting task. This solutions reference network
design (SRND) provides design and implementation guidelines for building a redundant, scalable
enterprise data center.
Target Audience
This publication provides solution guidelines for enterprises implementing Data Centers with Cisco
devices. The intended audiences for this design guide include network architects, network managers, and
others concerned with the implementation of secure Data Center solutions, including:
• Cisco sales and support engineers
• Cisco partners
• Cisco customers
Document Organization
This document contains the following chapters
Table 1 Document Organization:
Chapter or Appendix Description

Chapter 1, “Data Center Overview — Provides an overview of the data center architecture
Infrastructure Architecture” including an overview of infrastructure services.
Chapter 2, “Data Center Infrastructure” Provides design recommendations for designing and
deploying a data center.

956513 vii
Preface
Document Conventions
Chapter or Appendix Description

Chapter 3, “Enhancing Server to Server Provides design and implementation guidance on how to
Communications” enhance server to server communications in the data center.
Chapter 4, “Addressing Spanning-Tree Provides an overview of the limitations of the spanning tree
Limitations” protocol and identifies work arounds to overcome those
limitations.
Appendix A, “Configurations” Provides all the configurations used for the data center
infrastructure design.
Document Conventions
This guide uses the following conventions to convey instructions and information:
Table 2 Document Conventions
Convention Description
boldface font Commands and keywords.
italic font Variables for which you supply values.
[ ] Keywords or arguments that appear within square brackets are optional.
{x | y | z} A choice of required keywords appears in braces separated by vertical bars. You must select one.
screen font Examples of information displayed on the screen.
boldface screen Examples of information you must enter.
font
< > Nonprinting characters, for example passwords, appear in angle brackets.
[ ] Default responses to system prompts appear in square brackets.
Note Means reader take note. Notes contain helpful suggestions or references to material not
covered in the manual.
Timesaver Means the described action saves time. You can save time by performing the action
described in the paragraph.
Tips Means the following information will help you solve a problem. The tips information might
not be troubleshooting or even an action, but could be useful information, similar to a
Timesaver.
Caution Means reader be careful. In this situation, you might do something that could result in
equipment damage or loss of data.

viii 956513
Preface
Obtaining Documentation
Obtaining Documentation
The following sections explain how to obtain documentation from Cisco Systems.
World Wide Web

You can access the most current Cisco documentation on the World Wide Web at the following URL:
Translated documentation is available at the following URL:
http://www.cisco.com/public/countries_languages.shtml
Documentation CD-ROM
Cisco documentation and additional literature are available in a Cisco Documentation CD-ROM
package, which is shipped with your product. The Documentation CD-ROM is updated monthly and may
be more current than printed documentation. The CD-ROM package is available as a single unit or
through an annual subscription.
Ordering Documentation
Cisco documentation is available in the following ways:
• Registered Cisco Direct Customers can order Cisco product documentation from the Networking
Products MarketPlace:
http://www.cisco.com/cgi-bin/order/order_root.pl
• Registered Cisco.com users can order the Documentation CD-ROM through the online Subscription
Store:
http://www.cisco.com/go/subscription
• Nonregistered Cisco.com users can order documentation through a local account representative by
calling Cisco corporate headquarters (California, USA) at 408 526-7208 or, elsewhere in North
America, by calling 800 553-NETS (6387).
Documentation Feedback
If you are reading Cisco product documentation on Cisco.com, you can submit technical comments
electronically. Click Leave Feedback at the bottom of the Cisco Documentation home page. After you
complete the form, print it out and fax it to Cisco at 408 527-0730.
You can e-mail your comments to bug-doc@cisco.com.

956513 ix
Preface
Obtaining Technical Assistance
To submit your comments by mail, use the response card behind the front cover of your document, or
write to the following address:
Cisco Systems
Attn: Document Resource Connection
170 West Tasman Drive
San Jose, CA 95134-9883
We appreciate your comments.

Cisco provides Cisco.com as a starting point for all technical assistance. Customers and partners can
obtain documentation, troubleshooting tips, and sample configurations from online tools by using the
Cisco Technical Assistance Center (TAC) Web Site. Cisco.com registered users have complete access to
the technical support resources on the Cisco TAC Web Site.
Cisco.com
Cisco.com is the foundation of a suite of interactive, networked services that provides immediate, open
access to Cisco information, networking solutions, services, programs, and resources at any time, from
anywhere in the world.
Cisco.com is a highly integrated Internet application and a powerful, easy-to-use tool that provides a
broad range of features and services to help you to
• Streamline business processes and improve productivity
• Resolve technical issues with online support
• Download and test software packages
• Order Cisco learning materials and merchandise
• Register for online skill assessment, training, and certification programs
You can self-register on Cisco.com to obtain customized information and service. To access Cisco.com,
go to the following URL:
Technical Assistance Center

The Cisco TAC is available to all customers who need technical assistance with a Cisco product,
technology, or solution. Two types of support are available through the Cisco TAC: the Cisco TAC
Web Site and the Cisco TAC Escalation Center.
Inquiries to Cisco TAC are categorized according to the urgency of the issue:
• Priority level 4 (P4)—You need information or assistance concerning Cisco product capabilities,
product installation, or basic product configuration.
• Priority level 3 (P3)—Your network performance is degraded. Network functionality is noticeably
impaired, but most business operations continue.

x 956513
Preface
• Priority level 2 (P2)—Your production network is severely degraded, affecting significant aspects
of business operations. No workaround is available.
• Priority level 1 (P1)—Your production network is down, and a critical impact to business operations
will occur if service is not restored quickly. No workaround is available.
Which Cisco TAC resource you choose is based on the priority of the problem and the conditions of
service contracts, when applicable.
Cisco TAC Web Site

The Cisco TAC Web Site allows you to resolve P3 and P4 issues yourself, saving both cost and time. The
site provides around-the-clock access to online tools, knowledge bases, and software. To access the
Cisco TAC Web Site, go to the following URL:
http://www.cisco.com/tac
All customers, partners, and resellers who have a valid Cisco services contract have complete access to
the technical support resources on the Cisco TAC Web Site. The Cisco TAC Web Site requires a
Cisco.com login ID and password. If you have a valid service contract but do not have a login ID or
password, go to the following URL to register:
http://www.cisco.com/register/
If you cannot resolve your technical issues by using the Cisco TAC Web Site, and you are a Cisco.com
registered user, you can open a case online by using the TAC Case Open tool at the following URL:
http://www.cisco.com/tac/caseopen
If you have Internet access, it is recommended that you open P3 and P4 cases through the Cisco TAC
Web Site.
Cisco TAC Escalation Center

The Cisco TAC Escalation Center addresses issues that are classified as priority level 1 or priority
level 2; these classifications are assigned when severe network degradation significantly impacts
business operations. When you contact the TAC Escalation Center with a P1 or P2 problem, a Cisco TAC
engineer will automatically open a case.
To obtain a directory of toll-free Cisco TAC telephone numbers for your country, go to the following
URL:
http://www.cisco.com/warp/public/687/Directory/DirTAC.shtml
Before calling, please check with your network operations center to determine the level of Cisco support
services to which your company is entitled; for example, SMARTnet, SMARTnet Onsite, or Network
Supported Accounts (NSA). In addition, please have available your service agreement number and your
product serial number.

956513 xi
Preface

xii 956513
C H A P T E R 1
Data Center Overview — Infrastructure
Architecture
Data Centers, according to the report from the Renewable Energy Policy Project on Energy Smart Data
Centers, are “an essential component of the infrastructure supporting the Internet and the digital
commerce and electronic communication sector. Continued growth of these sectors requires a reliable
infrastructure because … interruptions in digital services can have significant economic consequences”.
According to the META Group, the average cost of an hour of downtime is estimated at $330,000.
Strategic Research Corporation reports the financial impact of major outages is equivalent to US$6.5
million per hour for a brokerage operation, or US$2.6 million per hour for a credit-card sales
authorization system.
Virtually every Enterprise has a Data Center, yet not every Data Center is designed to provide the proper
levels of redundancy, scalability, and security. A Data Center design lacking in any of these areas is at
some point going to fail to provide the expected services levels. Data Center downtime means the
consumers of the information are not able to access it thus the Enterprise is not able to conduct business
as usual.
Benefits of Building Data Centers

You can summarize the benefits of a Data Center in one sentence. Data Centers enable the consolidation
of critical computing resources in controlled environments, under centralized management, that permit
Enterprises to operate around the clock or according to their business needs. All Data Center services
are expected to operate around the clock. When critical business applications are not available, the
business is severely impacted and, depending on the outage, the company could cease to operate.
Building and operating Data Centers requires extensive planning. You should focus the planning efforts
on those service areas you are supporting. High availability, scalability, security, and management
strategies ought to be clear and explicitly defined to support the business requirements. Often times,
however, the benefits of building Data Centers that satisfy such lists of requirements are better realized
when the data center fails to operate as expected.
The loss of access to critical data is quantifiable and impacts the bottom line: revenue. There are a
number of organizations that must address plans for business continuity by law, which include federal
government agencies, financial institutions, healthcare and utilities. Because of the devastating effects
of loss of data or access to data, all companies are compelled to look at reducing the risk and minimizing
the impact on the business. A significant portion of these plans is focused on Data Centers where critical
business computing resources are kept. Understanding the impact of a Data Center failure in your
Enterprise is essential. The following section introduces the Data Center role in the Enterprise network.

956513 1-1
Chapter 1 Data Center Overview — Infrastructure Architecture
Data Centers in the Enterprise
Data Centers in the Enterprise

Figure 1-1 presents the different building blocks used in the typical Enterprise network and illustrates
the location of the Data Center within that architecture.
Figure 1-1 Enterprise Network Infrastructure
Internet
PSTN SP1 SP2 VPN Partners
Data Center
DMZ
AAA Internet server farm

Internet edge
RPMS Core
switches
Remote access
Extranet server farm
Private WAN
Intranet server farm

76435
Campus
The building blocks of the typical Enterprise network include:

• Campus

1-2 956513
Data Center Architecture
• Private WAN
• Remote Access
• Internet server farm
• Extranet server farm
• Intranet server farm
Data Centers house many network infrastructure components that support the Enterprise network
building blocks shown inFigure 1-1, such as the core switches of the Campus network or the edge routers
of the Private WAN. Data Center designs however, include at least one type of server farm. These server
farms may or may not be built as separate physical entities, depending on the business requirements of
the Enterprise. For example, a single Data Center may use a shared infrastructure, resources such as
servers, firewalls, routers, switches, etc., for multiple server farm types. Other Data Centers may require
that the infrastructure for server farms be physically dedicated. Enterprises make these choices
according to business drivers and their own particular needs. Once made, the best design practices
presented in this paper and subsequent design documents can be used to design and deploy a highly
available, scalable, and secured Data Center.

The architecture of Enterprise Data Centers is determined by the business requirements, the application
requirements, and the traffic load. These dictate the extent of the Data Center services offered, which
translates into the actual design of the architecture. You must translate business requirements to specific
goals that drive the detailed design. There are four key design criteria used in this translation process
that help you produce design goals. These criteria are: availability, scalability, security, and
management. Figure 1-2 shows the design criteria with respect to the Data Center architecture:
Figure 1-2 Architecture Layers and Design Criteria
y bility
ilit i l ity ty e a
a b b r i g
vail c ala ecu ana
A S S M
Aggregation Layer
Front-end Layer
Application Layer
76443
Back-end Layer
Storage Layer
Metro Transport Layer

956513 1-3
The purpose of using availability, scalability, security, and manageability as the design criteria is to
determine what each layer of the architecture needs to meet the specific criteria. For instance, the answer
to the question “how scalable the aggregation layer should be?” is driven by the business goals but is
actually achieved by the Data Center design. Since the answer depends on which functions the
aggregation layer performs, it is essential to understand what each layer does.
Your design goals and the services supported by the Data Center dictate the network infrastructure
required. Figure 1-3 introduces the Data Center reference architecture.
Figure 1-3 Data Center Architecture
Campus Internet
Campus
core edge
Distribution Aggregation layer
Access Front-end layer
Access Application layer
Access Back-end layer
DWDM
FC
Storage layer
76447

1-4 956513
The architecture presents a layered approach to the Data Center design that supports N-Tier applications
yet it includes other components related to other business trends. The layers of the architecture include:
• Aggregation
• Front-end
• Application
• Back-end
• Storage
• Metro Transport
Note The metro transport layer supports the metropolitan high-speed connectivity needs between distributed
data centers.
The following sections provide a detailed description of these layers.
Aggregation Layer
The aggregation layer provides network connectivity between the server farms and the rest of the
Enterprise network, provides network connectivity for Data Center service devices, and supports
fundamental Layer 2 and Layer 3 functions. The aggregation layer is analogous to the campus network
distribution layer. Data Center services that are common to servers in the front-end or other layers should
be centrally located in the aggregation layer for predictability, consistency, and manageability. In
addition to the multilayer switches (aggregation switches) that provide the Layer 2 and Layer 3
functionality, the aggregation layer includes, content switches, firewalls, IDSs, content engines, and SSL
offloaders, as depicted in Figure 1-4.

956513 1-5
Figure 1-4 Aggregation Layer
Campus Internet
edge
Campus
core
Multilayer switches: L2-L5

Aggregation layer
Firewalls
Layer 3
Content engines
SSL offloading
Intrusion detection system
Front-end layer
Layer 2
76444
Front-End Layer
The front-end layer, analogous to the Campus access layer in its functionality, provides connectivity to
the first tier of servers of the server farms. The front-end server farms typically include FTP, Telnet,
TN3270, SMTP, Web servers, and other business application servers, in addition to network-based
application servers, such as IPTV Broadcast servers, Content Distribution Managers, and Call Managers.
Specific features, such as Multicast and QoS that may be required, depend on the servers and their
functions. For example, if live video streaming over IP is supported, multicast must be enabled; or if
voice over IP is supported, QoS must be enabled. Layer 2 connectivity through VLANs is required
between servers supporting the same application services for redundancy (dual homed servers on
different Layer 2 switches), and between server and service devices such as content switches. Other
requirements may call for the use of IDSs or Host IDSs to detect intruders or PVLANs to segregate
servers in the same subnet from each other.

1-6 956513
Application Layer
The application layer provides connectivity to the servers supporting the business logic, which are all
grouped under the application servers tag. Applications servers run a portion of the software used by
business applications and provide the communication logic between front-end and the back-end, which
is typically referred to as the middleware or business logic. Application servers translate user requests
to commands the back-end database systems understand.
The features required at this layer are almost identical to those needed in the front-end layer. Yet,
additional security is typically used to tighten security between servers that face users and the next layer
of servers, which implies firewalls in between. Additional IDS may also be deployed to monitor different
kinds of traffic types. Additional services may require load balancing between the web and application
servers typically based on Layer 5 information, or SSL if the server-to-server communication is done
over SSL. Figure 1-5 introduces the front-end, application, and back-end layers in a logical topology.
Figure 1-5 Front-End, Application, and Back-End Layers
Aggregation
layer
Front-end
Layer 2 switches
Layer 2
Web and client

facing servers
Application
Firewalls
Layer 2 Layer 2 switches

Application servers
Back-end
Firewalls
Layer 2 Layer 2 switches

Database servers
76445

956513 1-7
Back-End Layer
The back-end layer provides connectivity to the database servers. The feature requirements of this layer
are almost identical to those of the application layer, yet the security considerations are more stringent
and aimed at protecting the Enterprise data. The back-end layer is primarily for the relational database
systems that provide the mechanisms to access the enterprise's information, which makes them highly
critical. The hardware supporting the relational database systems range from medium sized servers to
mainframes, some with locally attached disks and others with separate storage.
Storage Layer
The storage layer connects devices in the storage network using Fibre-Channel (FC) or iSCSI. The
connectivity provided through FC switches is used for storage-to-storage communications between
devices such as FC attached server and disk subsystems of tape units. iSCSI provides SCSI connectivity
to servers over an IP network and is supported by iSCSI routers, port adaptors, and IP services modules.
FC is typically used for block level access, whereas iSCSI is used for file level access.

The metro transport layer is used to provide a high speed connection between distributed data centers.
These distributed Data Centers use metro optical technology to provide transparent transport media,
which is typically used for database or storage mirroring and replication. This metro transport
technology is also used for high speed campus-to-campus connectivity.
The high speed connectivity needs are either for synchronous or asynchronous communications, which
depends on the recovery time expected when the primary data location fails. Disaster recovery and
business continuance plans are the most common business driver behind the need for distributed data
centers and the connectivity between them. Figure 1-6 presents a closer look to the logical view of the
layer between the back-end and the metro transport.

1-8 956513
Data Center Services
Figure 1-6 Metro Transport Topology
Primary Data Center Distributed Data Center

Back-end layer Back-end layer
GE GE
Storage layer Metro Transport Layer Fibre channel Storage layer

Fibre channel
switch switch
FC FC
ONS 15xxx ONS 15xxx

FC FC
ESCON ESCON
ESCON
76446
Distributed Data Centers
Distributed Data Centers provide redundancy for business applications. The primary Enterprise Data
Center is a single point of failure when dealing with disasters. This could lead to application downtime
leading to loss in productivity and lost business. Addressing this potentially high impact risk requires
that the data is replicated at a remote location that acts as a backup or recovery site, the distributed data
center, when the primary site is not longer operating.
The distributed Data Center, typically a smaller replica of the primary data center, takes over the primary
data center responsibilities after a failure. With distributed Data Centers, data is replicated to the
distributed data center over the metro transport layer. The clients are directed to the distributed Data
Center when the primary Data Center is down. Distributed data centers reduce application down time for
mission critical applications and minimize data loss.

The Data Center is likely to support a number of services, which are the result of the application
environment requirements. These services include:
• Infrastructure: Layer 2, Layer 3, Intelligent Network Services and Data Center Transport
• Application Optimization Services: content switching, caching, SSL offloading, And content
transformation
• Storage: Consolidation of local disks, Network Attached Storage, Storage Area Networks

956513 1-9
• Security: Access Control Lists, Firewalls, and Intrusion Detection Systems

• Management: Management devices applied to the elements of the architecture
The following section introduces the services details and their associated components.
Infrastructure Services
Infrastructure services include all core features needed for the Data Center infrastructure to function and
serve as the foundation for all other Data Center services. The infrastructure features are organized as
follows:
• Metro
• Layer 2
• Layer 3
• Intelligent Network Services
Metro Services
Metro services include a number of physical media access, such as Fibre-Channel and iSCSI, and metro
transport technologies such as Dense Wave Division Multiplexing (DWDM), Coarse Wave Division
Multiplexing (CWDM), SONET and 10GE. Metro transport technologies enable campus-to-campus and
distributed data centers connectivity for a number of applications that require high bandwidth and low
predictable delay. For instance, DWDM technology provides physical connectivity for a number of
different physical media such as Gigabit Ethernet, ATM, Fibre Channel, and ESCON concurrently. Some
instances where this connectivity is required are for long-haul Storage Area Networks (SAN) extension
over SONET or IP and short-haul SAN extension over DWDM/CWDM, SONET, or IP (Ethernet).
Layer 2 Services
Layer 2 services support the Layer 2 adjacency between the server farms and the service devices, enable
media access, provide transport technologies, and support a fast convergence, loop free, predictable, and
scalable Layer 2 domain. In addition to LAN media access, such as Gigabit Ethernet, and ATM; there is
support for Packet over SONET (PoS), and IP over Optical media. Layer 2 domain features ensure the
Spanning Tree Protocol (STP) convergence time for deterministic topologies is in the single digit
seconds and that the failover and fallback scenarios are predictable. The list of features includes:
• 802.1s + 802.1w (Multiple Spanning-Tree)
• PVST+802.1w (Rapid Per VLAN Spanning-Tree)
• 802.3ad (Link Aggregate Control Protocol)
• 802.1q (trunking)
• LoopGuard
• Uni-Directional Link Detection (UDLD)
• Broadcast Suppression

1-10 956513
Layer 3 Services
Layer 3 services enable fast convergence and a resilient routed network, including redundancy, for basic
Layer 3 services, such as default gateway support. The purpose is to maintain a highly available Layer
3 environment in the Data Center where the network operation is predictable under normal and failure
conditions. The list of available features includes:
• Static routing
• Border Gateway Protocol (BGP)
• Interior Gateway Protocols (IGPs): OSPF and EIGRP
• HSRP, MHSRP & VRRP
Intelligent Network Services

Intelligent network services include a number of features that enable applications services network wide.
The most common features are QoS and Multicast. Yet there are other important intelligent network
services, such as Private VLANs (PVLANs) and Policy Based Routing (PBR). These features enable
applications, such as live or on demand video streaming and IP telephony, in addition to the classic set
of enterprise applications. QoS in the Data Center is important for two reasons: marking, at the source,
application traffic and port based rate limiting capabilities that enforces a proper QoS service class as
traffic leaves the server farms. Multicast in the Data Center enables the capabilities needed to reach
multiple users concurrently or servers to receive information concurrently (cluster protocols).
For more information on infrastructure services in the data center, see the Data Center Networking:
Infrastructure Architecture SRND.
Application Optimization Services

Application optimization services, which is the focus of this SRND, include a number of features that
provide intelligence to the server farms. These features permit the scaling of applications supported by
the server farms and packet inspection beyond Layer 3 (Layer 4 or Layer 5).
The application services are:
• Server load balancing or content switching
• Caching
• SSL offloading
For more information about application optimization services, see the Data Center Networking:
Optimizing Server and Application Environments SRND.
Storage Services
Storage services include the storage network connectivity required for user-to-server and
storage-to-storage transactions. The major features could be classified in the following categories:
• Network Attached Storage (NAS)
• Storage Area Networks (SAN) to IP: Fibre Channel and SCSI over IP
• Localized SAN fabric connectivity (Fibre Channel or iSCSI)
• Fibre Channel to iSCSI Fan-out

956513 1-11
Storage consolidation leads to NAS and SAN environments. NAS relies on the IP infrastructure and, in
particular, features such as QoS to ensure the proper file over the IP network to the NAS servers. SAN
environments, commonly found in Data Centers, use Fibre Channel (FC) to connect servers to the
storage device and to transmit SCSI commands between them. The SAN environments need to be
accessible to the NAS and the larger IP Network.
FC over IP (FCIP) and SCSI over IP (iSCSI) are the emerging EITF standards that enable SCSI access
and connectivity over IP. The transport of SCSI commands over IP enables storage-to-IP and
storage-to-storage over an IP infrastructure.
SAN environments remain prevalent in Data Center environment, thus the localized SAN fabric becomes
important to permit storage-to-storage block access communication at Fibre Channel speeds. There are
other features focused on enabling FC to iSCSI fan-out for both storage-to-IP and storage-to-storage
interconnects.
Security Services
Security services include a number of tools used in the application environment to increase security. The
approach to security services in server farm environments is the result of increasing external threats but
also internal attacks. This creates the need to have a tight security perimeter around the server farms and
a plan to keep the security policies applied in a manner consistent with the risk and impact if the
Enterprise data was compromised. Since different portion of the Enterprise's data is kept at different tiers
in the architecture, it is important to consider deploying security between tiers so that the specific tier
has its own protection mechanisms according to likely risks.
Utilizing a layered security architecture provides a scalable modular approach to deploying security for
the multiple data center tiers. The layered architecture makes use of the various security services and
features to enhance security. The goal of deploying each of these security features and services is to
mitigate against threats, such as:
• Unauthorized access • Denial of Service

• Network reconnaissance • Viruses and worms
• IP spoofing • Layer 2 attacks
The security services offered in the data center include: Access Control Lists (ACLs), Firewalls,
Intrusion Detection Systems (IDS, Host IDS), Authentication, Authorization and Accounting (AAA)
mechanisms, and a number of other services that increase security in the data center. For more
information on security services in the data center, see the Data Center Networking: Securing Server
Farms SRND.
Management Services
Management services, also discussed in this SRND, refer to the ability to manage the network
infrastructure that provides the support of all other services in the data center. The management of
services in the Data Center include service provisioning, which depending on the specific service,
requires its own set of management considerations. Each service is also likely supported by different
organizational entities or even by distinct functional groups whose expertise is in the provisioning,
monitoring, and troubleshooting of such service.

1-12 956513
Summary
Cisco recommends that you have a network management policy in place that follows a consistent and
comprehensive approach to managing Data Center services. Cisco follows the FCAPS OSI management
standard and uses its management categories to provide management functionality. FCAPS is a model
commonly used in defining network management functions and their role in a managed network
infrastructure. The management features focus on the following categories:
• Fault management
• Configuration management
• Accounting management
• Performance management
• Security management
For more information on management services, see the Data Center Networking: Optimizing Server and
Application Environments SRND.
Summary
The business requirements drive the application requirements, which in turn drive Data Center design
requirements. The design process must take into account the current trends in application environments,
such as the N-Tier model, to determine application requirements. Once application requirements are
clear, the Data Center architecture needs to be qualified to ensure that its objectives are met and that
application requirements are met.
A recommendation to the Data Center design process is that you consider the layers of the architecture
that you need to support, given your specific applications, as the cornerstone of the services that you
need to provide. These services must meet your objectives and must follow a simple set of design criteria
to achieve those objectives. The design criteria include high availability, scalability, security, and
management, which all together focus the design on the Data Center services.
Achieving your design goals translates to satisfying your application requirements and ultimately
attaining your business objectives. Ensure that the Data Center design lets you achieve your current
objectives, particularly as they relate to your mission critical applications. Knowing you can, enables
you to minimize the business impact, as you would have quantified how resilient your Enterprise is to
the always dynamic business conditions.

956513 1-13
Summary

1-14 956513
C H A P T E R 2
Data Center Infrastructure
This chapter describes how to design and deploy the infrastructure of an enterprise data center. The
following are the design criteria:
• High Availability
– Avoid single point of failure
– Achieve fast and predictable convergence times
• Scalability
– Changes and additions without major changes to the infrastructure
– Predictable service scalability: can easily add new services
– The server farm must support up to 3000 dual-homed servers
• Simplicity
– Predictable traffic paths in steady and failover states
– Explicitly defined primary and backup traffic paths
• Must provide port density, and Layer 2 and Layer 3 connectivity
• Must support security services provided by ACLs, firewalls and IDSs
• Must support server farm services such as content switching, caching, SSL offloading
• Must integrate with mainframes and support mainframe services (TN3270, load balancing and SSL
offloading)
• Must integrate with multi-tier server farms
Keeping the above criteria in mind and following the design practices in this chapter helps ensure that
your data center infrastructure is able to meet the desired service level expectations. While the data
center infrastructure must be scalable and highly available, it should be simple to operate, troubleshoot,
and easily accommodate new demands. Given the critical nature of the data center services, a closer look
at the issues at each layer is important. For a high level overview of enterprise data centers, see
Chapter 1, “Data Center Overview - Infrastructure Architecture.”
By following the principles in this design guide, you can accommodate all the elements that pertain to
specific services, such as the server farms security services, with minimal changes. The focus of the
chapter is on how to design the Layer 2 and Layer 3 infrastructure, while keeping in mind the
requirements of the appliances that provide additional data center services.

956513 2-1
Chapter 2 Data Center Infrastructure
Data Center Infrastructure Topology Recommendations

Figure 2-1 presents an overview of the basic data center infrastructure. The data center architecture
follows the proven Cisco multi-layer design. The core of the network connects to the aggregation
switches with Layer 3 links. The aggregation switches, in turn, connect to the front-end switches.
Front-end switches provide direct connectivity to the server farms; therefore, they have direct impact on
the size of the server farm as they control the port density.
The aggregation switches serve two purposes: traffic aggregation and distribution between the server
farm and the rest of the network and as the aggregation point for service appliances (content switches,
firewalls, SSL offloaders). The front-end switches make the bulk of the Layer 2 domain and provide
Layer 2 connectivity to the aggregation switches.
Note The aggregation and front-end switches are the same as the distribution and access layers in the Cisco
multi-layer design.
Figure 2-1 Basic Data Center Infrastructure
Enterprise
Campus Core
Aggregation Layer
Mainframe
Front-end Layer Servers Servers

87436
Content Switch Firewall SSL

Cache Site IDS
offloader Sensor
Selector

2-2 956513
Each layer and each service device provides a specific function. Connecting servers to the aggregation
switches is possible but not optimal. Servers connected to the aggregation switches prevent scalability
and growth. Furthermore, once you reach the port capacity of the aggregation switches, new servers
force a major network overhaul with potential downtime when you deploy new higher density switches
or when you re-connect the servers to a new set of front-end switches.
Mainframes connect to the infrastructure via one or more optical server adapters (OSA) cards. If the
mainframe uses Enterprise System Connections (ESCON), it can be connected to a router with a Channel
Interface Processor (CIP/CPA). The CIP connects to the mainframes at the channel level. By using an
ESCON director, multiple hosts can share the same CIP router. Figure 2-1 shows the attachment for a
mainframe with an OSA card.
For the purpose of this design chapter, the transport protocol for mainframe applications is IP. You have
the choice of providing clients with direct access to the mainframe or you can build a multi-tiered
environment so clients can use browsers to run mainframe applications. The network not only provides
port density and Layer 3 services, but can also provide the TN3270 service from a CIP/CPA card. The
TN3270 can also be part of a multi-tiered architecture where the end client sends HTTP requests to web
servers, which, in turn, communicate with the TN3270 server. You must build the infrastructure to
accommodate these requirements as well.
The topology is fully redundant to avoid single point of failure problems. Achieving redundancy is a
comprehensive task. Use the following Layer 1 processes to help you achieve redundancy:
• Create channels between the aggregation switches
• Insert service appliances in pairs connected to different aggregation switches
• Spread servers providing the same service on different front-end switches
• Dual attach servers to different front-end switches
Redundancy at the component level, inside the same chassis, can also provide benefits.
Choosing the Spanning-Tree Algorithm

Spanning-tree ensures a logical loop free topology on the top of a physical “looped” topology. The
current recommendation is to use Rapid Per VLAN Spanning Tree(Rapid PVST+), which is a
combination of 802.1w and PVST+. For higher scalability, you can use 802.1s/1w, also called multi
instance spanning-tree (MST). 802.1s achieves higher scalability because you limit the number of
spanning-tree instances, but it is less flexible than PVST+ if you use bridging appliances. The use of
802.1w in both Rapid PVST+ and MST provides faster convergence than the traditional Spanning Tree
Protocol (STP). Other Cisco enhancements to STP, such as Loopguard and Unidirectional Link
Detection (UDLD), are still applicable and their use recommended in both Rapid PVST+ and MST
environments.
Routing in the Data Center

The routing functions occur between the campus core switches and the aggregation switches. If firewalls
are present, they can also participate in the routing process. Depending on the requirements and the
design, the boundary between Layer 2 and Layer 3 at the aggregation layer can be the MSFCs, the
firewalls, or the content switching devices. You can achieve routing either with static routes or with
routing protocols such as EIGRP and OSPF. This design guide covers routing using EIGRP and OSPF
only.

956513 2-3
Network devices, such as content switches and firewalls, often have routing capabilities. Besides
supporting the configuration of static routes, they often support RIP and sometimes even OSPF. Having
routing capabilities facilitates the task of the network design but you should be careful not to misuse this
functionality. The routing support that a content switch or a firewall provides is not the same as the
support that a router has, simply because the main function of a content switching product or of a firewall
is not routing. Consequently, you might find that some of the options that allow you to control how the
topology converges (for example, configuration of priorities) are not available. Moreover, the routing
table may not be big enough to accommodate as many routes as a router accommodates.
Remember, the routing capabilities of the MSFC, used in conjunction with the Catalyst supervisor,
provide traffic switching at wire speed in an ASIC. Equal cost routes load balancing is done in hardware
also. These capabilities are not available in a content switch or a firewall.
Cisco recommends that you take advantage of the routing features of the products to simplify the
configuration of the network, but at the same time to plan and design in such a way as to limit the number
of routes that such a device sees and minimize the weight of routing table recalculations. For example,
it is good practice to have a router as an area border router (ABR) and not another device.
The servers use static routing to respond to client's requests. The server configuration typically contains
a single default route (default gateway) pointing to either a router or a firewall or a load balancer.
Mainframes can be configured with static routing just like other servers, and support OSPF routing.
Unlike most servers, mainframes have several internal instances of Logical Partitions (LPARs) and/or
Virtual Machines (VMs) each containing a separate TCP/IP stack. OSPF routing allows the traffic to gain
access to these partitions and/or VMs via a single or multiple OSA cards.
Predictable Convergence Time

By following the guidelines in this chapter, you can design a redundant network that has a predictable
convergence time.Knowing the convergence time is key to bringing redundancy from the local data
center to geographically dispersed data centers (as is the case with data centers designed for disaster
recovery). When your applications are hosted in multiple data centers, you probably want the local data
center to answer local requests, and even in the event of a failure, you still want to give the local data
center priority before dispatching requests to other locations. This is only possible if you know when an
irresponsive data center is really down as opposed to just converging.
You can achieve disaster recovery by deploying a site selector that monitors the data center by means of
keepalives or probes as if the data center was a “single server.” These probes are Layer 4 or Layer 5
probes as a preference: if an outside device (such as the site selector) can establish a TCP connection
with a server farm represented by a Virtual IP address (VIP), you can assume that the data center is
available.
If a failure occurs, there is convergence at Layers 2, 3, and 4. Layer 2 convergence is the time it takes
for spanning-tree to unblock a port when a failure occurs that breaks the active topology. Layer 3
convergence is the time it takes for the routing protocols to detect a failure and determine the new Layer
3 topology. Layer 4 convergence is the time it takes for a content switch's VIP address to become
available after a failure that typically involves the reach ability of servers' Layer 4 ports.
An external probe from a site selector verifies that the network has converged at all layers. The solution
does not tolerate false alarms. You need to know the convergence time for the whole data center in order
to configure the site selector appropriately.
Figure 2-2 displays data center convergence and its impact on content routing. The convergence time for
failures between the access and the aggregation layers is mainly due to the convergence time required
for the STP at Layer 2. The convergence for failures between the aggregation and the core layer is mainly

2-4 956513
System Components
due to the routing protocols at Layer 3. Redundant service appliances communicate to determine which
device is active and which is standby. The configuration for these devices needs to give time for the
infrastructure to converge to avoid the flapping of active and standby service appliances.
The site selector monitoring the data center needs to give time for the Layer 2, Layer 3, and the service
appliances to converge before declaring the data center unavailable.
Mainframes typically attach to the aggregation switches with Layer 3 links. Mainframes also run OSPF,
which means that the convergence for failures related to the mainframe attached in Figure 2-2 depends
on the Layer 3 convergence time.
Figure 2-2 Convergence in the Data Center
Enterprise
Campus Core
Global Site Selector

Layer 3
convergence
Data Center Application
convergence
Content
Switch
Layer 2
convergence
Layer 4 and 5
convergence Layer 3
convergence
Mainframe
87437
Servers Servers
System Components
Hardware
The data center provides access ports for servers and service appliances as well as “slots” for service
modules. You can easily scale this design to 6,000 server ports to be split among Fast Ethernet and
Gigabit Ethernet NIC cards.

956513 2-5
System Components
Service appliances are external networking devices like:

• Content Services Switches (CSS)
• PIX Firewalls
• Secure Content Accelerators (SCA)
• IDS sensors
• Cache engines
The number of ports these appliances require strictly depends on how many appliances you use and how
you configure the Layer 2 and Layer 3 connectivity between the appliances and the infrastructure.
Service modules are cards that you plug into the Catalyst 6500 to provide content switching, firewalling,
intrusion detection functionalities from the Catalyst switch. Each card provides a different functionality
and takes one slot out of the Catalyst 6500. Examples of these modules are:
• Content Switching Module (CSM)
• Firewall Services Module (FWSM)
• SSL Services Modules (SSLSM)
Service modules communicate with the network through the Catalyst backplane. For more information
on integrating service modules, see the Data Center Networking: Integrating Security, Load Balancing,
and SSL Services using Service Modules SRND.
If you attach mainframes to the network using the OSA card, you can connect them to gigabit line cards,
if the mainframes have ESCON ports you can use a Cisco 75xx/72xx with CIP/CPA interfaces. Besides
providing physical connectivity, the Cisco CIP/CPA can also provide the TN3270 server functionality.
This section provides an example of how to allocate ports for a data center made of a few hundred
servers. For a data center hosting 400 Fast Ethernet attached servers, follow these calculations:
• 400 dual attached Fast Ethernet servers means 800 Fast Ethernet ports
• Three Fast Ethernet linecards per front-end switch for a total of 3 x 48 = 144 Fast Ethernet server
ports
• One slot is left empty in the front-end switch for future use
• The Fast Ethernet servers require 800 / 144 = 6 front-end switches
In the above scenario, you need 6 front-end switches.
The following calculation provides an example of how much hardware would be required for a data
center made of 400 servers:
Six access switches attach to each aggregation switch with a 1 Gigabit link
As result, there are 6 Gigabit ports allocated on the aggregation switches for connectivity with the access
switches
• A 4-port channel between aggregation switches is recommended
• Aggregation switches are connected to the core switches using either a Gigabit Etherchannel or by
separate Gigabit uplinks
• Aggregation switches need additional Gigabit ports for service appliances
• Aggregation switches need additional Gigabit ports for mainframe attachment
• Aggregation switches might have 2 slots reserved for the switch fabric modules or future use
• Aggregation switches need additional slots for service modules

2-6 956513
System Components
Aggregation Switches
Cisco recommends using the Catalyst 6500 family switches at the aggregation layer. This is because they
allow using service modules for load balancing and security. For instance, the CSM, SSLSM, FWSM are
all hosted in the Catalyst 6500 chassis. The chassis configuration depends on which specific services you
want to support at the aggregation layer, the port density of uplinks and appliances, and the need for
supervisor redundancy. Having dual supervisors is not a requirement, as such, is not covered in this
design guide.
The density at the aggregation switches is related to ports as well as slots. External service appliances
require ports at the aggregation switches. These appliances include PIX firewalls; content services
switches, secure content accelerators and content engines. You can attach other service appliances
directly to the backplane (slots), as is the case with the CSM and the FWSM.
Typically, you would also attach mainframes to the aggregation switches, especially if you configure
each connection to the OSA card as a Layer 3 link.
The supervisor 2 is recommended for performance and compatibility reasons. Memory
recommendations are as follows:
• 32MB of bootflash (allows room to store service module images and bigger Catalyst 6000 IOS
images)
• 128MB of RAM
• rommon version 7.1 or higher
The MSFC2 is recommended for performance reasons. Memory recommendations are as follows:
• 256 MB of DRAM
• 16 MB of bootflash
For more information about the memory and rommon configuration, refer to the following web page:
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12111bex/ol_2720.htm#xtocid3
The following is a possible physical configuration:
Chassis Slot Module Description

1 Catalyst 6000 Family Supervisor Engine 2
w/MSFC2
2 Reserved for a possible redundant supervisor
3 Reserved for Future use
4 16-Port Gigabit Ethernet Module
6 Reserved for Future use - SFM
7 Reserved for a possible Firewall Services Module
8 Reserved for a possible Content Switching
Module
9 Reserved for a possible SSL Module

956513 2-7
System Components
Access Switches
There are various types of platforms that you can use as access switches; this configuration guide uses
Catalyst 6506s. The advantage of using Catalyst switches (6500s or 4000s specifically) is that you can
install dual supervisors for maximum redundancy. This option is recommended if you have business
critical applications on mainframes or servers and you want to increase its availability.
The use of dual supervisors avoids the single point of failure at the component level and decreases the
impact of supervisor failures on service levels.
From the data center design perspective, the access layer, or front-end switches, must support 802.1s/1w
and Rapid PVST+ to take advantage of rapid convergence.
The supervisor 2 is recommended for performance and compatibility reasons. The memory
recommendations are as follows:
• 32MB of bootflash (allows room to store service module images and bigger Catalyst 6000 IOS
images)
• Rommon version 7.1 or higher
The following is a possible physical configuration for an access switch:
Chassis Slot Module Description
4 48-Port RJ-45 10/100 Ethernet Module
Software
Aggregation Switches
You can either use Native IOS or Catalyst OS as the operating system for the aggregation switches.
If you use Native IOS, you should run 12.1(13)E or higher: this software release introduces support for
Rapid PVST+ and support for the FWSM.
If you use the Catalyst OS, you should use 7.5 or higher. Starting with 7.5, the CatOS software introduces
the support for the service modules (CSM, SSLSM, FWSM) and introduces Rapid PVST+.
Access Switches
Cisco recommends software version 7.5 of CatOS software. CatOS allows you to support high
availability at the access layer by using dual supervisors with stateful failover for the Layer 2 protocols.
CatOS 7.5 also introduces Rapid PVST+.

2-8 956513
Design Details
Design Details
Layer 1
Using the design criteria introduced earlier in the chapter, the first goal is to address a fully redundant
data center. Achieving full redundancy implies the following design recommendations are applied:
• Two aggregation switches
• Channeling between the aggregation switches: use two or four ports out of two different modules
(for example, GigabitEthernet1/1 and GigabitEthernet4/15)
• Each access switch has an uplink to the aggregation switches, which can be either a single link or a
channel of two gigabit links to each aggregation switch
• One Layer 3 link (each one single or channel) from each aggregation to each core switch (for
example, aggregation1 connects to both core1 and core2)
Service Appliances
Service availability is critical; therefore Cisco recommends that you deploy redundant service
appliances. Distribute service appliances equally to both aggregation switches. Since you deploy service
appliances in pairs, it is recommended the active appliance be physically in the same aggregation switch
that performs the STP root and primary HSRP functions. You must connect the standby appliance to the
switch performing the secondary STP root and standby HSRP functions. When service appliances, such
as content engines or secure content accelerators are load balanced by a content switch, Cisco
recommends that you distribute service appliances across the aggregation switches to minimize impact
and reduce downtime. This keeps the failure of the aggregation switch from taking out all the services
of the connected service appliances.
Enable portfast on the Catalyst ports connected to a service appliance. You should enable trunkfast (more
details in the Layer 2 section) in case the service appliance receives multiple VLANs from the
aggregation switch.
Certain service appliances are available for the Catalyst 6500 switch as service modules. Service
modules do not have external ports, they plug into the Catalyst chassis: this solution offers a higher
integration than external appliances.
Redundant Supervisors
Router Processor Redundancy Plus (RPR+) is the implementation of supervisor redundancy available in
Catalyst IOS. Its richer counterpart in CatOS is high availability. In addition to RPR+ features, CatOS
high availability has stateful failover for the Layer 2 protocols, which means that upon the switchover
of the supervisor, high availability keeps the forwarding ports in forwarding state.
RPR+ keeps the standby supervisor fully booted and synchronizes the configurations at runtime. When
switchover occurs, RPR+ resets the ports and does not reset the line cards. In the meantime, the standby
supervisor comes online and restarts the protocols. You must reprogram the routing table. It takes about
30 to 60s for switchover to complete.
When a supervisor switchover occurs, the service modules are unaffected, which means that their OS
continues running and they can virtually forward traffic. This is true for CatOS because ports do not
change state during the switchover; it is not completely true for Catalyst IOS because the protocols
(including Spanning-Tree) have to restart on the newly active supervisor.

956513 2-9
Design Details
For more information about this specific feature, refer to the following web page:
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12_1e/swconfig/redund.htm
Channel Configuration
Configure a channel between aggregation1 and aggregation2. Use multiple ports from different line
cards to minimize the risk of losing connectivity between the aggregation switches. Use Link Aggregate
Control Protocol (LACP) active on aggregation1 and LACP passive on aggregation2. The following
example shows the channel configuration between the ports Giga1/1, Giga1/2, Giga4/15, and Giga4/16
for aggregation1.
interface GigabitEthernet1/1
description to_aggregation2
[…]
channel-group 2 mode active
channel-protocol lacp
[…]
[…]
[…]
The configuration for aggregation2 is the same with the exception that the channel mode is passive:
channel-group 2 mode passive
For more information about configuring channels, refer to:

http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12_1e/swconfig/channel.htm
There are channels that you do not configure, for example the channels that go to the service modules.
To verify information about those channels, use the: show etherchannel summary command.
aggregation1#sh etherchannel summary
Flags: D - down P - in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
Number of channel-groups in use: 3

Number of aggregators: 3
Group Port-channel Protocol Ports

------+-------------+-----------+-----------------------------------------------
2 Po2(SU) LACP Gi1/1(P) Gi1/2(P) Gi4/15(P) Gi4/16(P)
261 Po261(SU) - Gi8/1(P) Gi8/2(P) Gi8/3(P) Gi8/4(P)
275 Po275(SU) - Gi7/1(P) Gi7/2(P) Gi7/3(P) Gi7/4(P)
Gi7/5(P) Gi7/6(P)

2-10 956513
Design Details
The channels Po261 and Po275 go to the CSM and to the FWSM respectively.
When troubleshooting issues with the service modules, use the show spanning-tree interface
Port-channel command to determine if the channels are in forwarding mode. Be sure to use the right
channel number, which depends on the internally assigned channel number.
Remember, channels are like ports, which means that the configurations described later in terms of
trunking are applied to the channel as a whole. In the previous example, use the interface Port-channel
2 command to access information for Port-channel 2.
Also notice that the connectivity between aggregation1 and aggregation2 is a “channel + trunk”.
Layer 2
As previously described, the Layer 2 domain starts at the access layer and terminates at the device that
provides the default gateway. This Layer 2 domain is actually made of several VLANs that you assign
based on how you want to segregate the server farms. InFigure 2-3, on the left side, you can see several
VLANs used to connect the servers to the infrastructure.
There are also VLANs that do not reach the access switches; these VLANs are just used between the
aggregation switches. In Figure 2-3, on the right side, you can see two service appliances (SSL modules)
attached to the aggregation switches. These service appliances communicate on a VLAN that does not
need to reach the access switches. These “service” VLANs must be cleared from the trunks to the access
switches. The same needs to be done for Layer 3 VLANs (VLANs that are used for layer 3 connectivity
between the routers) and FT VLANs (VLANs used purely for the failover communication between
active/standby appliances). Do not disable STP for any of the VLANs.
Figure 2-3 Server VLANs Versus Service VLANs
Cisco recommends that you use RSTP and MST (802.1w and 802.1s) because of their stability and fast
convergence. Remember, 802.1w assigns roles to the ports, and you can control port assignment with
portfast and trunkfast. An edge port (which is a port that connects to a host or a service appliance) does
not need to change state from blocking, to learning, and finally to forwarding. To classify a port as edge,
you must enable portfast or trunkfast manually.
This section outlines how to configure Layer 2 in the data center to avoid loops and to minimize the
convergence time by using specific features available on the Catalyst switches.

956513 2-11
Design Details
VLAN Configuration
This data center reference architecture supports the use of 4096 VLANs starting from Catalyst IOS
12.1(11b) EX or CatOs 7.1. Cisco recommends that you have a one-to-one mapping between VLANs
and subnets. The following list is useful to correctly provision the number of subnets and VLANs that
you need in the data center. You need:
• Access VLANs (VLANs for servers mainly)
• Layer 3 VLANs (for communication between MSFCs, this is used to have a contiguous OSPF area)
• Service VLANs (used to forward traffic to the service modules, like the client VLAN of a content
switch)
• Fault tolerant VLANs (the VLANs used for redundancy by CSM, FWSM, CSS, etc.)
• Some additional VLANs are taken by the system for routed ports as well as WAN ports.
If you want to use 4000 VLANs, you have to enable the mac address reduction command as follows:
spanning-tree extend system-id
Mac address reduction makes it possible for you to configure 4,000 VLANs. The mac address reduction
feature modifies the bridge identifier in such a way that instead of 16 bits of bridge priority, you only
use 4 bits, 12 identify the VLAN (where the relation with 4,000 VLANs) and 6 bytes are for the bridge
mac address. While the implication in terms of bridge ID is unnoticeable, you see that the priority of a
root and secondary root are slightly different from what you were accustomed to without mac address
reduction:
• Root bridge priority: 24576 (instead of 8192)
• Secondary root bridge priority: 28672 (instead of 16384)
• Regular bridge priority: 32768
The specific spanning-tree configuration is explained later. In order to verify that mac address reduction
is enabled, use show spanning-tree summary command.
Remember to enable mac address reduction on all the switches in the data center. The configuration
command in CatOS is the following:
set spantree macreduction enable
Now that you enabled mac address reduction, you need to define the VTP mode for the switch. Use the
VTP transparent mode for the following reasons:
• Given that the VLANs live between aggregation and access switches only, there are no major
benefits from automatic VLAN configuration distribution between switches.
• VLAN misconfiguration errors are easily propagated through VTP, creating an unnecessary risk. For
instance, server VLANs accidentally removed from a switch can propagate resulting in entire server
farms being isolated. Extended VLANs are not advertised by VTP (VLANs greater than 1006).
Use the following steps to configure VTP:
vtp domain mydomain
vtp mode transparent
Use the same VTP domain name everywhere in the data center.
Type the following command to configure VTP on access switches running CatOS:
set vtp domain mydomain mode transparent

2-12 956513
Design Details
You can create VLANs either from the “config” mode or from the “vlan database”. Do not use the VLAN
database configuration mode. Using config mode is recommended for two reasons:
• Extended VLANs can only be configured in config mode
• If you are using RPR/RPR+, the VLANs defined in the “database” mode are not synchronized to the
standby supervisor
(config)VLAN 10
(config-vlan) name my_vlan
When configuring VLANs follow these guidelines:

• Allocate normal VLANs from 1 up
• Allocate extended VLANs from 4096 down
The reason is because of how the switches allocate internal VLANs. As described previously, VLANs
for routed ports and WAN ports are created internally without user intervention. They are carved from
the extended range of 1006 and up. If you allocate extended VLANs from 4096 down, you avoid
overlapping problems.
The following bullets summarize the ranges:
• 1 - 1005 normal VLANs
• 1002 FDDI, 1003 Token Ring, 1004 FDDI Net, 1005 Token Ring Net
• 1006 - 4094 internal VLANs
• 1006 - 4094 extended VLANs
• 4095 protocol filtering
For more information about extended and internal VLANs and how to monitor them, refer to the
following link:
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12_1e/swconfig/vlans.htm
Beside the standard 12 bits allocated to the VLAN ID, there are operational limitations that restrict the
number of active VLANs. In certain cases, service appliances support a smaller number of VLANs. Use
the following list as a reference for how many VLANs are supported in the following service appliances:
• CSM— 256 VLANs (2.2)
• FWSM — 100 VLANs (1.1)
Also consider that the need for VLANs on a specific module depends mainly on which device is
providing the default gateway to the servers. For example, if the firewall provides the default gateway,
you do not need many VLANs on the content switch and vice versa.
Access Switch Port Configuration

Access ports carry traffic for a single VLAN. You typically connect servers to access ports.
Additionally, devices like SSL offloaders, caches or the client side (and sometimes the server side) of a
CSS are connected to an access port. When a server is attached to an access port, or when a device other
than a switch is attached to an access port, Cisco recommends that you enable portfast with the following
command:
(config-if)# spanning-tree portfast
For more information about portfast see:

http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12_1e/swconfig/stp_enha.htm

956513 2-13
Design Details
The recommended configuration for the ports that provide access to the servers, if you are running
CatOS on your access switches, is:
set port host <a/b>
As explained in the “Spanning-Tree: 802.1w” section, it is important to assign the portfast and trunkfast
definition to the eligible ports because 802.1w categorizes ports into edge and non-edge. There are two
main advantages in doing this. First, if flapping occurs on edge-ports, 802.1w does not generate a
topology change notification. Secondly, an edge port does not change its forwarding state when there is
a topology recalculation.
Trunk Configuration
You can configure trunks between the following network devices:
• Aggregation switches (carrying basically all of the VLANs)
• Access switches and aggregation switches (carrying only the server VLANs)
• Aggregation switches and the service appliances
You can define the VLANs allowed on a trunk with the following command:
switchport trunk allowed vlan 10,20
You can modify the list of the VLANs allowed on a trunk with the following commands in Native IOS:
switchport trunk allowed vlan add <vlan number>
switchport trunk allowed vlan remove <vlan number>
You can modify the list of the VLANs allowed on a trunk with the following commands in CatOS:
clear trunk <mod/port> <vlan>
The recommended trunk encapsulation is 802.1q, mainly because it is the standard. The configuration in
Catalyst 6500 IOS is:
switchport trunk encapsulation dot1q
The configuration in CatOS:

set trunk <mod/port> dot1q
You can force a port to be a trunk by typing:

switchport mode trunk
The same command in CatOS is:

set trunk <mod/port> on
This mode puts the port into permanent trunk mode and sends Dynamic Trunking Protocol (DTP) frames
to turn the neighboring port into a trunk as well. If the trunk does not form, verify the VTP domain
configuration. VTP domain names must match between the neighboring switches.
Cisco recommends that you configure the switches to tag all traffic on trunks with the VLAN tag. In
802.1q, the native VLAN of a port is carried untagged, this might lead to misconfigurations. To avoid
misconfigurations, enable dot1q tagged all:
vlan dot1q tag native
In CatOS, type:
set dot1q-all-tagged enable all
Use 802.1q to configure trunks to external devices such as a CSS.

2-14 956513
Design Details
Configuration of trunks to service modules, such as a CSM or a FWSM, is either implicit or explicit.
Implicit means that the device sees all the VLANs by default and explicit means that the devices are
aided by using an explicit command. In the case of the FWSM the commands are:
firewall vlan-group <vlan-group-number> <vlan list>
firewall module <module> vlan-group <vlan-group-number>
In the case of the SSLSM the commands are:

ssl-proxy module <module> allowed-vlan <vlan list>
As explained in the “Spanning-Tree: Rapid PVST+” section, it is important to assign the portfast and
trunkfast definition to the eligible ports because 802.1s/1w categorizes ports into edge and non-edge.
For this reason, you should enable trunkfast on the trunks that go to a CSS or any external service
appliance connected to the aggregation switches via a trunk. Use the following command:
spanning-tree portfast trunk
A special instance of trunks is given by the ports used by service modules internal to the Catalysts to
connect to the switch backplane. These ports cannot be manually configured even if you can see the
channels. As of 12.1(11b)EX, the ports go into blocking, learning, and forwarding when topology
changes occur. This state transition brings up the convergence time at Layer 4 to 30s even if you are using
MST (see CSCdy00143 and CSCdy00148 for more details). Additionally, the autostate feature on the
MSFC adds 10 additional seconds to the convergence. For more information, see the Appendix.
Logical Ports
So far, you have seen how to configure up to 4,000 VLANs. The question is how many VLANs can you
realistically configure without creating too much load for the switch CPU? This depends on the number
of “logical ports.” A logical port is the sum of the number of physical port times the number of VLANs
that each port carries. This is what produces load on the CPU because each port carrying a VLAN has
to generate and process BPDUs. You can find the number of logical ports that you can realistically
configure at this web page:
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12_1e/ol_2310.htm#26366
These numbers depend on the spanning-tree algorithm used, the supervisor, and the operating system.
MST scales better than Rapid PVST+ because there are fewer instances of Spanning-tree. On the other
hand, Rapid PVST+ is more flexible when you have bridging devices attached to your aggregation
switches.
The recommended approach to deploying spanning-tree is to consider Rapid PVST+ first and monitor
how many logical ports you are using. You can monitor how much growth you have for additional
VLANs by comparing the numbers above with the output of the following command:
Show spanning-tree summary totals
If you realize that you are reaching the limits listed at the above URL, consider migrating to MST.

956513 2-15
Design Details
Figure 2-4 Choosing the Spanning-Tree Algorithm
Count the number

of logical ports
Is No
it < R-PVST Use MST
limits?
Yes
87439
Use Rapid
PVST+
Spanning-Tree: 802.1w
The 802.1w protocol is the standard for rapid spanning-tree convergence. 802.1w is on by default when
running spanning-tree in MST mode. The key features of 802.1w are the following:
• Convergence is accelerated by the handshake (proposal agreement mechanism)
• No need to enable backbonefast or uplinkfast
• Changes of status of an edge port does not cause a topology change
Because spanning-tree categorizes ports into edge and non-edge and this is based on the duplex
information as well as the assignment of portfast to a port, it is important to configure this feature on all
eligible ports. This makes the network more stable because it keeps a port in forwarding state during
topology changes. Failure to configure portfast has drastic effects on the convergence time: a non-edge
port connected to a device that does not speak spanning-tree cannot perform the handshake that shortens
convergence time. Consequently, a non-edge port connected to a server or a service appliance goes
through the blocking, learning, and forwarding steps; slowing down the convergence time by 30s. This
is still acceptable if it happens on a single server port (meaning this single server is going to be
unavailable for 30s). However, this slower convergence time has major effects if all of the servers in the
server farm have to go through this process and/or if the service modules are affected by this delay (all
the traffic has to traverse these modules).
In terms of convergence, the new spanning-tree is much faster, especially because of the proposal
agreement mechanism that allows a switch to decide new port roles by exchanging proposals with its
neighbors. BPDUs are still sent every 2s by default (hello time).
If three BPDUs are missed, spanning-tree recalculates the topology, which takes around 1s for the rapid
spanning-tree (802.1w). Therefore, you could say that the spanning-tree convergence time is around 7s.
Because the data center is made of point-to-point links, the only “real” failures are either physical
failures of the networking devices or of links. This means that the convergence time is around 1 or 2s
rather than 7s. The scenario where BPDUs are lost is actually likely to cause Layer 2 loops, and for this
specific problem, you can use Loopguard (described in a later section).
802.1w can be deployed in conjunction with either one of the following protocols:
• PVST+: the combination of the two is called Rapid PVST+
• 802.1s: the combination of the two is called MST
For more information about 802.1w, refer to the Understanding the Rapid Spanning Tree Protocol
(802.1w) at:

2-16 956513
Design Details
http://www.cisco.com/en/US/tech/tk389/tk621/technologies_white_paper09186a0080094cfa.shtml
Spanning-Tree: Rapid PVST+

Rapid PVST+ combines the fast convergence of 802.1w with the flexibility of Cisco Per VLAN STP. For
information about Rapid PVST+ refer to the following URL:
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12_1e/swconfig/spantree.htm#1082480
One of the reasons for using Rapid PVST+ rather than MST is the fact that Rapid PVST+ allows you to
have a redundant pair of bridging appliances attached to aggregation switches without the need to filter
BPDUs. This is explained in the following paragraphs.
In a data center environment it is common to have bridging appliances like firewalls or load balancers.
These devices can attach to an aggregation switch as you can see in Figure 2-5. Normally, a bridging
device such as a firewall or a load balancer is deployed in pairs, with only one pair active. This concept
is depicted in Figure 2-5 where you can see the active firewall or load balancer represented as a bridge
in a circle. The redundant pair is attached to the aggregation switch on the right of the picture.
The standby pair does not forward any traffic. The active devices merge VLANs on the aggregation
switch: in the example you can see VLAN 10 merged with VLAN 20. Merging VLANs means bridging
the traffic from one VLAN to the other VLAN including the BPDUs. For PVST+, the two VLANs are
equivalent to two separate switches (see the right of the picture). As a result of this, The layer 2 network
that is equivalent to the physical network is the one that you see at the bottom of Figure 2-5.
Figure 2-5 Rapid PVST+ with Bridging Devices
Aggregation switches DP VLAN 10

Active Standby
Aggregation 1
VLAN 20
RP
Aggregation 1
Root for Vlan10

87440
DP VLAN 10
DP DP
VLAN 20
RP RP RP
As you can see, there is no loop because only one device of the redundant pair is active. If both firewalls
or load balancers become active you have a loop, but Spanning-Tree would intervene and block one port.
If the bridging devices drop BPDUs, it is possible that a malfunctioning active/standby election process
between firewalls and load balancers makes both firewalls / load balancers active. This would create a
loop that Spanning-Tree would not be able to block because the BPDUs are not bridged.

956513 2-17
Design Details
As a result, when operating with bridging appliances, it is advised bridging BPDUs on these appliances.
Previous to the availability of 802.1w, it was sometimes recommended to filter BPDUs from being sent
to a bridging load balancer or a Layer 2 firewall because some convergence scenario could introduce a
30s delay due to the blocking, listening, learning, and forwarding steps. With 802.1w, this is not a
concern any longer because the proposal agreement mechanism makes the convergence extremely fast.
In summary:
• Attaching a Layer 2 firewall or a load balancer that operates in bridge mode to the aggregation
switches is a valid option if Rapid PVST+ is present.
• Having Layer 2 firewall and load balancers pairs operating in active/standby mode does not
introduce loops
• A scenario where both devices of a redundant pair become active as a result of a misconfiguration
or malfunctioning of the devices themselves does not bring down the data center because
Spanning-Tree would break the loop.
• If the firewalls and load balancers drop BPDUs, the above scenario could bring down the network
because Spanning-Tree would not be able to detect the loop.
• Passing BPDUs on Layer 2 appliances is recommended and by using 802.1w, has no impact on the
convergence time.
To configure Rapid PVST+ in native IOS do the following:
spanning-tree mode rapid-pvst
To configure Rapid PVST+ in CatOS do the following:

set spantree mode rapid-pvst
Cisco recommends that you have one single spanning-tree topology in the data center. In the event that
you need to load balance traffic to the uplinks from the access to the aggregation switches, you would
then assign to different priorities for even VLANs and odd VLANs.
The first configuration step is to assign the root and the secondary root switches. Because Cisco's
recommendation is not to do uplink load balancing in the data center, the examples in this design chapter
show one aggregation (aggregation1) switch as the root for all the instances and the other aggregation
switch as the root for all the other instances (aggregation2). The configuration for Native IOS on
aggregation1 is as follows:
spanning-tree vlan 3,5,10,20,30,100,200 root primary
The configuration for CatOS on aggregation1 is as follows:

set spantree root 3,5,10,20,30,100,200
The configuration on aggregation2 for Native IOS is as follows:

spanning-tree vlan 3,5,10,20,30,100,200 root secondary
The configuration on aggregation2 for CatOS is as follows

set spantree root secondary 3,5,10,20,30,100,200
The priorities are assigned as follows:

• Root bridge priority: 8192
• Secondary root bridge priority: 16384
• Regular bridge priority: 32768

2-18 956513
Design Details
Spanning-Tree: 802.1s
This section addresses key features of 802.1s. The main advantage of using 802.1s is the fact that it
scales better in presence of many VLANs and trunks, see the Logical Ports section for more information.
The main disadvantage of using 802.1s is because of its sub optimal behavior in the presence of Layer
2 appliances.
Consider Figure 2-6 where a Layer 2 firewall or a bridging load balancer bridges VLAN 10 with VLAN
20. On the right of the illustration, you can see the converged topology: one of the two ports will be
blocking. The reason is the fact that from the point of view of 802.1s, this topology is equivalent to a
loop. In case of PVST+, both ports are forwarding (as explained in the Spanning-Tree: Rapid PVST+
section) because each VLAN is considered a separate instance, and there really is no loop.
The workaround to this problem is filtering BPDUs so that 802.1s does not think there is a loop. The
drawback of this is the fact that Spanning-Tree would not detect a real loop caused by a redundant pair
if both appliances were to go active.
Figure 2-6 802.1s and Layer 2 Appliances
Aggregation switches Instance 0

Active
VLAN 10 belongs to instance 1

VLAN 20 belongs to instance 1
VLAN 10
87441
VLAN 20
The key features of 802.1s are:

• One single BPDU has the information about all instances and the IST
• Up to 16 instances are possible
The 802.1s (MST) protocol is very close to the Cisco implementation of MISTP. This protocol allows
for the mapping of multiple VLANs to a single spanning-tree instance thus alleviating the load on the
CPU of maintaining the Layer 2 topology even when many VLANs are configured. In 802.1s, one default
instance is always present, called Internal Spanning Tree (IST) or the MST instance 0.
The switch uses IST to build a shared tree for compatibility with regions that run Common Spanning
Tree (CST). The IST information is carried on BPDUs on all the ports. Do not map VLANs to the IST:
the IST is only for compatibility with other regions.
The Cisco recommendation is to create a separate instance for the data VLANs (different from MST 0)
and map VLANs to that MST instance.
The association between VLANs and instances is defined in the spanning-tree region configuration. A
region for spanning-tree is defined by an alphanumeric identifier, by a revision number, and by a table
that maps the VLANs to their respective instance. The region information in the data center switches
must match; otherwise, they will belong to different regions. The reason for the region concept is to make

956513 2-19
Design Details
sure that you have consistent mapping between VLANs and MST instances. If you notice that you have
ports categorized by spanning-tree as a boundary port, the problem is likely related to an inconsistent
region configuration.
The following are the configuration steps using Catalyst 6500 IOS:
spanning-tree mst configuration
name data_center_mst
revision 10
instance 1 vlan 1-1000
The same name and same revision number, as well as the instance mapping, must match on all data center
switches. Notice that you do not map any VLAN to instance 0.
Use the following steps to configure the MST region in CatOS with the following steps:
set spantree mst config name data_center_mst revision 10
set spantree mst 1 vlan 1-1000
set spantree mst config commit
Cisco recommends that you have one single spanning-tree topology in the data center. In the event that
you need to load balance traffic to the uplinks from the access to the aggregation switches, you would
then configure two instances in addition to the IST. The conclusion is that the total number of instances
ranges between 2 and 3 depending on the configuration. This is still a small number when compared to
the number of spanning-tree instances that the switch had to maintain with PVST+.
The configuration steps to assign the root and the secondary root switches are the same as PVST+.
Because Cisco's recommendation is not to do uplink load balancing in the data center, the examples in
this design chapter show one aggregation (aggregation1) switch as the root for all the instances and the
other aggregation switch as the root for all the other instances (aggregation2). The configuration on
aggregation1 is as follows:
spanning-tree mst 0 root primary
spanning-tree mst 1 root primary
The configuration on aggregation2 is as follows:

spanning-tree mst 0 root secondary
spanning-tree mst 1 root secondary
Because of mac address reduction, the above macros assign priorities as follows:
• Root bridge priority: 24576 (instead of 8192 without mac address reduction)
• Secondary root bridge priority: 28672 (instead of 16384 without mac address reduction)
• Regular bridge priorit:y 32768
Protection From Loops

Spanning-tree is designed to be used on both shared media as well as point-to-point links and assumes
that missing BPDUs are a symptom of a device failure or of a link failure. In the case of point-to-point
links, this is not true: a neighboring switch immediately detects the physical failure of a link or of a
network device because a link is down. In normal conditions, BPDUs should always be received. Missing
BPDUs are a symptom of the following other problems:
• Link oversubscription and queues dropping packets
• A link that becomes unidirectional because of the failure of a transceiver
• BUGs on a neighboring device

2-20 956513
Design Details
If a port becomes forwarding because of missed BPDUs, it is likely to cause a Layer 2 loop that brings
down the network. The feature that fixes this problem is Loopguard. As you can see from Figure 2-7,
spanning-tree alone cannot tell the difference between Failure 1 (bug, over subscription or unidirectional
link) and Failure 2, a broken link between the two aggregation switches.
Figure 2-7 Failures That Cause Port 3/21 to go into Forwarding (Regular Spanning Tree)
In this failure the upstream switch In this failure there is still

no longer sends BPDUs an exchange of BPDUs
aggregation2 Failure 1: aggregation2 stops aggregation2 Failure 2: the link between

sending BPDUs on port 3/11 aggregation2 and aggregation1 fails
3/11 3/11 B
agg PDU:
BPD rega
X U: is ro tion2
RP 3/10 RP 3/10 ot
access access
3/21 3/21
X
vlan 99 vlan 99
3/22 3/22
RP RP
DP 3/10 DP 3/10
DP vlan 99 DP vlan 99
3/12 3/12
76493
aggregation1 aggregation1
Spanning-tree alone would transition port 3/21 into forwarding causing a Layer 2 loop. Loopguard
prevents the transition on port 3/21.
The configuration of loopguard follows (this command enables loopguard globally on the switch):
(config)#spanning-tree loopguard default
The corresponding command in CatOS is:

set spantree guard loop <mod/port>
The recommendation is to enable loopguard on all the ports both at the aggregation and at access
switches.
With the rapid spanning-tree, UDLD can still be used to prevent loops caused by bad wiring of fiber
links. UDLD cannot detect loops that occur after the topology has already converged: a link that
suddenly becomes unidirectional causes the spanning-tree topology to converge within 7s (6s to detect
missing BPDUs and 1s to send the proposal and receive an agreement). UDLD can detect a
unidirectional link in 21s with a message interval of 7s, which is more than the time it takes for
spanning-tree to converge.
The conclusion is that loopguard and UDLD complement each other, and therefore UDLD should also
be enabled globally:
aggregation2(config)#udld enable
Besides failures that can cause loops, another common cause of loops especially during configuration,
are devices that bridge VLANs, like content switches used in bridge mode. Content switches typically
do not forward BPDUs, which means that when two of them are active and bridge the same VLAN, a
loop occurs.

956513 2-21
Design Details
If you are using such a device, make sure that, before bridging VLANs, the two devices can see each
other and agree on their active / backup role. On the CSM, you can achieve this by configuring the Fault
Tolerant VLAN first and then by assigning the client and server VLAN and bridging them.
Layer 3
The Layer 3 portion of the data center design changes slightly depending on whether this is a Internet
data center or an intranet data center. Figure 2-8 shows on the left the physical topology of an intranet
data center, and on the right the logical topology.
Figure 2-8 Intranet Data Center Physical and Logical Diagrams
Enterprise Enterprise
Campus Core Campus Core
core 1 core 2 core 1 core 2
Aggregation Layer
Mainframe
Mainframe
87442
Front-end Layer Servers Servers
As you can see, on the right of Figure 2-8, the Layer 3 switches of the aggregation layer map to two
routers connected to multiple segments; each segment is represented with a different line style. The
logical representation of the Intranet Data Center shows that the data center is a spoke of a
hub-and-spoke topology. As such, there is very little dynamic routing. All you need to do is to inject a
default route into the data center and advertise the data center subnets to the core.

2-22 956513
Design Details
Figure 2-9 Internet Data Center Physical and Logical Diagrams
ISP1 ISP1
ISP1 ISP2
Core 1 Core 2
Enterprise
Campus Core Border Router Border Router Border Router Border Router
Aggregation Layer
87443
Front-end layer Servers Servers
Figure 2-9 represents an Internet data center, the left portion of the figure shows the physical diagram,
while the right portion represents the logical diagram. The firewall is a key element in this topology
because it is used to create a special zone, the demilitarized zone (DMZ). As for the aggregation routers,
they are placed in the middle between the firewall and the core routers.
Border routers provide Internet connectivity. These routers connect to the service providers. These
routers are also called edge routers. For providing Internet access, these routers typically advertise a
default route into the core in order to draw traffic destined to the external address space.
This design guide addresses designs with routing protocols with special attention to EIGRP and OSPF.
The key characteristics of the designs are:
• Use Layer 3 links between routing devices, when possible
• Summarization occurs from the data center to the core
• Inside the data center, passive all VLANs except one used to keep a Layer 3 escape route with the
neighboring MSFC
• Passive VLANs where you do not need to establish neighbor relationship between routers
• As much as possible, provide the default gateway at the MSFC via HSRP
• Alternatively the default gateway is provided by content switches or firewalls
The routing between the data center and the core is typically performed on the MSFC.

956513 2-23
Design Details
OSPF
The routing design for the data center should make the Layer 3 convergence as independent as possible
from the routing recalculations that occur in the core. Similarly, the data center routing recalculations
should not affect the other areas.
The data center should be a separate area; the placement of the area border routers (ABRs) can be either
in the core or in the aggregation switches.
Figure 2-10 LSA Propagation in case of Failure in the Data Center
Area 1 Area 1
Type 3 Type 1
ABR ABR
Core Core
Area 0 Area 0
Type 3 Type 3
Data Center Data Center

ABR ABR
Type 3
Type 1 Area 2 Area 2
87444
Figure 2-10 shows the effect of a link failure inside the data center and the effect of a link failure on an
area outside the data center. In the left portion of Figure 2-10, the router that detects the failure floods
an LSA type 1 in the data center area (Area 2), the ABR (in this case the MSFC, which is also probably
the router that detected the failure) generates an LSA Type 3 and sends it to the core (Area 0). The LSA
type 3 reaches another ABR and goes into Area 1. A similar sequence of events occurs in the right
portion of Figure 2-10.
As you can see from this diagram, if you use just regular OSPF areas, local failures propagate to the rest
of the network causing unnecessary shortest path first (SPF) calculations.
There are two solutions to this problem from a data center perspective:
• Limit the number of LSAs that the data center receives: you can achieve this by using OSPF stub
areas
• Limit the number of LSAs that the data center sends: you can achieve this by using summarization.

2-24 956513
Design Details
Figure 2-11 OSPF Stub Areas
Stub default Area 0
Type 3, Type 4
Totally Stubby Area 0
default
NSSA
Type 5 Area 0
Type 7
87445
Type 3, Type 4
Figure 2-11 shows the possible options that you have in terms of OSPF areas. In this picture, you can
see the LSA types that can propagate between areas depending on the stub type. A stub area can receive
Type 3 and Type 4 LSA, but it does not pass Type 5 LSAs. A totally stubby area only receives a default
from the ABR. In Figure 2-10, the ideal solution would be to configure area 2 as a totally stubby area.
If you configure a data center stub area and you need to redistribute static routes (an example could be
Route Health Injection), or you need to originate a default route (this is typically the case of the DMZ);
then you need a Not-So-Stubby Area (NSSA). When you configure an area as NSSA, the ABR does not
send a default route, because you typically want to originate the default route and send it to the core.
In an Internet data center, deployment of the edge router would originate a default as an external Type
7, the ABR translates this LSA into a Type 5 before sending it to the core.
Stub areas protect the data center routers from receiving too many LSAs. In order to protect your campus
from the effect of flapping links inside a data center, you should configure summarization. Because of
summarizing the data center subnets, the flapping of a single subnet does not cause the ABR to generate
a Type 3 LSA.
Figure 2-12 shows a possible design where the core routers are the ABRs. The ABRs are configured for
summarization and the data center area is a stub, totally stubby or NSSA.

956513 2-25
Design Details
Figure 2-12 OSPF Design in the Data Center
Summary Summary
Area 0
ABR ABR
L3 link
core 1 core 2
Default Default
L3 link
L3 link L3 link
L3 link
aggregation 1 aggregation 2
76494
OSPF-STUB AREA
The following capture shows the entries in the routing table in the case of a totally stubby area.
Remember that in a totally stubby area, the only Inter-Area route is the default route:
aggregation1#sh ip route
Gateway of last resort is 10.21.0.2 to network 0.0.0.0
172.26.0.0/26 is subnetted, 1 subnets

C 172.26.200.128 is directly connected, Vlan802
10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
C 10.20.30.0/24 is directly connected, Vlan30
O 10.20.20.0/24 [110/11] via 10.20.30.6, 00:00:05, Vlan30
O 10.21.0.12/30 [110/2] via 10.20.30.3, 00:00:05, Vlan30
[110/2] via 10.21.0.6, 00:00:05, GigabitEthernet4/8
O 10.21.0.8/30 [110/2] via 10.21.0.2, 00:00:05, GigabitEthernet4/7
[110/2] via 10.20.30.3, 00:00:05, Vlan30
C 10.21.0.4/30 is directly connected, GigabitEthernet4/8
O*IA 0.0.0.0/0 [110/2] via 10.21.0.2, 00:00:06, GigabitEthernet4/7
[110/2] via 10.21.0.6, 00:00:06, GigabitEthernet4/8
Core1 is 10.21.0.2 and Core2 is 10.21.0.6.

The following is the necessary configuration on the aggregation switches for a totally stubby area:
router ospf 20
log-adjacency-changes
area 20 stub no-summary
passive-interface Vlan5
network 10.20.0.0 0.0.255.255 area 20
network 10.21.0.0 0.0.255.255 area 20

2-26 956513
Design Details
The summarization of the data center networks occurs on the ABRs, which in this case are the core
routers:
router ospf 20
area 20 range 10.20.0.0 255.255.0.0
area 20 range 10.21.0.0 255.255.0.0
network 10.0.0.0 0.0.0.255 area 0
network 10.20.0.0 0.0.255.255 area 20
network 10.21.0.0 0.0.255.255 area 20
EIGRP
The goal of advertising default routes to the data center and summarizing the data center networks to the
core can be achieved with EIGRP using the ip summary-address eigrp command on a per-interface
basis.
Figure 2-13 EIGRP Routing in the Data Center
L3 link
core 1 Default Default core 2
Summary Default Default Summary

L3 link
ry Su L3 link
mma mm
Su ar
y
L3 link
aggregation 1 EIGRP aggregation 2
76495
Data Center
The following example is the configuration on the router core1. Notice the cost of the ip
summary-address route.
router eigrp 20
network 10.0.0.0 0.0.0.255
network 10.20.0.0 0.0.255.255
no auto-summary
no eigrp log-neighbor-changes
!
ip address 10.20.0.2 255.255.255.252
ip summary-address eigrp 20 0.0.0.0 0.0.0.0 200
end
!
ip address 10.20.0.10 255.255.255.252
end

956513 2-27
Design Details
It is important to configure a cost of 200 for the default route that the core advertises to the data center.
The reason is that the core router installs automatically a NULL0 route for the same default route. In
case an edge router advertises a default route (to access the Internet for example), this route has to take
precedence over the NULL0. If you do not force the cost to be 200, this does not happen and any traffic
that reaches the core and does not match any route is black holed instead of being pushed to the edge
routers.
On the aggregation router, passive all the VLAN interfaces but one to keep a Layer 3 route between the
aggregation routers. Configure summarization on the specific Layer 3 links that connect to the core.
The configuration for aggregation1 follows:
router eigrp 20
network 10.20.0.0 0.0.255.255
no auto-summary
no eigrp log-neighbor-changes
!
description to_mp_core1_tserv3
ip address 10.21.0.1 255.255.255.252
end
VLAN Interfaces
On the aggregation switches, you can assign IP addresses either to a Layer 3 interface or to a VLAN
interface. This type of VLAN interface is called a switched VLAN interface (SVI). Even when assigning
an IP address to an interface, a VLAN is internally allocated from the extended range.
The number of SVIs that you can configure on a Catalyst 6500 is 1000 despite the fact that the total
number of VLANs that can be switched is 4000.
When running dynamic protocols, it is a good practice to “passive” the VLAN interfaces to reduce the
number of VLANs that the aggregation switch MSFCs has to become a neighbor. A single Layer 3
VLAN is required to ensure that the area is contiguous. This VLAN is kept non-passive.
HSRP
It is not the purpose of this chapter to explain HSRP, which, like spanning-tree, is one of the basics of
the Campus design. For the purpose of this design chapter, it suffices to remember that HSRP provides
a “virtual interface” from a pair of routers. If either router fails, the remaining one can still receive traffic
destined to the HSRP IP address. HSRP sends hello packets every 3s and a neighboring router interface
is considered unavailable after 10s, or three HSRP hellos are lost.
HSRP provides the “preempt” option, which forces the router interface with the highest priority to
become master regardless of which interface became active first. Cisco's recommendation is to use the
preempt option only to force the topology to look like designed at configuration time and then eventually
remove the option once the network converges to the expected topology.
Another scenario when you may want to use the preempt option is when you configure HSRP track. In
this case, you want to make sure that bringing down the priority for one interface and make sure that the
peer interface takes over. In the example design, HSRP track is not used so there is no need for the
preempt option. If you configure static routing, it might be necessary to configure HSRP tracking.
You can use HSRP in two scenarios:

2-28 956513
Design Details
• Providing the default gateway for servers

• For static routing
The Supervisor2 on the 6500 allows you to configure a maximum of 15 HSRP groups. This limitation is
easily addressed by reusing the same HSRP group for all the VLANs configured on the same 6500
switch. More information is available at the following web page:
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12111bex/ol_2720.htm#xtocid17
IBM Mainframes
IBM networking has gone through a significant evolution from subarea System Network Architecture
(SNA), to Advanced Peer-to-Peer Networking (APPN) and eventually IP. If your mainframe hosts a
legacy SNA application, it is very likely that you now use Data Link Switching (DLSw) to bridge SNA
traffic from the branch office to the data center, by encapsulating it into TCP.
This design guide assumes that your mainframe is attached to the network with an OSA card (Gigabit
Ethernet) and that the clients gain access to the mainframe applications (whether SNA based or IP based)
via TCP/IP. You can also use 75xx/72xx with CIP/CPA interface cards to provide ESCON attachment.
You can give access to SNA applications on an IP network by using:
• Enterprise Extenders (EE): An EE is device close to the terminal, which tunnels traffic from the
terminal into high performance routing (HPR) over IP. The end-points of the Rapid Transport
Protocol (RTP) session (RTP is the equivalent of TCP in the APPN world) can be an OSA attached
mainframe and a Cisco router supporting EE in the branch office, or a channel interface processor
(CIP) attached mainframe and a Cisco router supporting EE. EE software runs on both the branch
router and the VTAM in the mainframe. Depending on the attachment option, the Virtual
Telecommunication Access Method (VTAM) receives IP traffic either on the top of the channel
protocol or from the OSA card.
• TN3270: TN3270 give access to legacy SNA applications from a regular PC. The software running
on the TN3270 server translates the characters from the format used on terminals to the ASCII
format and vice-versa. It converts clients' characters in SNA data streams. Typically a TN3270
server emulates a cluster of physical units (PUs) and logical units (LUs) that allow the mainframe
to create a System Services Control Point (SSCP) sessions as well as a LU-to-LU session. Each LU
is assigned to a different telnet session. Notice that you can run the TN3270 server on the CIP/CPA.
You can also build multi-tier server farms and front-end mainframes with web servers. A mainframe
could be attached with an OSA card and configured with a TN3270 server. The client uses HTTP as the
access method to the mainframe applications. Middleware applications running on the web server
provide translation between HTTP requests and TN3270 commands. A mainframe could also attach to
the network with an ESCON connection, front-ended with 75xx/72xx which provide the TN3270
functionalities. The client once more uses HTTP as the access method and the middleware software acts
as the translator.
Mainframes can also host IP based applications. All of the mainframe operating systems; z/OS, z/VM,
LINUX, VSE, and even TPF, have robust TCP/IP protocol stacks, the ability to run at least one routing
protocol, and access to Gigabit Ethernet interfaces. On one single mainframe you can host a number of
virtual Linux servers. For more information about consolidating Linux servers on mainframes, refer to:
http://www.redbooks.ibm.com/redpapers/pdfs/redp0222.pdf
The following sections cover topics that relate to integrating mainframes into the Layer 2/ Layer 3
infrastructure specifically from the IP addressing and routing point of view.

956513 2-29
Design Details
Attachment Options
Mainframes can attach to the data center infrastructure in many different ways which mainly derive from
the use of either the OSA card (a Gigabit Ethernet card) or from the use of the ESCON connection (a
serial connection on fiber). Figure 2-14 shows the different combinations of these two attachment types.
Figure 2-14 Mainframe Attachment Options
Enterprise
Campus Core
core 1 core 2
ESCON director
87446
Mainframe 1 Mainframe 2 Mainframe 3 Mainframe 4 Mainframe 5
Mainframe 1 is attached to the gigabit port of an access switch with an OSA card. Mainframe 2 has two
OSA cards and attaches to both aggregation switches. Each OSA card belongs to a different IP subnet.
The links from the mainframe are Layer 3 links. On the Catalyst 6500, assign a separate VLAN to each
link or assign an IP address to each Gigabit port that attaches to the mainframe.
For more information about attaching mainframes to Cisco 6500 with OSA cards, refer to:
http://www-1.ibm.com/servers/eserver/zseries/networking/cisco.html
Mainframe 3 has an ESCON connection to a 75xx/72xx router. The router in turn attaches to the
aggregation 6500s with two Layer 3 links. On the Catalyst 6500, assign a separate VLAN to each link
or assign an IP address to each port that connects to the router. Mainframe 4 and 5 attach to an ESCON
director which in turn connects to a 75xx/72xx router. Connectivity from the router to the aggregation
switches is similar to Mainframe 3's router.
A single mainframe can use multiple attachment types at the same time. For more information, refer to
the redbook “Networking with z/OS and Cisco Routers: An Interoperability Guide” at:
http://www.redbooks.ibm.com/pubs/pdfs/redbooks/sg246297.pdf

2-30 956513
Design Details
IP Addressing
Mainframes attach either to the access switches or the aggregation switches, like any other server. The
main differences between mainframes and normal servers is how the mainframe handles multiple
interface cards and how LPARs and VMs come into play for the IP addressing.
The easiest approach to configuring a dual attached mainframe is to have the OSA interfaces backing up
each other. If you have, for example, a mainframe with two OSA cards attached to the same VLAN, you
could have one OSA with IP address 10.0.2.1 and the other one with IP address 10.0.2.2. If the OSA with
IP address 10.0.2.1 fails, the other OSA would take over the same IP address 10.0.2.2 besides its own IP.
The above approach is similar to configuring dual attached servers. Mainframes offer more options than
this. Mainframes have the concept of static virtual IP address (VIPA), which is an IP address not
associated with any card. The mainframe can process the traffic with destination IP of VIPA from any
interface card.
Additionally, inside each mainframe there are several logical partitions, sharing access to either the
channel or the OSA cards. When the mainframe implements a TCP/IP stack, each LPAR has its own IP
address on each interface adapter. Each LPAR can also be configured with a static VIPA.
Figure 2-15 exemplifies the case where a mainframe attaches to a channel attached router and to an
Ethernet switch. The LPARs have an IP address for the channel connection as well as the OSA
connection. As an example LPAR1 has IP address 10.0.0.1 on the ESCON adapter and 10.0.1.1 on the
OSA adapter. Similar addressing applies to the other LPARs: each LPAR has two IP addresses, one on
the 10.0.0.x subnet for the ESCON connection and one on the 10.0.1.x subnet for the OSA connection.
Whether you have an ESCON adapter or an OSA adapter is irrelevant: with two OSA adapters you would
still have two “internal” subnets.
Figure 2-15 LPARs, Attachment, and Addressing
Mainframe
LPAR1 LPAR2 LPAR3
VIP A 10.0.3.1 10.0.3.2 10.0.3.3
10.0.1.1 10.0.0.1
10.0.1.2 10.0.0.2
10.0.1.3 10.0.0.3
OSA ESCON
87447
If you want to give an IP address to an LPAR that does not belong to a specific interface, you can
configure a static VIPA. In Figure 2-15 the static VIP for LPAR1 is 10.0.3.1, for LPAR2 is 10.0.3.2 and
for LPAR3 is 10.0.3.3. The static VIPA can be advertised by the LPAR via OSPF or RIP with a next hop
that equals the IP address of the LPAR on the 10.0.0.x subnet and 10.0.1.x subnet. Should one physical
adapter fail, routers can forward traffic destined to the VIPA to the remaining interface.

956513 2-31
Design Details
OSPF Routing on the Mainframe

The routing configuration of regular servers is typically limited to a default route pointing to the default
gateway. On mainframes it makes more sense to use dynamic routing because of the number of IP
addresses hosted on a single machine and the presence of multiple interfaces. OSPF is the preferred
choice.
Figure 2-16 Mainframe Attachment with OSA Cards and OSPF Routing
Enterprise Area 0
Campus Core
Layer 3
ABR ABR
core 1 core 2 Layer 3
Layer 3 default default Layer 3
Layer 3
Layer 3 Layer 3 Layer 3 Layer 3

Layer 3 Layer 3
OSPF-STUB AREA
87448
Mainframe 1 Mainframe 2 Mainframe 1 Mainframe 2
Figure 2-16 shows, on the left, the physical attachment of mainframes with dual OSA cards. The logical
topology is on the right of the drawing. The presence of mainframes in the data center architecture does
not change the OSPF recommendations given in the OSPF section. The recommendation is to have a stub
area or totally stubby area and to make either the core routers or the aggregation routers (MSFCs) the
ABRs. Figure 2-16 shows the core routers being used as ABRs. Another recommendation is to make sure
that the mainframe does not become the default route for a given segment, which can be achieved by
correctly configuring the OSPF priority on the aggregation switches.
For more information about attaching mainframes to Catalyst 6500 and using OSPF, refer to the
document titled “OSPF Design and Interoperability Recommendations for Catalyst 6500 and
OSA-Express Environments” available at:
http://www-1.ibm.com/servers/eserver/zseries/networking/pdf/ospf_design.pdf
Sysplex
The sysplex is a clustering technology to virtualize a number of mainframes as if they were a single
machine. If one of the mainframe components fails (for example, an LPAR), the system works around
the failure and distributes new requests to the remaining machines. The components can belong to a
single mainframe as well as to separate physical machines.

2-32 956513
Design Details
The parallel sysplex uses the Cross Coupling Facility (XCF) as the means to exchange messages between
systems.
In Figure 2-17 you can see what a sysplex looks like. The sysplex main components are the following:
• Up to 32 components (like LPARs) which can operate as a single virtual
• XCF: allows programs to communicate within the same system or across systems, as if it was shared
memory between processors. The coupling facility is a processor running a specific piece of
software and connecting to the systems on separate machines via an optical connection.
• Sysplex Timer is used to synchronize the operations
• Workload Manager: This component runs in every LPAR and provides metrics that can be used to
decide how incoming requests should be distributed.
• Sysplex Distributor: This component acts as a load balancing device for TCP/IP connections. Return
traffic goes directly to the client bypassing the distributor.
Figure 2-17 Sysplex
XCF
ESCON
Director
Channel router Access switch

87449
In terms of IP addressing, you typically configure the sysplex with static VIPA, dynamic VIPA (DVIPA),
and distributed DVIPA. The topic of static VIPA has been already covered in the IP Addressing section.
The DVIP is categorized as:
• Dynamic Virtual IP Address (DVIPA) refers to the capability of an LPAR (for example, LPAR2) to
take over a VIPA (DVIPA to be more accurate) from another LPAR (for example, LPAR1) in the
event of failure (of LPAR1).
• Distributed DVIPA is a “virtual” IP address that identifies one application that is running on multiple
machines or multiple LPARs. The sysplex distributor sends incoming connection to the available
TCP/IP stacks based on several parameters including the load information derived from the
Workload Manager.
Alternatively, rather than forwarding packets through multiple mainframe TCP/IP stacks, you can use
Sysplex Distributor to send the load information to a Forwarding Agent (FA) in 6500 switches or 7xxx
routers, using Multi Node Load Balancing (MNLB).

956513 2-33
Design Details
Figure 2-18 DVIPA and Static VIPA
Mainframe
LPAR1 LPAR2 LPAR3
VIPA 10.0.3.1 10.0.3.2 10.0.3.3
DVIPA 10.0.80.1 10.0.80.1 10.0.80.1
10.0.1.1 10.0.0.1
10.0.1.2 10.0.0.2
10.0.1.3 10.0.0.3
OSA ESCON
87450
Figure 2-18 shows a mainframe with 3 LPARs. Each LPAR runs its own TCP/IP stack and can be
accessed either through the ESCON channel or the OSA adapter. Clients can connect to LPAR1 by
10.0.1.1 and to LPAR2 by 10.0.1.2 through the OSA adapter. You can access LPAR1 and LPAR2 also
through 10.0.0.1 and 10.0.0.2 through the ESCON channel.
Typically you do not care which interface is used to access the LPAR. This is why the concept of static
VIPA was introduced: VIPA is equivalent to a loopback address on the LPAR. If you want to access
LPAR1, you connect to 10.0.3.1, the routing devices downstream of the mainframe receive OSPF
advertisements for 10.0.3.1 with a next-hop equal to 10.0.0.1 and 10.0.1.1. LPAR2 and LPAR3 can be
accessed by 10.0.3.2 and 10.0.3.3 and the downstream router has two routes for each IP, each route
pointing to either the IP address of the OSA adapter or the IP address of the ESCON channel.
Note Figure 2-18 shows a clustered DVIPA, but DVIPA may have only a single instance. With one instance
of DVIPA, the IP address is moved around within the cluster, when an application fails.
Now imagine that you run the same application on the three LPARs, and while the user mainly connects
to LPAR1, if LPAR1 fails, you want that new incoming requests go to LPAR2. For this to happen, you
need to share an IP address between LPAR1 and LPAR2. This IP address is called DVIPA. As you see
in Figure 2-18, the DVIPA is the same on the three LPARs.
This example can be extended to a number of mainframes coupled with sysplex. If each system runs one
single LPAR, the static VIP provides high availability for each mainframe in case of failure of one
interface adapter. The DVIPA instead provides high availability for the applications because the fact that
is shared across multiple machines.
For more information about the Sysplex environment please refer to the redbook TCP/IP in a Sysplex:
http://www.redbooks.ibm.com/pubs/pdfs/redbooks/sg245235.pdf

2-34 956513
Summary
Summary
A good infrastructure design is critical in order for any data center to be able to meet the desired service
level expectations. The infrastructure has many components. A good the Layer 2 and Layer 3
infrastructure, while keeping in mind the requirements of the appliances that provide additional data
center services, is crucial to a good data center design. This data center architecture follows the proven
Cisco multi-layer design.

956513 2-35
Summary

2-36 956513
C H A P T E R 3
Enhancing Server to Server Communications
Traffic in a data center can be categorized as either client-to-server traffic or server-to-server traffic. An
example of client-to-server traffic is traffic that flows between a client’s browser and a web-server.
Server-to-server traffic is traffic that flows between two servers belonging to the same organization, for
example: a web-server communicating with an application server or a server backing up its data to
another server.
Normally, when server farms use content switching technology to scale performance, all traffic in the
data center flows through a content switch, which performs load balancing and traffic state information
tracking. It is essential that all the client-to-server traffic go through a content switch to take advantage
of the aforementioned features. However, when dealing with server-to-server traffic, a some traffic
requires load balancing and some does not. The server-to-server traffic that does not need load balancing
also goes through the content switch because they are configured as the server’s default gateway.
Note When server-to-server traffic that needs to be load balanced and both the source and destination are on
the same VLAN source-NAT is required
Figure 3-1 displays server-to-server communication in a typical normal data center. Servers belonging
to different subnets in the same data center exchange traffic through the content switch. Servers located
in different buildings of the same intranet communicate with servers in the data centers through content
switches.

956513 3-1
Chapter 3 Enhancing Server to Server Communications
Figure 3-1 Normal Traffic Path for Server-to-Server Communication
Internet
Client
Intranet
Backup
server
Content Switch Content Switch

module 1 module 2
76363
Web servers Application servers
Note Figure 3-1 presents a logical view of the Content Switching Module (CSM) as a separate device from
the Catalyst 6500. This is done to illustrate the point. The CSM reside in a slot in the Catalyst 6500
chassis.
This chapter describes how to route server-to-server traffic that does not require load balancing through
the backbone infrastructure instead of the content switch without affecting client-to-server
communications that require load balancing. This requires directly connected subnets, such as the
VLANs where the servers are located, to communicate directly bypassing the CSM. All the traffic that
matches a default route, such as traffic that needs to go back to the Internet or intranet, goes through the
CSM. The network administrator controls which traffic flows through the backbone and which traffic
goes through the CSM by changing the distribution switches routing table.
Figure 3-2 depicts, with the dotted line, the traffic path for client-to-server communications. The dashed
line identifies the traffic path for server-to-server communications after applying the configuration
recommended in this application note. The key feature used to achieve this result is Policy Based
Routing (PBR), specifically the “ip default next-hop” option. The feature does not have a performance
impact on the Catalyst 6500 because it is implemented in the Layer 3 ASIC, which processes traffic at
wire-speed.
Note Server-to-server communication may or many not require load balancing. This chapter optimizes the
case where you want servers to speak directly without going through a load balancer, such as with
application servers talking to data base servers or for any server tier communicating with another tier
directly. Multi-tier environment servers often have a stub that keeps communication open with a number
of servers that belong to a further tier. This design chapter ensures that the Catalysts, instead of the
content switches, switch the connections opened between these tiers.

3-2 956513
Figure 3-2 Optimized Traffic Path
Internet
Client
Intranet
Backup
server
Content Switch Content Switch

module 1 module 2
Client-server
communication
76364
Server-to-server
communication Web servers Application servers
Benefits
There are multiple benefits to using the configuration described in this chapter. PBR makes configuring
the content switch much easier. The content switch is no longer required to process unnecessary traffic
and server-to-server communication performance (wire-speed) increases because of the use of the
infrastructure instead of the content switch appliance.
The following list details the advantages to implementing the server-to-server communications outlined
in this chapter:
• Wire-speed server-to-server communication
• Simplifies the content switch configuration
• Default gateway using Hot Standby Router Protocol (HSRP) provided on the Multilayer Switching
Feature Card (MSFC) on the Catalyst 6500
• Relieves the content switching module from processing unnecessary traffic
• No ACL configuration required
• No Client NAT required on the content switch
• Very simple additional configuration (compared to the standard content switch configurations)

956513 3-3
System Components
Restrictions
This chapter requires that your subnet is not present in the routing table of the distribution switches in
the data center. The assumption is that the routing table contains only routes specific to the data center.
Push traffic that leaves the data center to the core by means of the default route.
Because of the previous restriction, terminating BGP on the distribution switches breaks the normal
behavior because it causes the clients’ traffic to bypass the content switches.
This design chapter is about enhancing server-to-server communications and does not address
client-to-server communications. Client-to-server communication still flows through content switches.
For example, streaming traffic originated by a server in the data center does not bypass the content
switch. When configuring health checks on the CSM, do not use Layer 2 probes like ARP because the
CSM is not Layer 2 adjacent to the servers.
Note ARP “probes” are not user configurable. When you do not define a probe, the CSM relies on ARPs to
decide whether a device is available or not. Cisco recommends useing ICMP probes or higher layer
protocols to verify that a server is reachable.
System Components
Hardware Requirements
The equipment needed to support wire-speed server-to-server communications is as follows:
• Catalyst 6500s with supervisor 2 configured with MSFC2 to achieve wire speed PBR.
• A pair of CSMs installed in the Catalyst 6500s.
Note The principles described in this chapter also apply to Content Services Switch (CSS) family
environments with similar topologies as the one depicted in Figure 3-1 and Figure 3-2.
Software Requirements
The configuration requires that you use the “ip default next-hop” option, which is available (operating
at wire speed) in the following IOS releases:
• Catalyst 6500 IOS 12.1(11b)E (used as supervisor IOS)
• CatOS 7.2 (the MSFC running IOS 12.1(11b)E)
For more information about the changes introduced in 12.1(11b)E, refer to the bug id CSCdw28489 and
for CatOS 7.2, refer to the bug id CSCdw39689.

3-4 956513
System Components
Features
Policy Based Routing

This configuration relies on PBR. PBR supports traffic routing based on policies rather than the
destination IP address. Examples of policies are:
• Incoming VLAN
• Source IP address
• Protocol
You can apply policy routing on a per-VLAN basis using route-maps. The route-map is like a routing
table. It defines the next-hop for traffic that matches the policy. The next-hop can be on a different VLAN
than where the traffic originated. The key message from this section is that policy routing allows you to
route traffic based on the incoming VLAN.
Note PBR is implemented in hardware ASICs on the Catalyst 6500 if used with Supervisor 2.
Default Next-Hop
This design chapter uses a specific feature of PBR called “default next-hop.” The MSFC performs
routing table lookups on incoming server traffic as normal. If the destination IP address matches a route
in the main routing table, the MSFC forwards the traffic accordingly. If the destination IP address does
not match any route but the default route, and default next-hop is enabled on the incoming VLAN, the
MFSC uses the route-map next-hop. In this case, the next-hop defined by PBR is the CSM. Figure 3-3
illustrates the algorithm.

956513 3-5
Network Topology
Figure 3-3 IP Default Next-Hop Algorithm
Incoming traffic on Vlan 10
Route-map?
No Yes
Lookup routing table Lookup routing table
Yes
Matches a route?
No
Route based on Route to the route-map
76365
Destination IP next-hop
Static Routes
Static routes are routes defined with the command ip route <ip address> <mask> <next-hop IP
address>. To implement the proposed design that allows direct communication between local and remote
servers, you must add a static route on the MSFCs in the data center for the IP address of the remote
server(s). The next-hop IP address is the address of the core routers.
Network Topology
Topology Description
The basic topology is described in Figure 3-4. The two routers represent the two MSFCs in the
aggregation layer. These two routers have several VLAN interfaces. Figure 3-4 identifies two VLANs:
VLAN 10 (the server VLAN) and VLAN 5 (the CSM VLAN). The MSFCs provide the default gateway
for the servers, which is an HSRP address. The MSFCs also provide the default gateway for the CSM,
which is again an HSRP address.
Note The CSM is used in a one-arm configuration with a single VLAN instead of the traditional Client and
Server VLANs.

3-6 956513
Network Topology
Figure 3-4 Topology Overview
Alias IP
VLAN 5
Content Switch HSRP address Content Switch
module 1 module 2
To the core
MSFC-AGG1 MSFC-AGG2
HSRP address
PBR PBR
Vlan 10
ACC1
76366
Notice the CSMs are one hop away from the servers. It is essential that load balanced traffic go through
the routers before getting to the CSM. The CSM offers a virtual address as a next-hop for the router. This
address is called an “alias” on the CSM and is equivalent to an HSRP address on a router.
The horizontal arrows in the diagram represent redundancy protocol communication, which takes place
on a specific VLAN between the peer routers (HSRP) or between the pair of CSMs (CSRP). These
devices are configured to agree to display a common IP address to their clients, which eliminates the
single point of failure on the CSMs or the routers. If the master device fails, the backup takes over and
is reachable with the same IP address.
Apply PBR to VLAN 10 and define the CSM as the default next-hop. As Figure 3-4 depicts, PBR pushes
traffic from VLAN 10 to VLAN 5. In fact, the default next-hop is on VLAN 5 even whenPBR is applied
to VLAN 10.
Configure a static route, pointing to the CSMs, on the MSFCs to push traffic destined to a Virtual IP
address (VIP). Clients use the VIP as the IP address to connect to the services offered by the server farm.
The CSM assigns a client connection, which is destined to a VIP, to a specific server in the server farm.
The load balancing algorithms, otherwise known as predictors, defined on the CSM select a server to
which the content switch sends the client request.
Note Static routers are not regularly required. In this instance, they give you the flexibility of multiple VIP on
different subnets.
This topology does not show the VLAN used by the CSMs for the redundancy protocol or other VLANs
that provide access to additional servers in the server farm. The Configuration Description section
provides additional details.

956513 3-7
Network Topology
Traffic Paths for Client-to-Server Traffic

This section describes the path taken by traffic coming from the Internet or intranet to the data center. In
this example, the client is browsing www.example.com, which resolves to the IP address <VIP
ADDRESS>.
Once the traffic reaches the data center, the MSFC routes it to the CSM by means of a static route.
Source IP = Client IP address - Destination IP = VIP address - Destination MAC = CSM ALIAS
MAC
The CSM load balances this traffic to the appropriate server. The destination IP and MAC addresses are
rewritten using the server IP address and the HSRP virtual MAC address of the MSFC on VLAN 5 (the
CSM default gateway).
Source IP = Client IP address - Destination IP = Server IP address – Destination MAC =
MSFC VLAN 5 HSRP MAC
The MSFC routes the traffic to the server.

Source IP = Client IP address – Destination IP = Server IP address – Destination MAC =
Server MAC
Server-to-Client Traffic
Now the server puts the traffic back and sends it to its default gateway, the MSFC.
Source IP = Server IP address – Destination IP = Client IP address – Destination MAC =
MSFC HSRP VLAN 10 MAC
The MSFC performs a route table lookup and finds that the client IP address does not match any route
except the default route. Because the server VLAN has a route-map, the MSFC sends this traffic back to
the CSM. Otherwise, the MSFC sends the traffic directly to the client:
Source IP = Server IP address – Destination IP = Client IP address – Destination MAC = CSM
ALIAS MAC
The CSM rewrites the source IP address to the VIP address.

Source IP = VIP address – Destination IP = Client IP address – Destination MAC = MSFC HSRP
VLAN 5 MAC
Server-to-Server Traffic (Servers Directly Connected)

A server belonging to VLAN 10 (Server A) in the data center decides to communicate with a server
belonging to VLAN 20 (Server B) in the same data center. Therefore, server A sends traffic to the MSFC.
Source IP = Server A IP address in VLAN 10 - Destination IP = Server B IP address in VLAN
20 - Destination MAC = MSFC HSRP VLAN 10 MAC
The MSFC does a routing table lookup. Since it finds that Server B in VLAN 20 belongs to a directly
connected subnet, it routes this traffic directly to Server B.
Source IP = Server A IP address in VLAN 10 - Destination IP = Server B IP address in VLAN
20 - Destination MAC = Server B MAC in VLAN 20
Note If you need to load balance server-to-server communication, you must configure source NAT on the CSM
for server originated connections. This design is meant to optimize “direct” server-to-server
communication.

3-8 956513
Configuration Description
Server-to-Server Traffic (Remote Servers)

The question now is how to bypass a content switch when a server needs to communicate with a remote
server (a backup sever for example). You can bypass the content switch as long as there is a specific route
to the remote server in the MSFC routing table. If the route is not present, you must configure a static
route to the remote server with the core routers as the next-hop.
Configuration Sequence and Tasks

Perform the following steps to configure direct server-to-server communications.
Step 1 Define a route-map that contains the CSM alias IP address.

Step 2 Apply the route-map to the server VLANs.
Step 3 Define a “server VLAN” on the CSM.
Step 4 Define the same VLAN as the “server VLAN” on the MSFC for “MSFC – CSM” communication.
Step 5 Define the MSFC as the default gateway on the CSM.
Step 6 On the MSFC, define a static route for the Virtual IP address pointing to the CSM “alias IP address”.
Compared to a regular configuration the only additional steps are:

• Define a route-map that contains the CSM IP address
• Apply the route-map to the server VLANs
The previous configuration steps simplify the configuration of the CSM because the following
components are no longer required on the CSM:
• A client VLAN
• Multiple server VLANs (you only need to configure one)
Configuring the Server VLANs on the MSFC

The configuration for PBR on the MSFC is as follows:
route-map server-client-traffic permit 10
set ip default next-hop <CSM ALIAS IP ADDRESS>
There is no requirement to have additional route-maps unless there are additional pairs of content
switches. Apply the above route-map to each VLAN used by the load-balanced servers, for example
VLAN 10 in Figure 3-4.
interface VLAN10
description server_side_VLAN
ip address <IP ADDRESS VLAN10>
ip policy route-map server-client-traffic

956513 3-9
Configuring the “MSFC-to-CSM” VLAN

You can configure the CSM to use a single VLAN thus not requiring a client and a server VLANs. Even
if this VLAN is referred to as the “server VLAN,” it is just one VLAN. All servers are one hop away
from the CSM. PBR takes care of returning the traffic to the CSM. Under the “server” VLAN
configuration, define the MSFC as the “gateway” for the CSM. The VLAN between the CSMs in
Figure 3-4 is VLAN 5. Configure VLAN 5 as the “server” VLAN on the CSMs.
ip slb VLAN 5 server
ip address <CSM MAIN IP ADDRESS>
gateway <MSFC IP ADDRESS>
alias <CSM IP ADDRESS>
On the MSFC, define the same VLAN to exchange traffic with the CSM.
interface VLAN5
description msfc_to_csm_VLAN
ip address <MSFC IP ADDRESS>
On the MSFC, configure a static route pointing to the CSM alias IP address:
ip route <VIP ADDRESS> <CSM ALIAS IP ADDRESS>
Sample Configurations
This section provides a detailed description of the topology that was tested to support this application
note.
Infrastructure Details
This section covers the topology represented in Figure 3-5. As you can see, this design uses the
traditional Cisco multi-layer design. For more details, refer to
http://www.cisco.com/warp/public/cc/so/neso/lnso/cpso/gcnd_wp.htm. In this design chapter, the focus
is on the access and aggregation layers.

3-10 956513
Figure 3-5 Detailed Server Farm Topology
To the core To the core
VLAN 5, 10, 20, 100

5, 100 5, 100
AGG1 AGG2 Content Switch

Content Switch Vlan 20
module 1 module 2
Vlan 10
B B
ACC1 ACC2
10.14.0.12 10.14.0.13 10.15.0.12 10.15.0.13

gw 10.14.0.1 gw 10.14.0.1 gw 10.15.0.1 gw 10.15.0.1
CSM1 ALIAS Master 10.6.0.6 CSM2 ALIAS Backup 10.6.0.6
HSRP Master for 10.14.0.1 on VLAN 10 HSRP Backup for 10.14.0.1 on VLAN 10
AGG1 HSRP Master for 10.15.0.1 on VLAN 20 AGG2 HSRP Backup for 10.15.0.1 on VLAN 20
76367
HSRP Master for 10.6.0.1 on VLAN 5 HSRP Backup for 10.6.0.1 on VLAN 5
Servers are connected to VLAN 10 and VLAN 20. Servers in VLAN 10 are load balanced by the content
switch, whereas servers in VLAN 20 are not load balanced in this topology.
The access switches (ACC1 and ACC2) carry VLAN 10 and VLAN 20, respectively. Uplinkfast and
backbonefast are enabled on these switches andportfast is enabled on the server ports. The uplinks from
the access switches connect to the aggregation switches (AGG1 and AGG2). The uplinks can be trunks
if they need to carry more than one VLAN.
The CSM is the load-balancing appliance used in this application note. In Figure 3-5, the CSM is
represented as an external appliance to simplify the description of the configuration. In reality, the CSM
is a card in the Catalyst 6500. The CSM uses two VLANs: VLAN 5 and VLAN 100.
VLAN 5 provides communication between the routers and the CSM as well as for regular traffic. VLAN
100 is the fault tolerant VLAN. The two CSMs use the fault tolerant VLAN to exchange redundancy
information that identifies which CSM is active and which is backup.
The aggregation switches (AGG1 and AGG2) are trunking the access VLANs (10,20), and the CSM
VLANs (100, 5) on an Etherchannel. AGG1 and AGG2 are the root and the secondary root switches
respectively for all of the VLANs. Backbonefast is enabled. VLAN 5 and VLAN 100 are trunked
between AGG1 and AGG2; they do not need to be carried to the access layer. The MSFCs provide the
default gateway support to the servers by means of HSRP (10.14.0.1 and 10.15.0.1 in this drawing). The
MSFCs also provide the default gateway support to the CSM appliance by means of HSRP (10.6.0.1).
Apply PBR on VLAN 10 and VLAN 20 on both AGG1 and AGG2 and use the default next-hop option
to push traffic to the content switch.

956513 3-11
Configure a static route on the MSFC to point the VIP address for the server farm to the alias address of
the CSM (10.6.0.6).
PBR Details
Apply route-maps to the VLANs where there are servers load balanced by the content switch. In the
test-bed, the only servers load balanced by the CSM are in VLAN 10. The next-hop IP address belongs
to any subnet regardless of the subnet where you apply the route. The next-hop, in this topology, is the
CSM alias IP that does not belong in the subnet where the route-map is applied.
set ip default next-hop 10.6.0.6
!
The route-map is applied to VLAN 10:
interface VLAN10
ip address 10.14.0.3 255.255.255.0
no ip redirects
standby 1 ip 10.14.0.1
standby 1 priority 105
standby 1 preempt
standby 1 track GigabitEthernet1/1
Traffic coming from any other VLAN is routed normally based on the routing table. The following is
routing table output:
AGG1#sh ip route
C 10.15.0.0/24 is directly connected, VLAN20
S 10.10.10.150/32 [1/2] via 10.6.0.6
S 10.200.1.20/32 [1/0] via 10.10.20.10
S* 0.0.0.0/0 [1/0] via 10.10.20.10
According the algorithm, the routing table, and the route-map, and traffic coming from VLAN 10 take
the following routes:
• Destination IP = 10.15.0.12 (in the routing table) goes to 10.15.0.12 (bypassing the CSM)
• Destination IP = 171.68.173.10 (not in the routing table) goes to 10.6.0.6 (CSM)
• Destination IP = 10.200.1.20 (in the routing table) goes to the next-hop router 10.10.20.10
(bypassing the CSM)
The static host route is 10.200.1.20. This address represents a remote server in the test-bed. The static
route was purposefully configured to bypass the CSM.
CSM1 Configuration
ip address 10.6.0.4 255.255.255.0
gateway 10.6.0.1
alias 10.6.0.6 255.255.255.0
!
ip slb serverfarm SERVERFARM1
nat server
no nat client
real 10.14.0.12
inservice

3-12 956513
real 10.14.0.13
inservice
!
ip slb vserver TEST
virtual 10.10.10.150 tcp 0
serverfarm SERVERFARM1
sticky 10
replicate csrp sticky
replicate csrp connection
persistent rebalance
inservice
!
ip slb ft group 1 VLAN 100
priority 20
preempt
!
Note There is a single VLAN (the server VLAN). The default gateway is the MSFC HSRP address on VLAN
5. There is no need to add more VLANs when you define more server farms, or virtual servers (vservers).
CSM2 configuration
ip address 10.6.0.5 255.255.255.0
gateway 10.6.0.1
alias 10.6.0.6 255.255.255.0
!
ip slb serverfarm SERVERFARM1
nat server
no nat client
real 10.14.0.12
inservice
real 10.14.0.13
inservice
!
ip slb vserver TEST
virtual 10.10.10.150 tcp 0
serverfarm SERVERFARM1
replicate csrp sticky
replicate csrp connection
persistent rebalance
inservice
!
ip slb ft group 1 VLAN 100
priority 10
preempt
!
Note This configuration is basically the same as CSM1 except for the IP address on VLAN 5 and the priority
for the “fault tolerant” VLAN.
MSFC-AGG1 Configuration
ip address 10.10.20.11 255.255.255.0
no ip proxy-arp
!
description to_agg2
switchport

956513 3-13

no ip address
channel-group 1 mode on
!
description to_acc1
switchport
no ip address
!
description to_acc2
switchport
no ip address
!
description to_agg2
switchport
no ip address
!
interface VLAN5
description internal_csm_VLAN
ip address 10.6.0.2 255.255.255.0
no ip redirects
standby 1 preempt
!
interface VLAN10
ip address 10.14.0.6 255.255.255.0
no ip redirects
standby 1 preempt
!
interface VLAN20
ip address 10.15.0.6 255.255.255.0
no ip redirects
standby 1 preempt
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.10.20.10
ip route 10.200.1.20 255.255.255.255 10.10.20.10
ip route 10.10.10.150 255.255.255.255 10.6.0.6
no ip http server
!
!

3-14 956513
The MSFC configuration shows a trunk plus channel between AGG1 and AGG2. VLAN 5 sends traffic
to the CSM. The server farm is on VLAN 10, therefore, the route-map applies to VLAN 10. The
application servers use VLAN 20. There is no need to configure a route-map because those application
servers are not load balanced by the CSM.
Note A static route points the traffic destined to the VIP address (10.10.10.150 in this example) to the CSM
“alias” IP address (10.6.0.6 in this example).
Note You may need to configure HSRP tracking for the server VLANs depending on how routing is configured
in the Data Center.
MSFC-AGG2 Configuration
ip address 10.10.11.21 255.255.255.0
no ip proxy-arp
!
description to_agg1
switchport
no ip address
!
description to_acc1
switchport
no ip address
!
description to_acc1
switchport
no ip address
!
description to_agg2
switchport
no ip address
!
interface VLAN5
ip address 10.6.0.3 255.255.255.0
no ip redirects
standby 1 preempt
!
interface VLAN10
ip address 10.14.0.3 255.255.255.0
no ip redirects

956513 3-15
standby 1 preempt
!
interface VLAN20
ip address 10.15.0.3 255.255.255.0
no ip redirects
standby 1 preempt
!
ip route 0.0.0.0 0.0.0.0 10.10.11.20
ip route 10.200.1.20 255.255.255.255 10.10.11.20
ip route 10.10.10.150 255.255.255.255 10.6.0.6
!
!

3-16 956513
C H A P T E R 4
Addressing Spanning-Tree Limitations
Spanning-Tree ensures that only one forwarding logical path is present at any given time for a specific
VLAN despite having multiple physical paths for redundancy. If multiple paths were present and
Spanning-Tree was disabled, you would have a Layer 2 loop which means that packets would be
replicated an infinite number of times.
Even with Spanning-Tree enabled there are some failure scenarios where a Layer 2 loop is possible. This
chapter addresses Spanning-Tree limitations and identifies how to prevent loops by using the features
available on Cisco switches.
Spanning Tree Overview

The original IEEE standard for spanning-tree is called 802.1d. Recently IEEE introduced improvements
to 802.1d:
• IEEE 802.1s (also called Multiple Spanning-Tree) defines multiple instances of Spanning-Tree
• IEEE 802.1w (also called Rapid Spanning-Tree Protocol) is a standard to improve the convergence
time of Spanning-Tree
Spanning-tree is now available on Cisco products in three main implementations:
• Cisco Per VLAN Spanning-Tree Plus (PVST+) is the Cisco implementation of Spanning-Tree based
on 802.1d. A separate instance of Spanning-Tree runs for each VLAN, which allows building
different logical topologies on a per-VLAN basis.
• Multi Instance Spanning-Tree (MST) is the combination of 802.1s and 802.1w
• Rapid PVST+: which is the combination of PVST+ and 802.1w
For more information about the spanning-tree algorithm and which spanning-tree to choose for your
environment, refer to Chapter 2, “Data Center Infrastructure.”
There are failures from which no spanning-tree implementation can prevent loops, specifically cases
where BPDUs are lost because of unidirectional links or software bugs.
Loopguard is Cisco proprietary optimization for the standard Spanning Tree Protocol. Loopguard
protects Layer 2 networks from loops caused by malfunctioning NICs, busy CPUs, and any reason that
prevents the normal forwarding of BPDUs. Loopguard is useful in switched networks where switches
are connected by point-to-point links, as it is the case in most modern campus and data center networks.
Loopguard was introduced on Cisco Catalyst switches starting from CatOs 6.2 and IOS 12.1(11b)EX,
and on the 2950/3550s starting from 12.1(9)EA1.

956513 4-1
Chapter 4 Addressing Spanning-Tree Limitations
Protocols Overview
Spanning tree is a link management protocol that provides path redundancy while preventing undesirable
loops in the network. For ethernet networks to function properly, only one active path must exist between
two stations.
When you create fault-tolerant networks, you must have a loop-free path between all nodes in a network.
The spanning-tree algorithm calculates the best loop-free path throughout a switched network. Switches
send and receive spanning-tree frames (BPDUs) at regular intervals. Switches do not forward these
frames, but use them to construct a loop-free path that yet connects every two pair of LANs within the
network.
Root
Each switch has a unique numerical identifier (switch ID). The switch with the lowest switch ID is the
Root. In 802.1d the root switch is in charge of generating BPDUs at regular intervals. The downstream
switches forward those BPDUs upon receiving them from the root. In 802.1w every switch generates
BPDUs at regular intervals, as opposed to just forwarding BPDUs generated by the root switch.
Designated Bridge
On each LAN segment one switch is in charge of forwarding frames from the segment towards the root.
This switch is the Designated Bridge. The designated bridge is also responsible for sending BPDUs on
a given segment.
Root Port and Designated Ports

Every switch selects a port as the Root Port, which is the port with the lowest cost path to the root. The
switch receives BPDUs from the root port and forwards (or generates) BPDUs out of the designated
ports. Designated ports are those ports that connect the switch to segments for which it is the designated
bridge.
Forwarding and Blocking

The root ports and the designated ports are placed in spanning-tree forwarding mode; all the other ports
are placed in blocking mode. When a port is in blocking mode it does not forward or receive any traffic.
All the ports of the root bridge are designated ports, thus they are all in forwarding mode, except from
self-looped ports.
BPDU Format
Each BPDU contains the following information: the root bridge id, the cost of the path to the root, and
the bridge id of the bridge that is forwarding/generating the BPDU. When a switch receives a BPDU, it
compares the content of the BPDU with the local information.
Hello Time and MaxAge

In 802.1d-like implementations, the root switch generates BPDU packets every hello time seconds,
which according to the IEEE standards, should be 2s. Each port of the switch has a timer associated with
the BPDU information and receiving the BPDUs refreshes this timer. The information associated with a

4-2 956513
port is considered to be stale if the timer reaches MaxAge. The default MaxAge is 20s. When a switch
stops receiving BPDUs from its root port and the MaxAge expires, the switch looks for a new root port,
from the pool of blocking ports. If no blocking port is available, it claims itself to be the root on the
designated ports.
In spanning-tree implementations that use 802.1w, every switch generates BPDUs every hello time
seconds (2s). In 802.1w, three missing three hello packets from a neighbor is considered a network
failure. As a result, it takes 6s for a switch to detect a failure which does not involve a link-down.
Port States
In spanning-tree implementations based on 802.1d, when a transition occurs from the blocking state into
the forwarding state, the port goes through a listening phase (where the port simply listens to BPDU to
find out if it has to go back to blocking mode) and a learning phase (where a port still blocks frames but
it also learns the MAC addresses). The port eventually moves to the forwarding state. The time it takes
to transition from one state to another one is called forwarding delay (15s).
In spanning-tree implementations that adopt 802.1w, there is no listening state. The states are
categorized as follows:
• Discarding, which is when the switch does not learn MAC addresses and drops traffic. For the
purpose of this discussion, discarding and blocking are equivalent terms.
• Learning state occurs when a port learns MAC addresses but drops traffic.
• Forwarding is when a port learns MAC addresses and forwards traffic.
Under normal circumstances, the transition of the state of a port in 802.1w takes very few seconds, so
you should not see the 30s delay that you normally experience with 802.1d. There are exceptions to this
behavior for non-edge ports connected to a device (a hub for example) which does not speak 802.1w. In
this case the switch port takes 30s before going forwarding.
A designated discarding port which does not receive an agreement after having sent a proposal, slowly
transitions to the forwarding state, falling back to the traditional 802.1d sequence. This is called slow
convergence.
In a well designed network, you should not experience slow convergence.
Convergence Time
The convergence time of 802.1d is around 50s: 20s for the MaxAge timer to expire plus 30s for the
transition of the port from blocking to forwarding. Cisco implemented the following features for PVST+
which make the convergence faster:
• Uplinkfast: which makes it possible to converge in 3 to 5s for uplink failures at the access switches
• Backbonefast: which optimizes the convergence by 20s (MaxAge) time for indirect failures.
With 802.1w you do not need to enable uplinkfast nor backbonefast. 802.1w makes it possible to have
network convergence in seconds thanks to several mechanisms. It uses BPDUs as hello packets and, after
missing 3, triggers a topology recalculation. It relies on a mechanism of proposal and agreement between
switches to quickly transition ports on point-to-point links into the forwarding state, if their role so
requires. It includes a convergence mechanism very similar to UplinkFast, which transitions an alternate
port into forwarding when a switch loses its root port.

956513 4-3
Spanning Tree Limitations
Aggressive Spanning-Tree Timers

Even though it is not a recommended practice, 802.1d spanning-tree timers can be changed to enhance
spanning-tree convergence time. By lowering MaxAge, it is possible to detect indirect failures much
faster. Having a hello time of 1s can also speed up network convergence if MaxAge and forwarding delay
are changed properly. For example, a network of diameter 2 could converge in 14s, a network of diameter
3 in 16.5s … and a network of diameter 7 in 30.5s as opposed to the typical 50s convergence time.
In a multilayer switched environment, a specific macro: set spantree root, allows enhancing the
convergence time of the network by changing the diameter (2 or 3 hops) and the hello time (1s).
As a drawback, the CPU utilization for spanning-tree can increase.
With 802.1w there is no reason to change the default timers because the forwarding delay and the
MaxAge do not have a direct effect on the calculation of the convergence time with the exception of slow
convergence scenarios.
A Simple Spanning-Tree Failure

This example shows how missing BPDUs on a link can cause a Layer 2 loop on a spanning-tree enabled
network.
Failure 1
Consider Figure 4-1.
Figure 4-1 Failure 1 Aggregation 2 Stops Sending BPDUs on Port 3/11
aggregation2 Failure 1: aggregation2 stops

sending BPDUs on port 3/11
3/11
BPD
X U:
RP 3/10
access
3/21
vlan 99
3/22
RP
DP 3/10
DP vlan 99
3/12
87490
aggregation1
Aggregation1 is the root for VLAN 99 and aggregation2 is the secondary root. For some reason,
aggregation2 stops sending BPDUs on port 3/11. In 802.1d after 20s, the MaxAge timer expires on the
access switch for port 3/21. In 802.1w, after three lost BPDUs, the access switch considers aggregation2
dead. As a result, port 3/21 eventually transitions into forwarding.

4-4 956513
This process takes 50s with regular spanning-tree (can be 14s if aggressive timers are used) and, in case
of 802.1w, typically takes 36s. The reason for 36s with 802.1w is because after 3 missed BPDUs the
access switch sends a proposal but never receives an agreement from aggregation 2. As a result port 3/21
takes 30s to become forwarding.
If, at any time, aggregation2 sends BPDUs again, port 3/21 goes returns to the blocking state.
If aggregation2 no longer sends BPDUs on port 3/11, this port goes into forwarding mode, at which point
a Layer 2 loop is created.
Possible Reasons for Spanning Tree Instability

As showed in the previous example, it is possible to have Layer 2 loops in a network where spanning-tree
is enabled. Layer 2 loops can be caused by BPDUs that are lost or not sent on a link for enough time to
cause a spanning-tree recalculation.
There are real situations that can prevent a switch from receiving BPDUs on a link, such as:
• A faulty transceiver that becomes unidirectional
• A switch with a busy CPU that is incapable of sending BPDUs
• A congested link where BPDUs are dropped
The likelihood of the last two events can be argued because their occurrence can only be inferred, but
the occurrence of the first event has been proven to be a cause of several network meltdowns.
Possible Spanning-Tree Failures in the Aggregation and Access Layers

Taking a closer look at the other possible failures that can occur in the access and aggregation layer helps
to explain how Loopguard is useful.
The following examples show spanning-tree correctly taking care of broken links, and examples where
missing BPDUs are causing Layer 2 loops.
The spanning-tree algorithm used in these examples is PVST+ but the results of the examples equally
apply to Rapid PVST+, with the exception of the convergence time which is faster with Rapid PVST+.
The first failure example is available in the A Simple Spanning-Tree Failure section.
Failure 2
Consider Figure 4-2:

956513 4-5
Figure 4-2 Failure 2-Link Between Aggregation1 and Aggregation 2 Fails
aggregation2 Failure 2: the link between

aggregation2 and aggregation1 fails
3/11 B
agg PDU:
rega
is ro tion2
RP 3/10 ot
access
3/21
X
vlan 99
3/22
RP
DP 3/10
DP vlan 99
3/12
87491
aggregation1
In this example, the link between aggregation1 and aggregation2 is cut. This is direct failure, so the
MaxAge timer on port 3/10 expires immediately. As a consequence, aggregation2 sends a BPDU out of
the Designated Port 3/11 claiming to be the root switch.
On the access switch, port 3/21 receives inferior BPDUs for 20s (MaxAge). After 20s, the port goes into
listening mode and sends BPDUs with aggregation1 as the root. After 50s, port 3/21 on the access switch
goes into forwarding mode. If backbone fast is used, the MaxAge expires almost immediately upon
receiving and inferior BPDU from aggregation2.
This failure scenario is contemplated in the regular spanning tree behavior and does not need any
optimization.
Failure 3
Figure 4-3 Failure 3-Aggregation1 Stops Sending BPDUs on Port 3/10

3/11 B
agg PDU:
rega
RP 3/10 is ro tion2
ot access
3/21
vlan 99
3/22
X
RP
DP 3/10
DP vlan 99
3/12
87492
aggregation1

4-6 956513
Design Guidelines
In this scenario, aggregation1 (which is the root) stops sending BPDUs to aggregation2. As a
consequence, on port 3/21 the MaxAge timer expires first (because the Message Age will be 1s older
than message age on port 3/10), putting the port into a listening state and sending BPDU with
aggregation1 as the root. 1s later, aggregation2 expires the MaxAge on port 3/10 and turns 3/11 into root
port and 3/10 into designated port, causing a Layer 2 loop.
Failure 4
Figure 4-4 Failure 4-Aggregation1 Stops Sending BPDUs on Port 3/12

3/11
RP 3/10
access
3/21
vlan 99
3/22
RP
DP 3/10 vlan 99
DP X
3/12
87493
aggregation1
In this example, aggregation 1 (which is the root) stops sending BPDUs out of port 3/12. Port 3/22
MaxAge expires after 20s and turns port 3/21 into a Root Port, while port 3/22 becomes designated port.
This creates a Layer 2 loop.
Summary
Other obvious possible failures in the access layer were not covered, but this is enough to notice that
there is a difference between the regular and unexpected failures. Regular failures are those caused by a
disconnected or broken link and unexpected failures are those where BPDUs are no longer received or
sent on a link.
The difference is the presence of BPDUs themselves. Spanning tree on the access switch behaves in the
same way when BPDUs are not received on port 3/21 and when inferior BPDUs are received on port
3/21, even though they identify two different failure scenarios.
In PVST+ (802.1d based) it takes around 50s for the loop to happen unless you use aggressive timers. In
Rapid PVST+ (which uses 802.1w) it takes from 7s to 37s for the loop to happen.
Design Guidelines
Using Loopguard to Address STP Limitations
This section provides a description of the main design details.

956513 4-7
Design Guidelines
Loopguard
The Loopguard option operates with spanning-tree to prevent an alternate port or a root port from
assuming a designated role due to the absence of BPDUs. When loopguard does not receive BPDUs from
a root port or a blocking port, it puts or keeps the port in a blocking state and marks the port as
loop-inconsistent.
By default, loopguard should be enabled both at the aggregation and access layer in a switched
environment. Loopguard should never be enabled on shared links.
An alternate port is a blocked port that provides an alternate path to the root. A backup port is a blocked
port because it is connected in a loopback fashion.
Figure 4-5 Alternate Port vs. Backup Port
Alternate Port vs Backup Port
DP
RP Alternative
port
RP
Root switch
Backup port
87494
DP
Examples
Let's get back to the examples and analyze the behavior of Loopguard used in conjunction with PVST+:
• In Failure 1, port 3/21 on the access switch is not receiving BPDUs. With loopguard enabled, this
port is kept in blocking state so that no loop can occur.
• In Failure 2, spanning tree behaves as usual because port 3/21 keeps receiving BPDUs from
aggregation2, therefore loopguard does not kick in even if it is enabled.
• In Failure 3, port 3/10 on aggregation2 does not receive BPDUs from the root switch. As a
consequence, the access switch does not receive BPDUs from port 3/21. MaxAge expires first on
the access switch, so loopguard forces port 3/21 into loop-inconsistent state.
On aggregation2, loopguard puts port 3/10 in blocking state and marked as loop-inconsistent.
Immediately after, aggregation2 starts sending BPDUs claiming to be the root. The access switch
port 3/21 leaves the loop-inconsistent state upon reception of the new BPDUs and spanning tree
takes over.

4-8 956513
Design Guidelines
• In Failure 4, port 3/22 on the access switch does not receive BPDUs from the root switch. If
loopguard is enabled, this port is put into blocking state and marked as loop-inconsistent. Port 3/21
becomes forwarding and assumes a root port role. There is no Layer 2 loop.
Automatic Recovery
When a port is in loop-inconsistent state, it recovers from this state as soon as it receives a BPDU, and
spanning tree takes over.
How this is useful is seen in the Failure 3, where the unidirectional link connects the aggregation
switches. In this example, MaxAge expires first on the access switch so the uplink port goes into
loop-inconsistent state. Aggregation2 actually sends BPDUs again on this link 1s later, and it claims to
be the root. Since the link between aggregation2 and the access switch is not unidirectional, port 3/21
on the access switch leaves the loop-inconsistent state.
Key Benefits
Loopguard has the following key benefits for a Layer 2 network:
• It protects against Layer 2 loops that spanning-tree cannot handle
• It works together with spanning-tree, so there is no additional protocol traffic on the link
• Loopguard takes care of Layer 2 loops even when spanning-tree aggressive timers are used
• Loopguard is also going to be available on the CatOS code with the new 802.1s, 802.1w standards
Interoperability with Other Features
PVST+ - Uplinkfast
Loopguard is compatible with Uplinkfast. If Loopguard puts a root port into a blocking state, uplinkfast
puts a new root port into forwarding state. Also, Uplinkfast does not select a loop-inconsistent port as a
root port.
PVST+ BackboneFast
Loopguard is compatible with Backbone Fast. Backbone fast is triggered by the reception of an inferior
BPDU coming from a designated bridge. Since BPDUs are received from this link, loopguard does not
kick in, therefore backbone fast and loopguard are compatible.
Rapid PVST+ - 802.1w

Loopguard is compatible with 802.1w.
Rootguard
Rootguard and loopguard are mutually exclusive. The reason being that a “Rootguarded” port is forced
to be a designated port all of the time. A loopguarded port is either a root port or a blocking port.

956513 4-9
Implementation Details
PagP and LACP

Channeling works with loopguard enabled, and all of the ports of the channel must be loopguard enabled
for the channel to be formed. Since the BPDUs travel only on the first operational port of the channel,
if BPDUs are no longer forwarded on this link, the whole channel is put in loop-inconsistent state.
UDLD
Unidirectional link detection (UDLD) is a Layer 2 protocol that detects unidirectional links on regular
ports (as opposed to ports that are members of a channel). The main usage for UDLD is to prevent users
from creating loops as a consequence of inadequate wiring (especially on fibers that connect very far
away switches). This is useful at linkup. If the link is unidirectional, the port is shut down.
UDLD can also be used to detect unidirectional failures after a port has been up and bi-directional for a
certain time. The time it takes for UDLD to detect a unidirectional failure is around 2.5 * msg_interval
+ 4s (or 6s). UDLD message interval, by default, is 15s. If the msg_interval is lowered to 7s, the total
time would be around 21s. This could not be fast enough to prevent a loop if spanning-tree if you use
802.1w.
UDLD can be used in two modes: normal and aggressive. The normal mode protects against Layer 2
loops and the aggressive mode does the same but also helps to prevent traffic black holing.
At port linkup, loopguard does not protect against Layer 2 loops. UDLD can detect a unidirectional link
before spanning-tree puts the port into forwarding state.
As a result, the recommendation is to enable both UDLD and loopguard as follows:
udld enable
spanning-tree loopguard default
Where to Enable Loopguard

Loopguard should be configured on all the switches in the Layer 2 domain in a switched network (a
network with only point-to-point links).
Loopguard should not be enabled on shared links. The reason is that on a shared link, the absence of
BPDUs is not a symptom of a unidirectional link.
Consider the following diagram:

4-10 956513
Figure 4-6 Where Not to Enable Loopguard
aggregation2 - secondary root

3/11
RP 3/10
DP 3/10
DP
87495
3/12
aggregation1 - root
If Loopguard was enabled on port 3/11, and the link from port 3/12 was lost. The host on the shared
medium becomes isolated because port 3/11 does not transition into forwarding. If there is a shared
medium and a link from a switch is lost, there is still connectivity at a physical layer. BPDUs are not
received, but this is not a symptom of a unidirectional failure.
How to Configure Loopguard

You can enable loopguard globally by using the following command in IOS:
Use this command Catalyst IOS:

set spantree global-default loopguard enable
How to Test Loopguard

Testing loopguard is easy. It is simply necessary to prevent a port from sending BPDUs (for example,
port 3/11 on aggregation2 in the previous examples). Starting from CatOs 6.2, a new feature is available
that prevents portfast enabled ports from sending BPDUs. This feature is called bpdu-filtering.
Bpdu-filtering can only be enabled on a per-switch basis.
For example, the user could disable port 3/11 on aggregation2 from sending BPDUs and observe that
port 3/21 on the access switch goes into loop-inconsistent state.
The commands follow:
aggregation-2 (enable) set spantree portfast 3/11 ena
Spantree port fast start should only be enabled on ports connected to a single host.
Connecting hubs, concentrators, switches, bridges, etc. to a fast start port can cause
temporary spanning tree loops. Use with caution.
Spantree port 3/11 fast start enabled.
aggregation-2 (enable) set spantree portfast bpdu-filter ena
Spantree portfast bpdu-filter enabled on this switch.
2001 Feb 06 13:32:14 %SPANTREE-4-LOOPGUARDBLOCK: No BPDUs were received on port 3/21 in
VLAN 99. Moved to loop-inconsistent state

956513 4-11
Summary and Recommendations
The above example used the Catalyst IOS syntax, but you can follow similar steps using the
bpdu-filtering commands in IOS. The command that you would use in IOS is
(config-if) spanning-tree bpdufilter ena
Summary and Recommendations

The spanning-tree protocol prevents loops in your network, however it has some limitations. Cisco offers
two main tools to prevent Layer 2 loops, specifically loopguard and UDLD. You should enable both
features globally on your aggregation and access switches to protect your network.

4-12 956513
A P P E N D I X A
Configurations
The physical topology of the tested network is displayed in Figure A-1.
Figure A-1 : Physical Topology for the Example Configurations
Core1 Core2
1/1 1/1
1/2 1/2
4/15 4/15
4/16 4/16
4/8 4/7
4/7 4/8
4/13 4/14 4/13 4/14

1/1 1/1
1/2 1/2
4/15 4/15
4/16 4/16
Aggregation1 4/2 4/1 Aggregation2
4/1 4/2
B
1/2
B
1/1
1/2 1/1
access1 access2
87451
Data Center
Figure A-2 displays the Layer 3 configuration.
Book Title
78-xxxxx-xx A-1
Appendix A Configurations
Figure A-2 Layer 3 Topology for the Example Configuration
Core1 Core2
10.0.0.1 10.0.0.2
Summary-address Summary-address
0.0.0.0.0.0.0.0 10.21.0.10 10.21.0.6 0.0.0.0.0.0.0.0
10.21.0.2 10.21.0.14
10.21.0.1 10.21.0.13
10.21.0.5 10.21.0.9
Aggregation1 10.20.5.2 VLAN 10.20.5.3 Aggregation2

10.20.10.2 VLAN 10.20.10.3
10.20.20.2 VLAN 10.20.20.3
B
access1
87452
Data Center
In the following configurations, the italicized topics of configuration are topics described earlier in the
document. In addition, the bolded text represents fields that differ between aggregation1 and
aggregation2.
The configuration for the access switch is based on CatOS.
Aggregation1 - Rapid PVST+ - OSPF

version 12.1
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname aggregation1
!
boot system flash slot0:c6sup22-jsv-mz.121-11b.EX1
boot bootldr bootflash:c6msfc2-boot-mz.121-8a.E5
!
vtp domain mydomain
udld enable
ip subnet-zero
!
!
no ip domain-lookup
!
no mls ip multicast aggregate
no mls ip multicast non-rpf cef
Book Title
A-2 78-xxxxx-xx
mls qos statistics-export interval 300

mls qos statistics-export delimiter |
!
!
spanning-tree vlan 3,5-6,10,12,80,100 priority 8192
!
redundancy
mode rpr-plus
main-cpu
auto-sync running-config
auto-sync standard
!
!
vlan 3
name L3_RTR_NEIGH
!
vlan 5
name csm_vip vlan
!
vlan 10
name servers_group1
!
vlan 20
name servers_group2
!
vlan 30
name FW_outside
!
vlan 100
name CSM_fault_tolerant
!
vlan 802
name mgmt_network
!
interface Loopback0
ip address 3.3.3.3 255.255.255.255
!
interface Port-channel2
no ip address
switchport
switchport trunk allowed vlan 3,5,10,20,30,100,1002-1005
!
no ip address
logging event link-status
switchport
!
no ip address
switchport
Book Title
78-xxxxx-xx A-3

!
description to_access1
no ip address
switchport
switchport trunk allowed vlan 5,10,20
!
no ip address
switchport
!
description to_core1
ip address 10.21.0.1 255.255.255.252
!
ip address 10.21.0.5 255.255.255.252
!
no ip address
switchport
!
no ip address
switchport
!
interface Vlan3
description L3_VLAN_FOR_RTR_NEIGH
ip address 10.20.3.2 255.255.255.0
!
interface Vlan5
ip address 10.20.5.2 255.255.255.0
no ip redirects
!
interface Vlan10
ip address 10.20.10.2 255.255.255.0
no ip redirects
Book Title
A-4 78-xxxxx-xx
standby 1 ip 10.20.10.1
!
interface Vlan20
ip address 10.20.20.2 255.255.255.0
no ip redirects
standby 1 ip 10.20.20.1
!
router ospf 20
network 10.20.0.0 0.0.255.255 area 20
network 10.21.0.0 0.0.255.255 area 20
!
ip classless
ip route 171.0.0.0 255.0.0.0 172.26.200.129
no ip http server
!
!
alias exec csm5 show module csm 5
alias exec csmrun show run | begin module ContentSwitchingModule
alias exec showport show run | begin interface GigabitEthernet
alias exec showport4 show run | begin interface GigabitEthernet4
alias exec rtr show run | begin router
!
Aggregation2 - Rapid PVST+ - OSPF

!
!
!
vtp domain mydomain
udld enable
ip subnet-zero
!
!
!
!
!
spanning-tree vlan 3,5-6,10,12,80,100 priority 16384
!
redundancy
mode rpr-plus
main-cpu
auto-sync standard
!
Book Title
78-xxxxx-xx A-5

!
vlan 3
name L3_RTR_NEIGH
!
vlan 5
name csm_vlan
!
vlan 10
name servers_group1
!
vlan 20
name servers_group2
!
vlan 30
name FW_outside
!
vlan 100
!
vlan 200
name FORNAX_ft
!
vlan 802
name mgmt_network
!
!
interface Loopback0
ip address 4.4.4.4 255.255.255.255
!
no ip address
switchport
!
no ip address
switchport
!
no ip address
switchport
!
no ip address
switchport
Book Title
A-6 78-xxxxx-xx

!
no ip address
switchport
!
ip address 10.21.0.9 255.255.255.252
!
ip address 10.21.0.13 255.255.255.252
!
no ip address
switchport
!
no ip address
switchport
!
interface Vlan3
description L3_vlan_for_RTR_neigh
ip address 10.20.3.3 255.255.255.0
!
interface Vlan5
ip address 10.20.5.3 255.255.255.0
no ip redirects
!
interface Vlan10
ip address 10.20.10.3 255.255.255.0
no ip redirects
standby 1 ip 10.20.10.1
!
interface Vlan20
ip address 10.20.20.3 255.255.255.0
no ip redirects
standby 1 ip 10.20.20.1
!
interface Vlan802
Book Title
78-xxxxx-xx A-7
ip address 172.26.200.136 255.255.255.192

!
router ospf 20
network 10.20.0.0 0.0.255.255 area 20
network 10.21.0.0 0.0.255.255 area 20
!
ip classless
no ip http server
!
!
Access - Rapid PVST+

!
#stp mode
set spantree mode rapid-pvst+
!
#vtp
set vtp domain mydomain
set vtp mode transparent
set vlan 1 name default type ethernet mtu 1500 said 100001 state active
set vlan 5 name csm_vlan type ethernet mtu 1500 said 100005 state active
set vlan 10 name servers_group1 type ethernet mtu 1500 said 100010 state active
set vlan 802 name mgmt_network type ethernet mtu 1500 said 100802 state active
set vlan 6
set dot1q-all-tagged enable
!
#spantree
#portfast
set spantree global-default bpdu-guard enable
!
#udld
set udld enable
!
# default port status is enable
!
!
#module 1 : 2-port 1000BaseX Supervisor
set port name 1/1 to_aggregation1
set port name 1/2 to_agg2
set udld aggressive-mode enable 1/1-2
clear trunk 1/1 1-4,6-9,11-19,21-1005,1025-4094
set trunk 1/1 desirable dot1q 5,10,20
clear trunk 1/2 1-4,6-9,11-19,21-1005,1025-4094
set spantree portfast 1/1-2 disable
set spantree guard loop 1/1-2
!
!
#module 4 : 48-port 10/100BaseTX Ethernet
set vlan 6 4/6
Book Title
A-8 78-xxxxx-xx
set vlan 10 4/1-5

set vlan 20 4/7-11
set vlan 802 4/48
set vlan 1400 4/12-24
set vlan 1673 4/25-47
set port speed 4/1-2,4/4-11 100
set port duplex 4/1-2,4/4-11 full
set port name 4/1 toserver_1
set port name 4/10 toserver_a
set port name 4/11 toserver_b
set spantree portfast 4/1-48 enable
set spantree guard none 4/1-48
!
Aggregation1 - MST - OSPF

version 12.1
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
!
!
vtp domain mydomain
udld enable
ip subnet-zero
!
!
no ip domain-lookup
!
!
spanning-tree mode mst
!
revision 10
!
spanning-tree mst 0 priority 24576
redundancy
mode rpr-plus
main-cpu
Book Title
78-xxxxx-xx A-9
auto-sync standard
!
!
vlan 3
name L3_RTR_NEIGH
!
vlan 5
name csm_vip vlan
!
vlan 10
name servers_group1
!
vlan 20
name servers_group2
!
vlan 30
name FW_outside
!
vlan 100
!
vlan 802
name mgmt_network
!
interface Loopback0
ip address 3.3.3.3 255.255.255.255
!
no ip address
switchport
!
no ip address
switchport
!
no ip address
switchport
!
no ip address
switchport
Book Title
A-10 78-xxxxx-xx
!
no ip address
switchport
!
ip address 10.21.0.1 255.255.255.252
!
ip address 10.21.0.5 255.255.255.252
!
no ip address
switchport
!
no ip address
switchport
!
interface Vlan3
description L3_VLAN_FOR_RTR_NEIGH
ip address 10.20.3.2 255.255.255.0
!
interface Vlan5
ip address 10.20.5.2 255.255.255.0
no ip redirects
!
interface Vlan10
ip address 10.20.10.2 255.255.255.0
no ip redirects
standby 1 ip 10.20.10.1
!
interface Vlan20
ip address 10.20.20.2 255.255.255.0
no ip redirects
standby 1 ip 10.20.20.1
!
router ospf 20
Book Title
78-xxxxx-xx A-11
network 10.20.0.0 0.0.255.255 area 20
network 10.21.0.0 0.0.255.255 area 20
!
ip classless
ip route 171.0.0.0 255.0.0.0 172.26.200.129
no ip http server
!
!
Aggregation2 - MST - OSPF

!
!
!
vtp domain mydomain
udld enable
ip subnet-zero
!
!
!
!
spanning-tree mode mst
!
revision 10
!
redundancy
mode rpr-plus
main-cpu
auto-sync standard
!
!
vlan 3
name L3_RTR_NEIGH
!
Book Title
A-12 78-xxxxx-xx
vlan 5
name csm_vlan
!
vlan 10
name servers_group1
!
vlan 20
name servers_group2
!
vlan 30
name FW_outside
!
vlan 100
!
vlan 200
name FORNAX_ft
!
vlan 802
name mgmt_network
!
!
interface Loopback0
ip address 4.4.4.4 255.255.255.255
!
no ip address
switchport
!
no ip address
switchport
!
no ip address
switchport
!
no ip address
switchport
!
Book Title
78-xxxxx-xx A-13
no ip address
switchport
!
ip address 10.21.0.9 255.255.255.252
!
ip address 10.21.0.13 255.255.255.252
!
no ip address
switchport
!
no ip address
switchport
!
interface Vlan3
description L3_vlan_for_RTR_neigh
ip address 10.20.3.3 255.255.255.0
!
interface Vlan5
ip address 10.20.5.3 255.255.255.0
no ip redirects
!
interface Vlan10
ip address 10.20.10.3 255.255.255.0
no ip redirects
standby 1 ip 10.20.10.1
!
interface Vlan20
ip address 10.20.20.3 255.255.255.0
no ip redirects
standby 1 ip 10.20.20.1
!
interface Vlan802
ip address 172.26.200.136 255.255.255.192
!
router ospf 20
Book Title
A-14 78-xxxxx-xx
network 10.20.0.0 0.0.255.255 area 20
network 10.21.0.0 0.0.255.255 area 20
!
ip classless
no ip http server
!
!
Access - MST
#mac address reduction
set spantree macreduction enable
!
!
#stp mode
set spantree mode mst
!
#vtp
set vtp domain mydomain
set vtp mode transparent
set vlan 1 name default type ethernet mtu 1500 said 100001 state active
set vlan 5 name csm_vlan type ethernet mtu 1500 said 100005 state active
set vlan 802 name mgmt_network type ethernet mtu 1500 said 100802 state active
set vlan 1002 name fddi-default type fddi mtu 1500 said 101002 state active
set vlan 1004 name fddinet-default type fddinet mtu 1500 said 101004 state active stp ieee
set vlan 1005 name trnet-default type trbrf mtu 1500 said 101005 state active stp ibm
set vlan 6
set vlan 1025
set vlan 1025 mistp-instance 1
set vlan 1003 name token-ring-default type trcrf mtu 1500 said 101003 state active mode
srb aremaxhop 0 stemaxhop 0 backupcrf off
set dot1q-all-tagged enable
!
#spantree
#portfast
set spantree global-default bpdu-guard enable
#MST (IEEE 802.1s)
#MST Configuration
set spantree mst config rollback force
set spantree mst config name data_center_mst revision 10
set spantree mst 1 vlan 1-1000
set spantree mst config commit
!
#udld
set udld enable
!
# default port status is enable
!
!
Book Title
78-xxxxx-xx A-15
#module 1 : 2-port 1000BaseX Supervisor

set port name 1/1 to_aggregation1
set port name 1/2 to_agg2
set udld aggressive-mode enable 1/1-2
clear trunk 1/1 1-4,6-9,11-19,21-1005,1025-4094
clear trunk 1/2 1-4,6-9,11-19,21-1005,1025-4094
set spantree portfast 1/1-2 disable
set spantree guard loop 1/1-2
!
!
#module 4 : 48-port 10/100BaseTX Ethernet
set vlan 6 4/6
set vlan 10 4/1-5
set vlan 20 4/7-11
set vlan 802 4/48
set vlan 1400 4/12-24
set vlan 1673 4/25-47
set port speed 4/1-2,4/4-11 100
set port duplex 4/1-2,4/4-11 full
set port name 4/10 toserver_a
set port name 4/11 toserver_b
set spantree portfast 4/1-48 enable
set spantree guard none 4/1-48
!
Book Title
A-16 78-xxxxx-xx
I N D EX
BGP 1-11, 3-4

Numerics
blocking 4-2
802.1q 1-10, 2-14 Border Gateway Protocol 1-11
802.1s 1-10, 2-11, 2-16, 2-19, 4-1 bpdu-filtering 4-11
802.1s/1w 2-3, 2-8 BPDU format 4-2
802.1w 1-10, 2-3, 2-11, 2-14, 2-16, 2-18, 4-1 BPDUs 4-1
802.3ad 1-10 Broadcast Suppression 1-10
building blocks 1-2
business continuance 1-8
A
business logic 1-7
AAA 1-12 business requirements 1-3
ABR 2-4, 2-25
ABRs 2-24, 2-27, 2-32
C
access layer 1-6
accounting management 1-13 cache engines 2-6, 2-13
ACLs 1-12, 2-1 caching 1-11, 2-1
Advanced Peer-to-Peer Networking 2-29 Call Managers 1-6
aggregation layer 1-5 campus-to-campus connectivity 1-8
application layer 1-7, 1-8 Catalyst 6500 2-7
Application Optimization Services 1-9 Channel Interface Processor 2-3
application requirements 1-3 CIP/CPA 2-3, 2-6, 2-29
APPN 2-29 Cisco multi-layer design 2-2
area border router 2-4 Cisco PVST+ 4-1
area border routers 2-24 client-to-server traffic paths 3-8
ARP probes 3-4 Coarse Wave Division Multiplexing 1-10
ASICs 3-5 Common Spanning Tree 2-19
asynchronous communications 1-8 configuration management 1-13
availability 1-3 Content Distribution Managers 1-6
content engines 1-5
content routing 2-4
B
Content Services Switch 2-6
Backbonefast 4-3, 4-9 content switches 1-5
back-end layer 1-8 content switching 1-11, 2-1

956513 IN-1
Index
Content Switching Module 2-6, 3-2 Enhancing Server-to-Server Communications

core switches 1-3 benefits 3-3
Cross Coupling Facility 2-33 configuration description 3-9
CSCdw28489 3-4 configuring MSFC-to-CSM VLAN 3-10
CSCdw39689 3-4 configuring server VLANS on MSFC 3-9
CSCdy00143 2-15 CSM1 configuration 3-12
CSCdy00148 2-15 hardware requirements 3-4
CSM 2-6, 2-7, 3-2, 3-6 MSFC-AGG2 configuration 3-15
CSRP 3-7 network topology 3-6
CSS 2-6 pbr details 3-12
CWDM 1-10 restrictions 3-4
sample configurations 3-10
software requirements 3-4
D
Enterprise Extenders 2-29
Data Link Switching 2-29 Enterprise System Connections 2-3
default next-hop 3-5 ESCON 1-10, 2-3, 2-29, 2-30, 2-34
demilitarized zone 2-23 Extranet Server Farm 1-3
Denial of Service 1-12
Dense Wave Division Multiplexing 1-10
F
designated bridge 4-2
designated port 4-2, 4-7 fault management 1-13
disaster recovery 1-8 FCAPS 1-13
discarding 4-3 federal government agencies 1-1
distributed data centers 1-9 Fibre Channel 1-11
distribution layer 1-5 Fibre-Channel (FC) 1-8
DLSw 2-29 financial institutions 1-1
DMZ 2-23 Firewalls 1-12
DTP 2-14 firewalls 1-5, 1-7, 2-1, 2-18
DVIPA 2-33, 2-34 Firewall Services Module 2-6
DWDM 1-10 forwarding 4-2, 4-3
Dynamic Trunking Protocol 2-14 Forwarding Agent 2-33
dynamic VIPA 2-33 forwarding logical path 4-1
Dynamic Virtual IP Address 2-33 front-end layer 1-6
FTP 1-6
FT VLAN 2-11
E
FWSM 2-6, 2-7
EE 2-29
EIGRP 1-11, 2-3, 2-23, 2-27

IN-2 956513
Index
Layer 3 1-10, 1-11

H
Layer 4 1-11
healthcare 1-1 Layer 5 1-7, 1-11
hello packets 4-3 learning 4-3
hello time 4-2 Link Aggregate Control Protocol 1-10, 2-10
high availability 1-1, 2-1 link oversubscription 2-20
high speed connection 1-8 LINUX 2-29
Hot Standby Router Protocol 3-3 load balancing 2-1
HSRP 1-11, 2-9, 2-23, 2-28, 3-3, 3-7, 3-8 Logical Partitions 2-4
HSRP group 2-29 LoopGuard 1-10
HTTP 2-29 Loopguard 2-21, 4-5, 4-8, 4-10
Loopguard compatibility 4-9
loop-inconsistent state 4-9
I
LPAR 2-31, 2-34
ICMP probes 3-4 LPARs 2-4, 2-33
IDS 1-7, 1-12
IDSs 1-5, 2-1
M
IDS sensors 2-6
IGP 1-11 mainframe 2-29
incoming VLAN policy 3-5 mainframes 1-8, 2-5
Infrastructure 1-9 mainframe services 2-1
Intelligent Network Services 1-10 Management 1-10
Interior Gateway Protocols 1-11 management 1-1, 1-3
Internet Server Farm 1-3 Management services 1-12
Intranet Server Farm 1-3 MaxAge 4-2, 4-4, 4-9
ip default next-hop 3-4 medium sized servers 1-8
ip-default next-hop 3-2 Metro 1-10
ip route 3-6 metro optical 1-8
IP spoofing 1-12 metro transport layer 1-8
ip summary-address eigrp 2-27 MHSRP 1-11
IPTV Broadcast servers 1-6 middleware 1-7
iSCSI 1-8 missing BPDUs 2-20
MISTP 2-19
MNLB 2-33
L
monitoring 1-12
LACP 2-10, 4-10 MSFC 3-3, 3-8
Layer 2 1-6, 1-10 MSFC2 2-7
Layer 2 attacks 1-12 MST 2-3, 2-11, 2-19, 4-1
Layer 2 loop 4-1, 4-4 Multicast 1-6

956513 IN-3
Index
Multi Instance Spanning-Tree 4-1 PVST 4-9

multi instance spanning-tree 2-3 PVST+ 2-16, 2-20, 4-5, 4-7
Multilayer Switching Feature Card 3-3
Multi Node Load Balancing 2-33
Q
Multiple Spanning-Tree 4-1
multi-tier server farms 2-1 QoS 1-6, 1-11
N R
NAS 1-11 Rapid Per VLAN Spanning 2-3
Native IOS 2-8 Rapid PVST+ 2-3, 2-8, 2-15, 2-17, 4-1, 4-5, 4-7, 4-9
Network Attached Storage 1-11 Rapid Spanning-Tree Protocol 4-1
network reconnaissance 1-12 Rapid Transport Protocol 2-29
Not-So-Stubby Area 2-25 regular bridge priority 2-12, 2-18, 2-20
NSSA 2-25 relational database 1-8
Remote Access 1-3
Renewable Energy Policy Project 1-1
O
replication 1-8
optical server adapters 2-3 revenue 1-1
OSA 2-3 RIP 2-31
OSA card 2-6, 2-7, 2-29, 2-30 root 4-2
OSPF 1-11, 2-3, 2-4, 2-5, 2-23, 2-24, 2-31 root bridge priority 2-12, 2-18, 2-20
OSPF priority 2-32 Rootguard 4-9
OSPF stub area 2-24 root port 4-2
Route Health Injection 2-25
route-map 3-9
P
Router Processor Redundancy Plus 2-9
PagP 4-10 RPR+ 2-9
PBR 1-11, 3-5, 3-10 RPR/RPR+ 2-13
performance management 1-13 RSTP 2-11
Per VLAN Spanning-Tree Plus 4-1 RTP 2-29
PIX 2-6, 2-7
policy based routing 1-11, 3-5
S
portfast 2-13, 2-15
port states 4-3 SAN 1-11
Private VLANs 1-11 SCA 2-6
Private WAN 1-3 scalability 1-1, 1-3, 2-1
protocol policy 3-5 secondary root bridge priority 2-12, 2-18, 2-20

IN-4 956513
Index
Secure Content Accelerator 2-6 System Network Architecture 2-29

Security 1-10 System Services Control Point 2-29
security 1-1, 1-3, 1-7
security management 1-13
T
Security services 1-12
Server load balancing 1-11 Telnet 1-6
server-to-client traffic 3-8 TN3270 1-6, 2-3, 2-6, 2-29
server-to-server 3-1 TPF 2-29
server-to-server traffic 3-8 troubleshooting 1-12
service provisioning 1-12
shortest path first 2-24
simplicity 2-1
U
SMTP 1-6 UDLD 1-10, 2-3, 2-21, 4-10
SNA 2-29 Unauthorized access 1-12
SONET 1-10 unidirectional 2-20
source IP address 3-5 Unidirectional link detectio 4-10
spanning-tree 4-1 Uni-Directional Link Detection 1-10
SPF 2-24 Unidirectional Link Detection 2-3
SSCP 2-29 Uplinkfast 4-3
SSL 1-7 user-to-server 1-11
SSL offloaders 1-5, 2-13 utilities 1-1
SSL offloading 1-11, 2-1
SSL Services Modules 2-6
SSLSM 2-6 V
static routes 3-6 VIP 3-7
static routing 1-11 VIPA 2-31
Storage 1-9 virtual IP address 2-4, 2-31, 3-7
Storage Area Networks 1-11 Virtual Machines 2-4
storage layer 1-8 Virtual Telecommunication Access Method 2-29
storage mirroring 1-8 viruses and worm 1-12
Storage services 1-11 VLANs 1-6
storage-to-storage 1-11 VRRP 1-11
SVI 2-28 VSE 2-29
switched VLAN interface 2-28 VTAM 2-29
synchronous communications 1-8
Sysplex 2-34
sysplex 2-32 W
Sysplex Distributor 2-33
Web servers 1-6
Sysplex Timer 2-33

956513 IN-5
Index
wire-speed server-to-server communication 3-3

Workload Manager 2-33
XCF 2-33
z/OS 2-29
z/VM 2-29

IN-6 956513

Cisco Systems - Data Center Networking - Infrastructure Architecture

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cisco Systems - Data Center Networking - Infrastructure Architecture

Uploaded by

Copyright:

Available Formats

Data Center Networking: Infrastructure

Customer Order Number: 956513

Data Center Networking: Infrastructure Architecture

CHAPTER 1 Data Center Overview — Infrastructure Architecture 1-1

Data Center Networking: Infrastructure Architecture

Security Services 1-12

CHAPTER 2 Data Center Infrastructure 2-1

Data Center Networking: Infrastructure Architecture

CHAPTER 3 Enhancing Server to Server Communications 3-1

CHAPTER 4 Addressing Spanning-Tree Limitations 4-1

Data Center Networking: Infrastructure Architecture

Convergence Time 4-3

Data Center Networking: Infrastructure Architecture

Table 1 Document Organization:

Chapter or Appendix Description

Data Center Networking: Infrastructure Architecture

Chapter or Appendix Description

Table 2 Document Conventions

Data Center Networking: Infrastructure Architecture

World Wide Web

Data Center Networking: Infrastructure Architecture

Obtaining Technical Assistance

Technical Assistance Center

Data Center Networking: Infrastructure Architecture

Cisco TAC Web Site

Cisco TAC Escalation Center

Data Center Networking: Infrastructure Architecture

Data Center Networking: Infrastructure Architecture

Benefits of Building Data Centers

Data Center Networking: Infrastructure Architecture

Data Centers in the Enterprise

Figure 1-1 Enterprise Network Infrastructure

AAA Internet server farm

Extranet server farm

Intranet server farm

The building blocks of the typical Enterprise network include:

Data Center Networking: Infrastructure Architecture

Data Center Architecture

Figure 1-2 Architecture Layers and Design Criteria

Metro Transport Layer

Data Center Networking: Infrastructure Architecture

Figure 1-3 Data Center Architecture

Distribution Aggregation layer

Access Front-end layer

Access Application layer

Access Back-end layer

Metro Transport Layer

Data Center Networking: Infrastructure Architecture

The following sections provide a detailed description of these layers.

Data Center Networking: Infrastructure Architecture

Figure 1-4 Aggregation Layer

Multilayer switches: L2-L5

Intrusion detection system

Data Center Networking: Infrastructure Architecture

Figure 1-5 Front-End, Application, and Back-End Layers

Web and client

Layer 2 Layer 2 switches

Layer 2 Layer 2 switches

Data Center Networking: Infrastructure Architecture

Metro Transport Layer

Data Center Networking: Infrastructure Architecture

Figure 1-6 Metro Transport Topology

Primary Data Center Distributed Data Center

Storage layer Metro Transport Layer Fibre channel Storage layer