DATACENTER DESIGN & INFRASTRUCTURE

LAYOUT
OF
PATNI COMPUTER SYSTEMS LTD.

PROJECT BY

1
 Name of the Learner: Mr.
Registration No:
Program Name: PGDBA
Address:
Contact No:

PART – I
Summery

S No. Description Page No.(s)

1. Introduction 8 - 11
2. Objectives And Scope 11 - 15
3 Limitations 15 - 18
4 Theoretical Perspective 18 - 19
5 Methodology And Procedure 19 - 41
6. Analysis of Data 42 – 42
7. Findings, Inferences And Recommendations 42 - 52
8. Conclusions 53 –55

2
 PART – II
Overview of the Organization

S No. Description Page No.(s)

1. An Overview Of The Organization 55 – 56
2. Patni Computer Systems Ltd. 56 - 57
3. Operational Statistics 57 – 59

PART – III
Project Overview

S No. Description Page No.(s)

1. Patni Data Center Design 59 – 69
2. Patni Data Center Infrastructure Layout 69 - 80
3. Bibliography 80

3
 NO OBJECTION CERTIFICATE

This is to certify that Mr. (SCDL Regn: 200621368) is an employee of this organization for
the past 2 Years.

We have no objection for him to carry out a project work titled “Datacenter Design &
Infrastructure Layout” in our organization and for submitting the same to the Director,
SCDL as part of fulfillment of the Post Graduate Diploma in Business Administration
(PGDBA) program.

We wish him all the success.

Place: Noida, UP Signature of the Competent Authority

Of the Organization

Date: 15th Dec, 2009

4
 DECLARATION BY THE LEARNER

This is to certify that carried out this project work myself in part fulfillment of the Post
Graduate Diploma in Business Administration (PGDBA) program of SCDL.

The work is original, has not been copied from any where else and has not been submitted to
any other University/Institute for an award of any Degree/Diploma.

Place: Faridabad Signature:

Name:
Date: 15th Dec, 2009 SCDL Regn:

5
 CERTIFICATE OF SUPERVISOR (GUIDE)

Certified that the work incorporated in this project report “Datacenter Design &
Infrastructure Layout” submitted by Mr. (SCDL Regn:) is his original work and completed
under my supervision. Material obtained from other sources has been duly acknowledged in
the Project Report.

Place: Noida, UP Signature of Supervisor

Date: 15th Dec, 2009 Designation:

Email:

1. Introduction

1.1 Data center is home to the computational power, storage, and applications necessary to support an
enterprise business. The data center infrastructure is central to the IT architecture, from which all

6
content is sourced or passes through. Proper planning of the data center infrastructure design is critical,
and performance, resiliency, and scalability need to be carefully considered.

Another important aspect of the data center design is flexibility in quickly deploying and supporting
new services. Designing a flexible architecture that has the ability to support new applications in a
short time frame can result in a significant competitive advantage. Such a design requires solid initial
planning and thoughtful consideration in the areas of port density, access layer uplink bandwidth, true
server capacity, and oversubscription, to name just a few.

The data center network design is based on a proven layered approach, which has been tested and
improved over the past several years in some of the largest data center implementations in the world.
The layered approach is the basic foundation of the data center design that seeks to improve
scalability, performance, flexibility, resiliency, and maintenance. Figure 1-1 shows the basic layered
design.

A Data Center Architecture includes the layout of the boundaries of the room (or rooms) and the
layout of IT equipments within the room. Most users do not understand how critical the floor layout is
to the performance of a data center, or they only understand its importance after a poor layout has
compromised the deployment. The floor plan either determines or strongly affects the following
characteristics of a data center:

• The number of rack locations that are possible in the room

• The achievable power density
• The complexity of the power and cooling distribution systems
• The predictability of temperature distribution in the room
• The electrical power consumption of the data center

7
There are five core values that are the foundation of a data center design philosophy:Simplicity,
flexibility, scalability, modularity, and sanity. The last one might give you pause, but if you’ve had
previous experience in designing data centers, it makes perfect sense.
Design decisions should always be made with consideration to these values.

1.2 Keep the Design as Simple as Possible

A simple data center design is easier to understand and manage. A basic design makes it simple to do
the best work and more difficult to do sloppy work. For example, if you label everything—network
ports, power outlets, cables, circuit breakers, their location on the floor—there is no guess work
involved. When people set up a machine, they gain the advantage of knowing ahead of time where the

8
machine goes and where everything on that machine should be plugged in. It is also simpler to verify
that the work was done correctly. Since the locations of all of the connections to the machine are pre-
labeled and documented, it is simple to record the information for later use, should the machine
develop a problem.

Figure 1.2 Simple, Clean, Modular Data Center Equipment Room

1.3 Design for Flexibility

Nobody knows where technology will be in five years, but it is a good guess that there will be some
major changes. Making sure that the design is flexible and easily upgradable is critical to a successful
long-term design.
Part of flexibility is making the design cost-effective. Every design decision has an impact on the
budget. Designing a cost effective data center is greatly dependent on the mission of the center. One
company might be planning a data center for mission critical applications, another for testing large-
scale configurations that will go into a mission critical data center. For the first company, full backup
generators to drive the entire electrical load of the data center might be a cost-effective solution. For
the second company, a UPS with a 20-minute battery life might be sufficient. Why the difference? If
the data center in the first case goes down, it could cost the company two million dollars a minute.
Spending five million on full backup generators would be worth the expense to offset the cost of
downtime. In the second case, the cost of down time might be $10,000 an hour. It would take 500
hours of unplanned downtime to recoup the initial cost of five million dollars of backup generators.
9
1.4 Design for Scalability

The design should work equally well for a 2,000, 20,000, or 2,000,000 square foot data center. Where
a variety of equipment is concerned, the use of watts per square foot to design a data center does not
scale because the needs of individual machines are not taken into consideration. This book describes
the use of rack location units (RLUs) to design for equipment needs. This system is scalable and can
be reverse engineered.

1.5 Use a Modular Design

Data centers are highly complex things, and complex things can quickly become unmanageable.
Modular design allows you to create highly complex systems from smaller, more manageable building
blocks. These smaller units are more easily defined and can be more easily replicated. They can also
be defined by even smaller units, and you can take this to whatever level of granularity necessary to
manage the design process. The use of this type of hierarchy has been present in design since
antiquity.

1.6 Keep Your Sanity

Designing and building a data center can be very stressful. There are many things that can, and will, go
wrong. Keep your sense of humor. Find ways to enjoy what you’re doing. Using the other four values
to evaluate design decisions should make the process easier as they give form, order, and ways to
measure the value and sense of the design decisions you’re making. Primarily, they help to eliminate
as many unknowns as possible, and eliminating the unknowns will make the process much
less stressful.

2 Objectives & Scope

2.1 Objectives:
• To ensure that your data center is built and meet to your business requirement
• Meeting to known Data Center Standard and best practices
• To meet the principal goals in data center design, which are flexibility, and scalability,
where these shall involve site location, building selection, floor layout, electrical system
design, mechanical design and modularity.
2.1.1 Provide all design as mentioned in the scope of services
• Many factors go into the design of a successful data center.

10
Careful and proper planning during the design phase will ensure a successful implementation resulting
in a reliable and scalable data center, which will serve the needs of the business for many years.
Remember the times when you had to keep your computer in a cool, dust-free room? Your personal
computer may not need such a cozy environment anymore, but your precious servers do. The
datacenter is what houses your organization’s servers.

The benefits of a datacenter are many. But a datacenter may not be necessary for every organization.
Two things dictate the requirement of a datacenter: the number of servers, network devices and your
ability to manage the datacenter. If you find monitoring and managing a datacenter overwhelming, you
need not forsake the datacenter. Instead, opt for a hosted one, where your datacenter is kept within the
premises of a vendor who also takes care of its management and security.
Ideally, large businesses and the mid-size ones that are growing rapidly should go for datacenters. For
small businesses, single servers or cluster servers can provide all that they need.
Now, how does a datacenter help your business?

• It offers high availability. A well-managed datacenter ensures that business never suffers
because of one failure somewhere.
• It is highly scalable. The datacenter offers support as the business needs change.
• It offers business continuity. Unexpected problems and server failures don’t deter the
functioning of your business in any way.

2.1.2 One of the biggest criteria of owning a datacenter is your ability to manage it. Datacenter
management, however, does not necessarily depend on you. You can hire professionals to manage it
for you. In fact, you don’t even need to keep the datacenter in your premises; it can be kept within the
premises of the vendor.

IT operations are a crucial aspect of most organizational operations. One of the main concerns is
business continuity; companies rely on their information systems to run their operations. If a system
becomes unavailable, company operations may be impaired or stopped completely. It is necessary to
provide a reliable infrastructure for IT operations, in order to minimize any chance of disruption.
Information security is also a concern, and for this reason a data center has to offer a secure
environment which minimizes the chances of a security breach. A data center must therefore keep
high standards for assuring the integrity and functionality of its hosted computer environment. This is
accomplished through redundancy of both fiber optic cables and power, which includes emergency
backup power generation.

11
2.1.3 When designing a large enterprise cluster network, it is critical to consider specific objectives.
No two clusters are exactly alike; each has its own specific requirements and must be examined from
an application perspective to determine the particular design requirements. Take into account the
following technical considerations:

•Latency—In the network transport, latency can adversely affect the overall cluster
performance. Using switching platforms that employ a low-latency switching architecture
helps to ensure optimal performance. The main source of latency is the protocol stack and NIC
hardware implementation used on the server. Driver optimization and CPU offload techniques,
such as TCP Offload Engine (TOE) and Remote Direct Memory Access (RDMA), can help
decrease latency and reduce processing overhead on the server.

Latency might not always be a critical factor in the cluster design. For example, some clusters
might require high bandwidth between servers because of a large amount of bulk file transfer,
but might not rely heavily on server-to-server Inter-Process Communication (IPC) messaging,
which can be impacted by high latency.

•Mesh/Partial mesh connectivity—Server cluster designs usually require a mesh or partial

mesh fabric to permit communication between all nodes in the cluster. This mesh fabric is used
to share state, data, and other information between master-to-compute and compute-to-
compute servers in the cluster. Mesh or partial mesh connectivity is also application-
dependent.

•High throughput—The ability to send a large file in a specific amount of time can be critical
to cluster operation and performance. Server clusters typically require a minimum amount of
available non-blocking bandwidth, which translates into a low oversubscription model between
the access and core layers.

•Oversubscription ratio—The oversubscription ratio must be examined at multiple

aggregation points in the design, including the line card to switch fabric bandwidth and the
switch fabric input to uplink bandwidth.

•Jumbo frame support—Although jumbo frames might not be used in the initial
implementation of a server cluster, it is a very important feature that is necessary for additional
flexibility or for possible future requirements. The TCP/IP packet construction places

12
additional overhead on the server CPU. The use of jumbo frames can reduce the number of
packets, thereby reducing this overhead.

•Port density—Server clusters might need to scale to tens of thousands of ports. As such, they
require platforms with a high level of packet switching performance, a large amount of switch
fabric bandwidth, and a high level of port density.

• High-Availability— All data center designs are judged by their ability to provide continuous
operations for the network services they support. Data center availability is affected by both
planned (scheduled maintenance) and unplanned (failures) events. To maximize availability,
the impact from each of these must be minimized and/or eliminated.

All data center designs are judged by their ability to provide continuous operations for the
network services they support. Data center availability is affected by both planned (scheduled
maintenance) and unplanned (failures) events. To maximize availability, the impact from each
of these must be minimized and/or eliminated.

All data centers must be maintained on a regular basis. In most data center designs,
scheduled maintenance is a planned event requiring network downtime. For this reason,
general maintenance is often forgone, leaving long-term availability to chance. In robust data
center designs, concurrently maintainable systems are implemented to avoid interruption to
normal data center operations.
To mitigate unplanned outages, both redundancy and fault-tolerance must be incorporated into
the data center design. High-availability is accomplished by providing redundancy for all,
major and minor, systems, thereby eliminating single points of failure. Additionally, the data
center design must offer predictable uptime by incorporating fault-tolerance against hard
failures. (A hard failure is a failure in which the component must be replaced to return to an
operational steady state.)
A data center achieves high-availability by implementing a fully redundant, fault-tolerant, and
concurrently maintainable IT and support infrastructure architecture in which all possible hard
failures are predictable and deterministic.

13
2.2 Scope

An important distinction to make at this point is what really constitutes the elements of a data center.
When we talk about the data center, we are talking about the site, the Command Center (if one is to
be added), the raised floor (if one is to be added), the network infrastructure (switches, routers,
terminal servers, and support equipment providing the core logical infrastructure), the environmental
controls, and power. Though a data center contains servers and storage system components
(usually contained in racks), these devices are contents of the data center, not part of the data
center. They are transient contents just as DVDs might be considered the transient contents of a DVD
player. The data center is more of a permanent fixture, while the servers and storage systems are
movable, adaptable, interchangeable elements. However, just as the DVD is of no value without the
player and the player is of no value without the DVD, a data center without equipment is an expensive
empty room, and servers with no connection are just expensive paper weights. The design of the data
center must include all of the elements. The essential elements are called the criteria.

Most often, it is the project scope that determines the data center design. The scope must be
determined based on the company's data center needs (the desired or required capacities of the system
and network infrastructure), as well as the amount of money available. The scope of the project could
be anything from constructing a separate building in another state with offices and all the necessary
utilities, to simply a few server and storage devices added to an existing data center. In either case,
those creating the project specifications should be working closely with those responsible for the
budget.

3. Limitation
The primary components necessary to build a data center include rack space (real estate), electrical
power and cooling capacity. At any given time, one of these components is likely to be a primary
capacity limitation, and over the past few years, the most likely suspect is power. The obvious
requirement is power for the servers and network equipment, but sometimes less obvious is the power
required to run the air handling and cooling systems. Unless you have built your data center right next
to a power station, and have a very long contract in place for guaranteed supply of power at a nice low
rate per kilowatt-hour, you are likely seeing your data center costs rise dramatically as the cost of
electricity increases.

Power costs have become the largest item in the OpEx budgets of data-centre owners and, for the first
time, now exceed the IT hardware cost over its average service-life of 3-4 years. This has resulted in
14
the pursuit (both real and spun) for higher efficiency, sustainable, ‘green’ designs. Majority of the
losses occurs in a data centre as per the below mentioned ratio:

COMPONENT Losses %
Chip Set 34%
SMPS losses 9%
Server Fans 6%
Room Fans 4%
Pumps 2%
Compressors 18%
Condenser Fans 4%
Humidification 1%
Plantroom Cooling 1%
Anticilliary Power 7%
Security, Controls & Comms 1%
UPS & Distribution Losses 6%
Transmission Losses 7%

Many IT professionals are finding themselves in the awkward position of begging their data center
hosting providers for increased electrical capacity, or they are facing the unpalatable option of
relocating their data center facilities.

While the industry is very aware of the issue of equipment power consumption, and the major
manufacturers are already designing in power reduction and conservation features to their latest
products, we are far from turning the corner on the demand for more data center electrical power. The
demand for computing capacity continues to rise, which implies a demand for “more, better and
faster” systems. This results directly in higher demands for rack space, power and cooling in
your data center, and related increased costs.

The increasing costs are receiving a lot of executive management attention, especially given current
economic conditions. Server rationalization is the buzz word of the day. Do we need to buy a new
server, or can we re-use one we already have? Are we using the servers we have to their full
capability? A typical data center has very high storage utilization (as no one voluntarily throws old
data away), and server utilization is low (as different functions within a company don’t want to share).
Server virtualization has become one of the hottest topics in IT, as it has become a means of
ensuring higher utilization of servers, lowering data center costs and providing flexibility for
shuffling systems’ workload. But there are limits to what can be virtualized, and of course, the
overall utilization of a server pool has an upper bound, as well. But there are other components needed
to build a data center, including: network equipment and communication circuits, power distribution

15
equipment (e.g., distribution panels, cable and outlets), power backup equipment (including
generator(s) and uninterruptible power supplies (UPS)), cable trays (for both network and power
cables), and fire suppression systems.

And, yes, believe it or not, sometimes these other components can become the constraining factor in
data center capacity. “Cable trays can be a limiting factor?” you ask. Yes, just two years ago, we ran
into a situation where an older data center couldn’t add capacity to a specific cage because the weight
of the cable already in the tray was at design load limits. We couldn’t risk the tray ripping out of the
ceiling by laying in more cable, and we couldn’t shut the network and power down long enough to pull
out all of the old cables and put new ones back in without excessive downtime to the business. A very
expensive migration to a new cage in the data center became the only feasible option. Though it may
seem hard to believe, there are many IT professionals who have never seen the inside of a data center.

Over the years, having a glass-walled computer room as a showcase in your corporate headquarters
became problematic for a number of reasons including, security, high-priced real estate for servers that
could be better accommodated elsewhere, as well as loss of the glass wall space for other purposes
(e.g., communications punch-down blocks, electrical distribution panels). Besides, the number of
blinking lights in the data center has steadily decreased over the years. And you don’t see white lab-
coated workers pushing buttons and handling tapes. So, what’s there to see? (Not much, especially in a
“lights out” facility.)

But the interesting part is: when corporate executives and managers do occasionally visit a data center
facility, they still expect to see nice clean rows of equipment, full racks of blinking lights and servers
happily computing away. Instead, we now see large amounts of unused floor space and partially filled
racks. As servers have become more powerful per cubic inch of space occupied, power and cooling
capacity have become increasingly scarce, not the rack space for the equipment. The decreasing server
footprint relative to the higher energy per cubic inch requirement is often referred to as a “power-
density” problem.

You should ask your data center manager for a tour. Standing behind a full rack of 40+ servers
consuming 200 watts (or more) of power each is an amazing experience, likened to having someone
turn six 1200-watt hair dryers directly toward you, running full blast.

While the heat load behind a server rack has its shock effect, the more interesting cognitive dissonance
for many executives is seeing the empty racks and floor space. When called upon to explain, my
16
simple example has been this: if the power capacity (power cap) in your cage (or computer room) is
limited to 100 kWh, it doesn’t matter whether you have 10 full racks that consume 10 kWh each, or 20
partially filled racks that only consume 5 kWh each. If you have only one supercomputer that
consumes all 100 kWh sitting in the middle of the room taking up only 20 square feet and there is 500
square feet of unused space all around it, it may look very odd, but you’re still out of power.

It would be wonderful if the data center design would “come out even,” with exactly the right
amount of full racks, power and cooling to look like the space is being well-utilized, but that is not
a common occurrence these days. Even if you can optimize one cage, it’s extremely difficult to
optimize across the entire data center floor.

Building out a new data center space requires careful planning and engineering of all the
components. And even then, changes over time in server utilization, increases (and decreases) in
the business’ needs, Virtualization and server technology all conspire to generate re-work of
your carefully thought-out and well-balanced design.

4. Perspectives
IT operations are a crucial aspect of most organizational operations. One of the main concerns is
business continuity; companies rely on their information systems to run their operations. If a system
becomes unavailable, company operations may be impaired or stopped completely. It is necessary to
provide a reliable infrastructure for IT operations, in order to minimize any chance of disruption.
Information security is also a concern, and for this reason a data center has to offer a secure
environment which minimizes the chances of a security breach. A data center must therefore keep high
standards for assuring the integrity and functionality of its hosted computer environment. This is
accomplished through redundancy of both fiber optic cables and power, which includes emergency
backup power generation.

Information stewardship is one key perspective. Information stewardship calls for holistic data
management in the enterprise: defining and enforcing policy to guide the acquisition, management,
and storage lifecycle of data, and the protection of data from theft, leak, or disaster. Our research
shows that enterprises that manage these intertwined issues as a set are more successful dealing with
them than those that treat them as disjoint.

The IT executives we are speaking with in our current research on security and information protection
frequently cite the rising importance of a second key perspective: risk management.
17
In the past, we mainly heard about risk management in two specific contexts: disaster planning and
security. In disaster planning, risk assessment (where risk equals the cost to the business of major IT
outages times the likelihood of the natural or manmade disasters that would lead to those outages)
dictates how much the enterprise should spend on back-up IT infrastructure and services. In security,
IT would often focus on specific threats and specific defensive technologies, and use risk assessment
mainly to help decide where to spend money on security tools, or how to dedicate IT security staff
time.

Now many of the people we speak with tell us they use risk as a lens through which they view all their
systems, processes, and staffing. Risk is not subordinate, as a component in the calculations of
security and business continuity planners; instead, security and business continuance have
become facets of risk management.

5. Methodology and Procedure of Work

5.1 Sizing the Data Center
Nothing has a greater influence on a Data Center's cost, lifespan, and flexibility than its size—even the
Data Center's capability to impress clients. Determining the size of your particular Data Center is a
challenging and essential task that must be done correctly if the room is to be productive and cost-
effective for your business. Determining size is challenging because several variables contribute to
how large or small your server environment must be, including:

• How many people the Data Center supports

• The number and types of servers and other equipment the Data Center hosts
• The size that non-server areas should be depending upon how the room's infrastructure is
deployed

Determining Data Center size is essential because a Data Center that is too small won't adequately
meet your company's server needs, consequently inhibiting productivity and requiring more to be
spent on upgrading or expansion and thereby putting the space and services within at risk. A room that
is too big wastes money, both on initial construction and ongoing operational expenses.
Many users do not appreciate these effects during data center planning, and do not establish the floor
layout early enough. As a result, many data centers unnecessarily provide suboptimal performance.

Below explained how floor plans affect these characteristics, and to prescribe an effective method for
developing a floor layout specification.

18
5.2 Role of the Floor Plan in the System Planning Sequence

Floor plans must be considered and developed at the appropriate point in the data center design
process. Considering floor plans during the detailed design phase is typical, but simply too late in the
process. Floor plans should instead be considered to be part of the preliminary specification and
determined BEFORE detailed design begins.

It is not necessary for a floor layout to comprehend the exact location of specific IT devices. The
effective floor plans only need to consider the location of equipment racks or other cabinets, and to
target power densities. These preliminary floor layouts do not require knowledge of specific IT
equipment.

For most users it is futile to attempt to specify particular IT equipment locations in advance – in fact,
racks may ultimately house equipment that is not even available on the market at the time the data
center is designed.

Figure 5.2 – The floor plan is a key input in the system planning sequence

19
The reasons that floor plans must be considered early, as part of the preliminary specification, and not
left until the later detailed design include:
• Density is best specified at the row level, so rows must be identified before a density
specification can be created.
• Phasing plans are best specified using rows or groups of rows, so rows must be identified
before an effective phasing plan can be created.

20
 • The floor grid for a raised floor and the ceiling grid for a suspended ceiling should be aligned
to the rack enclosures, so rows must be identified before those grids can be located.
• Criticality or availability can (optionally) be specified differently for different zones of the
data center – rows must be identified before a multi-tier criticality plan can be created.

Density and phasing plans are a key part of any data center project specification, and both require a
row layout. Detailed design can only commence after density, phasing, and criticality have been
specified.
Therefore, a floor plan must be established early in the specification phase of a project, after
SYSTEM CONCEPT but well before DETAILED DESIGN (see Figure 5.2).

5.3 Floor Planning Concepts

A data center floor plan has two components: the structural layout of the empty room and the
equipment layout of what will go in the room. Note that for many projects the room is pre-existing
and the only option is to lay out the equipment within the room. A key rule of data center design is that
there is a potentially huge advantage in efficiency and density capacity if planners can lay out the
room boundaries at the outset. Wherever possible, an attempt should be made to influence the
structural room layout using the principles established.

5.3.1 Structural layout

The room layout includes the location of walls, doors, support columns, windows, viewing windows,
and key utility connections. If the room has a raised floor, the height of the raised floor and the
location of access ramps or lifts are also part of the structural layout. If the room has a raised floor or a
suspended ceiling, the index points for the floor or ceiling grid are critical design variables, and must
also be included in the structural layout. Room measurements will be described in units of tiles, where
a tile width is equal to 2 feet (600 mm) or one standard rack enclosure width.

5.3.2 Equipment layout

The equipment layout shows the footprint of IT equipment and the footprint of power and cooling
equipment. IT equipment can usually be defined as rack locations without regard for the specific
devices in the cabinets, but other equipment such as tape libraries or large enterprise servers may have
form factors that are different from typical racks and must be called out explicitly. In addition, IT
equipment in a layout must be characterized by its airflow path. In the case of typical IT racks, the

21
airflow is front-to-back, but some devices have other airflow patterns such as front-to-top. Power
and cooling equipment must also be accounted for in equipment layouts, but many new power and
cooling devices are either rack mountable or designed to integrate into rows of racks, which simplifies
the layout.

5.4 The Effects of Floor Plans on Data Center Performance

Several important data center characteristics are affected by floor plans. To understand effective floor
layout methods, it is important to understand the consequences.

5.4.1 Number of rack locations

The floor layout can have a dramatic affect on the number of rack locations that are possible in the
room. Although, on average, the number of IT rack locations possible can be estimated by dividing the
room area by 28 sq ft / rack (2.6 sq meters / rack)1, the actual number of racks for a particular data
center can vary greatly from this typical value.

The basic principle of floor planning is to maximize the number of rack locations possible. Small
variations in the location of walls, existing IT devices, air conditioners, and power distribution units
can have a surprisingly large impact on the number of possible rack locations. This effect is magnified
when high power densities are required. For this reason, a careful and systematic approach to floor
planning is essential.

5.4.2 Achievable power density

The floor plan can have a major impact on the achievable power density. With certain cooling
architectures, a poor layout can decrease the permissible power for a given rack by over 50%.
This is a huge performance compromise in a modern data center, where new technologies have power
densities that are already stressing the capabilities of data center design. In many data centers, users
may want to establish zones of different power density. These density zones will be defined by the
equipment layout. The floor plan is therefore a critical tool to describe and specify density for data
centers.

5.4.3 Complexity of distribution systems

The floor plan can have a dramatic affect on the complexity of the power and cooling distribution
systems. In general, longer rows, and rows arranged in the patterns, simplify power and cooling

22
distribution problems, reduce their costs, and increase their reliability.

5.4.4 Cooling performance

In addition to impacting the density capability of a data center, the floor plan can also significantly
affect the ability to predict density capability. It is a best practice to know in advance what density
capability is available at a given rack location and not to simply deploy equipment and “hope for the
best,” as is a common current practice. An effective floor plan in combination with row-oriented
cooling technologies allows simple and reliable prediction of cooling capacity. Design tools such
as APC InfraStruXure Designer can automate the process during the design cycle, and when layouts
follow standard methods, off-the-shelf operating software such as APC InfraStruXure Manager can
allow users to monitor power and cooling capacities in real time.

5.4.5 Electrical Efficiency

Most users are surprised to learn that the electrical power consumption of a data center is greatly
affected by the equipment layout. This is because the layout has a large impact on the effectiveness
of the cooling distribution system. This is especially true for traditional perimeter cooling
techniques. For a given IT load, the equipment layout can reduce the electrical power consumption of
the data center significantly by affecting the efficiency of the air conditioning system.

• The layout affects the return temperature to the CRAC units, with a poor layout yielding a
lower return air temperature. A lower return temperature reduces the efficiency of the
CRAC units.
• The layout affects the required air delivery temperature of the CRAC units, with a poor
layout requiring a colder supply for the same IT load. A lower CRAC supply temperature
reduces the efficiency of the CRAC units and causes them to dehumidify the air, which in turn
increases the need for energy-consuming humidification.
• The layout affects the amount of CRAC airflow that must be used in “mixing” the data center
air to equalize the temperature throughout the room. A poor layout requires additional mixing
fan power, which decreases efficiency and may require additional CRAC units, which draw
even more electrical power.
A conservative estimate is that billions of kilowatt hours of electricity have been wasted due to
poor floor plans in data centers. This loss is almost completely avoidable.

23
5.5 Basic Principles of Equipment Layout

The existence of the rack as the primary building block for equipment layouts permits a standardized
floor planning approach. The basic principles are summarized as follows:

• Control the airflow using a hot-aisle/cold-aisle rack layout.

• Provide access ways that are safe and convenient.
• Align the floor or ceiling tile systems with the equipment.
• Minimize isolated IT devices and maximize row lengths.
• Plan the complete equipment layout in advance, even if future plans are not defined.
Once these principles are understood, an effective floor planning method becomes clear.

5.5.1 Control of airflow using hot-aisle/cold-aisle rack layout

The use of the hot-aisle/cold-aisle rack layout method is well known and the principles are described
in other documents, such as ASHRAE TC9.9 Mission Critical Facilities, “Thermal Guidelines for
Data Processing Environments” 2004, and a white paper from the Uptime Institute titled
“Alternating Cold and Hot Aisles Provides More Reliable Cooling for Server Farms.” The basic
principle is to maximize the separation between IT equipment exhaust air and intake air by
establishing cold aisles where only equipment intakes are present and establishing hot aisles where
only equipment hot exhaust air is present. The goal is to reduce the amount of hot exhaust air that is
drawn into the equipment air intakes. The basic hot-aisle/cold aisle concept is shown in Figure 5.5.1.
Figure 5.5.1 – Basic hot-aisle/cold-aisle data center equipment layout plan

24
In the above figure, the rows represent the IT equipment enclosures (racks). The racks are arranged
such that the adjacent rows face back to back, forming the hot aisles.
The benefits of the hot-aisle/cold-aisle arrangement become dramatic as the power density increases.
When compared to random arrangements or arrangements where racks are all lined up in the same
direction, the hot-aisle/cold-aisle approach allows for a power density increase up to 100% or more,
without hot spots, if the appropriate arrangement of CRAC units is used. Because all cooling
architectures (except for fully enclosed rack-based cooling) benefit dramatically from hot-
aisle/cold-aisle layout, this method is a principal design strategy for any floor layout.

5.5.2 Align the floor and/or ceiling tiles with the equipment

In many data centers the floor and ceiling tile systems are used as part of the air distribution
system. In a raised floor data center, it is essential that the floor grille align with racks. If the racks and
the floor grid do not align, airflow can be significantly compromised. It is also beneficial to align any
ceiling tile grid with the floor grid. This means the floor grid should not be designed or installed
until after the equipment layout is established, and the grid should be aligned or indexed to the
equipment layout according to the row layout options.
Unfortunately, specifiers and designers often miss this simple and no-cost optimization opportunity.
The result is that either (1) the grid is misaligned with the racks, with a corresponding reduction in

25
efficiency and density capability, or (2) the racks are aligned to the grid but a suboptimal layout
results, limiting the number of racks that can be accommodated.

5.5.3 Pitch – the measurement of row spacing

The row length in a hot-aisle/cold aisle layout is adjustable in increments of rack width, which
provides significant flexibility. However, the spacing between aisles has much less flexibility and is a
controlling constraint in the equipment layout. The measurement of row-to-row spacing is called pitch,
the same term that is used to describe the repeating center-to-center spacing of such things as screw
threads, sound waves, or studs in a wall. The pitch of a data center row layout is the distance from one
mid-cold-aisle to the next mid-cold-aisle (Figure 5.5.3).

Figure 5.5.3 – Pitch of a row layout

26
5.5.4 Minimize isolated IT devices and maximize row lengths

The control of airflow by separating hot and cold air, as described above, is compromised at
the end of a row where hot air can go around the side of the end rack and return to IT equipment air
intakes on the back.
Therefore, the theoretical ideal design of a data center is to have no row ends – i.e. rows of infinite
length. Conversely, the worst case situation would be rows of one-rack length – i.e., isolated single
racks. In addition, the ability to effectively implement redundancy is improved with longer rows. The
goal of row layout is to maximize row length consistent with the goals of providing safe and
convenient access ways. In general, a layout that provides longer row lengths is preferred, and a row
layout that generates short rows of 1-3 racks should be avoided.

5.5.5 Special considerations for wide racks

Standard-width racks (2 ft or 600 mm) conveniently align with the width of raised-floor tiles. When
underfloor cables must be distributed to such a rack, a hole is typically created in the tile directly
below the rack to run the cables; if that particular rack is then re-located or removed, the tile is simply
replaced with a new one.

Wide racks that do not align with the standard raised floor tile width are creating a new challenge,
because a rack may occupy two or even three tiles. If such a rack is removed, no longer can the tile
simply be replaced with a new one, since the tile is partially underneath the neighboring rack as well.
These issues can be avoided altogether by overhead power and data cable distribution.

5.5.6 Plan the complete floor layout in advance

The first phase of equipment deployment often constrains later deployments. For this reason it is
essential to plan the complete floor layout in advance.

5.5.7 Minimize isolated IT devices and maximize row lengths

When row lengths are three racks or less, the effectiveness of the cooling distribution is impacted.
Short rows of racks mean more opportunity for mixing of hot and cold air streams. For this reason,
when rooms have one dimension that is less than 15-20 feet it will be more effective in terms of
cooling to have one long row rather than several very short rows.

27
5.5.8 Standardized room dimensions

There are preferred room dimensions for data centers, based on the pitch chosen. Given an area or
room that is rectangular in shape, free of the constraints imposed by support columns (described
earlier), the preferred length and width are established as follows:
• One dimension of the room should be a multiple of the hot-aisle/cold-aisle pitch, plus a
peripheral access-way spacing of approximately 2-4 tiles
• The other dimension of the room is flexible and will impact the length of the rows of racks

When one of the dimensions of the room is not optimal, the performance of the room can be
dramatically reduced, particularly if the room is smaller. The most obvious problem is that the number
of equipment racks may be lower than expected because some space cannot be used. The second, and
less obvious, problem is that when the ideal layout cannot be achieved, the power density and
electrical efficiency of the system is reduced.

To understand the effect of room dimension on the number of racks, consider a room with a fixed
length of 28 feet and a variable width. In such a room, the length of a row would be 10 racks, allowing
for 2 tiles (4 feet) at each row-end for access clearance. The number of racks that could fit in this room
will vary as a function of the width of the room as shown in Figure 5.5.8.

Figure 5.5.8 shows that the number of installable racks jumps at certain dimensions as new rows fit
into the room. Furthermore, the chart shows that certain numbers of racks are preferred because the
even row number permits a complete additional hot-aisle/cold-aisle pair to be installed. The preferred
width dimensions are indicated by the arrows, for the pitch (the most compact pitch A in this case) and
perimeter clearances (2 tiles) defined.

Figure 5.5.8 – Impact of room dimension on number of rows

28
5.5.9 Location of support columns in room boundary layout
The location of support columns in the room can dramatically affect the equipment layout, as
previously illustrated. Therefore, when an option exists to locate room boundaries, the following
guidelines apply:
• For smaller rooms, arrange the room boundaries, if possible, so that no support columns are
in the equipment area.
• Rooms should be rectangular, where possible. Unusual shapes, niches, and angles often
cannot be effectively utilized and/or create a reduction in power density or electrical
efficiency.
• For situations where columns are unavoidable but boundaries are flexible, the floor plan
should be laid out as if no columns existed, based on the standardized dimensions of the
room, and the pitch(es) required. Columns should then be located directly over any one
particular rack location, preferably at a row end.
• For very large rooms, the location of the walls in relation to the columns is typically
inflexible.
29
When a column is located directly over a particular rack location, as the third bullet above suggests, it
is important to block off any openings between the column(s) and the neighboring racks. If these gaps
are not blocked with a filler panel, mixing of hot and cold air streams can occur and cooling
performance can be compromised.

5.5.10 Phased deployments

When a phased deployment is planned, there are two strategies that can be beneficial. These are:
• Creating area partitions
• Advance layout of future rows
When a future phase has a very large uncertainty, area partitions or walls that subdivide the data center
into two or more rooms can be used. The benefits are:
• Ability to re-purpose areas in the future
• Ability to perform radical infrastructure modifications in one area without interfering with the
operation of another area
• Ability to defer the installation of basic infrastructure (such as piping or wiring) to a future
date

The advent of modular row-oriented power and cooling architectures has reduced the need to provide
radical infrastructure modifications during new deployments, and has greatly reduced the cost and
uncertainty associated with installing base wiring and plumbing infrastructure. Therefore, the
compelling need to partition data centers has been dramatically reduced. Nevertheless, retaining
options such as future re-purposing of area is valuable for some users. The key to successful
partitioning is to understand that partitions should NEVER be placed arbitrarily without first
performing an equipment layout scenario analysis. This is because the floor layout can be seriously
compromised by a poor choice of a partition position.

During the setting of partitions or walls within a data center room, the same principles should be
applied as those used when establishing the overall perimeter room boundaries. The standard spacing
of rows must be considered. Failure to do this can result in problems (See Figure 5.5.10).
Note that the location of the wall in the bottom scenario has caused row 5 of the equipment layout to
be lost, representing 10 racks out of the 80 rack layout, or 12% of the total – a significant loss of rack
footprint space. Although the wall was only offset by a small amount, this loss occurs because the
wall-to-wall spacing does not permit appropriate access ways if row 5 is included. Furthermore, the

30
access way between row 6 and the wall has become a hot aisle. This reduces the confining effect of the
hot-aisle/cold-aisle design and will result in a reduced power capacity for row 6. Furthermore, because
the primary access path between row 6 and the wall is now a hot aisle, this creates an uncomfortable
zone for personnel. These factors taken together demonstrate how serious a small change in a wall
location can be when partitioning a data center.
Figure 5.5.10 – Impact of portioning placement on number of rack locations

5.6 Floor Planning Sequence

Using the rack as the basic building block for floor layout, and the row-pair “pitch” as the spacing
template, a standardized floor layout approach is achievable. Starting with a floor plan diagram for the
room, the basic principles are summarized as follows:

31
5.6.1 Identify and locate the room constraints

First, identify and locate all physical room constraints:

• Columns – verify the exact as-built dimensions
• Doorways
• Existing fixed equipment – breaker panels, pipe connections, fire suppression equipment,
cooling equipment

5.6.2 Establish key room-level options

Next, identify what additional equipment will be placed in the room, and the options available for
delivering/installing that equipment within existing room constraints:
• Identify additional equipment (besides the IT equipment or in-row power and cooling
equipment) that will be placed in the room, including any additional cooling equipment, fire
suppression equipment, power equipment, or user workstations.
• If the room uses a raised floor, determine the length(s) of the access ramp(s) and identify all
possible options for locating the ramps It is critical at this stage to know if the facility will have
a raised floor. Many new high-density data centers do not use a raised floor, so a raised floor
should not be automatically assumed. Sometimes it is even appropriate to remove a raised floor
from an existing site for new deployments.

5.6.3 Establish the primary IT equipment layout axis

Every room has two primary layout axes, or directions that the rows can be oriented. The axis
selection is one of the most critical decisions in a data center plan and has a large impact on
performance and economy. When using a hot-aisle/cold-aisle row pair arrangement in the pitch
determined necessary or preferred, test the two primary axis orientation layouts to establish if either
has an obvious advantage. When performing the test layouts, ensure that:
• Columns are not located in main access ways
• (If no raised floor) Rows are aligned to the ceiling grid so that the cold aisles contain
complete tiles
• There is sufficient clearance at row-ends and between rows and walls
• There is sufficient clearance/access around any fixed equipment in the room
• Access ramps, if required, are present and have been optimally located
• Any open areas or areas for another purpose face a cold aisle, not a hot aisle

32
 • Locations have been found for any additional equipment identified in the room level options
above
• Rows that are separated by an access way should not reverse the direction they face
• All rows align with the same axis (i.e., all a rows are parallel, with no perpendicular rows)
• The entire room is laid out in the floor plan, even if no immediate plans are in place to deploy
some sections of the room

To determine the preferred layout, the following factors should be considered:

• Which axis most effectively keeps support columns out of the main access ways?
• Which axis allows for the most racks?
• Which axis works best with the preferred hot-aisle/cold-aisle pitch?
• Which axis has hot-aisle/cold-aisle row pairs and does not end up with an odd number of
rows?
• Which axis has the fewest short rows or isolated racks?
• Which layout provides the desired aesthetic layout of the data center for purposes of viewing
or tours, if that is a consideration.

Different users may weigh the above criteria differently. It is common for users to choose a layout axis
that meets aesthetic considerations without concern for the data center performance, and later regret
their choice. The preferred method is to test both axes during planning and carefully decide the axis
selection based on an understanding of the consequences.

5.6.4 Lock the row boundaries

The process of selecting the primary layout axis typically establishes the row locations accurately.
With the row locations established, it is critical to establish and validate the row boundaries. This
includes setting the row end boundaries, and verifying the boundaries between fronts and/or backs of
rows with respect to other equipment, columns, or walls.

Access must be provided between row-ends and other obstructions using the following guidelines:
• For plain walls, a minimum of 2 tiles is an acceptable spacing for a row-end; larger data
centers often prefer 3 tiles to provide better accessibility.
• For some layouts, it may be desired to end a row at a wall. However, this creates a dead-end
alleyway which may limit the length of the row based on code requirements.

33
 • For long rows of over 10 racks, local regulations may require that breaks be placed in rows to
allow personnel to pass through. This may also be of practical concern for technicians who
need access to both sides of a rack without having to walk a long distance.
• The spacing between the row front (cold aisle) or the row back (hot aisle) and other
equipment must be carefully checked to ensure that access ways are sufficient, and that any
access required to those other devices for service or by regulation is sufficient and meets code.
• It must be verified that any other equipment that has been located as part of the floor plan is
not constrained by piping, conduits, or access restrictions.

The above restrictions and boundaries must be marked on the room layout before the axis selection
and row layout are confirmed.

For small data centers – i.e., up to 2 rows of racks – this floor planning process can occur as a paper
study. As the room size grows, computer-aided tools that ensure consistent scale become necessary in
order to accurately plan the floor layout. Ideally, the row layout and boundary areas should also be
marked out using colored masking tape in the actual facility. This step is quite feasible for many
smaller fit-out designs and for retrofits, and often identifies surprise constraints that were not realized
during the conceptual plans.

5.6.5 Specify row/cabinet density

Once the row boundaries and the orientation of the row axis have been established, the
enclosure/cabinet layout can be performed. This begins with the partitioning of rows by buildout
phase. For each phase, multiple zones or areas may exist, each with a unique density requirement.

5.6.6 Identify index points (for new room)

If the data center has a pre-existing raised floor, then the actual location of the floor grid relative to the
wall is pre-established and will have been comprehended in an earlier process step. However, for new
rooms, the raised floor grid location is controlled by the floor layout. An index point for the raised
floor grid should be established in the plan, and clearly and permanently marked in the room. It is
absolutely essential that the contractor installing the raised floor align the grid to the index point
during installation. If this is not done, it may not be possible to shift the layout later to align with
the grid due to the boundary constraints. In a raised floor design, this can result in a massive
loss of power density capability and a dramatic reduction of energy efficiency. This is a

34
completely avoidable and terrible error that is commonly made during data center installations. Data
centers that use a hard floor rather than a raised floor do not have this concern.
If the data center uses a suspended ceiling for lighting and/or air return, aligning the index point to the
ceiling grid is also highly recommended, but less critical than aligning with the floor grid.

5.6.7 Specify the floor layout

The final step in the floor planning process is to specify the floor layout for subsequent design and
installation phases of the data center project. The specification is documented as a detailed floor layout
diagram, which includes all necessary room and obstruction measurements, all rack locations
identified, all unusable areas marked, and non-rack-based IT equipment that requires power and
cooling noted. Ideally, this specification diagram is created in a computer-aided tool such as APC’s
InfraStruXure Designer, which subsequently allows for the complete design of the data center’s
physical infrastructure, detailed to the rack level.

5.6.8 Identify index points (for new room)

If the data center has a pre-existing raised floor, then the actual location of the floor grid relative to the
wall is pre-established and will have been comprehended in an earlier process step. However, for new
rooms, the raised floor grid location is controlled by the floor layout. An index point for the raised
floor grid should be established in the plan, and clearly and permanently marked in the room. It is
absolutely essential that the contractor installing the raised floor align the grid to the index point
during installation. If this is not done, it may not be possible to shift the layout later to align with
the grid due to the boundary constraints. In a raised floor design, this can result in a massive
loss of power density capability and a dramatic reduction of energy efficiency. This is a
completely avoidable and terrible error that is commonly made during data center installations. Data
centers that use a hard floor rather than a raised floor do not have this concern.
If the data center uses a suspended ceiling for lighting and/or air return, aligning the index point to the
ceiling grid is also highly recommended, but less critical than aligning with the floor grid.

5.7 Common Errors in Equipment Layout

Many users attempt rudimentary floor layout planning, yet still have downstream problems. Here are
some of the most common problems observed by industries:

35
5.7.1 Failure to plan the entire layout in advance

Most data centers begin to deploy equipment without a complete equipment deployment plan. As the
deployment expands, severe constraints on the layout may emerge, including:
• Equipment groups grow toward each other and end up facing hot-to-cold instead of hot-tohot,
with resultant hot spots and loss of power density capability
• Equipment deployments grow toward a wall, and it is subsequently determined that the last
row will not fit – but would have fit if the layout had been planned appropriately
• The rows have a certain axis orientation, but it is later determined that much more equipment
could have been deployed if the rows had been oriented 90º the other way – and it is too late
to change it
• Equipment deployments grow toward a support column, and it is subsequently determined
that the column lands in an access way, limiting equipment deployment – but much more
equipment could have been placed if the layout had been planned in advance
• Equipment deployments drift off the standard floor tile spacing and later high-density
deployments are trapped, not having full tiles in the cold aisles, with a resultant loss of power
density capability

Most existing data centers have one or more of the problems listed above, with the attendant
lossesof performance. In typical data centers routinely observed, the loss of available rack locations
due to these problems is on the order of 10-20% of total rack locations, and the loss of power density
capability is commonly 20% or more. These unnecessary losses in performance represent substantial
financial losses to data center operators, but can be avoided by simple planning.

5.8 Data Center Multi-Tier Design Overview

The multi-tier model is the most common model used in the enterprise today. This design consists
primarily of web, application, and database server tiers running on various platforms including
blade servers, one rack unit (1RU) servers, and mainframes.

5.8.1 Why Use the Three-Tier Data Center Design?

Why not connect servers directly to a distribution layer and avoid installing an access layer?
The three-tier approach consisting of the access, aggregation, and core layers permit flexibility in the
following areas:

36
 •Layer 2 domain sizing— When there is a requirement to extend a VLAN from one switch to
another, the domain size is determined at the distribution layer. If the access layer is absent, the
Layer 2 domain must be configured across the core for extension to occur. Extending Layer 2
through a core causes path blocking by spanning tree and has the risk of uncontrollable broadcast
issues related to extending Layer 2 domains, and therefore should be avoided.

•Service modules—An aggregation plus access layer solution enables services to be shared across
the entire access layer of switches. This lowers TCO and lowers complexity by reducing the
number of components to configure and manage. Consider future service capabilities that include
Application-Oriented Networking (AON), ACE, and others.

•Mix of access layer models—The three-tier approach permits a mix of both Layer 2 and Layer 3
access models with 1RU and modular platforms, permitting a more flexible solution and allowing
application environments to be optimally positioned.

•NIC teaming and HA clustering support—Supporting NIC teaming with switch fault tolerance
and high availability clustering requires Layer 2 adjacency between NIC cards, resulting in Layer 2
VLAN extension between switches. This would also require extending the Layer 2 domain through
the core, which is not recommended.

5.8.2 Why Deploy Services Switch?

When would I deploy Services Switch instead of just putting Services Modules in the
Aggregation Switch?
Incorporating Services Switch into the data center design is desirable for the following reasons:

•Large Aggregation Layer—If services are deployed in the aggregation layer, as this section
scales it may become burdensome to continue to deploy services in every aggregation switch. The
service switch allows for services to be consolidated and applied to all the aggregation layer
switches without the need to physically deploy service cards across the entire aggregation layer.
Another benefit is that it allows the aggregation layer to scale to much larger port densities since
slots used by service modules are now able to be deployed with LAN interfaces.

•Mix of service modules and appliances—Data center operators may have numerous service
modules and applicances to deploy in the data center. By using the service chassis model you can
deploy all of the services in a central fashion, allowing the entire data center to use the services
instead of having to deploy multiple appliances and modules across the facility.

•Operational or process simplification—Using the services switch design allows for the core,
aggregation, and access layers to be more tightly controlled from a process change prespective.

37
 Security, load balancing, and other services can be configured in a central fashion and then applied
across the data center without the need to provide numerous access points for the people operating
those actual services.

•Support for network virtualization—As the network outside of the data center becomes more
virtualized it may be advantageous to have the services chassis become the point where things like
VRF-aware services are applied without impacting the overall traffic patterns in the data center.

5.8.3 Determining Maximum Servers

What is the maximum number of servers that should be on an access layer switch? What is the
maximum number of servers to an aggregation module?
The answer is usually based on considering a combination of oversubscription, failure domain sizing,
and port density. No two data centers are alike when these aspects are combined. The right answer for
a particular data center design can be determined by examining the following areas:

•Oversubscription—Applications require varying oversubscription levels. For example, the web

servers in a multi-tier design can be optimized at a 15:1 ratio, application servers at 6:1, and
database servers at 4:1. An oversubscription ratio model helps to determine the maximum number
of servers that should be placed on a particular access switch and whether the uplink should be
Gigabit EtherChannel or 10GE. It is important for the customer to determine what the
oversubscription ratio should be for each application environment. The following are some of the
many variables that must be considered when determining oversubscription:

–NIC—Interface speed, bus interface (PCI, PCI-X, PCI-E)

–Server platform—Single or dual processors, offload engines

–Application characteristics—Traffic flows, inter-process communications

–Usage characteristics—Number of clients, transaction rate, load balancing

•Failure domain sizing—This is a business decision and should be determined regardless of the
level of resiliency that is designed into the network. This value is not determined based on
MTBF/MTTR values and is not meant to be a reflection of the robustness of a particular solution.
No network design should be considered immune to failure because there are many uncontrollable
circumstances to consider, including human error and natural events. The following areas of failure
domain sizing should be considered:

–Maximum number of servers per Layer 2 broadcast domain

–Maximum number of servers per access switch (if single-homed)

38
 –Maximum number of servers per aggregation module

–Maximum number of access switches per aggregation module

•Port density—The aggregation layer has a finite number of 10GigE ports that can be supported,
which limits the quantity of access switches that can be supported. When a Catalyst 6500 modular
access layer is used, thousands of servers can be supported on a single aggregation module pair. In
contrast, if a 1RU Catalyst 4948 is used at the access layer, the number of servers supported is less.
Cisco recommends leaving space in the aggregation layer for growth or changes in design.

The data center, unlike other network areas, should be designed to have flexibility in terms of
emerging services such as firewalls, SSL offload, server load balancing, AON, and future
possibilities. These services will most likely require slots in the aggregation layer, which would
limit the amount of 10GigE port density available.

5.8.4 Determining Maximum Number of VLANs

What is the maximum number of VLANs that can be supported in an aggregation module?
•Spanning tree processing—When a Layer 2 looped access topology is used, which is the most
common, the amount of spanning tree processing at the aggregation layer needs to be considered.
There are specific watermarks related to the maximum number of system-wide active logical
instances and virtual port instances per line card that, if reached, can adversely affect convergence
and system stability. These values are mostly influenced by the total number of access layer
uplinks and the total number of VLANs. If a data center-wide VLAN approach is used (no manual
pruning on links), the watermark maximum values can be reached fairly quickly.

•Default Gateway Redundancy Protocol— The quantity of HSRP instances configured at the
aggregation layer is usually equal to the number of VLANs. As Layer 2 adjacency requirements
continue to gain importance in data center design, proper consideration for the maximum HSRP
instances combined with other CPU-driven features (such as GRE, SNMP, and others) have to be
considered. Lab testing has shown that up to 500 HSRP instances can be supported in an
aggregation module, but close attention to other CPU driven features must be considered.

5.8.5 Importance of Team Planning

Considering the roles of different personnel in an IT organization shows that there is a growing need
for team planning with data center design efforts. The following topics demonstrate some of the
challenges that the various groups in an IT organization have related to supporting a “business ready”
data center environment:

39
 •System administrators usually do not consider physical server placement or cabling to be an
issue in providing application solutions. When the need arises for one server to be connected into
the same VLAN as other servers, it is usually expected to simply happen without thought or
concern about possible implications. The system administrators are faced with the challenge of
being business-ready and must be able to deploy new or to scale existing applications in a timely
fashion.

•Network/Security administrators have traditionally complied with these requests by extending

the VLAN across the Layer 2 looped topology and supporting the server deployment request. This
is the flexibility of having a Layer 2 looped access layer topology, but is becoming more of a
challenge now than it was in the past. The Layer 2 domain diameters are getting larger, and now
the network administrator is concerned with maintaining spanning tree virtual/logical port counts,
manageability, and the failure exposure that exists with a large Layer 2 broadcast domain. Network
designers are faced with imposing restrictions on server geography in an effort to maintain
spanning tree processing, as well as changing design methods to include consideration for Layer 2
domain sizing and maximum failure domain sizing.

•Facilities administrators are very busy trying to keep all this new dense hardware from literally
burning up. They also see the additional cabling as very difficult if not impossible to install and
support with current design methods. The blocked air passages from the cable bulk can create
serious cooling issues, and they are trying to find ways to route cool air into hot areas. This is
driving the facilities administrators to look for solutions to keep cables minimized, such as when
using 1RU switches. They are also looking at ways to locate equipment so that it can be cooled
properly.

These are all distinct but related issues that are growing in the enterprise data center and are creating
the need for a more integrated team planning approach. If communication takes place at the start,
many of the issues are addressed, expectations are set, and the requirements are understood across all
groups.

6 Analysis
Data center architectures are evolving to meet the demands and complexities imposed by increasing
business requirements to stay competitive and agile. Industry trends such as data center
consolidation, server virtualization, advancements in processor technologies, increasing storage
demands, rise in data rates, and the desire to implement "green" initiatives is causing stress on
current data center designs. However, as this document discusses, crucial innovations are emerging

40
to address most of these concerns with an attractive return on investment (ROI). Future data centers
architectures will incorporate increasing adoption of 10 Gigabit Ethernet, new technologies such as
Fibre Channel over Ethernet (FCoE), and increased interaction among virtualized environments.

Data Centers are specialized environments that safeguard your company's most valuable equipment
and intellectual property. A well-planned and effectively managed Data Center supports these
operations and increases your company's productivity by providing reliable network availability and
faster processing. In many ways your Data Center is the brain of your company. Your business' ability
to perceive the world (data connectivity), communicate (e-mail), remember information (data storage),
and have new ideas (research and development) all rely upon it functioning properly.
As businesses seek to transform their IT departments from support organizations into sources of
productivity and revenue, it is more important than ever to design and manage these specialized
environments correctly. A well-built Data Center does not just accommodate future growth and
innovation; it acts as a catalyst for them. Companies that know their Data Center is robust, flexible,
and productive can roll out new products, move forward with their business objectives, and react to
changing business needs, all without concern over whether their server environment is capable of
supporting new technologies, high-end servers, or greater connectivity requirements.

7. Findings, Inferences and Recommendations

Running out of data ports is perhaps the most common infrastructure shortcoming that occurs in server
environments, especially in those whose rows contain infrastructure tailored to support a specific
model of server. When it comes time to host different equipment, those cabinet locations must be
retrofitted with different infrastructure. Fortunately, lack of connectivity is one of the easier issues to
address. It can be remediated in one of two ways.

One option is added structured cabling. As long as the installer is careful to work around existing
servers and their connections, the upgrade can usually be completed without any downtime. This
cabling needs to terminate somewhere in the Data Center, however, either at a network substation or a
main networking row. These added ports might require more space than existing networking cabinets
can provide, so be aware that more floor space might need to be allocated for them, which in turn
reduces what is available to host servers.

41
A second option, particularly when the need is for copper connections, is to install either of two
networking devices—a console server or console switch—at the cabinet where more ports are needed.
These networking devices can send multiple streams of information over one signal, a process known
as multiplexing. For example, if you have several servers installed in a cabinet, instead of running a
dozen patch cords from those devices to the Data Center's structured cabling under the floor, you run
those patch cords to a console server or console switch and—thanks to multiplexing—you then run
just one patch cord from that device to the structured cabling. Installing these networking devices can
significantly expand the capacity of your Data Center's existing structured cabling.

This second approach is best used when:

• There is limited internal space within the Data Center's network cabinets. Installing these
networking devices in key server cabinets can reduce the space needed for additional patching
fields.
• Infrastructure within the Data Center plenum is chaotic, and installing more structured cabling
may either restrict airflow or pose a downtime risk. Increasing ports by way of these
networking devices involves only a fraction of the structured cabling than would otherwise be
required.
• The need for additional ports is temporary. Networking devices can be removed and reused
more easily than structured cabling.

As Data Center cabinets fill up with servers, you might discover that its cooling infrastructure isn't up
to the task of keeping the space cool. The overall ambient temperature of the server environment might
become too warm, or else hot spots might develop in areas where servers are tightly packed or a large
device emits a high amount of exhaust.

Presumably, you have already used the Data Center's floor tiles to good advantage, placing perforated
floor tiles so that cooling is directed at known hot spots and sealing unwanted openings to maintain air
pressure. There are many techniques for improving cooling in a server environment. Here are five:

• Relocate air handler temperature sensors— These devices are generally located at the
cooling unit itself, where temperatures are often lower. Placing the sensors deeper within the
room gives the air handlers more accurate readings about the Data Center's ambient
temperature and can cause them to provide cooling for longer periods.

42
 • Install ducted returns— These draw away more of the Data Center's heated air, channeling it
into each handler's normal cooling cycle, and can reduce temperatures in sections of the room
by a few degrees.
• Distribute servers— If tightly packed servers are causing hot spots, spreading such equipment
out is a sure way to prevent them. This solution obviously has limited value if your Data
Center also has space constraints. However, even if you don't have the option to only partially
fill server cabinets in your Data Center, at least try to strategically locate devices so that the
highest heat-producers aren't clustered together. It is easier to deal with several warmer areas in
a server environment than one that is very hot.
• Install self- cooling cabinets—Reinstall the Data Center's most prodigious heat-generating
servers into cabinets that are cooled by fans or chilled liquid. This eliminates hot spots at their
source and might lower the room's overall ambient temperature.
• Install additional air handlers— Finally, if all else fails you might need to put in another air
handler to increase how much cold air is being pumped in to the Data Center. Use the same
approach that you would when designing the room's cooling infrastructure from scratch—try to
place the handler perpendicular to server rows and create a buffer area around it so that short
cycling does not occur.

When performing any work on your Data Center's cooling system that might require air handlers to be
shut down, have multiple portable fans and spot coolers at the ready. Temperatures can rise quickly
when air handlers are turned off. You might need to prop open the Data Center doors and use fans to
blow hot air out of the room. For major cooling system work, try to schedule work to occur during
colder weather, such as at night or during winter months or both.

7.1 Paradigm Shifts

It is also possible that your Data Center has ample physical room and infrastructure available and yet
still begins having problems hosting incoming equipment. This occurs when servers, networking
devices, or other machines arrive that the server environment wasn't designed to accommodate.

Maybe server manufacturers alter their designs, making machines that need more physical space or
electrical power. Perhaps your company decides to pursue a different business goal, requiring
equipment that your Data Center never had to host in years past. It could also be that technology
changes, requiring new cabling media to accommodate it. Whatever the cause, this can be the hardest
of shortcomings to deal with in a server environment, because you might not be able to overcome it by
simply adding a few circuit panels or running more structured cabling. The physical layout of Data
43
Center rows might need to be changed, including the physical relocation of both servers and
infrastructure components—all while the server environment remains online.

If you are fortunate—more accurately, if you anticipated the need for future change—you designed
your Data Center infrastructure to be easily upgradeable. Flexible electrical conduits and lightly
bundled structured cabling, with additional slack provided, can enable you to reconfigure your under-
floor infrastructure quickly and concentrate power and data connectivity where it is needed.
Infrastructure components that enable you to use different media, such as multimedia boxes that can
accommodate multiple connector types and electrical whips pre-wired to terminate in several types of
receptacles, also make it easier to change elements of a server environment.

If your Data Center possesses these types of infrastructure components, retrofitting the room might be
as simple as having structured cabling and electrical conduits reterminated. More likely, however, you
are going to have to make more dramatic and intrusive changes to the server environment. This might
include rearranging server rows, removing and rerunning structured cabling, and either adding or
relocating power distribution units or air handlers.

Here are several tips to follow when making significant infrastructure changes to your existing Data
Center:

• Upgrade the room in phases, say a couple of server rows at a time, rather than trying to
overhaul it all at once. Retrofitting a live Data Center is like performing surgery on a conscious
patient. Breaking the task down into segments reduces the effect of downtime and makes it less
likely for something to go wrong on a large scale. If work can be completed over a long period
of time, you might even be able to coordinate infrastructure changes with server lifecycles, that
is as servers are decommissioned and replaced with newer models.
• When retrofitting the Data Center dictates physically moving equipment, take advantage of
devices that have dual power supplies. Strategically shifting power plugs from an old power
receptacle to a new one can enable you migrate a server to a new part of the Data Center, or
onto a different power source that won't be shut down, and enable you to avoid downtime.
• If work in the Data Center might produce debris or airborne particles, shut down the fire
suppression system to avoid setting it off accidentally.
• If new servers are installed in the Data Center while it is undergoing a retrofit, place them
according to how the room is going to be designed ultimately. Arrange them to be part of the
new layout, not another piece of equipment that must be relocated later.

44
An example of a recent paradigm shift for Data Centers is the emergence of 1U servers. At the start of
this century, server manufacturers began producing low profile servers that were high performing and
relatively inexpensive compared to earlier generations of devices.

IT departments in many companies opted to replace older, larger systems with dozens of 1U servers.
They are particularly popular for computing applications that can pool the power of multiple servers.
For these applications, a cluster of 1U devices provides both flexibility and redundancy: flexibility,
because exactly how many servers are dedicated to a task can be altered as needed and redundancy
because even if a server fails there are several others that continue processing. Low-profile servers are
also desirable for companies that lease Data Center space. Most hosting facilities charge based upon
how much floor space a client occupies, so using smaller servers can reduce those costs.

Despite their merits, 1U servers are difficult to host in many Data Centers. When clustered together,
they are heavy, draw large amounts of power, produce a lot of heat, and require a high amount of data
connections in a very small space—not what most pre-existing server environments were designed and
built to accommodate.

It isn't easy monitoring a distributed environment. Unchecked server growth fills data center space
rapidly. Heterogeneous platforms and operating systems make provisioning and inventorying a long
and laborious process. Power supply limitations can prevent cooling upgrades or curtail the addition of
servers. It is difficult to know what IT resources are available, making it extremely difficult to plan
effectively for future needs.

Capacity planners, therefore, have two key questions to answer: How do we use the existing
environment? And what is our total installed capacity? Most of the time there isn't an easy
response. The picture is muddled by the fact that the capacity planner is often being forced to compare
apples to oranges.

The rising energy costs of running a data center are gaining more and more attention as they are
already in the range of $3.3 billion annually, according to IDC. As a result, the Environmental
Protection Agency (EPA) and Department of Energy are now creating standard ratings for
energy efficiency benchmarks, forcing companies to be more conscious of their energy use and
environmental impact. Many companies, however, are wary of such new regulations and standards,
as they think it’ll mean incurring new costs to meet them.

45
This growing number of servers and data centers, naturally causes an increase in power and energy
consumption, quickly escalating the amount of resources needed and raising environmental concerns
even more. Thirty percent responded that they could find an additional 20 percent capacity. By
having the proper tools in place, companies can keep their data center lean, mean, and green.

Sometimes the driving force behind the retrofit of a Data Center is that the room has become so
disorganized and cluttered that it is problematic to install new equipment. Tangled patch cords,
unlabeled structured cabling and electrical whips, and poor cooling distribution can all make a server
environment vulnerable to downtime every time a new device is installed. Before embarking upon a
major construction project to add infrastructure or expand an overworked server environment, see if
any of the following can remediate the problems and make the space more usable:

Use the right length patch cords for the job— System administrators sometime plug in servers and
networking devices using whatever patch cords happen to be at hand rather than locating the correct
lengths of cable that are needed to make connections.
This results in 15 feet of cable being used to go 4 feet (4 meters of cable used to go 1 meter).
Excess cable length is left to dangle from the device or patch panel it is plugged into, perhaps
coiled with a tie wrap or perhaps not. Over time, this creates a spider web of cable that blocks
access to servers, pre-installed cabling ports, and electrical receptacles. This tangled web not
only reduces the usability of Data Center infrastructure, but it all presents a snagging hazard that
can cause accidental downtime. Replace overly long patch cords and power cables with those
that are the correct length. Also, remove cords and cables that aren't plugged in to functioning
servers and are simply leftovers from decommissioned equipment. Do this within cabinets and
under the raised floor.

Many servers come standard with power cables that are 6 feet (1.8 meters) long. This is useful for
reaching a power receptacle under a raised floor or in a ceiling-mounted raceway, but it is much longer
than necessary if you are installing the device into a server cabinet with its own power strips. You can
tie wrap the excess length, but even that still dangles somewhere within the server cabinet.

To reduce this problem, I stock power cables that are the same type provided with the servers, but just
2 feet (61 centimeters) long. The shorter cables are much less likely to become tangled or get snagged
when someone is installing or removing equipment into a cabinet.

46
• Add wire management— Even when Data Center users run the right length of patch cord,
you end up with hanging cables if adequate wire management is not provided. Install new or
larger cable management at cabinet locations where cable glut interferes with access to
infrastructure or equipment.

Together with replacing correct cable lengths, installing wire management can free up access
to existing data ports and electrical receptacles. These steps also make troubleshooting easier,
improve airflow around servers, and improve the overall appearance of the Data Center.

• Make sure that people are correctly using infrastructure— You can spend hundreds of
thousands of dollars on structured cabling, but it is worthless if Data Center users string patch
cords between server cabinets rather than use the infrastructure that is installed. Failing to
properly use the infrastructure in a server environment leads to disorganization, makes
troubleshooting difficult, and can create situations that are hazardous to both equipment and
Data Center users.

For example, imagine that someone installs a server into a cabinet within the Data Center, and
they don't understand that each cabinet location is provided with dedicated circuits from two
different power sources. Wanting redundant power for their server's dual power supplies, they
plug one power cable in to the power strip of their own server cabinet and then string the other
power cable to the strip in an adjacent cabinet. This adds unnecessary electrical draw onto the
adjacent power strip, making it more susceptible to tripping a circuit. It also ties the two
adjacent server cabinets together. If a time ever comes to relocate either cabinet, and whoever
is moving it is unaware that a power cord has been strung between them, an accident might
occur that could harm someone or damage a server.

• Install power strips with known electrical ratings— Do you know how much equipment
you can install into your server cabinets before you overload their power strips? If not, you are
either underutilizing server cabinet space or risking downtime every time you plug in new
equipment. Swap out any mystery power strips with ones whose amp ratings are known. If
possible, have the power strip rated for the same amperage as the circuit it is plugged into. This
enables you to maximize existing electrical circuits and server cabinet space.

• Redeploy floor tiles— If cooling is a problem and your Data Center has a raised floor, check
that floor tiles are deployed to best advantage. Close off unnecessary openings in the floor
surface and open perforated tiles closest to highly populated cabinets. Close or relocate floor
47
 panels that are especially close to air handlers and might be causing short cycling. Make sure
under-floor infrastructure isn't blocking air coming from the room's air handlers. If it is, try to
shift the location of structured cabling and electrical conduits to enable better circulation.

7.2 Below are the top ten Data Center Design Guidelines:
The following are the top ten guidelines selected from a great many other guidelines, many of which
are described throughout this book.

1. Plan ahead. You never want to hear “Oops!” in your data center.
2. Keep it simple. Simple designs are easier to support, administer, and use. Set
things up so that when a problem occurs, you can fix it quickly.
3. Be flexible. Technology changes. Upgrades happen.
4. Think modular. Look for modularity as you design. This will help keep things
simple and flexible.
5. Use RLUs, not square feet. Move away from the concept of using square footage
of area to determine capacity. Use RLUs to define capacity and make the data
center scalable.
6. Worry about weight. Servers and storage equipment for data centers are getting
denser and heavier every day. Make sure the load rating for all supporting
structures, particularly for raised floors and ramps, is adequate for current and
future loads.
7. Use aluminum tiles in the raised floor system. Cast aluminum tiles are strong
and will handle increasing weight load requirements better than tiles made of
other materials. Even the perforated and grated aluminum tiles maintain their
strength and allow the passage of cold air to the machines.
8. Label everything. Particularly cabling! It is easy to let this one slip when it seems
as if “there are better things to do.” The time lost in labeling is time gained when
you don’t have to pull up the raised floor system to trace the end of a single cable.
And you will have to trace bad cables!
9. Keep things covered, or bundled, and out of sight. If it can’t be seen, it can’t be
messed with.
10. Hope for the best, plan for the worst. That way, you’re never surprised.

48
Most existing data centers have one or more of the problems listed above, with the attendant
losses of performance. In typical data centers routinely observed, the loss of available rack locations
due to these problems is on the order of 10-20% of total rack locations, and the loss of power density
capability is commonly 20% or more. These unnecessary losses in performance represent substantial
financial losses to data center operators, but can be avoided by simple planning.

When row lengths are three racks or less, the effectiveness of the cooling distribution is impacted.
Short rows of racks mean more opportunity for mixing of hot and cold air streams. For this reason,
when rooms have one dimension that is less than 15-20 feet it will be more effective in terms of
cooling to have one long row rather than several very short rows.
 Use of best-practices air management, such as strict hot aisle/cold aisle configuration,
can double the computer server cooling capacity of a data center.
 Combined with an airside economizer, air management can reduce data center cooling
costs by over 60%1.
 Removing hot air immediately as it exits the equipment allows for higher capacity and
much higher efficiency than mixing the hot exhaust air with the cooling air being drawn
into the equipment. Equipment environmental temperature specifications refer
primarily to the air being drawn in to cool the system.
 A higher difference between the return air and supply air temperatures increases the
maximum load density possible in the space and can help reduce the size of the cooling
equipment required, particularly when lower-cost mass produced package air handling
units are used.
 Poor airflow management will reduce both the efficiency and capacity of computer
room cooling equipment. Examples of common problems that can decrease a Computer
Room Air Conditioner (CRAC) unit’s usable capacity by 50%2 or more are: leaking
floor tiles/cable openings, poorly placed overhead supplies, underfloor plenum
obstructions, and inappropriately oriented rack exhausts.
 Specify and utilize high efficiency power supplies in Information Technology (IT)
computing equipment. High efficiency supplies are commercially available and will
pay for themselves in very short timeframes when the total cost of ownership is
evaluated.
 For a modern, heavily loaded installation with 100 racks, use of high efficiency power

49
 supplies alone could save $270,000-$570,0002 per year and decrease the square-
footage required for the IT equipment by allowing more servers to be packed into a
single rack footprint before encountering heat dissipation limits.
 Cooling load and redundant power requirements related to IT equipment can be
reduced by over 10 – 20%, allowing more computing equipment density without
additional support equipment (UPSs, cooling, generators, etc.).
 In new construction, downsizing of the mechanical cooling equipment and/or electrical
supply can significantly reduce first cost and lower the mechanical and electrical
footprint.
 When ordering servers, power supplies that meet at least the minimum efficiency
recommendations by the SSI Initiative (SSI members include Dell, Intel, and IBM).
 When appropriate, limit power supply oversizing to ensure higher – and more
efficient – load factors.
 Select the most efficient UPS system that meets the data center’s needs. Among double
conversion systems (the most commonly used data center system), UPS efficiency
ranges from 86% to 95%. Simply selecting a 5% higher efficiency model of UPS can
save over $38,000 per year in a 15,000 square foot data center, with no discernable
impact on the data center’s operation beyond the energy savings. In addition,
mechanical cooling energy use and equipment cost can be reduced.
 For battery-based UPS systems, use a design approach that keeps the UPS load factor
as high as possible. This usually requires using multiple smaller units. Redundancy in
particular requires design attention; operating a single large UPS in parallel with a
100% capacity identical redundant UPS unit (n+1 design redundancy) results in very
low load factor operation, at best no more than 50% at full design build out.
 Evaluate the need for power conditioning. Line reactive systems often provide enough
power conditioning for servers, and some traditional double conversion UPS systems
(which offer the highest degree of power conditioning) have the ability to operate in the
more efficient line conditioning mode, usually advertised as ‘economy’ or ‘eco’ mode.

7.3 Below are recommendations to build Tier 3 data center:

• At least two access providers should serve this data center with the provider’s cabling from
their central offices or POPs (Points of Presence) separated by at least 66 feet along their route.

• Mantraps at all entrances to the computer room should control letting more than one person in
by the use of only one credential.
50
 • A signal reference grid (SRG) and lightning protection system should be provided.

• If the HVAC system’s air conditioning units are served by a waterside heat rejection system,
the components of these systems are to be sized to maintain design conditions, with one
electrical switchboard removed from service. The piping system or systems are dual path.

• Two independent sets of pipes are to be used for data centers using chilled water.

• Sufficient capacity and distribution should be available to simultaneously carry the load on
one path while performing maintenance or testing on the other path. If there are errors during
operation or any other unplanned activities, disruption will be caused.

Data center design should follow industry standards for best practices. Industry guidance is on the way
in the form of an emerging industry standard for data centers. This document, to be published as
Telecommunications Industry Association ANSI/TIA/EIA-942, Telecommunications Infrastructure
Standard for Data Centers, lists requirements and provides recommendations for data center design
and construction. TIA-942 helps consultants and end-users design an infrastructure that will last years
without forklift renovations. The standard also gives information on cooling, power, room sizing and
other information useful in data center design.

Below are proposals by World Network leader CISCO has given to follow:

● Realize the full potential of your data center investment by improving your network
performance, availability, security, and QoS.
● Adopt and combine technologies to create a data center network that continuously evolves to
sustain a competitive business.
● Proactively address potential data center issues before they affect operations.
● Achieve and maintain a comprehensive, end-to-end data center optimization solution.
● Make sure that the data center plays a strategic role in your efforts to protect, optimize, and
grow your business.
● Achieve operational excellence by providing informal training that prepares your staff to
knowledgeably manage data center technologies.

51
8 Conclusion
In many ways your Data Center is the brain of your company. Your business' ability to perceive the
world (data connectivity), communicate (e-mail), remember information (data storage), and have new
ideas (research and development) all rely upon it functioning properly.

A well-built Data Center does not just accommodate future growth and innovation; it acts as a catalyst
for them. Companies that know their Data Center is robust, flexible, and productive can roll out new
products, move forward with their business objectives, and react to changing business needs, all
without concern over whether their server environment is capable of supporting new technologies,
high-end servers, or greater connectivity requirements.

It is safe to assume that routers, switches, servers, and data storage devices will advance and change in
the coming years. They will feature more of something than they do now, and it will be your Data
Center's job to support it. Maybe they will get bigger and heavier, requiring more power and floor
space. Maybe they will get smaller, requiring more data connections and cooling as they are packed
tighter into the Data Center. They might even incorporate different technology than today's machines,
requiring alternate infrastructure. The better your server environment responds to change, the more
valuable and cost-effective it is for your business. New equipment can be deployed quicker and easier,
with minimal cost or disruption to the business.

Data Centers are not static, so their infrastructure should not be either. Design for flexibility. Build
infrastructure systems using components that are easily changed or moved. This means installation of
patch panels that can house an array of connector types and pre-wiring electrical conduits so they can
accommodate various electrical plugs by simply swapping their receptacle. It also means avoiding
items that inhibit infrastructure mobility. Deploy fixed cable trays sparingly, and stay away from
proprietary solutions that handcuff you to a single brand or product.

Make the Data Center a consistent environment. This provides stability for the servers and networking
equipment it houses, and increases its usability. The room's modularity provides a good foundation for
this, because once a user understands how infrastructure is configured at one cabinet location, he or
she will understand it for the entire room. Build on this by implementing uniform labeling practices,
consistent supplies, and standard procedures for the room. If your company has multiple server
environments, design them with a similar look and feel. Even if one Data Center requires

52
infrastructure absolutely different from another, use identical signage, color-coding, and supplies to
make them consistent. Standardization makes troubleshooting easier and ensures quality control.

Above all, your Data Center has to be reliable. Its overarching reason for existence is safeguarding
your company's most critical equipment and applications. Regardless of what catastrophes happen
outside—inclement weather, utility failures, natural disasters, or something else unforeseen—you want
your Data Center up and running so your business continues to operate.

To ensure this, your Data Center infrastructure must have depth: standby power supplies to take over
when commercial electricity fails, and redundant network stations to handle the communication needs
if a networking device malfunctions, for example. Primary systems are not the only ones susceptible to
failure, so your Data Center's backup devices might need backups of their own.

Additionally, the infrastructure must be configured so there is no Achilles Heel, no single component
or feature that makes it vulnerable. It does little good to have multiple standby power systems if they
are all wired through a single circuit, or to have redundant data connections if their cable runs all enter
the building at one location. In both examples, a malfunction at a single point can bring the entire Data
Center offline.

As IT organizations look for ways to increase the cost-effectiveness and agility of their data centers,
they need to understand the current state of their architecture and determine which changes can best
help them achieve their business and IT goals.

• Ensure more efficient use of data center facilities.

• Gain operational efficiencies and cost savings through standardization and asset consolidation.
• Increase asset utilization to increase flexibility and reduce costs.
• Reduce energy costs.
• Extend the working life of capital assets.
• Optimize use of space, power, and cooling infrastructure.
• Avoid or defer construction of new facilities.
• Reduce business impact of localized and large footprint disaster events.
• Improve productivity through enhanced application and data availability.
• Meet corporate and regulatory compliance needs.
• Improve data security and compliance.
• Extend desktop hardware lifecycles.
• Extend business continuity and disaster recovery to enterprise desktops.
53
Data Center solutions enables IT organizations to meet business continuance and corporate
compliance objectives, and provide benefits that include:
• Reducing business impact of localized and large footprint disaster events.
• Improving productivity through enhanced application and data availability.
• Meeting corporate and regulatory compliance needs and improving data security.

9. An overview of the organization

PATNI COMPUTER SYSTEMS LTD. is a provider of Information Technology services and
business solutions. The company employs over 13,600 people, and has 23 international offices across
the globe, as well as offshore development centers in 8 cities in India. PATNI's clients include more
than 200 Fortune 1000 companies. PATNI has registered revenues of US$ 719 million for the year
2008.

PATNI COMPUTER SYSTEMS LTD. was incorporated as Patni Computer Systems Private
Limited on February 10, 1978 under the Companies Act, 1956. In 1988, by virtue of Section 43A of
the Companies Act, the Company became a public limited company on September 18, 2003.

In 2004, PATNI came out with an initial public offering (IPO) of 18,724,000 equity shares in the price
of Rs 230 per share for a face value of Rs 2 each. In the same year, PATNI acquired Fremont,
California based Cymbal Corporation for a sum of US$78 mn. Cymbal's acquisition allowed PATNI
to enter $60 billion IT services market in the telecom vertical which was previously not available to
PATNI on their business landscape. This acquisition also allowed PATNI to spread its Non-GE
Business, and added a development center in Hyderabad, India.

In December 2005, PATNI COMPUTER SYSTEMS LTD. listed its ADRs on the New York Stock
Exchange (NYSE) under the ticker PTI.

10. PATNI COMPUTER SYSTEMS LTD.

PATNI COMPUTER SYSTEMS LTD. (BSE: PATNI COMPUT, NSE: PATNI, NYSE: PTI) is one of
the leading global providers of Information Technology services and business solutions. Over 13,600
professionals service clients across diverse industries, from 27 sales offices across the Americas,
Europe and Asia-Pacific, and 22 Global Delivery Centers in strategic locations across the world. We
have serviced more than 400 FORTUNE 1000 companies, for over two decades.

54
Our vision is to achieve global IT services leadership in providing value-added high quality IT
solutions to our clients in selected horizontal and vertical segments, by combining technology skills,
domain expertise, process focus and a commitment to long-term client relationships.

PATNI delivers high quality, reliable and cost-effective IT services to customers globally. We provide
world-class technology services by constantly exploring and implementing innovative solutions that
drive long-term value to our customers.

As industry leaders, we introduced offshore development centers, pioneered "follow the sun"
development and support frameworks, ensuring compressed delivery timeframes.

Today, our solutions provide strategic advantage to several most-admired organizations in the world.
We have long-standing and vibrant partnerships with over 300 companies across the globe.

The most successful companies in the world have two common characteristics: they focus on core
competencies, and flawlessly execute on a consistent basis, in every area of business. For all other
companies, these elusive qualities are a highly prized goal. To achieve these goals in an environment
with increasing costs and competition, many global organizations are turning to Customer Interaction
Services (CIS) & Business Process Outsourcing (BPO) for streamlining their operations and
increasing profitability.

PATNI’s suite of CIS & BPO services is a natural extension of our IT service offerings. Our CIS &
BPO services are built on a foundation of process and domain expertise, and are enabled by innovative
technologies. As in our other practices, PATNI’s CIS & BPO services are managed to meet high
quality standards. Clients rely on them to improve the bottom line and focus more effectively on their
core business, while maintaining a high quality of service.

PATNI provides customized global sourcing solutions to a diverse group of clients who rely on us for
vertical-specific processes, as well as shared corporate services. In addition to a broad range of
horizontal services including IT Helpdesk, Finance & Accounting, HR Services, Enterprise-wide
Service Desk and Product Support, we also provide a comprehensive suite of CIS & BPO services
for the Insurance, Financial Services, Telecom, Life Sciences vertical markets. For 401(K) Plan
administrators within the Insurance space, we also offer offshoring services of the entire Benefits
Administration Lifecycle.

PATNI offers clarity and expedience in delivering CIS & BPO solutions. We give clients a clear
roadmap that is designed to enhance productivity and reduce costs through process assessment,
55
process standardization and process re-engineering. To ensure the highest levels of information quality
and integrity, we have adhered to the BS7799 standard to establish a robust Information Security
Management System (ISMS) that integrates process, people and technology to assure the
confidentiality, integrity and availability of information. A global infrastructure helps us implement
CIS & BPO services that utilize the right combination of resources from offshore, onshore or onsite.
PATNI offers these CIS & BPO services through its state of the art BPO centers in India and across
the globe.

In the event of a business disaster, PATNI has you covered. We have in place a comprehensive model
for disaster recovery and business continuity in the event of any disruption.

10.1 Key Industry verticals

 Financial Services
 Manufacturing
 Telecom
 Product Engineering
 Life Sciences
 Independent Software Vendors(ISV)
 Retail
 Media & Entertainment
 Energy & Utilities
 Logistics & Transportation

10.2 Key Services

In addition to these, PATNI serves its customers through following Horizontal Business Units across
various industries:

 Application Development
 Application Management
 IT Consulting
 Embedded Software
 Infrastructure Management
 Enterprise Application Services
 Customer Intelligence Services & Business Process Outsourcing (CIS & BPO)
56
  BI & DW
 Enterprise integration
 Verification & Validation
 Process Consulting
 Customer Interaction Services & Business Process Outsourcing
 Engineering Services
 IT Governance
 Business Process Management
 Customized Learning Solutions
 User Experience Management

10.3 Global Locations

1. Americas

 Brazil
 Canada
 Mexico
 U.S.A.

2. Europe, Middle East & Africa (EMEA)

 Czech Republic
 Finland
 Germany
 South Africa
 Sweden
 The Netherlands
 UAE
 United Kingdom

3. Asia Pacific (APAC)

 Australia
 India
 Japan

57
  Singapore

10.4 Our Vision

“To be a trusted partner, powered by passionate minds, creating innovative options to excel.”

10.4.1 Our Vision & Values

 Passion
Zeal to exceed expectations by challenging the status quo.
 Honesty & Integrity
Driving all actions based on openness, trust and ethical conduct.
 Operational Excellence
Striving for excellence with an operating discipline, benchmarked with the best globally.
 Innovation
Creating innovative solutions for success.
 Accountability
Taking ownership and delivering on commitments.
 Customer Centricity
Ensuring customer delight. Always.

11. Patni Data Center Characteristics

Data center is designed for computers, not people. As a result, PATNI COMPUTER SYSTEMS LTD.
data center typically have no windows and minimal circulation of fresh air. PATNI Data center range
in size of 3500 sq. feet dedicated to housing servers, storage devices, and network equipment.

Data center room is filled with rows of IT equipment racks that contain servers, storage devices, and
network equipment. Data center include power delivery systems that provide backup power, regulate
voltage, and make necessary alternating current/direct current (AC/DC) conversions. Before reaching
the IT equipment rack, electricity is first supplied to an uninterruptible power supply (UPS) unit. The
UPS acts as a battery backup to prevent the IT equipment from experiencing power disruptions, which
could cause serious business disruption or data loss. In the UPS the electricity is converted from AC to
DC to charge the batteries. Power from the batteries is then reconverted from DC to AC before leaving
the UPS. Power leaving the UPS enters a power distribution unit (PDU), which sends power directly

58
to the IT equipment in the racks. Electricity consumed in this power delivery chain accounts for a
substantial portion of overall building load.
Electricity entering servers is converted from AC to low-voltage DC power in the server power
supply unit (PSU). The low-voltage DC power is used by the server’s internal components of, such
as the central processing unit (CPU), memory, disk drives, chipset, and fans. The DC voltage serving
the CPU is adjusted by load specific voltage regulators (VRs) before reaching the CPU. Electricity is
also routed to storage devices and network equipment, which facilitate the storage and transmission
of data.

The continuous operation of IT equipment and power delivery systems generates a significant amount
of heat that must be removed from the data center for the equipment to operate properly. Cooling in
data center is often provided by computer room air conditioning (CRAC) units, where the entire air
handling unit (AHU) is situated on the data center floor. The AHU contains fans, filters, and cooling
coils and is responsible for conditioning and distributing air throughout the data center. In most cases,
air enters the top of the CRAC unit and is conditioned as air passes across coils containing chilled
water pumped from a chiller located outside of the data center room. The conditioned air is then
supplied to the IT equipment (primarily servers) through a raised floor plenum. Cold air passes
through perforated floor tiles, and fans within the servers then pull air through the servers. The
warmed air stratifies toward the ceiling and eventually makes its way back to the CRAC unit intake.

Most air circulation in data center is internal to the data center zone. The majority of data centers are
designed so that only a small amount of outside air enters. Some data centers provide no ductwork for
outside air to directly enter the data center area. Instead, outside air is only provided by infiltration
from adjacent zones, such as office space. Other data centers admit a relatively small percentage of
outside air to keep the zone positively pressurized.

Data center use a significant amount of energy to supply three key components: IT equipment,
cooling, and power delivery. These energy needs can be better understood by examining the electric
power needed for typical data center equipment in and the energy required to remove heat from the
data center.

Table 11-1. Component Peak Power, Consumption for a Typical Server

59
 Component Peak Power (Watts)
CPU 80
Memory 36
Disks 12
Peripheral slots 50
Motherboard 25
Fan 10
PSU losses 38
Total 251

11.1 Data Centre Design & Layout

The main items for consideration in data centre design are broken down into four main groups:
 Structural
 Supporting Environment
 Cabling Infrastructure
 Security and Monitoring

We can’t ignore other sub items those play vital role in datacenter design:
 Air conditioning
 UPS
 Generators
 Gas Fire Suppression
 Air sampling systems
 Raised Floors
 Infrastructure Cabling
 Environmental Monitoring
 Security products
 Racks etc.

11.2 Data Centre Design Structural

The structure and the location of the building is an important consideration when it comes to the data
centre design. Recent new standards cover both the structural element of the room or building and its
geographic location and covers many different aspects of the premises.

60
11.2.1 These include:
 Location; both geographical and physically within the building i.e. within a flood plain
(building), below water level (building or room within a building)
 Construction methods; materials used in the construction of the building i.e. timber framed or
metal and concrete
 Access for deliveries
 Proximity to hazards; airports, oil refineries, train lines, motorways, power stations, chemical
works, embassies, military locations etc
You have to install all the elements necessary to ensure a working and standards compliant
environment, including:
 Fire rating of walls
 Floor loading
 Floor void
 Floor grid configuration
 Ceiling Height
 Door sizes; from the loading bay to the data centre
 Steps or ramps
 Lifts
All of these items, and more, must be taken into consideration at the design stage to ensure the room
meets all necessary requirements.

11.3 Data Centre Design Supporting Environment

Any data centre, computer room or server room is only as good as the supporting services which
maintain the environment within the specifications as laid down by the equipment manufactures. This
environment must be maintained 24/7 or the equipment within the room could fail with potentially
catastrophic consequences for the business which it supports. It is therefore essential that the data
centre is designed to ensure that redundancy and resiliency of equipment is catered for in the
initial plan and is not added as an after thought, therefore compromising space and other services.

You have to consider following elements for your data centre to ensure a working and standards
compliant environment:
 Electrical supply
 Lighting
 UPS
61
  Generator
 Air Conditioning
 Humidity Control
 Fire Suppression
There can be many variations to each of these services and the key to a successful implementation is
that all services are co-ordinated to co-exist in what can sometimes be very confined spaces.

The appointment of a properly qualified project manager can greatly improve the overall build time
and smooth implementation of any data centre project. Data Centre Standards Ltd has been involved in
the building and refurbishment of many Data Centers in the UK and Europe and can provide essential
first hand experience and guidance.As per the guidelines, PATNI has placed precision temperature
monitoring and control device in the Datacenter.

All the datacenter devices are power feeded by 2 UPS dedicated for Data center only to provide high
uptime. Both are of 160 KVA. There are 2 different power supply source have been provided in Data
center for redundancy purpose.3 central generators (1x3500 KVA and 2x1500 KVA) installed in the
facility to cater power requirement in case direct line power supply not available.

For data center and other Hub rooms we have provisioned separate split AC despite of central AC duct
feeded to all location just for redundancy purpose. Separate Fire fighting devices are placed in all Hub
rooms and Data center in addition with smoke/fire detection and prevention system.

11.4 Raised Floor

PATNI is using raised floor concept inside the data center. Raised floor is an option with very
practical benefits. It provides flexibility in electrical and network cabling, and air conditioning.

A raised floor is not the only solution. Power and network poles can be located on the floor and air
conditioning can be delivered through ducts in the ceiling. Building a data center without a raised floor
can address certain requirements in ISP/CoLo locations. Wire fencing can be installed to create cages
that you can rent out. No raised floor allows these cages to go floor to ceiling and prohibits people
from crawling beneath the raised floor to gain unauthorized access to cages rented by other businesses.
Another problem this eliminates in an ISP/CoLo situation is the loss of cooling to one cage because a
cage closer to the HVAC unit has too many open tiles that are decreasing subfloor pressure. However,

62
some ISP/CoLo locations have built facilities with raised floor environments, because the benefits of a
raised floor have outweighed the potential problems listed above.

Drawbacks to the no-raised-floor system are the very inefficient cooling that cannot easily be rerouted
to other areas, as well as the problems associated with exposed power and network cabling. A raised
floor is a more versatile solution.

PATNI data center a raised floor system where supply coming from tiles in the cold aisle. Hot
aisle/cold aisle configuration is created when the equipment racks and the cooling system’s air supply
and return are designed to prevent mixing of the hot rack exhaust air and the cool supply air drawn
into the racks. All equipment is installed into the racks to achieve a front-to-back airflow pattern that
draws conditioned air in from cold aisles, located in front of the equipment. The rows of racks are
placed back-to-back, and holes through the rack (vacant equipment slots) are blocked off on the intake
side to create barriers that reduce recirculation, as shown in the graphic below in figure 11.4.
With proper isolation, the temperature of the hot aisle no longer impacts the temperature of the racks
or the reliable operation of the data center; the hot aisle becomes a heat exhaust. The HVAC system is
configured to supply cold air exclusively to the cold aisles and pull return air only from the hot aisles.

Figure 11.4

The hot rack exhaust air is not mixed with cooling supply air and therefore can be directly returned to
the air handler through various collection schemes, returning air at a higher temperature, often 85°F or
higher. The higher return temperature extends economization hours significantly and/or allows for a
control algorithm that reduces supply air volume, saving fan power. In addition to energy savings,
higher equipment power densities are also better supported by this configuration.

63
Underfloor distribution systems should have supply tiles in front of the racks. Open tiles may be
provided underneath the racks, serving air directly into the equipment. However, it is unlikely that
supply into the bottom of a rack alone will adequately cool equipment at the top of the rack without
careful rack design.
For proper ventilation PATNI bought special designed racks so that the ideal air management system
would duct cooling air directly to the intake side of the rack and draw hot air from the exhaust side,
without diffusing it through the data center room space at all.

11.5 Data Centre Design Cabling Infrastructure

A data centre, computer room or server room by its very nature will house an abundance of
interconnecting cables. The cabling industry is now moving at such a pace that it is important to
select a cabling infrastructure that will cope with your day one requirements and any new
technologies which may be used within the data centre in the coming years.

PATNI COMPUTER SYSTEM LTD. is using copper cabling CAT6 UTP (Solid cables) in office
buildings, Data Center, and other installations to provide connectivity. Copper is a reliable medium for
transmitting information over shorter distances; its performance is only guaranteed up to 109.4 yards
(100 meters) between devices where solid cable provide better performance and are less susceptible to
interference, making them the preferred choice for use in a server environment.

For rest connectivity longer than 100 meters PATNI using fiber cabling which can handle connections
over a much greater distance than copper cabling, 50 miles (80.5 kilometers) or more in some
configurations. Because light is used to transmit the signal, the upper limits of how far a signal can
travel along a fiber cable is related not only to the properties of the cable but also to the capabilities
and relative location of transmitters.

Besides distance, fiber cabling has several other advantages over copper:

• Fiber provides faster connection speeds.

• Fiber isn't prone to electrical interference or vibration.
• Fiber is thinner and lighter weight, so more cabling can fit in to the same size bundle or limited
spaces.
• Signal loss over distance is less along optical fiber than copper wire.

64
PATNI is using multimode fiber to provide connectivity over moderate distances, such as in most Data
Center environments or among rooms within a single building. A light-emitting diode (LED) is its
standard light source. The term multimode refers to the several rays of light that proceed down the
fiber. PATNI use Fiber connectivity even for less than 100 meters distance for higher data rate
transmission like between switch to switch (Backbone connectivity).

11.6 Aisles and Other Necessary Open Space

Aisle space should allow for unobstructed passage and for the replacement of racks
within a row without colliding with other racks. The optimal space would allow for
the turn radius required to roll the racks in and out of the row. Also, rows should not
be continuous. Unbroken rows make passage from aisle to aisle, or from the front of
a rack to the back, very time consuming. Such clear passage is particularly important
in emergency situations. The general rule of thumb for free floor space is between 40
and 50 percent of the square footage.

Figure 11.6 Proper Aisle Space and Non-Continuous Rows

How aisle space is designed also depends upon air flow requirements and RLUs. When designing the
center, remember that the rows of equipment should run parallel to the air handlers with little or no
obstructions to the air flow. This allows for cold air to move to the machines that need it, and the

65
unobstructed return of heated air back to the air conditioners. Figure 11.6 is appropriate layout done in
PATNI.
Be sure to consider adequate aisle space in the initial planning stages. In a walls within-walls
construction where the data center is sectioned off within a building, aisle space can get tight,
particularly around the perimeter.

11.7 Network Redundancy

PATNI has built electrical redundancy provided to servers by electrical conduits running from more
than one power distribution unit, network redundancy is provided by structured cabling running from
more than one networking device. Whereas electrical conduits are hardwired into the PDUs, however,
structured cabling is standalone infrastructure that any networking devices can be plugged in to. Each
cable is essentially providing its own path, and you just need additional networking devices to make
them redundant.

As long as you provide abundant structured cabling throughout the Data Center, you increase
redundancy as much as you want by simply installing more networking devices at the network row
and network substations. If you want to provide a minimum level of redundancy over the entire Data
Center, install a second set of networking devices in the network row and patch to key components at
the network substations. If you want to provide an even greater level of redundancy, double the
networking devices at each network substation as done by PATNI COMPUTER SYSTEMS LTD.

Providing this redundancy may or may not require additional cabling infrastructure. It depends upon
how many network connections a given server requires, and how many servers are patched into a Data
Center's network devices. Most servers require a minimum of two connections, one for a primary
Ethernet connection and another for either a console connection or a secondary Ethernet connection.

11.8 Data Centre Design Security and Monitoring

Security and monitoring are two key factors to ensure that a data centre, computer room or server
room will run undisturbed. Good security will ensure that access is controlled and in so doing,
interference of important and sensitive equipment by untrained and unauthorized personnel is
prevented. Monitoring of the environment can also help to prevent incidents which could otherwise
disrupt or destroy equipment within the data centre.

There are various types of security and monitoring equipment available, much of it designed

66
specifically for the data centre environment. PATNI has installed access controllers (program based)
on each door within the facility with the appropriate access right to its employees. These access
controllers are centrally managed by access control team. For Data center access PATNI has
implemented 2 level of security, one is of Card level (Employee Identity) and other one is of PIN code
level security as mentioned in the figure below and Data center access is provided only for datacenter
team members. Other than this PATNI monitoring its campus facility with the help of digitals cameras
placed all over the facility. All points of access should be controlled by checkpoints, and coded card
readers. Below figure 11.8 shows one of the access controller type installed in PATNI.

Figure 11.8 Cipher Lock at Restricted Access Doorways

11.9 Data Center Network Architecture and Engineering

The Data Center network architecture is a key component of the Service Oriented Infrastructure. How
the network infrastructure is designed and implemented plays a key role in what level of service
availability and survivability the IT resources can offer. In many cases, the network is grown
organically with little consideration for future growth or physical/logical separation requirements. As
the application services infrastructure expands, it becomes more of a challenge to maintain and
purposefully plan for performance and availability of the network. Additionally, it is paramount to
understand not only the data center WAN and LAN infrastructure, but also the remote site WAN
infrastructure in order to match expected application performance and availability characteristics on an
end-to-end basis. PATNI follows a standard, proven approach that includes the following activities:
identification of touch points within the current environment, creation of configuration standards,
development of test plans (network and end user), development of integration and migration plans and
process for turnover to the operational environment. As-built documentation and knowledge transfer
are key deliverables. Design areas of focus are LAN Architectures (IP Routing Architecture, Layer 2
Switching), WAN Architectures (MPLS VPN, IPSEC VPN, and traditional VPN), WLAN

67
Architectures, Optical Architectures (DWDM, SONET), Content Delivery Architectures, IPv6 and
QoS Architectures.
PATNI is using most of the technology/topology mentioned above to run the Datacenter. The detail of
the same is given in below sections.

PATNI COMPUTER SYSTEMS LTD. new facility at Noida, SEZ caters corporate and staff for
different processes. This new facility have connectivity to PATNI’s Center-I facility in addition to two
Data centers in US, Billerica and US San Jose and one in UK.

Other the Data Centers are connected to SEZ over MPLS backbone from a Service Provider. MPLS
SP has placed dual CPE at SEZ, and other Data Centers. CPEs have dual local-loop to different PEs
for redundancy.Center-I will be connected to SEZ via Ethernet links. PATNI has procured 3x10mbps
links between Center-I and SEZ. These links have L-3 connectivity between Core switches at both
locations. This solution will enable PATNI to route Voice and data between SEZ and Noida Center-I
facility.

PATNI COMPUTER SYSTEMS LTD. has also procured links from ISP for providing internet access
to its users at SEZ and wants to have load balancing and redundancy on multiples ISP links. PATNI
has procured AS number space. PATNI has also setup VPN capability for its remote and
telecommuters.

PATNI has designed a Tier 3 data center that is “concurrently maintainable” because the redundant
components are not on a single distribution path (as in Tier 2). The data center can have infrastructure
activity going on without disrupting the computer hardware operation in any way. This data center is
manned 24 hours a day—for maintenance, planned activities, repair and replacement of
components, addition or removal of capacity components, testing, etc.

PATNI data center is designed to be upgraded to Tier 4 when the business case justifies the cost of the
upgrade (additional protection). This datacenter has sufficient capacity and distribution available to
simultaneously carry the load on one path while performing maintenance or testing on the other path.

68
12. Network Topology
Figure 12.0 presents voice and data network architecture at PATNI:

Figure 12.0

12.1 Solution resiliency

12.1.1 LAN resiliency

69
For sake for understanding and explanation, PATNI SEZ network setup has been divided into two
areas that are:
 Internet / Public Area, for accessing public network
 Intranet / Corporate area, which will connect to PATNI COMPUTER SYSTEMS LTD.’s
international Data centres in US & UK.

In First Phase deployment is done at First Floor Fully. Second and Ground Floor each have one
Catalyst 2960-48 port switch. As depicted in network LAN figure 12.2 below:

12.2 Intranet Area network layout

Figure 12.2
NETWORK DIAGRAM – PATNI

SF Single Switch Connectivity

CPE – 1 CPE – 2 FIRST FLOOR
HSRP HSRP
Intranet Area

C3750-24

STACK
Towards Inside Leg of Pubic
Area Firewall
Public Area
APs
ASA -5520
Dual-uplink from each access switch
C2960-48

GF Single Switch Connectivity

C2960-48 HUB HUB
ROOM 1 ROOM 2 Public Area
Catalyst 6509
APs
Dual-uplink from
each access
A B switch
Public Switch L-3

3750 - POE
3750 - POE IP PHONE
Public Switch

C2960-48 HUB
ROOM 3
Dual-uplink from
each access
switch
Intranet
APs
Physical Connectivity to
3750 - POE
Public Area WLC
WLC 4402 Public Switch

Physical Connectivity to MUX Room

PUB

Public Area
SERVER
FARM
SUB APs
CCM

12.3 Intranet Area

 The new facility is having three floors, however in its first phase of project implementation
only first floor is covered.
70
 PATNI COMPUTER SYSTEMS LTD. has procured two Catalysts 6509 with Sup-720 engine
for its LAN campus switching network. Campus network is deployed as industrial standard
network. Core switches in PATNI campus network provides core and distribution level access
where are edge switches which include Catalyst 3750 and Catalyst 2960 switches serving
access layer connectivity.
 There are three hub rooms at each floor which will serve local area connectivity for that floor.
Server room is separate to hub rooms and accommodates all core LAN / WAN / Voice
equipments. Both the core switches are installed in the server room with redundant power
supplies. However there are no Sup level redundancy in each Catalyst 6509 box.
 For better through-put Voice and Data network are divided into logically separate VLANs
subnets. Layer – 3 routing between the VLANs is done at core switches.
 There is an ether-channel of 2 TenGig ports on Sup-engine between the core-switches. Ether-
channel will avoid downtime due to single link failure and also provide high speed data
transfer rates.
 All access switches have dual-uplinks to both the core switches. Redundant uplink is blocked
via STP. In case any uplink/Core switch goes down, the redundant uplink changes from
blocked to forwarding state and all traffic starts routing via this link.
 Core Switch-1 (ODD) is acting as root for all the odd VLANs (1,3,5,7…) & secondary root for
all even VLANs (2,4,6,8…) and Core Switch-2 (EVEN) is acting as root for all the even
VLANs & secondary root for all odd VLANs.
 Cisco 3750-POE switch is configured for IP Phone and agent PCs connectivity. However, in
case any access switch fails, all IP Phones and Agent PCs connected to that switch will become
unreachable causing loss of production. To resolve PATNI Tech team suppose to pre
configured 1 POE 3750 switch and 1 Edge switch 2960 to replace faulty so that the production
downtime can be minimized.
 Both the core switches have been procured with single sup-engine and dual-power supplies.
Any problem in Supervisory engine can cause service outage and all devices connected to the
switch may become unreachable. Recalculation of STP may also be triggered in the network in
case of Supervisory engine on any Core-switch fails.

71
PATNI COMPUTER SYSTEMS LTD. has various application servers which are deployed in server
room. For isolation for server farm all servers are connected to high port density access switches
dedicated to server farm only and those server farm switches are connected to core switches. Server
farm is also logically separate as on different VLAN.

12.4 Internet Area network layout

Figure 12.4 shows Internet area layout.

Figure 12.4

12.5 Internet Area

 For internet access PATNI has procured bandwidth from VSNL and Second ISP is Verizon.
These two links are connected to Cisco 3825 routers. These routers are serving as first level of
defence towards internet. These routers are also have hardened ACL configuration to secure
corporate network. These two routers are connected to Cisco 3750 layer -3 switches these
switches termed as outside switches of the network. As a matter of traffic load balancing,
72
 PATNI has procured AS number space from its service providers. Multi-homing and load
balancing is achieved via BGP configuration on outside switches.
 In internet area there is internet visitor’s area from where visitors are allowed to access
internet. For this purpose PATNI has procured 4400 series WLC and 1200 light weight
wireless access points. WLC – Wireless LAN controller is directly connected to outside
switches and registers APs in visitor area for providing access to network.
 PATNI COMPUTER SYSTEMS LTD. has installed Video Conferencing unit, which
established connections via public network. The VC (Life Science) is used for video
conferencing within PATNI and clients meeting.

 In internet area one pair of Firewall is installed in active standby fashion, this serve as
perimeter security for corporate network. Various site to site VPN with client and between
PATNI offices has been configured and establish on this perimeter security firewall. PATNI
has various application servers like WEB Server, Mail server, Proxy server, FTP server and
client application base servers which are deployed in DMZ zone area of perimeter security
firewall.
 Default gateway for perimeter security firewall is layer three switches (Outside Switches).
 Inside leg of perimeter firewall is connected to corporate network firewall’s outside interface.
This will also serve internet access to corporate network users. So for internal users, there is 2
layer of security. PATNI has configured firewalls to allow only client specific and business
required applications. Hence restriction on firewall is port, source and destination based. There
is no direct access to internet for internal users, its only through the proxy where PATNI has
deployed content based filtering with the help of WEBSENSE.

12.6 Backbone resiliency in intranet area

12.6.1 PHASE-I
PATNI COMPUTER SYSTEMS LTD. has procured 3x10 MB of Ethernet link connectivity between
Center-I and SEZ facility. These links were primary path for connectivity to US & UK data centres in
first phase of deployment of SEZ facility at Noida. But now US & UK data centres are directly access
through MPLS cloud. Theses ethernet links are configured as Active as well as redundant to each
other.

73
In order to achieve the objective both site’s core switch comprising Cisco Catalyst 6509. For SEZ
PATNI has procured 2 NOS of Catalyst 6509 switches. Physical connectivity is as per network
diagram below in figure 12.6.2

12.6.2 SEZ to Center-I connectivity network layout

Figure 12.6.2

Center -I AND SEZ CONNECTIVITY @ PATNI

Center -I SEZ

Catalyst 6509 Catalyst 6509

Core A Core A

Layer – 2 Layer – 2
EIGRP DOMAIN
Domain Domain
C - 28 SEZ

Core B Core B

 2 Nos of links are connected to Catalyst 6509 “A” switch and remaining 1 Nos is connected to
Catalyst 6509 “B” switch. All three links are connected in Layer 3 domain in three different
subnets.
 There will be 1 Nos of Layer –3 connectivity between Catalyst 6509 switches in addition to
ether-channel at both facilities.
 As shown all four switches are in single EIGRP domain.
 For all voice subnet Core switch “A” at SEZ facility is primary path and Core switch “B” is
secondary path. Also will assure that no load balancing will happen between the links for voice
traffic and that the voice traffic will traverse through single path by implementing Policy Base
74
 routing with redundancy through other available path from any of the Core switch with priority
high set which will insure better voice quality.
 Data traffic routes as per the available path learned through EIGRP dynamic routing.

12.6.3 PHASE-II

 PATNI COMPUTER SYSTEMS LTD. has procured two MPLS link connectivity from two
different ISPs for its SEZ facility’s connectivity to US & UK Data centres. For local end
redundancy PATNI has procured local loop connectivity from two different service provider.
MPLS CPE routers are placed at SEZ facility only.
 Primary link from each service provider is Ethernet handoff and secondary link is serial.
 In corporate network pair of ASA5520 firewall is configured to secure corporate network from
MPLS cloud.
 PATNI has MPLS connectivity to its US & UK Data centres over Service Provider network.
Any outage in service provider MPLS network is handled and managed by the SP.
 There are two MPLS CPEs placed at SEZ facility. Both the CPE routers are connected to
Cisco-3750 Layer-3 switch.
 All corporate data and voice traffic flows between data centres and SEZ facility through MPLS
cloud.

12.7 LAN Switching

12.7.1 Design Requirements and Assumptions

 No. of LAN connections to be deployed in first phase at first floor w.r.t to each hub room. On
the basis of data and voice connections VLAN domains are defined.
 Total no. of Production Site for this LAN design is SEZ.
 For now in first phase at first floor 8 NOS of Catalyst 2960-48 switch has been installed in
each hub room along with 1 NOS of Catalyst 3750-48 POE switch in each hub room.
 At ground floor and Second floor each have 1 NOS of Catalyst 2960-48 port switch.
 1 NOS of Catalyst 2960-8 port switch is installed in each hub room and 1 NOS of Catalyst
2960-8 port has been installed at Ground floor. This is used for direct public network access in
premises.
 In first phase of installation 100 IP Phone connections are deployed at first floor.

75
  All access layer switches have dual leg fibre LC to LC connectivity to core switches for
redundancy purpose.
 Each core switch is having one number of SUP-720-3B module, there is no redundancy with
respect to Supervisory module redundancy available on Core switches. However, dual-power
supplies are available on both the switches.

12.7.2 Technical Prerequisites

 Telnet passwords - PATNI has Cisco TACACS server in place through which all network
devices are authenticated which is integrated with AD.
 As mention in above sections traffic load balancing or load sharing is configured for the
internet connection with the help of ISP.

 PATNI has installed Syslog server for capturing log of all network devices. PATNI has
installed DC, DHCP, DNS,WSUS, File server, FTP and Proxy/Websense Servers in the
network to authenticate end user to gain network access, IP address assignment, name
resolution, patch update, File management, File transfer and Internet access management with
content filtering respectively.
 All switches are in single VTP domain for the ease of configuration within switching domain.
VTP domain is password protected for the security prospect.

12.7.3 VLAN ID and Subnet Info

VLAN would be assigned as per the process requirement. Every process or department would be in
separate VLAN. Inter VLAN routing will be configured on Core switches.

12.7.4 Rack layout

Following is the Device Rack layout.
U
No. Rack # 1 Rack # 2 Rack # 3 Rack # 4
42
41
40
39
38
37
36
35 Public Switch
Primary(3750-
76
 24TS-E)
DMZ Primary
Switch(3750G-24TS-
34 S1U)
Public Switch
Secondary(3750-
33 24TS-E)
DMZ Secondary
External Firewall
Switch(3750G-24TS-
Primary(ASA 5520)
32 S1U)
External Firewall
Secondary(ASA
31 5520)
Wireless
Internal Firewall
External(WLC4402-
Secondary(ASA 5520)
30 12-K9)
Secure Access
29 Control (ACS-113)
Client Switch(WS- Internal Firewall
Wireless
28 C3750-24TS-S) Primary(ASA 5520)
Packet Shaper(3500) Internal(WLC4402-
Client Switch(WS- MPLS Switch
50-K9)
27 C3750-24TS-S) Primary(3750-24TS-E)
26
MPLS Switch
Client Bharti MPLS
Secondary(3750-
Router(2800)
25 24TS-E)
Client Reliance
24 MPLS Router(2800)
Verizon MPLS Primary
23 Router(Cisco-2800)
Verizon MPLS
Secondary
22 Router(Cisco-2800)
21 VSNL MPLS
Core Switch - ODD Core Switch – EVEN
Secondary
(WS-6509-E-Series) (WS-6509-E-Series)
20 Router(Cisco-2800)
19
18 VSNL MPLS Primary
17 Router(Cisco-2800)
16 VSNL Public
Router(CISCO-
15 3825)
14 Verizon Public VG Router(CISCO-
Router(CISCO- 3845-MB)
13 3825)
12
11
10
9
8
7
6
5
4
3
77
 2
1
Figure 12.7.4

PATNI has to arrange the Rack Space as per the following ‘U’ space of devices:

Equipment U- Space Required

Catalyst3750 Switch 1U
Catalyst 6509-E 13U
Cisco 3825 router 2U
Cisco WLC 1U
Cisco ASA 5520 Firewall 1U
Table 12-7-4
12.7.5 POWER CONSIDERATION

Cisco equipment minimum power points require is listed as follows:

Equipment Power 5/15A Req.

Catalyst 3750 Switch 1
Catalyst 6509-E 2
Cisco 3825 router 2
Cisco WLC 2
Cisco ASA 5520 1
Table 12-7-5

PATNI has provisioned efficient power and power point in the Data center to feed every device. To
achieve this PATNI has setup 2x160 KVA UPS dedicated for Data center in redundant fashion.

Proper earthing (1V- 3V) has been done for whole facility.

12.8 Spanning Tree Design

Spanning-tree is required in the LAN campus network infrastructure to ensure a loop-free forwarding
topology in presence of redundant Layer 2 paths. Spanning tree should never be disabled even if the
topology is designed to be loop free.
In PATNI network Rapid-PVST spanning tree algorithm is deployed for fast convergence. Root
switches will be segregated between the VLANs for traffic load sharing.

12.8.1 Trunking

Configured 802.1Q (dot1q) trunks between core switches & between core-access switches on the
uplink interfaces. Trunk is nothing but a point-to-point link to carry the traffic of multiple VLANs &
also allows us to extend VLANs across an entire network.
802.1Q has been chosen because this is an industry-standard trunking encapsulation whereas
78
 ISL being Cisco proprietary has some limitations like it is not supported on following
switching modules as on date:

· WS-X6502-10GE
· WS-X6548-GE-TX,WS-X6548V-GE-TX,WS-X6548-GE-45AF
· WS-X6148-GE-TX, WS-X6148V-GE-TX, WS-X6148-GE-45AF

12.8.2 EtherChannel

An EtherChannel bundles individual Ethernet links into a single logical link that provides the
aggregate bandwidth of up to eight physical links.

12.8.3 Spanning Tree Protocol

The IEEE 802.1w standard was developed to take 802.1D’s principle concepts and make the resulting
convergence much faster. This is also known as the Rapid Spanning Tree Protocol (RSTP). RSTP
defines how switches must interact with each other to keep the network topology loop free, in a very
efficient manner. Like 802.1D, RSTP’s basic functionality can be applied as a single or multiple
instances.

Core Switch-1 acts as root for all Odd VLANs & secondary root for all Even VLANs and Core
Switch-2 acts as root for all Even VLANs & secondary root for all Odd VLANs.

12.8.4 HSRP

This is configured so as to provide redundancy for L3 path. Important point to note is that L3 path for
particular VLAN traffic will converge with L2 STP path so that even in case of device failure, L2 &
L3 convergence occurs together on same device. A single HSRP group is used between both Core
switches.
Core-1 Switch is responsible or L3 active for all Odd VLANs and standby for all Even VLANs.
Like wise, Core-2 switch is responsible for L3 Active for all Even VLANs and standby for Odd
VLANs.

12.8.5 Helper Address

79
Configured switch for helper address so that the end user machines can get the IP address from DHCP
server sits in Server VLAN.Each L3 VLAN interface except server VLAN configured for helper
address.

12.8.6 Wireless

PATNI COMPUTER SYSTEMS LTD. has procured two 4400 series WLC controller and 20 APs for
its wireless network. One WLC controller has been installed in Public area which has direct
connectivity to Public network and provides wireless access to visitors at PATNI.
Second WLC controller has been installed in corporate area which is directly connected to private
network facility of PATNI. This provides wireless access to PATNI users. As of now, 8 internal and 8
External APs are installed for accesses to internet as well as corporate network access. In this scenario
Internal APs are associated to corporate area WLC, have non-broadcasted SSID and other 8 external
APs are associated directly to internet area WLC, have broadcasted SSID with share key access
configured.

Corporate SSID is associated to exiting Active domain controller of PATNI for user level
authentication. While External SSID is preshared.

Bibliography

• “Grow A Greener Data Center”, by Douglas Alger

• “Data Center Projects Plan”, by Neil Rasmussen & Wendy Torell
• “Enterprise Data Center Design and Methodology”, by Rob Snevely

• “Enterprise Data Center Design and Methodology”, by Rob Snevely

• “Enterprise Data Center Design and Methodology”, by RobSnevely
• “High performance Data Centers”, A Design Guidelines

80