Data Center Facility Global

Standards

Facilitated by

Where Education Meets IT, Facilities & Design

Presenter Information
 Outline

What’s a data center?

Data center elements

Availability & Reliability

Standards & Benchmarks

Redundancy

The Tier system
 What’s a Data Center?

Sole – use for information processing or
network operations

Independent of size & Tier rating

Must have a generator

With/without raised floor

Defined by functionality and systems
 Data Center Elements

The Physical Layer = The Built
Environment

Site Systems

Building Systems

Utility Systems
 The Site

Location & site risk assessment

Site massing

Weather

Seismic

Physical security

Staff impacts

Incentives
Linked to DR/BC plans
 Planning & Architecture

Solves most of the site’s risk profile

Meets IT flow and adjacencies issues

The data center and the supporting
facilities are designed to maximize IT staff
and system efficiency.
 Program Flow Diagram SINGLE UTILITY
FEEDER
DATA CENTER ADJACENCY DIAGRAM
SERVICE YARD

INCOMING UTILITY SERVICE SATELLITE

FARM
TEMPORARY LOCAL WELL
COOLING THERMAL
CHILLER MUNICIPAL
TOWERS STORAGE FIBER FIBER
CONNECTION WATER SUPPLY
FUEL OIL TEMPORARY UTILITY SERVICES SERVICES
STORAGE & GENERATOR DISTRIBUTION ENTRANCE #1 ENTRANCE #1
CONDITIONING CONNECTION EQUIPMENT

INCOMING UTILITY SERVICE

TEMP/EMERGENCY GENERATOR SERVICE

GENERATOR INPUT TO DATA CENTER

GENERATORS 12 Kv SERVICE
DAY TANKS UPS SYSTEMS ENTRANCE CHILLERS
BATTERY/D.C. SYSTEMS NORMAL & EMERGENCY WATER TREATMENT DOMESTIC WATER
INCOMING FIBER INCOMING FIBER
UPS OUTPUT SWITCHGEAR LOCAL HEATING FIRE PROTECTION
SERVICE POP #1 SERVICE POP #2
SWITCHGEAR ENERGY PLANT & HUMIDIFICATION & MAKE-UP WATER
POWER MONITORING C/E SUBSTATIONS & PUMPING SYSTEMS
UPS INPUT

ELECTRICAL

DC

DC
Pre-Action DC
LOAD BANKS
ENGINEERING MAINTENANCE AND ADMIN SHOPS

OFFICE SPACE CRITICAL SERVICES

PRINT CRITICAL SERVICES

DATA CTR

PARTS & TOOL MDF CRITICAL SERVICES

MATERIAL MATS
STORAGE
HELP DESK CRITICAL SERVICES

OP CENTER CRIT SERVICES

CHEMICAL &
MATERIAL MATS
PAINT STORAGE FIBER /
OPERATIONS
M HELP DESK COPPER PRINT
CENTER
MDF
CRITICAL ENVIRONMENTS / PROCESS SPACE
DATA CENTER & PBX

NON-SECURE NON-SECURE P
MATS MATS
DOCK STORAGE

PRINT
SECURE / MIS-ONLY CIRCULATION STAGING &
DISTRIBUTION
SECURITY
MIS-SECURE PEOPLE & MATS

VENDOR VENDOR
LOADING
DOCK PRINT / PAPER
TRAP VENDOR
VENDOR SOFTWARE STORAGE
SECURE DEVELOPMENT
OFFICES LIBRARY
STORAGE
STAGING &
ASSEMBLY
ROSS & VENDOR

TEST BED/
SECURE OFFICE AREAS REQUIRING
CHANGE NON-CRITICAL
DOCK UPS AND/OR GENERATOR POWER
CONTROL OFFICE AREAS
OR 7X24 COOLING
 Utility Systems

Cooling plants

Ventilation systems

Thermal storage

Water storage

Controls

Monitoring

Utility services

UPS power

Generators

Power distribution

Lighting

Grounding/lightning protection

Fire alarm and suppression
 Availability & Reliability

Where do the definitions come from?

Are they the same?
 Reliability

Reliability = the ability of a system to
perform its intended function

This tends to be a topology function with
an arithmetic solution based on how a
system is constituted
 Availability

Availability = the long-term fraction of time
that a system is in service and is
satisfactorily performing its intended
function

Unlike reliability, this is a time-related
issued

Other factors affect availability such as
human error, operations and maintenance
 Outages

A failure may not lead to an outage, as
redundancy in the system may save
you.

An outage is a loss of service or
disconnection of the load (planned or
unplanned).

The biggest impact to availability is
PEOPLE! Human error is the largest
root cause to outages or failures.
 Redundancy

What’s redundancy?

Typical redundancy descriptions:

N

N+1

2N or System + Systems

2(N+1)

x- or multi-N or Distributed Redundant
 Standards & Benchmarks

What affects your design and facility?

Where did they come from?

Who maintains them?

Moving from definitions towards
performance-based standards
 Standards & Benchmarks

British Standards Institution (BS)

ANSI / TIA / EIA

The Uptime Institute

ASHRAE

BICSI

IEEE

ISO/IEC 24764 (New)
Standard Organizations (Typical – UK & EU)

ANSI/TIA/EIA-568-B.1 - Commercial Building
Telecommunications Cabling Standard, Part 1: General
Requirements

ANSI/TIA/EIA-568-B.2 - Commercial Building
Telecommunications Cabling Standard, Part 2: Balanced
Twisted Pair Cabling Components

ANSI/TIA/EIA-568-B.3 - Optical Fibre Cabling Components
Standard

ANSI/TIA/EIA-569-B - Commercial Building Standard for
Telecommunications Pathways and Spaces

ANSI/TIA/EIA-606-A - The Administration Standard for the
Telecommunications Infrastructure of Commercial Buildings

ANSI/TIA/EIA-607-A - Commercial Building Grounding and
Bonding Requirements for Telecommunications
Standard Organizations (Typical – UK & EU)

BS7671: 2001 Requirements for Electrical

Installations. (16th edition IEE Wiring Regulations)

BS 7273-1:2000 Code of Practice for the Operation
of fire protection measures

BS 6266:1992 - Code of Practice for fire protection
for electronic data processing installations

BS 5839-1: 1988 Fire detection and alarm systems
for buildings. Code of practice for system design,
installation and servicing
Standard Organizations (Typical – UK & EU)

BS EN 50091-1:1993 - Specification for Uninterruptible Power

Supplies (UPS). General and Safety Requirements

BS EN 12825: 2001 - Raised Access Floors

BS EN 1047: Physical security and environment
protection

BS 4434 - Safety and Environmental aspects in the design,
construction and installation of Refrigeration appliances and
systems

BS ISO 8995:2002 Lighting of indoor work places

TIA/ EIA - PN -3-0092 - Telecommunications Infrastructure
Standard for Data Centres (Draft document)
Standard Organizations

ASHRAE

Thermal Guidelines for Data Processing Environments

ANSI

T1.336 Engineering Requirements for a Universal
Telecommunications Frame (waiting publication)

EIA

EIA-310-D Cabinets, Racks, Panels and Associated Equipment

NFPA

NFPA 75 Standard for the Protection of Information Technology
Equipment

Telcordia Technologies

GR-63-CORE (NEBS) Physical Protection

CENELEC

Information Technology – Generic Cabling Systems Part 5: Data
Centers

TIA

TIA-942 Telecommunication Infrastructure for Data Centers

BICSI

BICSI 002 Data Center Design and Implementation Best Practices
 The Tier Rating System

Allows for the differentiation between
varying system reliabilities

Four Tiers – One, Two, Three & Four.

Newer standards offer clear availability
requirements that do not demand a certain
topology.

Fire protection redundancy not related to
Tier ratings.
 Use of Standards for Evaluations
System / Tier Tier I Tier 2 Tier 3 Tier 4

10.4 Electrical Systems Overview &

Design Criteria

Common Industry Description Single Path Redundant Component / Concurrently Maintainable Fault Tolerant
Single Path & Operable

Number of Power Delivery Paths to the Single Single Two, one active or one Two or more active
Critical Load passive or non-
non-UPS
power

Redundant System Components (UPS, No Yes Yes Yes

Gens, etc.)

Distinct UPS Sources (A, B, etc.) Single or N Single or N Single or more, depending At least two resulting in a
on the critical power load minimum of N+2

System allows concurrent maintenance and No Within some systems with Yes Yes
operations paralleled components,
but not consistent
throughout the electrical
system.

System allows fault tolerance and self-

self-heals No No In some cases Yes
failures?
 Use of Standards for Evaluations
System / Tier Tier I
10.4 Electrical Systems Overview &
Design Criteria
Computer & Telecom Equipment Power Single-
Single- or Dual-
Dual-Cord fed
Cords with no redundancy up to
critical power system
capacity. No ability to
switch automatically
between input sources via
static switch-
switch-based PDU
or panel.
Ability to Add Systems or Components No
Transparantly to Existing Loads
Single Points of Failure One or more single points
of failure for distribution
systems serving electrical
equipment or mechanical
systems
UPS Redundancy N N+1
UPS Topology Single Module or Parallel
Non-
Non-Redundant Modules
Labeling Yes
Tier 2 Tier 3
Single-
Single- or Dual-
Dual-Cord fed Single-
Single-, Dual-
Dual- or Poly-
Poly-
with no redundancy up to Cord Feed with either
critical power system 100% capacity on each
capacity. No ability to cord or adequate capacity
switch automatically on each cord mapped to
between input sources via the individual IT system
static switch-
switch-based PDU should a given source fail
or panel. (i.e. 5 to make 6 or 2 to
make 3 inputs).
No If planned during the initial
design
One or more single points No single points of failure
of failure for distribution for distribution systems
systems serving electrical serving electrical
equipment or mechanical equipment or mechanical
systems systems
N+1
Parallel Redundant Parallel Redundant
Modules or Distributed Modules or Distributed
Redundant Modules Redundant Modules or
Block Redundant System
Yes Yes
Tier 4
Single-
Single-, Dual-
Dual- or Poly-
Poly-
Cord Feed with either
100% capacity on each
cord or adequate capacity
on each cord mapped to
the individual IT system
should a given source fail
(i.e. 5 to make 6 or 2 to
make 3 inputs).
Yes
No single points of failure
for distribution systems
serving electrical
equipment or mechanical
systems
Multiple N, 2N, 2N+1 or
any configuration greater
than N+1 that does not
compromise redundancy
during failure or
maintenance modes of
operation
Parallel Redundant,
Distributed Redundant or
Block Redundant
Systems
Yes
 Electrical Power Flow

Utility Basic Power Flow Components

Standby Electrical Non-Critical

Power Distribution Load

UPS

Critical
Load
 The Tier 1 System
Single Path

Industry Description: Single Path

System Redundancy: N

UPS Source Available to the Critical Load: One

Ability to Be Maintained While Under Load: No

Ability to Recover from Failures: No

Resulting Definition: Single Path / Single Module /
Single Source

The Tier 1 data center is a basic data center with no
redundancy. It has a single path for power and
cooling distribution with no redundant components.
This system can not be maintained while it’s
operating and a failure will likely result in a loss of
electrical service to the load. This system has a
single electrical supply to the load and no source
diversity. This system can’t be maintained without
interruption to the load. Mechanical and vital house
loads would be also be supplied by non-redundant
power systems.
 The Tier 1 System
Single Path (cont)

Single points of failure are common throughout
the electrical system, and critical load
interruptions are likely during both routine
maintenance activities and unanticipated
failures.

Single path

No redundant components

Single points of failure exist

Load interruption during maintenance and failure
modes of operation
 Tier 1 Topology
Single Path (w/o Redundant Components)

 Single path
 Without redundant components

N
TIA-942 /
Tier 1
Technology

Tier 1
Spaces and
Functional
Areas

Tier 1
 The Tier 2 System
Single Path with Redundant Components

Industry Description: Single Path with Redundant

Components

System Redundancy: N

UPS Source Available to the Critical Load: One

Ability to Be Maintained While Under Load: At the
system level only, but not in the distribution system

Ability to Recover from Failures: Only at the system
level.

Resulting Definition: Single Source / Multiple
Module / Single Path
 The Tier 2 System
Single Path with Redundant Components (cont)

The Tier 2 system may experience a failure while

it’s operating due to a lack of redundancy in the
distribution system. Redundant components may
exist on an n+1 and paralleled basis in the UPS or
generator systems, but a Tier 2 system does not
offer redundancy in the distribution system. A
failure in the n+1 systems would not likely result in
a load failure, but would reduce the redundancy
level in the paralleled systems to “n”. This system
has a single electrical supply to the load and no
source diversity and any failure in the distribution
system will likely result in a loss of electrical service
to the load. Large-scale system maintenance can’t
be performed without interruption to the load.
Mechanical and vital house loads would be
supplied by non-redundant power systems.
 The Tier 2 System
Single Path with Redundant Components (cont)

Single points of failure are present in

the distribution system, and critical
load interruptions are likely during
both routine maintenance activities
and unanticipated failures.
 Tier 2 Topology
Single Path w/ Redundant Components
 Single path
 With redundant components

N+1
TIA-942 /
Carriers Carriers Carriers Carriers

Tier 2 Customer Owned

Maintenance Hole
(Tier 1 and higher)
Customer Owned
Maintenance Hole
(Tier 2 and higher)

Technology
Spaces and Tier
2

Functional Entrance Room

(Tier 1 and higher)

Areas Offices, Operations

Center, Support
Rooms
Computer
Room
Main Dist Area
Telecom Room
(Tier 1 and higher)

Horiz Dist
Area
Horiz Dist Horiz Dist
Zone Dist Area Area Area

Equip Dist Equip Dist Equip Dist

Area Area Area
 The Tier 3 System
Concurrently Maintainable and Operable

Industry Description: Concurrently Maintainable and Operable

System Redundancy: N+1, as a minimum

UPS Source Available to the Critical Load: One active and
one passive or not on UPS power.

Ability to Be Maintained While Under Load: Yes, with a
reduction of the system redundancy from N+1 or better to N
during maintenance activities.

Ability to Recover from Failures: At the plant and distribution
level, but with a reduction of the system or distribution
redundancy from N+1 or better to N after the failure and prior
to the recovery.

Resulting Definition: Multiple Source / “N” Rated Single or

Multi Module System / Dual or Multiple Path

 The Tier 3 System
Concurrently Maintainable and Operable (cont)

The Tier 3 system possesses redundancy in the

power paths to the critical load, albeit with only one
of those paths being on UPS power. The individual
critical power systems are rated for a portion of the
total load, with a common and centralized
dedicated UPS system providing the “redundant”
supply to the “line” systems. The “redundant”
system, similar to the “line” systems may possess
either a single or multiple modules.
 The Tier 3 System
Concurrently Maintainable and Operable (cont)

This concurrently and maintainable system provides load

source selection either via static transfer switches or by
the internal power supplies in the IT systems
themselves. There would be no single points of failure in
either the critical power system or the power systems
supporting the mechanical or vital house/support loads.
The Tier 3 system allows for complete maintenance
during normal operations, but loses redundancy or
reducing redundancy to “n” when a given system or
power path fails or for selected failure or maintenance
modes of operations.

All maintenance and failure modes of operation are
transparent to the load
 Tier 3 Topology
Concurrent Maintenance & Operations

 Multi Module System

 Dual or Multiple Path
 Assured dual power path
 Loss of redundancy during
failure or maintenance
 Carriers Carriers Carriers Carriers

TIA-942 / Customer Owned

Maintenance Hole
Customer Owned
Maintenance Hole

Tier 3 (Tier 1 and higher) (Tier 2 and higher)

Tie
r 3

Technology

1
Tier

3
Tier
2
Tier

Spaces and Entrance Room

(Tier 1 and higher)
Secondary
Entrance Room
(Tier 3 and higher)
Functional
Areas Offices, Operations

1
Tier
Center, Support r3
Rooms
Tie Computer
Room
Main Dist Area
Telecom Room
(Tier 1 and higher)

Horiz Dist
Area
Horiz Dist Horiz Dist
Zone Dist Area Area Area

Equip Dist Equip Dist Equip Dist

Area Area Area
 The Tier 4 System
Fault Tolerant

Industry Description: Fault Tolerant

System Redundancy: 2N, 2(N+1), xN or any
variety of topologies in excess of N+1 meeting the
performance standard.

UPS Source Available to the Critical Load: Two or
more active.

Ability to Be Maintained While Under Load: Yes,
with a reduction no worse than N+1 during
maintenance activities.

Ability to Recover from Failures: Yes, with a
reduction no worse than N+1 during after the failure
and prior to the recovery.

Resulting Definition: Multiple path, with redundant
components and systems and dual source critical
power – Multiple Source or Sources / Multiple “N” or
Better than “N” - Rated Single or Multi Module
System / Dual or Multiple Path.
 The Tier 4 System
Fault Tolerant (cont)

This system possesses redundancy in the power

paths, and there may be more than two
independent sources of UPS power to the critical
load. The individual critical power systems are
rated for the complete load for the 2N / System-
Plus-System option or multiple UPS systems for
larger loads, where the system diversity is
undertaken solely by the connection of the critical
loads to the various / numerous UPS systems.
 The Tier 4 System
Fault Tolerant (cont)

The UPS system would consist of paralleled UPS

module or single / high-kW rotary UPS systems. The
Fault Tolerant system provides load source selection
either via static transfer switches or by the internal power
supplies in the IT systems themselves. There would be
no single points of failure in either the critical power
system or the power systems supporting the mechanical
or vital house/support loads. The Tier 4 system allows
for complete maintenance during normal operations, and
does not lose redundancy during either failure or
maintenance modes of operations.

All maintenance and failure modes of operation are
transparent to the load.
 Tier 4 Topology
Fault Tolerant
 Multiple path
 Redundant components
 Assured dual source critical power
 No loss of redundancy during failure or maintenance

xN: Multi-System 2N: System+System

 Carriers Carriers Carriers Carriers

TIA-942 / Customer Owned Customer Owned

Tier 4
Maintenance Hole Maintenance Hole
(Tier 1 and higher) (Tier 2 and higher)

Tie
Technology r 3

Tier 1

Tier 3
2
Tier
Spaces and Entrance Room
Secondary
Entrance Room
Functional (Tier 1 and higher)
(Tier 3 and higher)

Tie
Areas r 4

Tier 1

Tier 4
Offices, Operations
Center, Support r 3
Rooms
Tie Computer
Room
Secondary Dist
Main Dist Area
Telecom Room Tier 4 Area
(Tier 1 and higher)
(Tier 4)

Horiz Dist
Area
Horiz Dist Horiz Dist
Zone Dist Area Area Area

Equip Dist Equip Dist Equip Dist

Area Area Area
Questions & Answers