Professional Documents
Culture Documents
NETWORK CABLING
BEST PRACTICES TO MEET THE
DEMANDS OF THE DIGITAL ECONOMY
SPONSORED BY
DATA CENTRE
NETWORK CABLING
CONTENTS
4 INTRODUCTION
5 WHY BEST PRACTICE?
7 THE IMPORTANCE OF
REDUCING CABLING RISK
8 DATA CENTRE NETWORK CABLING BEST
PRACTICE THE KEY ACTIVITIES
9 THE FIVE STEPS
9 PLANNING
15 MAPPING
19 PROCUREMENT
25 INSTALLATION
28 MANAGEMENT
30 WHAT TO DO NEXT
32 BIBLIOGRAPHY
2
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
FIGURE INDEX
Page 5 Page 21
Figure 1: Perception of How Inadequate Networking Will Figure 11: IEEE Technical Specifications for Copper Twisted
Impact Data Centres (2015 to 2016) Pair Cabling
Page 6 Page 22
Figure 2: The Deming Quality Improvement Cycle Stages Figure 12: Technical Specifications of Standards for
Multimode and Single Mode Optical Fibre Cabling
Page 9
Figure 3: How the Network Row Can Be Positioned Closer Page 23
to Server Rows Figure 13: How Data Centre Building Spaces Differ in their
Network Cabling Requirements
Page 11
Figure 4: The Different Spaces of a Data Centre Building Page 24
Figure 14: Reach and Speed Specifications of Copper and
Page 12 Fibre Network Cabling
Figure 5: The Cabling Categories That Cover A Data Center
Figure 15: Copper Network Cabling Usage in Different Local
Page 14 Distribution Connections
Figure 6: A Data Centres Network Cabling Requirement
Check List Page 26
Figure 16: A Typical Ultra-High Density Fibre Panel
Page 16
Figure 7: The Three-Tier Network Topology Page 28
Figure 17: Network Cabling Used to Enable a Range of
Page 17 Services
Figure 8: A Network Topology Updated for East-West Data
Centre Traffic Page 29
Figure 18: Examples of Secure Lock to be used on Copper or
Page 18 Fibre Patch Cords
Figure 9: Different Connection Configurations at the Local
Distribution Level Page 31
Figure 19: Nexans Delivery Capabilities
Page 20
Figure 10: The CENELEC EN 50600 Telecom Cabling Series
Standards
3
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
INTRODUCTION
As the importance of data centres grows, so too does the need for high-quality networks in
order for the demands of the digital economy to be met. The modern data centre relies on
quick and efficient exchanges of information, to and from customers and between the active
IT components it houses. As such, growth in the deployment of virtualization and of cloud
systems within the data centre mean that low latency, adequate bandwidth, connections and
ports are as important as the adequate provision of power and cooling.
The need for proper decision making and management in relation to network equipment,
including cabling, is driven by data centre operational requirements. As with the
equipment it connects, cabling should be modular, scalable and easily adaptable. It is able
to transmit more data, faster and more accurately without an equivalent increase in the
power or space required to do so. Improved design and management practices mean
less time and labour are spent maintaining, repairing or installing cabling. More than ever,
planning, installing and maintaining network cabling to standards of best practice gives
operators a greater chance of chaos avoidance in the data centre.
Data centres typically contain many kilometres of network cabling in addition to thousands
of connections and ports that can be affected by physical stress, electromagnetic noise,
poor installation and connection, heat or humidity. Without knowing fully the location and
role of each length of cabling, its properties, what it is meant to be connected to, and
without some prior planning to replace the faulty length then a temporary interruption to
communication within data centre operations can become a larger problem.
Therefore, the rules of best practice in relation to data centre network cabling will be
based on a number of key steps:
4
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
The key purpose of the modern data centre is to house and transmit data between active
IT equipment servers and storage and to and from users outside the data centre. The
digitisation of business and society has created an exponential growth in data traffic with
consequent pressure on data centre network requirements. According to Ciscos Global
Cloud Index (2015), global data centre IP traffic will grow at a CAGR of 25% between 2014
and 2019 and data centre workloads within traditional (as opposed to cloud) data centres
will more than double (1).
The importance of networks to the data centre means a greater management focus on
how well the networks underpinning the data centre will meet its requirements. A majority
of data centre owners and operators in 2015 (58.6%) expressed concern about the
threat that inadequate data centre networks pose to their data centre operations (2). This
positions the network only slightly behind operational costs and the shortage of suitably
skilled staff as a major source of concern for operators:
Figure 1: Perception of How Inadequate Networking Will Impact Data Centres (2015 to 2016)
5
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
The convergence of servers, storage and network systems in the The principle behind Quality Improvement Cycle (QIC) is to review
data centre further creates an impetus for scalability and flexibility your data centre networks to plan how they can be improved, and,
in cabling deployment. The growth of hybrid cloud systems and as part of that, to define the requirements/targets that need to be
virtualisation in meeting corporate met. The do stage will map the
IT requirements represents another network design to the requirements
pressure on data centre network THE FAILURE TO FOLLOW BEST and configuration of the data
cabling since, at 2015, the DCD PRACTICE CAN ALSO COST MONEY centre and to the network solutions
Intelligence Census indicated that THROUGH THE FAILURE TO RECOGNISE available for the task which will then
17% of global footprint is located AND CAPITALISE ON OPPORTUNITIES be installed into the data centre.
in virtualised or on-premise cloud FOR BETTER DATA CENTRE OPERATION. Checking includes the testing of
environments (3). These internal equipment as well as ensuring
systems require more complex, that it continues to meet network
higher specification and lower latency network systems. requirements. Further action may be required to add, replace or
modify equipment to ensure that requirements continue to be met.
These changes will impact how the requirements of data centre
network cabling will need to evolve in the future. Only the As with any data centre component, the case for best practice can be
application of best practice and the availability of new cutting-edge viewed in terms of reducing the risk to operation and performance,
network cabling solutions will enable the modern data centre to and as the opportunity to develop improvements to both:
deliver on its networking requirements.
It reduces the risk of downtime/interruption to availability
Any best practice model such as the Deming Cycle needs to and the associated costs of this income if a data centre is
be continuous when used to support the availability, efficiency used for selling or associated revenue generation activity, the
and performance requirements of a data centre. Even the best damage to reputation, the damage also to company morale
designed network system will fail if it is not properly installed, and the costs of restoration
maintained or managed. The process starts pre-commissioning and The failure to follow best practice can also cost money
ends only once de-commissioning is finished. through the failure to recognise and capitalise on
opportunities for better data centre operation through
reduced operating costs, better use of space, or the reduced
Figure 2: The Deming Quality Improvement Cycle Stages need to build or extend the data centre.
PLAN
CONTINUAL
ACT DO
IMPROVEMENT
CHECK
6
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
THE IMPORTANCE
OF REDUCING
CABLING RISK
The analysis and reporting of major data centre outages indicate that network cabling
is not often the cause of a major data centre outage or shutdown. Data centre outages
or shutdowns are typically a result of power distribution, power protection failures or of
thermal events caused by IT overload and/or malfunctioning cooling systems. However,
failures of network cabling equipment can have a negative impact, as demonstrated in
2013 by reports of a poorly designed switch that, when inserted, hit the reset button on
the back of the equipment, wiping data when plugged in. The Uptime Institute notes,
incidents that may threaten the operation of the data
centre do not always result in outages or shutdowns (4).
However, a number of sources suggest that network
cabling accounts for a high proportion of such incidents.
CABLING RUNS THROUGHOUT THE
Emerson/Ponemon Institute studies quoted by the Uptime DATA CENTRE, WHICH MEANS IF IT IS
Institute (5) indicate that incidents occur daily which POORLY INSTALLED OR MANAGED, IT
means that if the proportion of incidents indicated by DCD CAN BE A NUISANCE OR AN IMPEDIMENT
Intelligence research is followed, the identification and TO PROGRESS.
correction of network cabling system incidents happens
at least weekly in most data centres (6).
Cabling runs throughout the data centre, which means if it is poorly installed or managed,
it can be a nuisance or an impediment to progress. If there is network failure in the data
centre and it is not possible to quickly identify and access what needs to be replaced,
then additional time and effort will be required to do this. If the cabling is not readily
accessible because of poor installation then adjacent sections of the data centre
will need to be shut down while staff or contractors go hunting for the rogue link or
connection. In this situation it should be remembered that a considerable proportion of
unplanned downtime can be linked to planned downtime that overruns or creates new
problems in the data centre. Using a process of trial and error to identify the culpable link
or connection has the potential to add to the initial problem.
7
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
The key stages in a best practice program for data centre network cabling are:
1
In order to establish the network requirements of
Planning the data centre in order that the deployment is
consistent with current needs
2
objectives established through planning, through
Mapping mapping the requirement onto the actual data
centre using the most suitable network topology
design
3 Procurement
network in terms of density, speed, reliability,
efficiency, service and support, return on
investment etc.
4
Since a number of the problems and inefficiencies
Installation associated with cabling in the data centre can be
traced back to poor installation practices
5 Management
as its management and operation, particularly
when it needs to bridge a period of significant
technological change.
8
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
STEP 1 OF 5
PLANNING
THE CRITICALITY OF NETWORK
AND CABLING PLANNING
The planning stage essentially will specify the requirement for cabling and other network
equipment in the data centre. This will then be used in the decision-making processes for
equipment and for the network topography which will provide the blueprint as to how the
equipment is distributed through the data centre.
The planning stage should be adopted across all types of major data centre projects
new build, modular deployments, facility extension and refit/refresh projects. Each of
these different types of data centre investment will present different opportunities and
limitations in terms of network design. A greenfield new build will enable greater latitude
in the design and installation of cabling infrastructure as it will not need to take into
account existing facility or network limitations. New build will also allow the servers and
storage racks to be configured to ensure that they are sufficiently close to network racks
to minimise transmission disruption across the link.
Figure 3: How the Network Row Can Be Positioned Closer to Server Rows
9
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
In contrast, a migration project may simply mean disconnecting groups that design, deploy, plan or oversee IT traffic that passes
the network equipment in one facility and plugging it back in at through data centre networks. The responsibility for network
the new facility. According to DCD Intelligence estimates, the cabling has traditionally fallen between IT and facilities personnel.
majority of networking equipment investment is spent on existing It may also now involve personnel with specific responsibility
data centres (7), most usually in situations where the IT equipment for the network, for cloud or for IT operations. The increased
is in need of an upgrade or where space and/or capacity is being criticality of the network to the data centre should be reflected in
added to the data centre. the greater degree of responsibility taken by all parties to whom
its performance is relevant.
In this context the use of higher-specification
equipment and cabling cannot simply be added This whole of data centre team
into a data centre as it will change the entire THE STRATEGY FOR BEST approach will make it easier also to
network topography of the data centre. If establish and review performance
PRACTICE IN CABLING NEEDS
introducing new network infrastructure then this criteria for the reliability of the
IDEALLY TO BE PART OF THAT
requires a detailed inventory of installations and network and its infrastructure in
what is needed when cabling type, quantities,
WHICH COVERS THE WHOLE addition to (i.e. MTBF (Mean Time
connections, panels, ports, equipment
DATA CENTRE. Between Failure), utilisation, meeting
specification and speed requirements, patching company needs, cost of operation in
and housing are considered. Those involved addition to other factors.
in the planning process need also to consider
for future network cabling requirements network cabling may As network cabling systems are complex and new innovations
have a life of 10 years or more. Therefore the future planning of in design, manufacturing and installation are introduced, many
network cabling is important so that it is able to accommodate companies will enlist network and cabling experts to advise on
the two or more generations of active IT equipment that may be how best to plan their network cabling.
deployed over that time. The introduction of new network units
into an existing data centre will also need to look at any particular
design and installation considerations limitations on space use,
airflow requirements, the use of under floor or overhead supply
systems and current practices in using Top of Rack, Middle of
Rack or End of Row configurations.
10
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
HOW TO PLAN WELL cables are narrower, the length at which signal integrity can be
preserved is longer in addition to lower rates of interference from
What should be the basis of establishing a best practice plan, and electromagnetic noise and greater capacity to integrate within
on what basis should the decisions relating to it be made? Each structured systems.
company will have its own priorities but the underlying need is to
ensure that all components of the network including cables, cords, Therefore it is important also to establish the length of time for
switches, connectors, panels, housing, cards, routers, nodes are which the best practice strategy is designed to be effective
configured to connect the companys equipment within the data although one of the defining characteristics of best practice is
centre and to provide links outside the data centre in order that that regular reviews will identify further needs for change and
the companys IT objectives can be met. improvement.
Other factors that need to be considered as part of the planning THE IMPORTANCE OF THINKING
stage will include:
BEYOND THE WHITE SPACE
How the IT capacity of the data centre may need to increase Most data centres and server rooms are located within buildings
and also how the IT load may become more variable and that also house other commercial or industrial activities rather
unpredictable due to the demands of convergence, big data than in dedicated purpose-built structures of their own. This
etc. These changes will mean that the network will need means it may be good practice to plan for the whole building
to be flexible and scalable in line with the evolving data rather than just the data centre or server room space.
transmission requirements of the data centre.
The deployment of cloud and virtualization. Few data centres Usually, the space within the building can be divided into
in established markets only house physical racks. Server three areas:
virtualization has been a mainstream technology for a number
of years and has been followed by virtualization of users and
application. End-user data centres have also become the
1 white space which houses the IT equipment
in a data centre or server room
location for private and hybrid cloud systems as companies the technical space which houses the electrical,
2
prefer cloud options that allow them to keep an eye on their mechanical, gensets and power protection equipment
cloud footprint. The move to software-defined data centres in addition to the other technical systems the building
and networks, however, is relatively new but it will affect requires to function
the type of IT equipment found in data centres and the
connection requirements in future. As such, planning needs
to examine how the data centres mix of physical servers,
3 the office or control space which houses staff
and the equipment needed for their work
11
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
The technical space has low density requirements but the need
Figure 4: The Different Spaces of a Data Centre Building for its systems to reach throughout the building based on its role
as a monitoring and control centre for supporting the data centre.
The office space requires higher speed connections to connect
its commercial equipment, to enable its telecommunication/VOIP/
PoE systems, wireless networks and cabling between equipment
units. The computer room/data centre requires connections to
Office Space / Control Room Office Space / Control Room Office Space / Control Room
Office Space / Control Room and from the outside world as well as connections between
Office Space / Control Room
active IT server and storage equipment. For switch to switch links
Office Space / Control Room it requires medium reach and ultra-high bit-rate connections; for
switch-to-server and switch-to-storage connections, short reach
Office Space / Control Room
and very high bit-rates are required.
Computer Room Computer Room Technical Space Computer Room Technical Space Technical Space
(White Space) (White Space) (White Space)
Fibre cabling
Copper cabling
12
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
WORKING OUT THE NETWORK REQUIREMENT December, on-line sports coverage on weekend afternoons,
for example) or due to traffic connecting across a converged
It is critical to the process of planning the data centres cabling data centre system. This mirrors the growing trend to look
requirement that some accurate means of estimating how much more at the maximum rack density rather than the average
network capacity and therefore how much cabling equipment will rack density in the data centre. For this reason, it may be
be required. Like the provision of other resources and equipment sound strategy to divide the data centre into network zones,
into the data centre this is a component of capacity planning where certain racks of servers have the necessary capacity to
that will look to maintain the optimum position between under- deal with higher demand or to be able to network to virtual or
provisioning where there is insufficient network capacity for the cloud systems to cope with additional IT requirements.
data centre to do the work it is required to, and over-provisioning The requirement for network capacity needs should also
where relatively little of the network infrastructure is used in the take account of redundancy in the network, although on a
data centres operation. day-to-day basis this will follow the overall redundancy of the
data centre.
As with other data centre metrics, the method of calculating the Capacity estimation here needs also to factor in the number
network infrastructure requirement has become more complex of channels running through the network, the type of cabling
over time as it reflects the larger number of variables that may and its capacity as well as growth factors based on projected
impact on the calculation. The estimate of requirement on the usage growth.
basis of space is now largely little more than an initial ballpark
step since that was appropriate for an era of largely predictable The requirements for networking equipment is an exercise in
and low density IT loads. Some of the factors that need to be numbers and based on the numbers and the specifications of the
considered when working out requirement are: key server and storage equipment that will need to be connected
as well as the network distribution sub-systems that will link the
The IT capacity of the equipment the network will connect equipment through and within the data centre. An example of
and how this is projected to change. how such information may be collected and organised is shown in
Many data centre owners and operators are looking at the Figure 6. The key information sets include the numbers of server,
peak requirements that their network will have to carry rather storage and network rows and racks and of distribution areas and
than an average requirement across the facility. This is a patching frames as well as their height and width, and the number
function of an IT load that is more variable and unpredictable of connections at these points in the network.
either as a result of variable customer demand (on-line retail in
13
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
1 If available please provide the drawings of the DC layout Number of network racks per row maybe positioned at end of server
23
rows
If available please provide the drawings of the cable trays layout
Cable tray configuration 24 Number of rows of network racks
2
Trays located above the racks or under the raised floor? 25 Location of network racks in the computer room floor
Distance from tray to cabinet
26 Height (U numbers) of network racks
3 If available please provide the networking diagram
27 Width and depth of the network racks
4 Type of Data Centre e.g. Corporate, Colocation
28 Switches model and part number
Chapter IV B / Server racks data collection
29 Number and location of the switches
5 Number of server racks per row
30 Number of Ethernet connections to patching frame per rack
6 Number of rows of server racks
31 Number of fibre uplinks to patching frame per rack
7 Location of server racks in the computer room floor
32 Class of copper cabling if specified
8 Height (U numbers) of server racks
33 Grade of fibre cable and connectivity if specified
9 Width and depth of the server racks
Chapter IV E / Distribution area / patching frames
10 Number of copper connections per rack Note: The following data can be calculated by Nexans if all data requested
here above are provided
11 Number of fibre connections per rack
34 Number of copper cabling racks per distribution frame
12 Class of copper cabling if specified e.g. CAT6A, CAT7A
35 Number of fibre channel connections per distribution frame
Grade of fibre cable and connectivity if specified
13 36 Number of distribution frames
e.g. OM3-LC, OM3-SC etc.
Chapter IV C / Storage (SAN) racks data collection 37 Location of the distribution frames in the computer room floor
14 Number of storage racks per row for SAN fabric switches 38 Height (U numbers) of the distribution frames
15 Number of storage rows for SAN fabric switches 39 Width and depth of the distribution frame
16 Location of Storage racks in the computer room floor Number of copper cables to each rack (should correspond with
40
combined number in each server and network rack)
17 Height (U numbers) of storage racks
Number of fibre connections to each rack (should correspond with
41
18 Width and depth of the storage racks combined number in each server and network rack)
20 Number of fibre connections per rack 43 Grade of fibre cable and connectivity if specified
22 Grade of fibre cable and connectivity if specified 46 A. LANsense Intelligent Infrastructure Management
14
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
STEP 2 OF 5
MAPPING
MAPPING DATA
CENTRE NETWORK REQUIREMENTS
There are two main parts to the purchasing decision process to ensure the network
meets the requirements of the data centre:
MAPPING:
Designing a topography that maps the network architecture from the point(s)
1 at which it enters the data centre building to the point at which it connects with
the server or storage unit and which also allows connection between active IT
equipment.
PROCUREMENT:
2 Deciding on the cabling and networking equipment that will be deployed in the
data centre.
The mapping of the most suitable distribution structure and network architecture will be
based on the following mapping components:
Zones or areas the main distribution cabling sub-system which will be the
central area within the network topography as well as the access point of the
The switching and network topology (e.g. legacy, fat tree, DCell, BCube,
2 Scafadia) paying particular attention to the switch to switch and switch to
server/switch to storage interfaces
The provision of services to the equipment via Middle of Row (MOR), End of
3 Row (EOR) or Top of Rack (TOR) and the use of structured as opposed to point
to point cabling systems.
15
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
There are a number of factors that may influence the design of THE EVOLUTION OF NETWORK TOPOLOGIES
the network architecture as part of broader cabling planning,
including: The topologies that the network architecture has traditionally followed
have been hierarchical, and based around 3 layers of network
Scalability that additional data can be carried across the switches connecting servers via access switches to aggregate layer
network whether on a short-term load-peak basis or as switches which are then connected to each other (north-south) by
part of a broader data centre capacity expansion. As noted core layer switches. These also connect to the Internet. The hierarchy
previously, networking equipment is so pervasive in data (or tree) may not be scalable enough to deal with cloud computing
centres that such scalability needs to be easily enabled. and the design may suffer congestion at its higher levels. Therefore,
Designing according to physical constraints of the cabling this topology has evolved through, for example, its sub-division into
link transmission specification, physical flexibility and pods to enable more complex switching arrangements, greater
strength, minimization of interference in order to increase bandwidth and reduced network congestion.
robustness and to prevent any major interruption to
transmission Server-centric building block topologies such as DCell have
The bandwidth and latency requirements of the equipment been developed to offer greater scalability. There are additionally
within the data centre and the service levels expected of topologies based on cube structures, ring-shaped configurations
it. Thus, the numbers of servers that are virtualised will and more amorphous/free form topologies.
increase the complexity of traffic through the data centre
and in particular this will influence the network topology
for switching and routing. The growing emergence of the
Figure 7: The Three-Tier Network Topology
data centre fabric which can act as a unified system of
connections between switches and equipment in disparate CORE
locations across the data centre has acted as one solution
to this potential complexity. Yet as with any service that is Backbone
16
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
Three-layer designs are being execute the data transfer, causing the
superseded by leaf/spine THREE-LAYER DESIGNS ARE potential for congestion there. Leaf/
topologies. In part, this change BEING SUPERSEDED BY LEAF/ spine adds switches at the layer below
is based on the trend in network SPINE TOPOLOGIES. IN PART, THIS the access layer and then spread a
traffic patterns from north-south CHANGE IS BASED ON THE TREND spine of switches across rather than up
(in and out of the data centre IN NETWORK TRAFFIC PATTERNS the network. Leaf/spine is a standard
usually between hosts by the most for Ethernet fabric designs as it creates
FROM NORTH-SOUTH (IN AND OUT
direct route) to east-west traffic less distance and switching for hosts to
OF THE DATA CENTRE USUALLY
which travels across network communicate east-west, greater flexibility
interconnection points within the
BETWEEN HOSTS BY THE MOST in switching protocols than is possible
data centre. In particular this is DIRECT ROUTE) TO EAST-WEST with chassis switching systems. However,
a consequence of convergence TRAFFIC WHICH TRAVELS ACROSS this flexibility requires a higher number of
(servers talking to each other and NETWORK INTERCONNECTION switches and a greater amount of cabling.
to storage units) and the need POINTS WITHIN THE DATA CENTRE.
therefore for traffic between Each of the topologies described
servers/apps and storage, and of briefly here, and others, have positive
virtualization and cloud systems which move workloads via virtual characteristics and drawbacks. Decisions as to which to use will
machines between hosts within segments in the data centre. The depend upon the priorities assigned to the network within the data
increase in crossways east-west traffic within the data centre centre according to corporate IT priorities, and the extent to which
means that it may have to move up to the aggregation layer to the data centre requires scalability and low latency in its operation.
INTERCONNECTION INTERCONNECTION
SWITCH SWITCH
SERVER SERVER SERVER SERVER SERVER ACCESS SERVER SERVER SERVER SERVER
CABINET CABINET CABINET CABINET CABINET SWITCHES CABINET CABINET CABINET CABINET
17
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
IEEE studies published in March and July 2014 (9), and a CCCR
White Paper (10), although based on different cost comparison End of Row
criteria, all indicate cost savings of EOR over TOR connection.
This is based on the further level of connectors required at the
top of the rack for a TOR connection, the need of EOR connection
for fewer port aggregation switches and uplink ports, and savings
on redundant power supply. The structured cabling/EOR option
also achieves a higher level of switch port utilisation. Based on
the assumption of a 39 cabinet data centre at an average of 5kW-
6kW per cabinet and 14 servers per cabinet, the total costs are
worked out as $4,227,000 for ToR access and $2,075,000 for a
structured cabling system (10).
Network Row
Comparatively, ToR is perceived to offer the advantages of fewer
cables located under the floor, cost saving on horizontal copper
cabling and suitability for high density blade servers. EoR also
offers lower latency, a better control over oversubscription and
lower TCO than ToR.
Network Row
18
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
STEP 3 OF 5
CABLING
PROCUREMENT
DECIDING ON CABLING AND NETWORK EQUIPMENT
The decisions to be made in relation to the choice of the network equipment to be fitted
in the data centre need to consider a number of different factors:
19
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
THE IMPORTANCE OF STANDARDS Where there is no resilience and high risk such as
Class 1 with direct point to point (unstructured) cabling
Cabling is an equipment category that has developed subject to a raft
Where there is no resilience but fixed cabling
of standards that variously define the performance and capabilities of Class 2 systems are deployed
both copper and fibre cabling as well as offering guidelines as to best
practice in design, installation and management. Concurrently maintainable based on multi-path
Class 3 cabling systems
Of particular relevance are the new CENELEC EN 50600 series
standards which will impact on data centres and buildings that Concurrently maintainable based on diverse routed
house data centres. The series is separated into practices in Class 4 topologies. This is the lowest risk (but highest cost)
terms of building construction, power distribution, environmental proposition.
control, telecommunications cabling and security systems,
as well as the management and operational systems that The EN 50600 standard is important since it creates a Class
oversee all of these. It specifies a classification based around system for redundancy in network cabling. This applies a standard
risks including availability, security and energy efficiency. Four to the network topology not just to the component products
telecommunication cabling classes have been developed: (cables, switches, connectors) where the standards have largely
been concerned with performance, technical specification and
not risk.
Environmental Control No
No Concurrently Fault Tolerant
LOW MEDIUM Resilience Maintainable
EN 50600-2-3 ECS N+1 2(N+1)
(Single (Active/Passive)
Path)
Highest Risk
Highest Cost
Higher Cost
Higher Cost
Lowest Cost
20
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
21
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
The same principles of balancing transmission capacity and speed The gradation from OM3 through OM4 has occurred to increase
requirements against the distance at which the signal will maintain the modal bandwidth to provide extended distance support for
integrity are applied also to the specification of fibre cabling bandwidth limited applications. The proposed move to OM5 is
(shown on the grid below). Loss of signal between the near and designed to increase the rate of transmission and provide higher
far end of the line applies also to fibre cabling but depends on the data throughput.
transmission speed, the distance and the number of connections in
the link. Modal bandwidth indicates the capacity of multimode fibre
cabling to transmit a signal at a maximum rate over a given distance.
Figure 12: Supported Distances for Applications on Singlemode and Multimode Optical Fibre Cabling
APPLICATIONS AND PERFORMANCE
Nexans LANmark-OF OM4 MTP fibre systems are warranted to support following applications.
Guaranteed distances vary depending on application and number of connections.
# MTP-L-C 2 4 6 8
# MTP CONNECTIONS 2 4 6 8
# MTP-LC MODULES 2 4 6 8
22
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
COPPER, FIBRE OR BOTH? The choice between copper and fibre cabling is one of the most
debated in the data centre industry; the decision on which is more
The key cabling profiles of copper and fibre shown previously suitable will depend on a number of considerations. However, the
can be deployed to meet the different networking needs of the different technical characteristics and physical tolerances of each
different spaces in a data centre building. type, particularly in terms of the distance at which the link can
maintain integrity of transmission tends to define the decision.
The first requirement is to understand the bandwidth Fibre offers transmission over longer distances within and outside
requirements of each zone. This will depend upon the density the data centre particularly in single- rather than multi-mode. It
of the servers and the networking equipment within the data is seen to offer greater bandwidth and data rates, immunity to
centre, and in other areas of the building. This will determine the electromagnetic interference, being more space efficient, lighter
category of copper or fibre cabling required. It indicates also the and generally quicker to install.
maximum length of the cable that will transmit at the specified
bandwidth. Wireless links will be an option in the office or control Copper has been preferred within the data centre as the server
space where the amount of traffic is far lower than in the data link between the equipment and the access switching. Copper is
centre or server room. The development of wireless transmission used to connect the switch to the server and the network rows to
within and from the data centre is still very much at an early stage. the server rows as part of Middle of Row and End of Row network
Wireless technologies potentially offer the capability of short row connections. In this role it is used to link up to 30 metres
bursts of added network capacity at peak periods, and they can across server rows.
be switched on and off as required. (11) (12) This indicates the
importance of a strategy that differentiates between the different
requirements of different zones within a data centre building.
Figure 13: How Data Centre Building Spaces Differ in their Network Cabling Requirements
COMPUTER ROOM SPACE
SMART OFFICE SPACE OTHER SPACES
CHOICES
SWITCH TO SWITCH SWITCH TO SERVER
Low High Very High Low High Very High Low High Low High
Bandwidth
<10Gbs-1 <25Gbs-1 <40Gbs-1 <100Gbs-1 <10Gbs-1 <25Gbs-1 <40Gbs-1 <100Gbs-1 <1Gbs-1 <10Gbs-1 <10Gbs-1 <1Gbs-1 <10Gbs-1
LANmark LANmark LANmark LANmark LANmark LANmark LANmark LANmark LANmark LANmark LANmark
Copper
6A 7A 8 6A 7A 8 6 6A 7A 6 6A
Distance
100m 30m 100m 30m 100m 100m
(Copper)
Distance
350 85 300 85 350 85 300 85 900 340 340 900 340
OM3
Distance
550 120 400 120 550 120 400 120 930 520 520 930 520
OM4
Other issues
Roll-Out
Low High Very High Low High Very High Low High Low High
Speed
Roll-Out On-Site Plug and On-Site Plug and On-Site Plug and
Pre-Term Pre-Term Pre-Term Pre-Term
Technique Term Play Term Play Term Play
23
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
Copper can also be considered for End of Row uplinks where the
Building-based
distance is less than 30 metres. Using Category 8 as an upgrade 1Km
MUL
TIMO
Reach
path means switches can be upgraded in steps from 1G to 40G DE F
IBRE
without re-cabling and the number of switch ports can be reduced
Room-based
100m COPPER TWISTED PAIR
by a factor of 10. At 40G this can lower energy use and costs by as
much as 50% and quadruple link speed. The cost of EoR access
Row-based
using 40GBase-T (over copper cabling) is around one-third of that 30m
of EoR with 40GBase-SR (over fibre cabling), and lower again than COPPER Rack-based
TWINAX
ToR with 40GBase-T and ToR with DAC. Transceivers account 5m
for 60% to 70% of the costs of the latter three options, and the 10M 100M 1G 10G 100G
equipment costs are similar in all four options. The advantages in
Speed
this usage situation is that a Base-T Ethernet approach reduces the Source: Alan Flatman
cost of energy; transceivers are less expensive, the components
are high volume and relatively easy to manufacture. It is easy to
install using field termination and is available in pre-terminated
Figure 15: Copper Network Cabling Usage in Different Local
Distribution Connections
variances, backward compatibility and plug and play and the reach
will be adequate to a maximum of 100 metres for bandwidth up to
Overhead Row-to-Row/ Overhead In-Row Trunks
10Gb/s or 30 metres for bandwidth of more than 10Gb/s. Pod-to-Pod Trunks Generally Copper
Either Copper or Fibre depending Structured Cabling
In the modern data centre, the need to prepare and support on distance Structured cabling
several generations of active equipment mean the specification
and standard of the cabling and network equipment needs to plan
for future networking requirements. Some provision in the form of
spare connection points, patch panels, links and ports or through
devices that can scale quickly to a future requirement has to be
built into the topology.
24
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
STEP 4 OF 5
INSTALLATION
BEST NETWORK CABLING AND
EQUIPMENT INSTALLATION PRACTICES
The use of sub-contractors who are fully accredited and trained to the necessary
installation standards (e.g. EN 50174) and who are also conversant with the
specific needs and conditions of the individual data centre. The sub-contractors
3 should be in full contact with the manufacturer of the equipment, to ensure
the installation process is as seamless as possible. It is critical the process is
conducted according to a Statement of Work which is agreed and signed off
by all parties.
Proper expectations established with clients about the length of time required
for the installation process including testing, and proper preparation of the site
4 beforehand. While in many data centres planned downtime costs availability
and therefore money, the need to install fast may put undue pressure on the
people and lead to unwanted outcomes.
25
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
The key requirement of cabling installation is that it should be they can be installed quickly and easily through pre-installation
quick and easy, fully organised and accurate in its outcomes. The design rather than extensive on-site work. These solutions include:
process needs also to conform to the conditions of equipment
supplier warranties. A well-managed installation process will be Cassette format with neatly packaged fan-outs to avoid
conducted by specialist contractors who are aware of the physical tangled connections
limitations of cabling such as bend stress and who will avoid over- Pre-terminated cable assemblies
bundling or crushing cables as well as being fully updated on local Cables delivered at individually specified lengths
space, access and air flow requirements. The installation process Housing or locking as security is important even in a location
should also account for future cabling installations and also avoid with very limited or no public access such as a data centre.
installation of cables in hot areas or close to power cabling.
A number of particular innovations relate to the installation
One of the key aims of cabling installation is to standardise the of fibre cabling:
process for cables, switches, panels, cabling management as
well as for labelling, colours and documentation. This process A specialist containment system will protect fibre against
should also establish the pathways and conduits through the data damage
centre, the design and installation of connections between the Pre-installed MTP-LC modules. These can save as much
sequential distribution systems within the overall system (main to as 70% of time required on site compared to traditional
intermediate, intermediate to local). Largely, the standardisation termination methods
should be established at the design stage so that the work on- Patch panels that allow for modules and enable fingertip
site is a matter only of execution rather than making on-the-job access to patch cords in separate trays
decisions as to how to progress. Like any other work conducted Colour matching of fibre pair-flips.
in the data centre, the process needs to conform also to
occupational health and safety requirements. The manufacturer will also be able to recommend on the stocking
of spare cabling products. As installation progresses, remove old or
Recent innovations by network equipment designers and unwanted cabling and equipment from the data centre to reduce
manufacturers have created products that can assist in the the fire load and avoid the possibility that it might cause confusion
installation process. Ideally, solutions need to be designed so that in the future.
26
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
27
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
STEP 5 OF 5
MANAGEMENT
BEST DATA CENTRE NETWORK
CABLING MANAGEMENT PRACTICES
Among the key best practice components in data centre cabling management are
documentation and inventory control that can be updated with changes to the network
or to related equipment. Ideally, these components will allow relevant information to be
shared as necessary with staff or contractors. It will be important also to establish protocols
as to dealing with the network system equipment and its equipment once standardisation
is established as well as to how the information collected should be used, shared and
acted on. These should cover installation, repair, maintenance, upgrades and recovery.
28
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
Figure 18: Examples of Secure Lock to be Used on Copper or Fibre Patch Cords
29
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
WHAT TO DO NEXT
In the field of LAN cabling systems, Nexans offers a complete range of products and
value-added services providing improved reliability and reduced cost of ownership for
data centres, offices and campus networks.
Nexans LANmark cabling systems allow you to optimally balance copper and fibre, in
line with the performance you need, for any number of ports and any bandwidth. High-
performance LANmark connectivity with minimised insertion loss allows for greater lengths
and more connections without sacrificing quality or reliability.
Our connectivity solutions exceed the requirements of all relevant standards, ensuring your
data centre performs better and more reliably. Whats more, Nexans expertise and far-reaching
involvement in the development of new protocols and standards for data centres and cabling
means we find the right solution every time.
Our design and manufacturing innovation allows us to offer solutions that meet and exceed
the requirements of todays cutting-edge data centres in terms of cost, space, installation and
energy efficiency and that help future-proof the data centre in terms of future connectivity
requirements with a minimum of disruption.
30
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
PLAN
Keeping you up-to-date on the latest innovations and Reducing Total
Cost of Ownership
helping plan for future changes.
DEFINE
Sharing our expertise in defining state-of-the-art
specifications and solutions.
Enabling Meeting higher
DESIGN sustainability bandwidth needs
Supporting you in designing robust, flexible
and scalable systems.
DEPLOY
Blueprinting infrastructure for cost-effective and accurate
installation, assisting on-site and ensuring warranties.
USE
Offering training, monitoring and fast repair, Anticipating Improving
technology change network reliability
replacement and third-level support.
31
www.dcd-intelligence.com
DATA CENTRE
NETWORK CABLING
BIBLIOGRAPHY
(1) Cisco Global Cloud Index (GCI): Forecast and Methodology 2014 2019;
October 2015.
(2) DCD Intelligence Global Census 2014-2015; findings previously unpublished.
(3) Global Data Center Market Overview and Forecasts 2014-2020; DCD Intelligence;
April 2015.
(4) Uptime Institute: Operations and Management Annual: October 2013.
(5) Data Center Outages: The Ponemon Institute and Emerson Network Power:
2013 Study on Data Center Outages.
(6) Data Center Research Group; December 2011.
(7) DCD Intelligence Global Census 2014-2015; findings previously unpublished.
(8) DCD Intelligence Global Census 2014-2015; findings previously unpublished.
(9) IEEE P802,3bq 40GBASE-T Task Force, presented Beijing China; March 2014.
(10) Communications Cable & Connectivity Association; quoted Todd Harpel, RCDD,
Marketing Committee Chair; BISCI.
(11) Project Genome: Wireless Sensor Network for Data Center Cooling;
The Architecture Journal: January 2009.
(12) Relieving hotspots in data center networks with wireless neighborways;
IEEE; December 2014.
32
www.dcd-intelligence.com
DatacenterDynamics (DCD), part of the DCD Group, is an
international B2B media and publishing company. It offers a
comprehensive range of products and services to assist senior
professionals in the worlds most ICT-dependent organizations in FOR MORE INFORMATION:
their risk-based infrastructure and capacity decisions.