DATACENTERS Electrical Design

ENERGY EFFICIENT DATACENTERS
ELECTRICAL DESIGN
Michael Ryan, Brett Rucker, Dean Nelson, Petr Vlasaty, Ramesh KV, Serena DeVito, and Brian Day Sun Global Lab & Datacenter Design Services Sun BluePrints Online
Part No 820-6213-10 Revision 1.0, 3/11/09
Sun Microsystems, Inc.
Table of Contents
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 About This Article . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Traditional Datacenter Electrical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Utility Power and Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Switchgear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Backup Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Rack Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Datacenter Tier Levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Suns Approach to Electrical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Increasing Electrical Distribution Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 Component Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Equipment Power Supplies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Power Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Appropriate Levels of Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Flexibility Through Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Datacenter Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Extending the Datacenter Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Electrical Yard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Building Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Pod Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Suns Electrical Design At Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Santa Clara, California . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Broomfield, Colorado . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Sun Modular Datacenter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 About The Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Ordering Sun Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Accessing Sun Documentation Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Introduction
Chapter 1
Introduction
Good datacenter electrical design supports increased energy efficiency, stability, longterm growth, and business agility. As the following examples indicate, electrical design has become paramount in this equation: Electrical design that supports growth gives datacenters a longer life. Many datacenters built even as recently as 510 years ago have reached a premature end of life because their infrastructure was overtaken by density, power, and cooling demands that were not planned for. Modular designs, from the electrical yard to server racks themselves, can support both rapid growth and rapid change, giving organizations the agility they need to adapt to changing business conditions. System efficiency can be increased while not comprising availability by careful system design and component selection, such as choosing high-efficiency transformers and uninterruptible power supplies (UPS). Appropriate use and implementation of redundancy can raise efficiency and reduce construction and operational cost over the life cycle of the datacenter. Everyone wants a Tier 4 datacenter, but there is inherent capital and operational inefficiency in having two complete distribution systems. Instead, a flexible and modular electrical design can support multiple Tier levels in the same datacenter, allowing the cost of the higher service level to be spent only on the systems requiring it. Metering at the right places in the electrical distribution system is a key feature thats needed in order to assess operating conditions of the system and components. Good metering design practices are imperative in the datacenter to continuously assess system heath datacenter efficiency. Metering should be implemented starting at the utility feeds and cascading through the system down to the rack level.
About This Article

This Sun BluePrints article discusses the electrical design principles that Suns Global Lab & Datacenter Design Services (GDS) organization incorporated into its energyefficient, pod-based datacenters. Suns GDS organization is responsible for Suns technical infrastructure (datacenters, labs, server and communication rooms and wiring closets) worldwide, and has ushered in a new generation of energy-efficient datacenters at Sun. Chapter 2, Traditional Datacenter Electrical Design on page 3, discusses how datacenter electrical systems are typically designed. Chapter 3, Increasing Electrical Distribution Efficiency on page 8, highlights the opportunities that we have found to improve modularity and efficiency in datacenter electrical design.
Introduction
Chapter 4, Flexibility Through Modularity on page 17, shows how our design choices create flexible datacenters that contribute to business agility. Chapter 5, Extending the Datacenter Lifecycle on page 21, discusses the importance of coordinating expected datacenter lifecycles with equipment refresh cycles. The chapter shows the many ways in which datacenters can be built from day one to support future expansion Chapter 6, Suns Electrical Design At Work on page 24, provides a tour of our datacenter designs from the perspective of the one-line diagrams that describe their electrical systems. We focus on Suns Santa Clara, California datacenters, including our most recent addition of Sun Modular Datacenters at the site.
Traditional Datacenter Electrical Design
Chapter 2

From an electrical design standpoint, a datacenter is a hierarchy of electrical devices that transmit power from a utility feed to server racks (Figure 1). One or more feeds arrive from the electrical utility before or after their voltages are transformed to usable levels. Switchgear provides, among other things, a disconnect point for the utility feed. Where needed, UPS and generators provide transient and longer-term backup power. Conduit and wireways feed distribution systems in the datacenters themselves. Power distribution mechanisms deliver electricity to racks and standalone systems. Within racks, smaller distribution systems provide power to individual servers. This chapter describes the various steps in the distribution chain; the next chapter describes the ways in which efficiency, modularity, and flexibility can be introduced into each step.
FACILITY POWER
TRANSFORMER
SWITCHGEAR
UPS
480V SWITCHGEAR
PDUs
RACKS
GENERATOR
MORE THAN ONE OF EACH DEVICE INSTALLED AT EACH LAYER
Figure 1. A typical datacenter power distribution system transforms and delivers utility power to equipment racks.
Utility Power and Transformer

Power from the electric utility is usually delivered at voltages anywhere from 480V to 21 kV and higher. The first step in the distribution process is to transform this voltage to a level that is usable by the building and datacenter equipment. The transformer may be owned by the utility but installed on a pad provided by the customer or it may be owned by the customer. Both models are used at Sun, and Figure 2 illustrates transformers owned by Sun at Suns Santa Clara, California campus. In many parts of the world, a single transformer reduces medium voltage from the utility to 400/230V, with 230 the final voltage supplied to datacenter equipment. (The 400/230V notation indicates three-phase power where the first number indicates phase to phase and the second number indicates phase to neutral voltage). In the United States, voltage is reduced in two steps. A first step to 480/277V is used to supply devices
including HVAC equipment and lighting throughout the building. A second step to 208/120V brings the voltage down to that used by compute equipment. Each time voltage is reduced, some energy losses are introduced into the power distribution system.
Generators
Transformers
Figure 2. This view of the main switch yard in Suns Santa Clara, California facility shows four transformers connected to three separate utility feeds and switchgear, Backup generators are in the background.
Switchgear
In industrial electrical design terminology, medium voltage ranges from 1-33kV.
Datacenters use switchgear and distribution boards to safely distribute power from the utility to the datacenter floor. The switchgear includes circuit breakers and switches for managing medium and low voltages and they are typically used to distribute large amounts of power to various locations within datacenters and buildings. The circuit breakers installed in switchgear tend to be more sophisticated than breakers in distribution panels. Switchgear breakers typically have intelligence for monitoring and control, have configurable trip points, and can include ground fault interruption circuitry (depending on the design). The purpose of the breakers is to interrupt and disconnect the load from the power source in the event of abnormally high current conditions that could damage the electrical system or create unsafe operating conditions such as a phase-to-phase or phase-to-ground faults. Switchgear may also include tie breakers and cross-connection schemes that allow the power path to be reconfigured through the same set of switchgear, or around it in case of a fault. For example, in a dual-path power system design, it is important to be able to re-route
power around a fault in the system. Tie breaker schemes allow switchgear that would normally be fed from one source to be switched over to a secondary source. Automatic transfer switches can also be used to automate this process. At Sun, a tie breaker scheme allows power supplied to buildings to be doubled. By installing double-ended switchgear with internal tie breakers, an additional power feed can be incorporated by installing a new transformer at the opposite end of the switchgear. The tie breakers then separate the switchgear from one bus into two distinct buses.
Backup Power
Backup generators are used to power critical equipment in the event of a utility power failure. Backup power is sometimes supplied to the main switchgear, sometimes it is provided further down in the distribution system. Backup power generators cannot start instantly, so another form of backup power needs to continue to power datacenter equipment during the interval of eight seconds or less that is needed to start the generators and ready them to carry the datacenter and support loads. The traditional approach is to use battery backup to support the datacenter load during this time. Rotational UPS that draw power from a flywheel are starting to prove themselves superior to battery backup. Both approaches introduce losses that can be mitigated as discussed in the next chapter.
Power Distribution
Conduit and wireway distribute the conductors that run from the switchgear to the datacenter itself, where typically one of three techniques are used to distribute power to racks. The trade-offs between these approaches are discussed in detail in Datacenter Power Distribution on page 17. Distribution panel boards use standard wall-mounted distribution panels at the datacenters perimeter with any combination of conduit, wire mold, raceways, and armored cable whips running from the breaker panels to individual racks. Power is typically delivered underneath a raised floor in the datacenter, but also can be delivered overhead. Power distribution units (PDUs) typically consist of multiple distribution panel boards and can also include internal transformers. (PDUs without transformers are typically called remote power panels (RPPs). PDUs tend to have a significant amount of built-in intelligence and monitoring capabilities. PDUs are typically installed on the datacenter floor throughout the space. PDUs can be used in both slab and raised-floor datacenters.
Busway provides a modular track above or below rows of racks. Cans combine breakers and short whips that can be plugged into the busway to provide power directly to rack-mounted power strips. Busways are capable of distributing threephase power across the datacenter floor. The cans are designed to use any combination of the three phases, allowing different types of equipment to be powered by plugging in different cans, and for power consumption to be balanced across phases. Sun has found that busway power distribution provides the most efficient and modular way to distribute power to the racks and has standardized on technology for new datacenters.
Rack Power Distribution

Inside each rack is a power strip or rack-mounted power distribution unit (rPDU) that distributes power to the IT equipment installed in the rack. Sun has standardized on rPDUs that monitor power and allow each plug to be powered on and off over the network. These rPDUs provide data for power monitoring as well as allowing remote users the ability to power cycle equipment. As the datacenter continues to evolve, power monitoring and control of individual outlets will increase in importance. Knowing actual power loads at the server and rack level enables the datacenter manager to see the power utilization and distribution across the datacenter space. This information can then be used in the IT equipment provisioning process. It can also be used to dynamically adjust loads based on server utilization as compute load changes.
Datacenter Tier Levels

As defined by Uptime Institute's white paper Tier Classifications Define Site Infrastructure Performance, Tier levels range from 14, with Tier 4 being the most fault tolerant.
Datacenters typically are classified into four Tier levels that are defined by the levels of redundancy in their power and cooling systems. From the perspective of electrical design, the Tier levels are defined as follows: Tier 1 uses a single path for power distribution, without redundant components. Tier 2 uses a single path for power distribution. Redundant components include multiple UPS and generator power. Tier 3 uses multiple power distribution paths along with redundant components. This allows one distribution path to be taken out of service for maintenance or repairs without having to take the datacenter offline. Tier 4 uses fault tolerant multiple power system and system distribution paths, redundant components, and uses only equipment with dual power supplies so that any distribution path or complete system can fail completely without affecting datacenter uptime.
Suns Approach to Electrical Design

The Green Grids definition of PUE and DCiE is available in their white paper Data Center Power Efficiency Metrics at: http://www.thegreengrid.org
Suns approach is to use the opportunities that exist within a datacenters electrical design to increase efficiency, flexibility, and provide room for growth. Efficiency increases help to reduce power consumption and increase the datacenters Power Usage Effectiveness (PUE) or its reciprocal DCiE. Building in flexibility allows a datacenter to change rack power configurations in minutes, rather than days or months, contributing to the companys ability to quickly adapt to changing business conditions and product development efforts. Preparing for future growth extends datacenter life by anticipating rapid hardware refresh cycles and the increased power requirements that accompany them.
Increasing Electrical Distribution Efficiency
Chapter 3

A typical datacenter can easily lose 10 percent of its power in the power distribution system. Transformers, UPS, and distribution contribute to power loss. Using a number of the techniques discussed in this section, we have reduced the power distribution system loss in our Santa Clara, California software organization datacenter from ten percent to approximately four percent of the total datacenter facility load.
The calculation is as follows: 800 kW x .06 = 48 kW 48 kW x 8760 hours x $0.08 = $33,638
A savings of six percent may not seem particularly significant. Consider one of our datacenters that has an IT load of 800 kilowatts (kW). Reducing overhead by six percent is a savings of 48 kW, or $33,638 per year. Any loss due to inefficiency is converted to heat, so the savings is even greater when one considers that greater efficiency means less cooling is needed to remove that heat, another cost reduction.
Component Selection
The low-hanging fruit for increasing electrical system efficiency is choosing highefficiency components from transformers and UPS to server power supplies. This straightforward choice is often overlooked due to higher initial costs. However, if the total cost of ownership is evaluated, the payback period proves to be very short. With the current green initiative trend, these component costs are starting to come down, making the decision easier to justify.
High-Efficiency Transformers
In the United States, most datacenters have at least two power conversion steps that require transformers. Typical transformers impose about a 3 percent loss, making them around 97 percent efficient. Using a high-efficiency NEMA TP1-rated transformer can increase performance to the mid-98 percent efficient range. This not only saves energy in the electrical system but since all transformer loss are converted to heat it also reduces electrical room cooling loads.
High-Efficiency UPS
There are many factors that contribute to the efficiency of UPS systems. However, a common factor across most UPS systems is that the higher the utilization, the more efficiently they operate. For example, if we look at a typical UPS efficiency curve, the rated efficiency of the unit is usually stated for loads greater 70% of capacity. Because of UPS redundancy requirements, UPS system are rarely operated in such a high region. Performance of the UPS begins to drop off when loads are 50 percent or less of rated loads. At around 30 percent utilization, UPS efficiency drops very quickly as utilization is reduced. Since UPS systems are a major contributor to electrical system losses it is
important to make sure the UPS system is designed so that it can be operated at a higher utilization level and that the efficiency of the unit is maximized for the expected design operations.
APC white paper #1 describes UPS topologies. Please refer to: http://www.apc.com
There are many different UPS topologies that can be used in datacenter applications, including double on-line conversion, delta conversion, and line interactive. In our Santa Clara datacenter, we chose to use a delta-conversion UPS that uses a modular approach to support capacity increases. This allows us to right size the UPS for day one needs and then grow the UPS using 200 kW power modules as the load increases, all without major construction. This choice provides us with UPS efficiency into the 97 percent range which is typically about 34 percent better than the traditional doubleconversion UPS.
Alternatives to Battery-Based UPS

Battery-based UPS are costly to maintain. Batteries must be replaced over time, sometimes as early as when they are five years old; rooms must be vented to prevent hydrogen gas build up; catch pans must be in place to protect against acid spills; and battery-based UPS take a significant amount of space. Batteries are also not very environmentally friendly and require controls on the hazardous materials both when handling and disposing of them. A low-maintenance, more environmentally friendly approach uses the rotational inertia of a flywheel to generate electricity during the gap between a power failure and generator startup (Figure 3). There are two common approaches to rotational UPS: Figure 3. A line-interaction UPS is in use at Suns Broomfield, Colorado datacenter. The traditional approach uses a typical UPS topology, such as a double-conversion or line-interactive model, replacing the batteries with a flywheel that stores energy through rotational inertia. When utility power is lost, the energy stored in the flywheel powers the datacenter. It can substitute directly for a battery-based UPS within a building or datacenter. Another approach couples a diesel generator with an induction clutch. When power fails, the flywheel turns the generator until the diesel engine starts and comes up to speed. Flywheel-based UPS are finding increasing acceptance throughout the industry because they greatly reduce maintenance costs and require less space in the UPS room than traditional battery-based UPS systems. The biggest concern of datacenter operators is the relatively short ride-through time provided by the flywheel. Battery UPS system are typically sized to support loads for 1015 minutes while the generators come on line. A typical generator, however, is ready to accept full load in 810 seconds. If a generator does not start up in 810 seconds, there is typically very little that can be done to repair it during the 15 minutes the batteries can deliver power. While rotational UPS do require maintenance on mechanical parts such as bearings, they take far less
10
space and maintenance than battery backup systems. The current drawback is that flywheel UPS systems tend to have a higher initial cost as compared to a traditional battery UPS. Reduced maintenance costs usually offset the difference in a short time.
Equipment Power Supplies

The power supplies on server, storage, and networking equipment also contribute to power distribution losses. Power supplies are always sized to handle the worst-case scenario: a server with the maximum memory, internal disk, and peripheral cards installed and with the CPUs and fans running at full power. Many servers, however, actually operate at 50 percent or less of their name plate power (Figure 4), a range where historically, average power supplies can be relatively inefficient.
100
Efficient Percent Efficiency
Average
45 0 Percent of Name Plate Power Output 100
Figure 4. Power supplies such as those used by Sun are more efficient across a broader output range.
Information on the 80 Plus program is available at http://www.80plus.org.
With todays focus on energy efficiency, server vendors including Sun are striving to make their power supplies more efficient at low utilization levels. In the U.S., an electric utility-funded program called 80 Plus is promoting the use of power supplies that are at least 80 percent efficient at 20, 50, and 100 percent of their rated load. Most of the power supplies on servers from Sun exceed this specification, with some of them 93 percent efficient at an 80 percent load. Another issue to consider is the fact that systems with redundant power supplies operate at far less than 50 percent load, often putting them into an inefficient range of operation. If the application supported by the systems is designed to be resilient to single points of failure, then servers with single power supplies can be used. This allows their power supplies to run in a much more efficient portion of their operating range. Regardless of whether single or dual power supplies are used, care needs to be taken to ensure that each power supply is plugged into a dedicated circuit.
11
Power Characteristics
An organizations choice of power characteristics voltages and AC vs. DC has an impact on datacenter power distribution efficiency.
208V Vs. 230V

This topic is discussed in detail in the Green Grids white paper 16 titled: Quantitative Efficiency Analysis of Power Distribution Systems for Data Centers The paper is available at: http://www.thegreengrid.org
In the U.S., we typically reduce the utilitys medium-voltage supply in two steps, using two stages of transformers, down to 208/120V. Other parts of the world use higher voltages, such as single-phase 230V (line to neutral), yielding some efficiency benefits. The higher voltage can eliminate the inefficiency of a second transformer step, and it allows the use of less copper, as distribution paths internal to the datacenter can carry more power at lower amperage. Virtually all datacenter equipment is built to accept a range of input power from 100 240V, so the choice of which voltage to use is one that is dependent on comfort levels and local electrical codes. Had we had more time to evaluate, test and educate the project team when we built our datacenters in California, we may have been able to increase efficiency even further by deploying 230V throughout.
Alternating Vs. Direct Current

There is a long-running controversy regarding the relative merits of using high-voltage DC versus AC power in the datacenter. The argument is this: high-voltage (380575V) DC eliminates the requirement for large amounts of copper in the distribution system, making this aspect competitive with AC power. Using DC throughout the datacenter eliminates three AC-DC power conversions that add to inefficiency (Figure 5): 480 VAC is converted to direct current at the UPS to charge the batteries. Battery power is inverted to 480 VAC to supply the datacenter. The first stage of datacenter equipment power supplies converts incoming AC power to high-voltage DC, which is then regulated down to the low DC voltages used within the server, storage, or networking equipment.
Power Inefficiencies 380 VDC (typical)
Power Supply Unit (PSU)
Lead-Acid Batteries
UPS
PDU
Server
Figure 5. The argument for DC power in the datacenter is that using DC power throughout eliminates multiple inefficient AC-DC power conversions.
Internal Voltages
480 VAC Supply Power
AC/DC
DC/AC
AC/DC
DC/DC
12
The final report resulting from this project is available at: http://hightech.lbl.gov/documents/ DATA_CENTERS/DCDemoFinalReport.pdf
In conjunction with a California Energy Commission-sponsored project, Sun implemented a proof-of-concept that demonstrated the power savings of using DC in the datacenter environment. Sun created two identical datacenters, one with AC and one with DC power distribution. Off-the-shelf equipment from Sun, Cisco, and Intel was used, with the power supplies modified by hand to eliminate the power conversion circuitry that converts alternating to direct current. Incoming power was rectified, distributed at 380 VDC, and fed directly into the modified power supplies. This was a proof-of-concept demonstration only, as 380 VDC power supplies are not available as products from Sun. (Sun does offer 48 VDC-powered equipment through its Netra product line designed for communication-carrier environments). The results showed a 25 percent improvement in power supply efficiency, and up to a 1020 percent improvement when the use of a flywheel vs. a battery-based UPS was factored into the equation. We believe that the benefits demonstrated by this experiment are superseded by improvements in power supply efficiency and that similar improvements are possible without the use of DC power. The equipment used in this experiment used older equipment equipped with power supplies where a greater improvement could be shown compared to todays more efficient power supplies. Another approach that is being discussed is using the telco standard of 48 VDC. Regardless of which DC power approach is used, we see some key disadvantages: Equipment that accepts high-voltage DC directly is not available on the market today. There is a limited selection of equipment that can use 48 VDC. There is a relative lack of experience and comfort using DC power in the datacenter. Distribution systems, system power connections, and breakers are all different, and everyone from electrical contractors to system administrators would need training. Maintenance and arc-flash suppression has not been uniformly addressed. For most companies, the ability to leverage the newest and best server, storage, and network technology is key to maintaining a competitive edge. Today, the choice of DC power restricts equipment choices from almost all major manufacturers.
Appropriate Levels of Redundancy

Datacenter power efficiency can be increased by using redundancy in appropriate ways. First, choosing a level of redundancy that is matched to the organizations mission, and their level of accepted risk, increases efficiency by reducing the amount of redundant equipment that is necessary, and also by reducing layers of components (such as UPS) that add to inefficiency. Second, redundancy can be implemented so that equipment is used in a more efficient part of its operating range.
13
Choosing Tier Levels

Everyone wants to have a Tier 4 datacenter, with two complete power distribution systems so that either can be taken out of service without affecting datacenter uptime. In fact, most organizations only require a small portion of their datacenter to be built to Tier 4 specifications if any at all. The cost of implementing a Tier 4 design and continuous operation at that level can be more than five times greater than a Tier 1 facility. Although our electrical design approach can support any Tier level, our datacenters at Sun are used for internal product development and typically require Tier 1 or Tier 2 support. Our IT specific datacenters provide more critical services, and therefore require a Tier 3 level of service.
A pod, as defined in the Sun BluePrints Series article The Role of Modularity in Datacenter Design, is a small, self-contained group of racks and/or benches that optimize power, cooling, and cabling efficiencies. A typical pod is a collection of 2024 racks with a common hot or cold aisle along with a modular set of infrastructure for it to handle its own power, cooling, and cabling.
Our datacenters can support multiple Tier levels in the same room. Different configurations of our modular busway can feed multiple parallel power distribution paths down to individual racks. Since most of our equipment does not actually require UPS power, we can run two busways, one with UPS power and one without, to each of our pods. Racks that require redundancy and UPS power can connect to both busways. Those that dont require UPS power do not. In the end, only approximately 20 percent of our systems actually require UPS power, helping to increase efficiency and significantly reduce the capital and operating costs associated with uninterruptible power. Our pods are self-sufficient from a power, cooling, and cabling perspective, so a Tier 3 pod can be placed in the same room with Tier 1 and Tier 2 equipment. The same would be true if the corporation required Tier 4 as long as a dual-path system with or without redundancy was carried back to the central plant.
Using Distributed Redundancy

Using an appropriate level of redundancy in the electrical distribution system is one means to increase efficiency; the way in which you implement redundancy also has an impact on efficiency. 2N Redundancy A typical UPS configuration uses 2N redundancy, where one of two UPS systems can support the entire load should one UPS fail. This model operates each UPS at less than 50 percent utilization. Typical maximum utilization levels are 4050 percent so that when the UPS has to support 100 percent of the datacenter load it does not exceed its full capacity rating and trip offline. It should also be noted that UPS systems operating in this range are relatively inefficient especially if operated at less than expected full design loads. A hypothetical configuration illustrated in Figure 6 shows two UPS powering a datacenter consisting of three racks.
14
Figure 6. In a 2N redundant model, each UPS is sized to support 100 percent of the workload but operates at less than a 50 percent utilization level. N+1 Redundancy (Parallel Redundant) This model uses N+1 UPS to support a load where one UPS can fail and the remaining N are sufficient. The example in Figure 7 shows a three UPS N+1 UPS system, each UPS supports 33 percent of the power draw, but is sized to support 50 percent in the event that one fails. This puts each UPS at a more favorable 66 utilization level. Because each UPS is sized too small to power the entire server workload, the outputs must be synchronized to a common bus, and it is a single point of failure for this model. Distributed Redundancy A distributed redundancy model uses three or more UPS connected with multiple paths to servers. A distributed redundancy UPS system model can be used to achieve multiple distribution path redundancy where required without the need for a 2N system. (Figure 8). For example, a three-UPS distributed system can achieve a utilization level of 66 percent. This is the same utilization level as the N+1 model, but the configuration eliminates the need to tie outputs together and eliminates the bus as a single point of failure. The benefits can include lower capital cost as compared to a 2N system, increased efficiency, and dual power paths to servers.
15
Figure 7. In an N+1 redundant model, all UPS are connected to all servers, but the bus is a single point of failure.
Figure 8. In a distributed redundant model, each UPS supports one third of the servers and operates at a 66 percent utilization level. There is no single point of failure.
16
As with all approaches to UPS redundancy, distributed redundancy requires adjusting the balance of servers to UPS, and this requires a flexible, modular power distribution system that can adapt as a datacenter grows. Modularity and Flexibility are the topics of the next chapter.
17
Flexibility Through Modularity
Chapter 4

Modularity is a key datacenter design principle that enables flexibility and the ability to grow. At Sun, our modular, pod-based design creates energy-efficient building blocks that can be duplicated easily in any size or Tier-level datacenter worldwide. The pod design allows different modules to be selected and configured to best meet client requirements as a datacenter changes over time. Following our modular, pod-based approach, different pods in the same room can to be built to different Tier-level specifications because our electrical design can support it. The modular approach allows pods, and racks within pods, to have radically different densities, with a rack using 4 kW next to one that requires 30 kW. The modular approach is flexible so that a 4 kW rack can be replaced with one using 30 kW or vice versa in a matter of hours. The same modular design that supports brick-and-mortar datacenters also can be used to support Sun Modular Datacenters as discussed in Sun Modular Datacenter on page 34
Datacenter Power Distribution

For a more detailed discussion of the tradeoffs summarized here, see the Sun BluePrints article: Energy Efficient Datacenters: The Benefits of Modular Design.
The feature that most supports the day-to-day flexibility of our datacenters is our choice of power distribution on the datacenter floor itself. The challenge here is two-fold: the power distribution system must be able to support growth so that the datacenter can handle increased density at every equipment refresh cycle. The power distribution system also must be able to handle day-to-day equipment churn that significantly changes the power draw at the rack level. At Sun, this is extremely important, as spaces used for new hardware bring-up must allow for the next generation of products and even higher densities to be rolled in and powered up without long delays. Our power distribution system helps future Sun products to get to market faster because all outlet types and loads are supported. Our datacenters have products in them today that our customers won't see for two years.
Hard-Wired Power Distribution

The oldest, and the least-flexible power distribution system is one that uses circuit breaker panels at the datacenter perimeter with conduit, wire moulding, or armoredcable whips running from the breaker panels to individual racks or groups of racks. This approach limits flexibility, as a significant amount of electrical work is necessary in order to add or change service to a rack. Every time a panel is opened, all of the equipment attached to it is at risk.
18
Power Distribution Units

PDUs combine transformers and breaker panels in units that are placed on the datacenter floor. PDUs are typically installed throughout the datacenter floor, with individual flexible whips powering individual racks. PDUs occupy valuable datacenter floor space and add to the datacenters heat load. Whips are difficult to manage, and every time that a PDU is opened to add or change a configuration, it introduces the possibility of human error, putting everything the PDU powers at risk. Although we have used PDUs in some of our earlier designs, we now see it as a limiting factor in those datacenters flexibility. This is because all changes require an electrician, the PDU must be powered down, much more time is required for changes, and the cost can be up to ten times more over the life of the electrical system when compared to busway systems.
The Benefits of Modular Busway

Our pod design removes all transformers and power distribution units from the datacenter floor. Instead, we use a hot-pluggable busway overhead or under a raised access floor to distribute power to each rack (Figure 9).
Figure 9. Modular busway is installed overhead or under a raised access floor that is not used for air distribution. Each busway includes three phases, one neutral, and one ground conductors that are accessed by cans that tap into the right combinations for the power to be delivered. The busway supports an intermixing of 120V, single-phase power up to 208V, threephase power (or the equivalent geographical power configuration for datacenters
19
located outside of the U.S.). Despite its capacity to handle up to 100A through a single can, an electrician can install a new circuit in a matter of minutes. This flexibility is key in our datacenters where equipment churn occurs on a regular basis. This modular approach can be used to support different Tier levels of service on a podby-pod basis. A Tier 1 pod can be supported with two busways, one over each row of racks in a pod. Tiers 24 can be supported with four busways, with the second pair of busways providing access to a redundant power source. This allows us to provide the required level of service to our customers while saving the cost and inefficiency of providing redundant power to every single rack. The pod illustrated in Figure 10 shows four busways delivering redundant power to the pod.
Two busways above each rack Busway cans
Figure 10. A pod using four busways provides two redundant power paths that support Tier 2-4 levels of availability. We have found busway to work equally well whether used with the in-row cooling approach illustrated in Figure 10, or with overhead cooling units as illustrated in Figure 11.
20
Two busways above each rack Busway cans
Figure 11. Busway can be used with overhead cooling units as easily as with inrow cooling.
21
Extending the Datacenter Lifecycle
Chapter 5

The industry is seeing a new wave of datacenter construction that is driven by premature obsolescence of existing datacenters that has resulted from the high demand driven by the global consumption of technology and services. Facilities built during the late 1990s were expected to last 1520 years, and most have reached a premature end of life. While these facilities may have sufficient space, their power and cooling systems no longer can support the IT equipment in them. The well-known reason for this problem is a mismatch between the datacenters projected life span and the frequency and impact of equipment refresh cycles. If a datacenter is built to last 15 years, then it must be built to accommodate equipment refresh cycles that range from 35 years. For example, a high-density datacenter in the year 2000 consumed around 34 kW per rack. Now 34 kW per rack is below the industry average, and todays high-density racks are exceeding 2030 kW per rack! In our datacenters at Sun, we have incorporated expansion capacity to support threeyear equipment refresh cycles. From day one, we have built a modular system that can accommodate growth through the addition of new modules, and we have built extra capacity in terms of additional equipment locations and power paths. This future proofing of infrastructure required an additional investment of 1015 percent in pipe work for the mechanical/electrical systems. But this scalability enabled Sun's agility and paid for itself within the first year. This chapter follows the path of electricity from the utility to servers. It illustrates how datacenters can be built to support the rapid change and increases in density that occur over time without requiring major construction inside or outside of in the datacenter itself.
Electrical Yard
The electrical yard is where the utility power is provided to the datacenter. When building a datacenter, or retrofitting an existing building, its important to plan for the future with the electrical utility: be sure that long-term power needs can be satisfied by the utility and under what terms. In other words, design backwards from the amount of power you will require or can receive to reach your end state. Build your electrical yard with expansion in mind: Plan for additional transformers that connect to new utility feeds by preparing concrete pads for them from the beginning. Design switchgear so that new feeds can be installed and cut over seamlessly.
22
Configure switchgear so that there is room for growth. One technique is to use crossconnect (cross tie) schemes to allow a bus to be split in half and powered by two separate utility feeds in the future. Leave room in the switchgear for additional breakers, and run empty conduits sized to accommodate the largest possible wire from the electrical yard to future equipment locations in buildings. Trench and run conduit underground only once. At Suns Santa Clara facility, we have installed conduit from the electrical yard to datacenter spaces to accommodate growth within datacenters. We also have run conduit from the electrical yard to existing office spaces that may become datacenters in the future. The electrical yard is typically home to one or more backup generators; plan space for growth by allowing space for additional generator capacity to support your end state.
Building Infrastructure
Each datacenter has a significant amount of electrical distribution infrastructure that needs to be designed and sized with room for expansion: Prepare to expand UPS systems as electrical loads grow. This requires not only a UPS design that is modular and can be expanded, but also floor space for additional UPS modules and battery racks or flywheel cabinets. Use transformers, not PDUs. Our modular, pod-based design uses transformers in separate rooms rather than PDUs on the datacenter floor. Reserve space for additional transformers that growth in electrical loads will require. Design power distribution with room for expansion and preinstall conduit to support the future growth. For example, if the goal is to support 100 percent growth over the datacenter lifecycle, then design distribution boards to be 50 percent full on day one to allow for easy expansion over time. Empty conduits to support future expansion should also be run at the time of original construction to reduce the impact of the expansion on the operating datacenter. Use industrial busway to provide power from the distribution panels to the pods. This allows power feeds to be changed as power consumption within pods grows.
Pod Power Distribution

Each of our pods use a modular overhead busway to deliver power from the distribution system to the equipment racks. Modular busway contributes to a datacenters agility by making it possible to reconfigure equipment racks for completely different power requirements voltage, phase, and amperage in a matter of minutes rather than hours, days, weeks, or months. Busway also enables a datacenters expansion capacity by supporting increasing power draws as equipment is replaced gradually over time. Rather than sizing the overhead busway to the maximum expected power draw, we have designed a strategy that can double the power provided to a pod in two phases allowing the power draw to increase by a factor of four over time.
23
For example, consider a pod with two busways, one above each row of racks (Figure 12). Our initial configuration is to jumper the two busways together through a conduit bridge and provide a single power feed for each pair of busways.
Phase One: One feed for two busways using jumper Phase Two: One feed for each busway
Phase Three: One feed for each split busway
Figure 12. As a pods power draw increases, its supply can be increased in two phases, each phase doubling the previous power capacity. The pods power feed can be doubled by removing the jumper and providing a power feed to each busway. The pods power feed can be doubled again by splitting each busway in half and providing a separate feed for each half. This expansion strategy avoids the up-front cost of purchasing high-amperage busway, while providing a way to support a four-fold increase in power consumption over time with minimal disruption or construction costs. This level of expansion in a traditional datacenter electrical distribution system would require a complete repowering of the datacenter space along with significant downtime and expense. Keep in mind that this strategy requires your conduit to be larger and pathways to be pre-defined to support the end state quantity and size of conductors. Choosing one size busway that supports the final load will allow the re-use of cans over the functional life of the datacenter. These busways also support higher voltages as discussed in the previous chapter.
24
Suns Electrical Design At Work
Chapter 6

The design principles that we have discussed to this point are embodied in our datacenter designs worldwide. This chapter provides a top-to-bottom overview of our Santa Clara, California datacenters followed by a discussion of Suns newest facility in Broomfield, Colorado, and an overview of Suns use of Sun Modular Datacenters at the Santa Clara site.
Santa Clara, California

Our electrical design tour of our Santa Clara datacenter is through a series of one-line diagrams. These diagrams illustrate the various components and how they are connected without the detail of showing each wire for each phase hence the term one line diagram.
Electrical Yard
Our Santa Clara project, one of the largest datacenter consolidation efforts in Suns history, allowed us to construct an entirely new electrical service yard to supply power to new datacenters and buildings. This electrical yard receives two 12 kV power feeds from Silicon Valley Power, each of which allows us to draw 4.5 MVA of continuous power. To ensure that we have power to grow over time, we have negotiated with the local utility to provide up to 24 MW of additional power to the electrical yard through additional 4.5 MVA feeders. A set of medium-voltage switchgear allows the additional feeds to be incorporated into the electrical yard without disrupting existing service. The medium-voltage switchgear feeds three similar sets of 480V switchgear through step-down transformers, one of which is illustrated in Figure 13. The switchgear supplies 480V power to office buildings and datacenters on the Santa Clara campus. The switchgear includes breakers for existing spaces and for future expansion, with conduit runs already installed to the future spaces. We accommodate expansion at this level of the distribution hierarchy by supporting each of the three sets of switchgear with a single transformer. Each set of switchgear is configured with two halves that are connected with a tie breaker that is currently closed. When the time comes to double the power supply, a second transformer will be connected to a new utility power feed, and the tie breaker will be opened. Now each half of the switchgear has a 3 MW continuous operating capacity.
25
Existing transformer supplies switchgear with 480/277V
Future transformer allows a doubling of capacity
Tie breaker remains closed until future transformer is installed
Breakers, future breaker locations, and power feeds to buildings and datacenters
Figure 13. Each of three sets of 480V switchgear can support a doubling of power by adding a second transformer and opening a tie breaker. A photograph of the Santa Clara switch yard gives a visual perspective (Figure 14). In the foreground are three rows of similar switchgear. From right to left is the 12 kV switching upstream of the transformer. Next is the transformer, followed by the 480V switchgear. Note the empty space at the end of two of the rows to accommodate an additional transformer and switchgear. One already has been installed. Behind the 480V switchgear is the medium-voltage, 12 kV switchgear that accommodates existing and future utility feeds. Also visible in the photograph are two diesel backup generators.
Load Bank
Generators 12 kV Utility Feeds and Switchgear
Expansion Locations
480V Switchgear
12kV Switchgear and Transformers
Figure 14. A view of the Santa Clara electrical yard shows the 12 kV to 480V transformers and switchgear in the foreground.
26
Backup Generators
Because the majority of our datacenters are for research and development efforts, the pods in them are designed to Tier 12 specifications. For 20 percent of the load, we deploy UPS to support more critical operations. As operational risks increase, we deploy generators behind the UPS to provide continuous power. Figure 15 shows a portion of the one-line diagram showing our backup generator configuration. Two backup generators are currently installed, with space, conduit, and breaker positions for a third. This provides 2 MVA of power today, with 3 MVA when the third generator is installed. The generators provide power to a set of paralleling switchgear that combines the generator outputs. A 1000 kW load bank is installed to provide a full load when testing the generators. Regular 100 percent load testing of generators is important to make sure the generators are operating correctly and always ready to accept load. The paralleling switchgear currently feeds one 800 kVA UPS with provisions for future growth. In the event of a failure, the feed is switched through automatic transfer switches (ATS). Two automatic transfer switches (ATS) switch the power feed for the UPS from utility power to generator power in the event of a power failure. A third ATS supplies power to keep the cooling system operating.
Feed from 480V switchgear
Existing two generators Automatic transfer switch
Future generator
Supply to UPS
Figure 15. Our Santa Clara electrical yard includes three generators that provide backup power to UPS.
27
Battery-Based UPS
Our Santa Clara site is equipped to accommodate a fourth UPS to bring the total to 3.2 MVA. The site uses three 800 kVA UPS capable of providing up to 2.4 MVA of uninterruptible power. We chose APC Symmetra MW units for the Santa Clara site for their modularity and efficiency. The APC UPS is unique in that it enables the UPS system to grow in 200 kW increments. At the Santa Clara site, two of the three UPS are sized with 800 kW frames with only 400 kW of power modules installed, right sizing the UPS to our day one needs. As our loads grow, we will add additional power modules that grow the UPS to its full 800 kW capacity with only minimal construction in the UPS room to match battery capacity to UPS capacity.
Power supply from automatic transfer switch
To transformers and distribution panels
Overhead busways providing uninterruptible power to pods
Figure 16. An 800 kVA UPS supplies power to three transformers, each of which supplies 208/120V uninterruptible power to a set of busways. The UPS illustrated in Figure 16 takes its input power from the automatic transfer switch illustrated in Figure 15. Its output feeds a distribution panel that connects to three transformers that step down the 480/277V power to the 208/120V power that
28
feeds the busways. One transformer is illustrated in Figure 16. It supplies power to a distribution panel that feeds a set of 225A and 400A busways inside the datacenter spaces.

For the majority of our pods that only require Tier 1 service, a single 480V feed from the switchgear supports multiple transformers that bring the voltage down to the 208/120V supplied to pods. The example shown in Figure 17 illustrates pairs of busway jumpered together as described in Pod Power Distribution on page 22.
Supply from 480V switchgear
480V transformer
Figure 17. Power is distributed to non-critical loads through transformers that feed 208/120V power to overhead busways.
Metering and Power Management

One of the most important aspects of datacenter electrical design is an integrated metering and energy management system. These systems are critical because they provide the ability to evaluate how power is being used in the datacenter, power quality or disturbance events, how efficiently the datacenter is operating, and what impact any energy-saving measures have on overall efficiency.
29
As we discussed in the Sun BluePrints Series article Energy Efficient Datacenters: The Role of Modularity in Datacenter Design, power usage effectiveness (PUE) is the ratio of total facility power to the IT equipment load. In order to obtain this ratio, and to be able to observe energy consumption of the various datacenter subsystems, appropriate metering capabilities must be placed at strategic locations cascaded through the electrical system. In our Santa Clara datacenters, we have metering and power management in place at the following locations: The utility switchgear All major switchgear All UPS, ATS, and generators Each overhead busway Each rack is equipped with a rack-mounted power distribution unit (rPDU) that monitors overall power consumption and also provides remote on/off per-outlet control. These rPDUs can be monitored over the network, providing fine-grained data on IT equipment loads.
Broomfield, Colorado
Our Broomfield, Colorado location is the site of our most recent datacenter construction project. Spurred by the need to consolidate the former StorageTek site into our Broomfield campus, there are two significant differences between this datacenters electrical design and the Santa Clara design described above: No electrical yard. In Broomfield, the utility provides 480V power through a set of primary voltage transformers located at the campus perimeter. For redundancy purposes, these transformers are loop fed through utility company-owned switch cabinets. This is an example of where the utility owns the primary voltage feeders, transformers, and switch cabinets. Flywheel UPS. The Broomfield campus is the largest installation of rotational UPS at Sun. The double-conversion UPS used in Broomfield supplies 900 kVA of backup power, sufficient to power critical loads between the time of a power failure and diesel generator startup. The UPS pictured in Figure 18 provides 15 seconds of backup power to critical loads, sufficient to cover the 8 seconds between a power failure and diesel generator startup.
Main Power Distribution

Power to datacenter equipment located on floors 14 of the Broomfield building originates from utility transformers located at one end of the building. One transformer feeds power to a 4000A, 480V main distribution board that serves equipment loads on the first and fourth floors. The second transformer feeds power to a 5000A, 480V main distribution board that serves equipment loads on the second and third floors. The two electrical services combined are capable of delivering 7.1 MW to
30
datacenter equipment on floors 14. A transient voltage surge suppression device is installed at each service entry point to protect the building electrical system from voltage surges generated outside the building.
Figure 18. The rotational UPS used in Broomfield supplies 960 KW kVA of backup power, sufficient to power critical loads between the time of a power failure and diesel generator startup. The main switchboards are equipped with spare circuit breakers and spaces with conduit runs to the secondary electrical rooms located on the second and fourth floors for future expansion. A one-line diagram illustrating this design is illustrated in Figure 19. The central plant transformers are fed from a new utility transformer located northeast of the building. It feeds power to a 2500A, 480V main distribution board which serves mechanical loads supporting the datacenter. The electrical service is capable of delivering 2.2 MW to mechanical load in the central plant.

The two main distribution switchboards distribute 480V power to a series of 225 kVA and 500 kVA high-efficiency dry-type transformers located in secondary electrical rooms on each floor. We sized these transformers so the would have a much greater load than the industry standard of 30 percent. High-efficiency transformers were selected because at a higher load factor, efficiency tends to remain constant and at a higher level. In contrast, the efficiency of standard dry-type transformers decline sharply beyond the 30 percent load factor.
31
Figure 19. Broomfield datacenter main power distribution Each transformer feeds a large 120/208V distribution panelboard also located in the secondary electrical rooms. A transient voltage suppression system (TVSS) device is installed at each distribution panelboard which protects sensitive datacenter equipment downstream from voltage surges internally generated within the building. The distribution panelboards are equipped with spare circuit breakers, spaces, and conduits routed to locations of future datacenter panelboards. The distribution panelboards feed 120/208V panelboards that distribute power to all 225A busways in the datacenter. The diagram shown in Figure 20 illustrates how pod power is distributed from a typical secondary electrical room to datacenter busway.
32
Figure 20. Pod power is supplied from a 500 kVA transformer located in a secondary electrical room.
Rotational UPS
A 960 kW/1200 kVA, 480V centralized rotational UPS provides approximately 15 seconds of back-up power for critical datacenter equipment located in the mainframe, server room and one pod in the CTC lab located on the second floor. The UPS has four modular 240 kW/300 kVA flywheels in which three of the four flywheels are needed to support the critical loads. This N+1 design approach provides redundancy where one flywheel can be completely taken offline for maintenance purposes. The UPS is equipped with a full-maintenance wrap-around bypass so that it can be taken out of service without interrupting power to critical loads. At 97 percent, the efficiency of the rotational UPS is higher than a traditional battery-based system due to the fact that the two electrical conversions do not occur under normal operating conditions only when the flywheel is carrying the critical load. The UPS illustrated in Figure 21 takes its input and maintenance by-pass power from a 1600A, 480V emergency distribution panel. This distribution panel is equipped with a TVSS device which protects sensitive datacenter equipment downstream from voltage surges internally generated within the building. The UPS output feeds a 1600A, 480V main distribution panel that distributes power to a large 225 kVA and 500 kVA highefficiency dry-type step-down transformers. The transformers feed 120/208V panelboards that distribute power to all 225A busway in the datacenter.
Back-up Generator
Emergency back-up power to the datacenter and central plan is served from an existing 1250 kW/1562.5 kVA diesel generator located outside the building. The back-up generator feeds three emergency main distribution boards located in main electrical room which supports life safety loads and critical secondary mechanical equipment
33
that supports cooling to critical datacenter and the rotational UPS. It also feeds an emergency main distribution board that supports critical primary mechanical equipment in the new central chiller plant.
Metering and Power Management

The Broomfield datacenter has metering and power management throughout. Threephase energy and power measurement with data logging and power quality analysis can be obtained locally from each meter location or remotely from the campus central control room or off campus from a Web browser. Metering has been installed at the following locations: The two 480V main distribution switchboards are monitored to determine total power consumption for the four floors of the datacenter. Each branch distribution feeder in the 480V main distribution switchboards is monitored.
Figure 21. The Broomfield rotational UPS are integrated into the power design so that any one of them can be taken out of service without disrupting loads.
34
Input and output distribution panelboards monitor the flywheel UPS power and quality. Each overhead busway incorporates power monitoring, allowing a per-pod evaluation of power consumption of equipment. Each rack is equipped with an rPDU that monitors overall rack power consumption. These in-rack PDUs can be monitored over the network, providing fine-grained data on IT equipment loads.
Sun Modular Datacenter

For more information on the Sun Modular Datacenter, please visit http://www.sun.com/sunmd
Recent expansion at our Santa Clara site has used two Sun Modular Datacenters (Sun MDs). This unit is a self-contained containerized datacenter configured in an enhanced 20-foot shipping container with integrated high-efficiency cooling and power infrastructure. Sun MD is available in 2 configurations: the original eight-rack S20 model, and a new seven-rack D20 edition. Both support a 200 kW capacity in only 160 square feet of space. The Sun MD can typically be deployed much more quickly than a traditional datacenter; all that is required is to provide appropriate chilled water, networking and power. This section describes the product power features and an overview of the external power requirements for deploying a Sun MD.
Power Overview
The Sun MD is available in two different electrical configurations. The low-voltage version operates on 190 to 220 VAC, three-phase, 50 or 60 Hz electrical service. The highvoltage version operates on 380 to 415 VAC, three-phase, 50 or 60 Hz electrical service
Power Distribution in the Sun MD

Power is routed into the Sun MD through two electrical feeder entrances, one on each side of the container, and can be provided by one or two external power sources. The Sun MD has a primary and a secondary internal power distribution panel corresponding to the two electrical feeder entrances (Figure 22). The internal power panels provide redundant power to each of the racks. Each panel supplies power through a set of circuit breakers to each of the racks and to each of the HX fan control units in the Sun MD.
Supplying Power to the Sun MD

Power can be supplied to a Sun MD through a commercial utility, an on-site utility power generator, or a standalone power generator in any combination. There are several factors to consider in designing and deploying external power infrastructure for the Sun MD, including requirements for redundancy, uptime, mobility, and existing site power infrastructure.
35
Figure 22. Sun Modular Datacenter power distribution panels. This view of a lowvoltage unit is from the back of the Sun MD, with A panel on the left and the B panel on the right. Uptime is typically a primary concern in the deployment of a datacenter. In order to provide the highest levels of service, a means of maintaining power in the event that an external power source fails is highly recommended. This involves a design using an automatic transfer switch to switch from the primary source of power to a secondary or backup source of power such as a generator. A UPS is used to supply power to the Sun MD during the time interval when the primary power fails and when the secondary source comes online. While the actual configuration of the electrical power distribution system will vary, a typical external power infrastructure will have many of the following components: Commercial electric utility service Site distribution panel Automatic transfer switch (if required) Chiller/heater sub-panel (if required by building code) Transformer (if required to match line voltage) Generator (optional) Uninterruptible power supply (optional) UPS power distribution panel (if required by building code) When we deployed Sun Modular Datacenters at our Santa Clara site, we utilized the flexibility of the electrical infrastructure described in previous sections and literally snapped in the modular datacenters. We installed high-efficiency transformers outside next to each container to convert 480V to 208V. Installing the transformers outside
36
eliminated the need to provide cooling for the transformers. The containers in Santa Clara are installed downstream of our UPS system which is currently providing power to both the Sun MDs as well as our building-based datacenters.
37
Summary
Chapter 7
Summary
A datacenters electrical design plays a critical role whose influence ranges from protecting real estate investments to reducing day-to-day power consumption. Traditional electrical designs can limit growth, reduce business agility, and add, rather than subtract, from skyrocketing utility bills. In contrast, the datacenter designs that we have used at Sun facilitate growth, increase agility, and provide significant savings in energy consumption. Rather than being a static aspect of datacenter design that hampers business operations, our datacenter electrical design flexibility helps to better align our datacenters and business operations. Our modular electrical design provides greater flexibility that helps our datacenters better support our business operations. Our modular design allows our engineering organization to roll in and power a next-generation server in a matter of minutes. The same modularity that allows us to support different Tier levels within the same datacenter also saves costs by delivering only the required amount of redundancy. Our philosophy of preparing for future growth from day one has a dramatic impact on the datacenter lifecycle. By building a datacenter electrical system that can support future generations of IT equipment, we have better matched equipment refresh cycles with our planned datacenter life cycles. This not only helps prepare our business for the future, it helps to protect the immense investment in real estate by increasing the longevity of our datacenters. This agility has become a competitive advantage for Sun. Ultimately, a datacenter electrical system must deliver power from the electric utility to each piece of IT equipment. Taking advantages of the opportunities to increase efficiency along the way, making sure not to overprovision for day-one capacity, and through careful monitoring, we have been able to achieve about a six percent greater efficiency that amounts to an annual savings of $33,638 in one datacenter alone. From energy-efficient transformers to appropriate implementations of redundancy, datacenter electrical designs offer abundant opportunity to reduce power consumption and overall operating costs. One of the best aspects of our datacenter electrical design principles is that they are straightforward and easy to implement, so organizations outside of Sun can reap similar benefits. While our designs initially seemed radical compared to traditional designs, the choices that we have made are now embodied in mainstream practices at Sun. These practices are also starting to become prevalent in the industry.
38
Summary
About The Authors

Michael Ryan
Mike Ryan is a Senior Staff Engineer in Suns Global Lab & Datacenter Design Services organization. Mike graduated from San Jose State University with a degree in mechanical engineering. He is a licensed professional mechanical engineer and has been involved in the design, construction, and operation of mission-critical facilities for the past 18 years. His experience spans numerous industries such as gas turbine cogeneration, semi-conductor manufacturing, and mission-critical datacenters. For the last seven years he has focused on the design and operation of mechanical and electrical infrastructure systems supporting high-availability datacenters. Mike joined Sun as a Staff Engineer for the GDS organization in March of 2006. Mike is the primary author of the GDS physical standards and was the lead design engineer for the Santa Clara, California datacenter project, one of the largest and most complex in Sun's history. He also provided supporting engineering services for the Broomfield, Colorado datacenter project. Mike's design work supports the Act portion of Sun's Eco strategy, and was showcased at Sun's Eco Launch in August 2007. Mike is a member of ASHRAE (American Society of Heating, Refrigeration, and Airconditioning Engineers). He also participates in the Data Center Pulse and the Critical Facilities Round Table group. Mike was the recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg Papadopoulos in July of 2008.
Dean Nelson
Dean is the Senior Director of Global Lab & Datacenter Design Services (GDS) in the Work Environments (WE) business unit of Sun Microsystems. The GDS organization bridges the gap between Facilities, IT, and Engineering, and is responsible for managing Suns multi-billion dollar global technical infrastructure portfolio including datacenter design, standards and strategy. The GDS work resides in the Act portion of Suns Eco strategy, and was showcased at Suns Eco Launch in August, 2007, and the Colorado Data Center grand opening in January, 2009. Dean has been in the technology industry for 19 years, of which 16 have been with Sun. He spent four years in Sun manufacturing in roles ranging from component-level debug to managing quality. This included helping the drive to achieve ISO 9002 certification. Dean joined the Sun Engineering community in 1993. He led systems and network administration support for some of Suns largest and most complex R&D lab environments. Dean left Sun in 2000 to join a networking startup company called Allegro Networks. At Allegro, he built a world-class QA team, state of the art global R&D lab environments and fully integrated automation system.
39
Summary
In 2003, Dean returned to Sun joining the newly formed N1 Software organization. He orchestrated the integration of Terraspring and Center Run R&D labs and the merger of Sun Cluster software into N1 technology. In mid-2003, Dean took over management all of the N1 organizations R&D labs, build engineering, automation, and capital budget responsibilities world-wide. In 2004, he became a leading member of the Global Lab & Datacenter Design Services (GDS) team tasked with creating a strategy to standardize Suns multi-billion dollar technical infrastructure portfolio. He was the architect of the GDS operating model and lead design engineer for lab and datacenter projects worldwide. In 2006, he became the GDS Director. Since taking over the group, Dean has delivered the GDS strategy, executing the largest technical infrastructure consolidation in Suns history compressing the 1.4 million square foot portfolio by 48 percent, a $250 million dollar investment. Dean holds numerous industry technical and business board positions including Founder of Data Center Pulse (http://datacenterpulse.org) - an exclusive datacenter owner community. Dean was featured in Contrarian Minds, a focus on the engineers, scientists and dreamers of Sun Microsystems. Dean was also the recipient of Suns prestigious Innovation Award presented by Suns CEO, Jonathan Schwartz and CTO, Greg Papadopoulos in July of 2008. Dean lives in Northern California with his wife and daughter.
Serena Devito
Serena DeVito is a Datacenter Design Engineer in Suns Global Lab & Datacenter Design Services organization. Serena joined Sun over ten years ago and has spent the bulk of her career as a system administrator within the IT organization where she designed, deployed, and supported critical infrastructure for the company. She then moved to the Software organization and managed the beta testing project for the Sun Cluster product group. Serena acted in a customer capacity running early releases of Sun Cluster software on production home directory clusters with over one hundred active users as well as clustered web services. Serena identified, documented, and filed bugs while supporting one of Sun's largest and complex engineering lab environments. Serena has been a core member of the GDS team since 2004. In December 2005 she joined the GDS organization to drive lab and datacenter consolidation projects worldwide. Along with building numerous labs and datacenters in the Americas, Serena was the technical lead and customer engagement engineer on the Santa Clara, California datacenter consolidation project, one of the largest and most complex datacenter consolidations in Sun's history. For this work, Serena was the recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg Papadopoulos in July of 2008. Serena is a graduate of the University of Adelaide (Australia) with a Bachelors of Architectural Studies and a focus in Computer-Aided Design. She is a Sun Certified
40
Summary
Solaris Administrator for the Solaris 7, 8 and 9 Operating Systems. She also loves traveling and riding horses.
Ramesh KV
Ramesh has been in the technology industry for over 13 years and has held a variety of technical & managerial roles since joining Sun in 2001. Ramesh began his career as a customer support engineer at Kumdev computers, where his clients included companies in the manufacturing, pharmaceuticals, telecom and IT industries. He later joined SAS (now know as SASKEN) as a Systems Engineer, where he supported the NORTEL infrastructure. He then moved on to TATA Elxsi as a Design and Development Specialist, where he was involved in various key infrastructure projects. Ramesh spent the initial years at Sun working for the RPE organization as Global Lab Manager, where he drove the global lab model and designed, deployed, and supported critical infrastructure across USA, EMEA & India. In early 2006, Ramesh moved into the role of IT/ Lab Council head for the India Engineering Centre (IEC) reporting to the site VP as well as the Global Lab & Datacenter Services (GDS) lead for the APAC region. In this role Ramesh led the physical and technical design for the Bangalore, India Datacenter consolidation project and supported all APAC lab and datacenter projects. In 2007 Ramesh became a Regional Work Place Manager (RWPM) covering Asia South and India in addition to his GDS responsibilities. In this role he has been responsible for all workplace resources activities across Asia South & India, including portfolio management, real estate services & facilities management. Ramesh has been a critical member of the GDS technology team since 2004 and has designed multiple labs and datacenters across Asia Pacific. Ramesh is the recipient of a research disclosure on Web Based Multi OS Installation, PC Quest IT Implementation project of the year award in 2007 and has received 12 internal awards globally including Sun's leadership award in 2005 and 2007. Ramesh was the recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg Papadopoulos in July of 2008. Helives in Bangalore, India with his wife and daughter.
Petr Vlasaty
Petr has been working in the IT industry for more than 12 years. His experience that started as an implementation engineer and project coordinator in 1996 has grown into a variety of people, project, process and service management areas of datacenter operations and technical support services. Petr worked for ANECT to define and develop the System Support and Maintenance department responsible for systems implementations and support for key customers including the Czech Ministry of Finance, Commercial Bank, and others. Petr formed a 12-person team that defined internal and external processes, established contracts, and represented support services to customers. In November 2004, Petr joined the Sun Software organization as
41
Summary
R&D Datacenter Manager and Team Lead for the Prague Datacenter Management Team. Petr designed and implemented two internal datacenter projects in Prague in partnership with the GDS team. He applied the GDS processes and standards for the new Prague datacenter expansion that was completed in June 2006. Due to Petr's outstanding performance in Prague, he was assigned as the technical lead for the Louisville, Colorado consolidation project in August of 2006. He spent four months in Colorado developing the very detailed and complex plan. Petr joined the GDS organization full time in January of 2007. He is responsible for the Europe, Middle-East and Africa (EMEA) region and continues to act as technical lead for the Colorado datacenter consolidation project, the largest and most complex in Sun's history. Petr is a Staff Engineer, Datacenter Architect, and the GDS Technical Infrastructure Lead for the EMEA region. He loves sports like mountain hiking (year round, especially winter) and rock climbing. Petr lives with his wife Vera and 10 month-old son Erik in Prague, Czech Republic. Petr is a CCSE (Check Point Certified Security Engineer), CCSA (Check Point Certified Security Administrator) and SCSA (Sun Certified Solaris Administrator). Petr was the recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg Papadopoulos in July of 2008.
Brett Rucker
Brett Rucker is a Staff Engineer in Suns Global Lab & Datacenter Design Services organization where he focuses on physical environment issues. Brett has been in the Corporate Engineering, Construction and Management business for 25 Years. A Graduate of the University of Colorado, Boulder, and a licensed Professional Engineer, Brett began his career with Ball Aerospace Systems Division, a rich opportunity that gave him complete design and management responsibility for all construction projects. He supervised products including building additions, clean rooms, X-Ray rooms, datacenters, laboratories, anechoic chambers, secure areas, gallium arsenide facilities, high elevation satellite dish structural mounts, RF shielded rooms, and office, conference, and customer show areas. Highlights of Bretts time at Ball include design and construction of large secured program area in 7-days to support critical Corp. project, stick build design and construction of MAMA Lab Clean Room facility for 1/3 market cost (still in operation today with one of Ball's premier products), metal finish Area with a $30,000 per year energy savings in 1992 base, and creation of the Facility Engineering Program at Ball Aerospace which grew to take over Ball corporate facilities. In 1992, Brett joined a newly forming engineering group at StorageTek to address facility backlog and project issues. Brett and his team created corporate building specifications, drawing standards, and quickly eliminated all backlog. Brett drove the high tech areas, and HVAC infrastructure as well as general projects. He represented StorageTek on refrigerant conversion and reported on this project at an ASHRAE
42
Summary
function. Brett became manager of the New Project Management Group in 2000 directing projects at headquarters and in the field. His responsibilities soon included StorageTeks energy management program, which deployed processes and delivered high-quality, low-cost projects with consistent high metrics with limited staff support. This group lead all project designs, construction and project budgets. The energy management group delivered ISO Certification, installed monitoring systems, metrics, monthly log and issue reviews, best payback energy projects and commodity contracts that beat the market consistently. Brett joined Suns GDS team in 2006 as a design engineer and is the physical design lead for the largest and most complex datacenter consolidation in Sun's history, the StorageTek move from Louisville, to Broomfield, Colorado. Brett is a member of ASHRAE, and AEE, and he was the recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg Papadopoulos in July of 2008. Brett is active in developing youth programs and coaching his daughters competitive fast-pitch team with great support from his wife and two daughters.
Brian Day
Brian Day is a Senior Program Manager in Suns Global Lab & Datacenter Design Services organization. Brian has been working in the technology industry for over 10 years, of which more than five have been with Sun. He spent over two years in Suns Systems group, developing automation programs for server, storage, and software QA functions. This work included testing, configuring, and administering some of Sun's largest systems at the time, including the Sun Enterprise 10000 and Sun Enterprise 6500 servers, and Sun StorEdge D1000 storage device. Brian then served in a similar role for iPlanet software, developing server side Java technology-based tools that automated core team processes (ticketing system, equipment tracking, on-call notification). Brian left Sun in 2000 to pursue his MBA full time. He then joined Adobe Systems in 2001 as a Program Manager, leading the development of several Digital Imaging plugins for Adobe Acrobat. In 2003 he took over full time program management of the Photoshop Elements product, releasing an all new version (3.0) in the fall of 2004. This included the coordination of all key product teams (engineering, QA, marketing, localization, documentation) and ensuring a successful release for the critical holiday shopping season. Brian returned to Sun in early 2005 to program manage the GDS initiative. As Senior Program Manager & Chief of Staff for GDS, Brian is responsible for first engagement with new customers, ensuring world-wide GDS projects are resourced, deliverable expectations are set, and commitments honored. Brian's work in GDS has significantly contributed to Sun's modular design and the teams success. Brian was the recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan
43
Summary
Schwartz ad CTO, Greg Papadopoulos in July of 2008. Brian holds a Bachelor of Science in Computer Science and a Masters of Business Administration.
Acknowledgments
The authors would like to thank Steve Gaede, an independent technical writer and engineer, for his probing questions, deep insights, and ability to create a coherent story. Steve is a frequent contributor to Sun Microsystems technical documents, including many Sun BluePrints articles. He is a member of ACM and USENIX, and is active in the Boulder Colorado professional community, having been a coordinator of the Front Range UNIX Users Group since 1984. Thanks to Mark Monroe for providing input on power supply efficiency trends, and to James Monahan for his detailed review comments. Thanks also to the Sun MD team members who spent time contributing to this paper, including Maurice Cloutier, Liz From, Brian Kowalski, and Bob Schilmoeller.
References
More information on Suns energy-efficient datacenters can be found at:
http://www.sun.com/datacenterdesign.
Ordering Sun Documents

The SunDocsSM program provides more than 250 manuals from Sun Microsystems, Inc. If you live in the United States, Canada, Europe, or Japan, you can purchase documentation sets or individual manuals through this program.
Accessing Sun Documentation Online

The docs.sun.com web site enables you to access Sun technical documentation online. You can browse the docs.sun.com archive or search for a specific book title or subject. The URL is http://docs.sun.com/ To reference Sun BluePrints OnLine articles, visit the Sun BluePrints OnLine Web site at: http://www.sun.com/blueprints/online.html
Energy Efficient Datacenters: Electrical Design
On the Web sun.com
Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com
2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, BluePrints, iPlanet, Java, N1, Solaris, StorEdge, Sun Enterprise, Sun Fire, and SunDocs are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the U.S. and other countries.Information subject to change without notice. Printed in USA 3/2009

DATACENTERS Electrical Design

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DATACENTERS Electrical Design

Uploaded by

Copyright:

Available Formats

ENERGY EFFICIENT DATACENTERS

Part No 820-6213-10 Revision 1.0, 3/11/09

Sun Microsystems, Inc.

Sun Microsystems, Inc.

About This Article

Sun Microsystems, Inc.

Traditional Datacenter Electrical Design

Sun Microsystems, Inc.

Traditional Datacenter Electrical Design

MORE THAN ONE OF EACH DEVICE INSTALLED AT EACH LAYER

Utility Power and Transformer

Traditional Datacenter Electrical Design

Sun Microsystems, Inc.

Traditional Datacenter Electrical Design

Sun Microsystems, Inc.

Traditional Datacenter Electrical Design

Sun Microsystems, Inc.

Rack Power Distribution

Datacenter Tier Levels

Traditional Datacenter Electrical Design

Sun Microsystems, Inc.

Suns Approach to Electrical Design

Increasing Electrical Distribution Efficiency

Sun Microsystems, Inc.

Increasing Electrical Distribution Efficiency

Increasing Electrical Distribution Efficiency

Sun Microsystems, Inc.

Alternatives to Battery-Based UPS

Increasing Electrical Distribution Efficiency

Sun Microsystems, Inc.

Equipment Power Supplies

Efficient Percent Efficiency

45 0 Percent of Name Plate Power Output 100

Increasing Electrical Distribution Efficiency

Sun Microsystems, Inc.

208V Vs. 230V

Alternating Vs. Direct Current

Power Supply Unit (PSU)

480 VAC Supply Power

Increasing Electrical Distribution Efficiency

Sun Microsystems, Inc.

Appropriate Levels of Redundancy

Increasing Electrical Distribution Efficiency

Sun Microsystems, Inc.

Choosing Tier Levels

Using Distributed Redundancy

Increasing Electrical Distribution Efficiency

Sun Microsystems, Inc.

Increasing Electrical Distribution Efficiency

Sun Microsystems, Inc.

Increasing Electrical Distribution Efficiency

Sun Microsystems, Inc.

Flexibility Through Modularity

Sun Microsystems, Inc.

Flexibility Through Modularity

Datacenter Power Distribution

Hard-Wired Power Distribution

Flexibility Through Modularity

Sun Microsystems, Inc.

Power Distribution Units

The Benefits of Modular Busway

Flexibility Through Modularity

Sun Microsystems, Inc.

Two busways above each rack Busway cans