Whitepaper

Data Center Site Selection

Cost, Risk, and Operational implications Written by: Mark Mac Auley mark@blunthammer.com

Data Center Site Selection is the most important process a business can go through. The reasons, while numerous, all tie back to the two most important criteria when selecting a facility where you will put valuable IT equipment and data 1. Cost 2. Uptime Virtually every single decision in the process is vetted against these two things with a common denominator of risk mitigation. Dont believe us? Why isnt every single data center a Tier IV data center? They are the most secure, most resilient, and allegedly best data centers on the planet because a Tier IV is the highest Tier in the industry. The answer is simply - Cost. Everyone wants a Tier IV facility until they see the price tag. If everyone were willing to pay for a Tier IV facility then data centers would all be Tier IV facilities. You could still put a Tier IV facility in the wrong place (fault line), and youll have a really expensive inoperable, or irreparable facility just the same. Site selection matters for many industries, not just data centers. Ask Okuma, Fukushima residents. This whitepaper is focused on site selection within the United States. We will cover at a high level the things we pay attention to when we work with clients. Our decision criteria is not a snowflake model like every data center on the planet is, it is a standard methodology that has been honed over 20 years and made to be inclusive of clients business needs while factoring in the degrees of risk that accompany each business decision no matter where a client ultimately ends up. Blunt Hammer is a consulting company that provides services to clients in the US who are trying to do three things well: 1. Consistently evaluate & scorecard sites for data center project requirements 2. Implement a disciplined strategy and underlying methodology for the future 3. Build the business case to support and prove a sound decision We have started data center companies, worked for owners, negotiated leases, quantified incentives, and we have developed the only consistent site selection guide that spans 20 years of experience. We are data center insiders with a network of experts supporting us to cut through the spin and hype, distill processes to help make sound decisions, and deliver real value. Our projects for 2013 include: Delivery of a 150 page business plan with comprehensive financial model to investors Site selection for 6 sites across the US Feasibility study for a municipality looking to attract another data center project Producing a detailed colocation pricing survey that included 26 vendors in 10 states

Table of Contents

Contents
Costs.......................................................................................................................................................... 4 Power .................................................................................................................................................... 4 Taxes ..................................................................................................................................................... 6 Equipment............................................................................................................................................. 6 Risks .......................................................................................................................................................... 8 Seismic .................................................................................................................................................. 8 Flooding................................................................................................................................................. 8 Tornado................................................................................................................................................. 9 Civil unrest ............................................................................................................................................ 9 Operational ............................................................................................................................................. 10

Costs
Every business decision is vetted against cost. Period. Costs for any data center are most influenced by 3 things: 1. Cost of electricity 2. Taxes 3. Equipment This is how the costs stack up for any facility anywhere in the US and need to be factored into ANY site selection. Will you find an exception? Perhaps, but we have not seen any meaningful variance from these three buckets. Measurable, sure. Meaningful, no. The other thing youre probably looking for that didnt make the list is the cost of Real Estate. In two recent projects, the cost of real estate was 2% and 1.8% of the overall cost to open the doors. Real estate costs didnt even move the needle. Again, measurable yes. Meaningful no.

Power
The cost of power is number one for a reason. Computers turn electricity into heat and the more computers we run, the more power it takes to run them and deal with the heat they produce. The increased heat we need to deal with adds to the power consumption over and above the power going to the computers, and helps make the cost of equipment #3 on the list. More on that later. Power costs vary widely in the US. It would be convenient to blame the cost disparity on deregulation or lack of a cohesive energy policy at the Federal level, but that doesnt change the reality that it costs more in certain states than others. The US Energy Information Administration has a handy website that tracks cost data, sales data, demand, capacity, and a lot of other data about the electricity market in the US - http://www.eia.gov/electricity/data.cfm - and they keep it up to date very well. This website is a great place to start to get an idea on a per kilowatt (KW) or per megawatt (MW) cost. What the site has shown us is where data centers should be located, and inversely, where you will pay a premium - based on the cost of electricity. In one project we had looked at New York Manhattan and Upstate - as an option for a client, within a day we ruled the NY option out because of power costs (and taxes). Gives us a little more insight as to why there are so many data centers in New Jersey it wasnt just 9-11-01 cost has a lot to do with it. We looked at several states across the US as part of a site selection study to determine a cost basis for the States on the cost of power, since the States and geographies had met criteria for other site selection items focused on environmental risk factors. We focused on the top four states for the study on cost of power and the four cheapest states were Utah, Virginia, Texas, and Georgia.

Average $/kW-mo by State

254 231 200 153

UT

VA

TX

GA

These costs were ranked against specific counties/areas, NOT ranked across the costs in all 50 states. The fact is - some states we wouldnt put a facility in for ANY reason (yes there are facilities in those states) so keep that in mind. So when we look at costs, these were the four states that it made sense to us to put a data center based on the cost of power and our standardized site selection criteria. Take a look at the EIA site for a lot more detail on power costs before your next data center strategy meeting. The next question might be so what? Or why should I care? Because data centers use a lot of electricity if you reduce the cost of the thing you use the most of in your business you save money or increase margins. Here is a quick example, in real numbers: Data Center Foxtrot uses 4 MW of power on a sustained basis at a cost of 12 cents per kwh. The electricity bill each month for the 4MW facility is ~$350,400. Data Center Tango uses 4MW of power on a sustained basis at a cost of 6.5 cents per kwh. The electricity bill for that facility is ~$189,800. That means 54% less every month. So in reality if a data center is located in Massachusetts vs. Utah, you can expect to pay more month to month every month for the term of the contract. Just for power. The assumptions here are that you can get metered power and pay for power separate from rent. Some places offer this, many dont, so ask. Since power is the single largest expense for a data center, then if you save money on the biggest expense they have (in a third party or multi-tenant facility) then you save money on the biggest. If you are building your own facility greenfield or retrofit the same rule applies. Google just bought a wind farm to cut their costs of sustainable power. You will use more electricity than anything else in running your data center so get that cost as low as possible.

Taxes
Taxes are the number two biggest expense, and by far the most cumbersome. There are many taxes that will kick in for an owner/operator AND for a tenant, and given their prominence as a cost factor, you want to make sure you perform due diligence on the taxes. The single biggest issue we have seen over and over in multi tenant facilities (third party facilities) is that they will negotiate and receive tax benefits, incentives, and reductions for themselves as the owner operators, but seldom do they get passed along to the tenants. Here is why taxes matter as much to tenants as to owner/operators: The value of the real estate is dwarfed by the value of a rack of blade servers. A single rack of 64 blade servers can run $300,000 in our example here. That is as much as a home in many parts of the country. A megawatt of electricity will be consumed by ~80-100 racks at average density. That is $24,000,000 30,000,000 of taxable equipment in a relatively small space. A lot more than your average washer/dryer in a home. As a tenant, you want to make sure that you are receiving the tax benefits, either as a pass through, or as a direct benefit from the state/county/city (taxing entity) if you are going to deploy data center equipment someplace. Its your equipment and the landlord gets the benefit. Why? The places that understand this are passing laws to combat the inequity. Servers are expensive, and we buy more of them every few years vs. fewer. That means there is more taxable income for municipalities where facilities are located. As an added bonus, the servers create no drain on resources (schools, police, fire department, public assistance, etc), and add to the tax rolls through sales tax on electricity in many places, gas tax for generator fuel, payroll taxes for good paying, albeit few, jobs and additional valuations for the cities and towns they locate in. We are not tax experts, but tax students. We study which taxes are in place and what they total, understand why they matter (or not), and coach our clients on how to ask the right questions when negotiating whether they are the landlord or tenant or owner/operator. We also know of tools and instruments that help companies and municipalities structure legitimate ways to craft a package that is equitable so a data center operator doesnt wind up contributing 23% of an entire countys tax base from one facility. Work with a CPA and your finance folks to get into detail on this that matters to your business, but as a rule, the property tax is where youll want relief because the other taxes in many cases are the smaller part of the tax cost overall.

Equipment
This is number three on the top 3 costs list. It is a big number because as we stated before, computers turn electricity into heat and the computers dont like a lot of heat for long periods of time and so you need expensive equipment to deal with it. The equipment number is tied to two buckets of equipment types mechanical (cooling) and electrical (distribution). Both of these buckets are heavily influenced by the design of a facility. One client we worked with had paid for two designs to be completed from two different respected firms. One was $9M/megawatt the other was $5M/megawatt. Same building, same footprint, same capacity. Very different design.

The first design was what we had referred to as a traditional design two 4MW phases. Large mechanical (cooling) footprint and cooling equipment deployed for 4MW at a time. The electrical gear included two 2.5 MW gensets that would scale to four units, big switchgear for 10MW, and 4MW of UPS at a time. While beefy and big and a decent design overall, what killed it was the inefficiencies in the design. The piping and cabling and conduit was roughed in for a full facility not a facility deploying its first set of cabinets and that stuff is expensive. Buying big gen sets and other distribution gear out of the gate is like someone in Scottsdale Arizona buying a dumptruck & highway plow just in case it snows in Scottsdale while they live there. Not very practical. The second design was VERY different. The approach was to chunk up the deliverable capacity into chunks that were optimized for the equipment vs. optimized for the future load. The scale was sized to deploy two megawatts at a time because that was the sweet spot for the size and cost of the equipment. Yes there was more equipment to buy over time, but there was never any stranded power or unused capacity that was being paid for and not being used by the operator or the tenant. It also provided more flexibility for maintenance and redundancy when needed, and for tenants who wanted their own dedicated infrastructure, it could be deployed in chunks for them too and fully optimized for cost. Either way the equipment is expensive. Its still millions of dollars per megawatt. When you add in costs of taxes on the equipment and the inefficiency of any equipment deployed (which will use more electricity than optimized equipment) it moves to the number three spot on the cost list. Adding to the equipment costs, the electricity cost and tax cost are added on top because the equipment uses electricity to run, and its expensive and the taxes add up. Mitigate costs by looking for inexpensive electricity, reasonable tax basis, and a properly sized backplane whether youre building the facility or not. Cost is cost and lower cost is obvious. Now that we have established what the top 3 costs of ANY data center are, lets look at risks. If you put a data center in wrong place, its still a data center in the wrong place. Here is a map of Presidential Disaster Declarations as a visual reference of where the action has been up until 2007.

Risks
1. 2. 3. 4. Seismic Flood Tornado Civil unrest

The similarities of all of these is that there is little if any meaningful warning. Others will (and do) argue that there are more, and depending on the client, and the business risk they understand and want to mitigate, there are more. These are the four horsemen we dont like across the board. And the fourth one is seldom considered, and most dangerous since people are involved.

Seismic
Seismic threats are at the top of the list because they have no warning and affect large geographic areas. We know you can retrofit a building, and put several counter measures in place at the facility to keep damage to a minimum and many do that. What happens outside the property line though? Did the utility company take the same care? What about the fiber conduits? Seismic events impact a large area, so while the facility may be fine, except for a knocked over bobblehead, or picture frame on the floor, what about the fiber vault 3 miles closer to the epicenter that you rely on for your telecommunications that wasnt seismically reinforced? Or the power lines that while buried coming into your facility, are aerial 5 miles away and sway like amber waves of grain during an earthquake and snap power lines like a tuna on 10 pound test line? Seismic events have little warning and while the facility may not be affected directly, the infrastructure leading into it from miles away likely will be. The other associated risk is that the facility may not be accessible in the event of a big one. Bridges pancake, roads get ripped up, debris can close highways making it difficult to get to the facility in the event you need to. Its not just the earthquake, its the collateral impacts you want to mitigate against as well. For clients with facilities in California we recommend a very different strategy for risk mitigation for seismic threats then those in New England.

Flooding
The recent floods in Colorado drive this point home, as did what happened during and after Hurricane Sandy. Flooding happens fast. Way faster than most people understand. A creek can turn into a river in under an hour, washing out roads, bridges, and critical infrastructure miles away from the facility youre in or you own. Water is relentless. It will sink to the lowest point which means you want nothing below sea level, and nothing important in a basement. Almost 15 years ago, way before Jim Cantore talked about superstorms, we walked through a facility in Houston Texas for a global concern. It was their main US data center, and it was in the basement of their building. The building was at sea level which meant the data center was below sea level. They built it there because people didnt like working in a basement and computers didnt mind so they put the computers there. That were important globally. When asked about the reliability of power the response was great unless it rains for more than two days, then we have problems. The people in Europe really get upset. There was the story too of fuel pumps in Manhattan being in the basement near the fuel tanks to pump diesel to the rooftop generators at some providers, and when the water from Sandy sank to the lowest

spot they had fuel, and they had generators, but the fuel couldnt get to the generators because the pumps were underwater and wouldnt pump the fuel to the roof where the generators were. Think about water in whatever design or facility you go into. Not being in a floodplain is a great start, not being below sea level or near a creek bed is even better.

Tornado
Tornados are another event that has little warning. There are seasons for tornadoes so the risk is contained somewhat by time of year, but the effects of climate change make the season length a moving target as well. Like seismic, mitigating against tornadoes means deploying a different strategy if you absolutely positively want to have a data center in a tornado prone area. One interesting data point I want to point out is that Virginia and Texas statistically have almost the same number of tornadoes. Texas on average has 155 tornadoes across the state while Virginia has 18. Based on that data you think immediately Virginia is safer. Texas is also 6.24 times the size of Virginia making the statistical comparison 118/155 in a Virginia/Texas tornado throw down. Both are mature data center regions in spite of this risk. The risk is not as much that a data center facility gets hit, its that the infrastructure between the facility and the point of interconnect gets hit and ripped out. If youre going to put servers in a tornado zone, make sure you have a few someplace else to hedge the risk.

Civil unrest
This is probably the most consistently overlooked risk in the data center industry. Civil unrest has a few different forms bomb threats, cable cuts, terrorism, riots, flash mobs, dirty bombs, social engineering, government policy, martial law, and peanut butter to name a few. Yes peanut butter. At one point or another I have experienced directly or indirectly the effects of all of these types of civil unrest. These are the social problems that come out of nowhere and can shut things down for hours, days, or for good. When I worked at one facility every March and April you could count on several bomb threats being called in. The data center was in the same building at the IRS and some people get crazy around tax time. The phoned in threats disrupt things because you have to leave the building, and you never know if they are real or not, so you treat them all as real. The building is still there and every March and April you can count on some extra hours outside waiting for the building to be swept and listening for the All Clear. Once I was in the same building and we dropped one of two physical connections to the networks. All of them. The facility had redundant conduits from different sides of the building so it didnt drop the facility, but it certainly created a stir in the Network Operations Centers across the US. Turns out someone had parked an official looking telephone company-like white truck in the street, popped the manhole cover, and gone underground with a saw and cut 6 out of the fiber feed, and left. In and out, a high tech smash and grab. The following week the manhole covers were welded in place. Lessons learned. In Boston, pressure cookers detonated by cowards at the finish line of the Boston Marathon shut the city down for 3 days. No warning, and martial law was the immediate result, which meant coming into the city was impossible - servers or no servers. I was in Los Angeles the day of the Michael Jackson funeral. Traffic was gridlocked to the point where I got out of my cab and walked close to two miles to

One Wilshire because it was faster than a cab. Thankfully I packed light and it was not an emergency or a permanent or long lasting situation. These are a few examples from real life that give us perspective that we can pass long to clients as they look at real risk. It also shows us what may be a popular decision may not be a good decision, and while there are some data center clusters in the US that are popular, a well placed rolled over tanker of hazardous material, a biological agent, or dirty bomb would affect close to half of all internet traffic worldwide. In one event in one spot. The reason we include peanut butter as a risk is because a 5 gallon bucket of peanut butter dumped into a manhole cover in most major cities would cripple the telecom infrastructure. Its not the peanut butter really, its the rats. Rats like to chew through cable insulation anyway and when we were working with one client, they pointed this out because they have seen just how many rats there are underground in at least three major US cities, and the damage they can do with or without peanut butter.

Operational Impact
The final point we will make in this whitepaper is about operational risk. We have alluded to it throughout the document and it warrants further discussion because even if you have a great facility in a great location, if you cant run it when things go south then no SLA will help you very much. Operational risk we assess by looking at how accessible a facility is day to day, and in the event of an emergency, what changes? Drawing on past real life experiences and following a set methodology in site selection, we look at operational risk. This is not a complex process but it requires thinking it through and discussing it at a minimum, up to and including a practice drill to test assumptions. Look at the policy and operations guides. Notice how you are screened before you enter a facility. Notice how much cardboard there is on the floor, in cages, or in hallways. A tour is designed to be the best case scenario operationally, so if its sloppy that day, its probably sloppier the other 364. Our rules of thumb at a minimum: Make sure the facility is a tank of gas away from an airport, preferably several. Make sure there is a route to travel that has as few bridges as possible Make sure you have provisions to keep your staff fed and hydrated during an emergency Dont lease space in major metros if you can avoid it. Cities have lots of people. And rats. Think through the common sense stuff first. Leave Hollywood scenarios for the big screen

Make sure if you have a facility in a high risk area like a basement in New Orleans or Houston that you have a replicated subset of critical applications and infrastructure someplace else where a simultaneous emergency is not likely to occur, and where you can have or get access to it quickly if needed. It keeps the whats going on? phone calls to a minimum during emergencies. This concludes the whitepaper and we hope you have gained some insight as to how to evaluate your next data center selection decision. If not we can help figure out whats important. Call us and ask us about or Site Selection Guide. It is our standard methodology we scorecard facilities against. mark@blunthammer or 617.209.9899