Add to the real estate generators, HVAC, etc. and you can charge a premium for conditioned colocation space

Support Infrastructure

“While today's information technology is incredibly reliable, computer systems still contain electronic circuitry that is highly sensitive to environmental conditions—for example, temperature, humidity, dust, and "dirty" power. And larger events, such as power outages and natural disasters, pose an even larger threat to your systems and data. Seemingly unnoticeable environmental, design, and structural elements can render your core systems inoperable in a heartbeat.”  -- Computer room design/build: Building Performance into your Data Center.  http://www.sequent.com/cs/computer_room.html

Common threads surrounding the build versus buy decision include core competence and expertise; should the company focus on its core competency, that which it does exceptionally well, and does the firm have the prerequisite expertise to execute? Next we’ll broadly describe each system and some issues in the implementation of the systems.

Security Systems Model

Security includes all systems protecting the physical assets in the Internet Data Center. This includes perimeter and in-building access control (biometrics, card keys, old school keys, magnetically sealed doors, etc.) and monitoring systems as well as physical add-ins like bulletproof glass and perimeter walls.

There is a key difference between outsourced data centers and those built and operated in-house. Shared Internet Data center Security Systems must handle the added complexity of multiple customer access. Solutions range from escorted access and supervised use to free (zoned) access. This complexity may not be required if built solely for the use of one company.  The common functionality is to keep non-authorized staff from the networked equipment.

The cost and scaling of the security depends more upon the application of the data center than on the scale of the center. For example, a small section of a  building that is essential to the operation (like the cashier office in a casino) may be equipped with armed guards, many surveillance cameras, etc. Likewise a shared facility may require more participant cages and authentication than a single user facility.

Having said all of that, economies of scale exist that allow the expense of circulating guards, bullet proof glass and Kevlar perimeter protection, that a smaller facility simply can’t afford.

The cost function shown below shows the financial build versus buy cost function. By sharing conditioned space (already complete with security systems) the cost is shared proportionately across all participants. As a result, the cost of security is fixed on a per rack basis. Conversely, building requires purchasing the entire security system and allocating the cost across the number of racks the company uses.

Figure 5 - Cost function of Security Systems

It was pointed out that customers tend to demand more security from a shared facility than they would require from their own facility simply because that which the collective money can afford, a single company can not individually afford. For example, integrated environmental (temperature, humidity, electric draw) and security (surveillance cameras on both sides of racks, cage access log) information feeds may be reasonable since the cost per rack is relatively low. The end result is a shared environment with greater than required security systems.

In the build case, restricting access to company employees (perhaps) reduces the access concerns and reduces the need for cages and therefore allows a higher rack density. Security can be built to company specifications, where buying may require accepting a level of security that is inappropriate for the application.

One practical implication of the shared data center security is access. When equipment must be repaired during off hours, some Internet Data Centers (particularly telephone company facilities) do not make it easy to gain access to the failed equipment. Others require escorts to continually monitor activities in “shared” areas; given a limited number of available escorts, repair technicians may be required to wait for an available escort.

HVAC Systems

Heating, Venting and Air Conditioning (HVAC) systems are critical for maintaining an operations environment for networking and computing equipment. According to IBM, computing machinery operates best between 18° C (65° F) and 29° C (85° F) with humidity between 20% and 62% relative humidity. Too much humidity can cause system failure. Too little humidity generates static electricity. IBM recommends an optimal temperature of 24° C (76° F) and 45% relative humidity.

HVAC infrastructure provides this conditioning and benefits from economies of scale. Large scale Internet Data centers can implement high cost and high efficiency HVAC systems and share the costs across a potentially large number of customers. These larger systems have the added benefits of requiring fewer units to perform the same task, resulting in higher Mean Time Between Failures and lower Mean Time To Repair. Sean Donelan argues that since fewer units are involved, there is a lower chance of more than one breaking at the same time before technicians can fix the broken unit(s).

For the financial analysis, assumptions include that the Internet Data Center has no windows, is well insulated in the walls and ceiling, has no office space, and operates in a lights out environment.

Cross over point at about 500 Racks where Water Cooled is Cheapest Per Rack Solution

Figure - Practical domain of HVAC system operations

From a practical perspective, DX systems can not be supported beyond 10 units. The fans must be continually checked and repaired, providing an undue maintenance issue beyond xx racks. Likewise, air cooled systems operate most effectively where 25-300 tons of cooling are required.

The build versus buy decision would therefore need to take into account the scale of operations and the comparative fixed and recurring HVAC costs.

The HVAC selection impacts the power requirements as well. Air cooled systems are more efficient than DX systems and water cooled systems operate more efficiently still under load.

Power Systems

The same economics for build versus buy and similar cost curves apply for power systems as did for HVAC systems, but the economies of scale are less dramatic.

The primary function of the power systems are to reduce the effect of short term and long term disruptions of utility power, and on occasion, the reduce the load on the utility grid during peak hours. The most common tool for handling short term interruptions and conditioning the power feeds is the Uninterruptible Power Supply (UPS), a conditioning system and a set of batteries brought on-line when the power feed is interrupted. In most Internet Data Centers the UPS is capable of supporting the full data center load for 15 minutes to an hour.

Long term interruptions are typically handled by diesel generators, switched on when there is a disruption in the power feeds to the data center. The UPS role is therefore simply a transition support until the generators start up. Fuel for the generators typically lasts a few days and up to a week. It is amusing to note that in extreme cases there is sufficient fuel to last a few months. Most data centers use refueling contracts to handle extended duration outage situations.

Increasingly, large scale data centers are leveraging flywheel systems which replace the diesel generator - UPS combination with a large constantly spinning flywheel that can handle disruptions in service using the electricity generated from induction from its continued spins. This provides the generators enough time to start up and take over the load.  There are religious arguments over which system is better.

General Note about Redundancy

All modern data centers have at least N+1 redundancy on these core infrastructure systems but the design of the N+1 systems require careful evaluation of each an every failure scenario. Consider the following N+1 “redundant” HVAC system below. While each chiller can support the entire operation, reparation of the broken unit may require a disruption in service to the operational unit.

While it may seem obvious that each system must undergo these fault analysis, in the field this represents a challenge as plans change during the construction process. Well established data center companies typically don’t make these mistakes as the experience curve kicks in. Diligence must be applied when evaluating outsourced and in-house constructed data centers to ensure that any single system component can be repaired without disrupting operations. 

The entire system economies of scale kick in for larger facilities that can allocate the incremental cost of the N+1 unit across a larger value of N.

Another factor kicks in when looking at the evolution approach of Data Center Design. If one builds a data center incrementally (starting with the minimal power systems) and adds components (generators for example) incrementally, it is possible to expand the system such that the N+1will fail since the ‘N’ component that fails could be a larger component than the set of smaller components.

Support Infrastructure for Internet Data Center

After an initial fixed cost deployment, we see that the cost of a DX system tends to approach a flat cost, in this case, about $525/rack.  At medium to large scale, alternative HVAC systems should be explored as they can realize much greater efficiency as shown below.

Support Infrastructure for Internet Data Center

Support Infrastructure for Internet Data Center