Critical Level data Center

A CRITICAL-LEVEL DATA CENTER

In articles and trade publications concerning data centers, most authors group all Data Centers into one large category and attempt to address them as being equal. This is not reality in the Data Center world. Data Centers’ final configurations are a direct result of a concerted Risk Analysis study and numerous discussions and can be generalized by the question “What events can I risk occurring given the amount of monies I have to spend?” It is indeed a rare occasion where there are not some risks accepted in the design and construction of a Data Center in an effort to minimize expenditures.

There are Data Centers that, by the very nature of their value and operational environment, must be as secure as possible in all ways and possess the redundancy to withstand almost any contingency. The vast majority of Data Centers does not fall into this high end group and should be designed to the level they are meant to operate within and with a certain acceptance of risks.

The Critical-Level Data Center (CLDC) that we talk about, in this Topic, is the highest level that one would normally find. There are others that go beyond what we will be discussing below but they are so rare that they only need to be addressed to a very small and special group. The CLDC, both the transacting and storage operations, should meet the following minimum standards:

Location:

Location, location, location… Location can be as critical as any of the other functions in that if outside events forces a shutdown of the CLDC or the facility becomes unreachable by its operating staff, it does not matter if it is the best facility money can build.

It should be as inconspicuous as possible so that it does not draw attention.
It should be sited so that it will not be affected by the release of hazardous materials from industrial facilities, main highways, railroads, or pipelines.
Access to the facility should be such that personnel have access to it during and after inclement weather.
The site itself should provide safe and comfortable access from transportation terminus to the building.
Flood plain surveys should be reviewed annually and constant vigilance for manmade changes that could adversely impact this information.
Surface and subsurface conditions in various areas of the country can cause major future structural issues if not recognized and dealt with during the design/construct.
If possible, the building should be fed from two different power company substations with two transformers that are located some distance from each other. Both transformers should be on loop feeds if possible. Underground service should be provided if at all possible.
Telecommunication feeds should be underground, from at least two diverse directions, and the entrances be located at opposite ends of the facility.

Facility:

The building structure itself has to be able to protect the Data Center from all types of natural and man made anomalies. This means that it should be designed and constructed to a safety level well beyond the normal office building.

As much as possible, a data transacting and storage facility should be physically isolated from other corporate functions where permitted by size and usage. This allows the building systems to concentrate on data handling needs.
A “Hardened Facility” is desired. This translates into constructing a building within a building where if the first physical level fails, there is a backup. This is especially critical in roofs which are very susceptible to wind and other damage and water intrusion.
The facility should be protected from being penetrated and/or damaged by errant vehicles.
All critical components should be designed to where they are protected from malicious damage.
Fresh air intakes should be positioned so that they cannot easily be tampered with.
Isolation of domestic water should be available to prevent hazardous intrusion.
Access control should be established to provide maximum protection to both the operation and personnel. Quite often adequate protection is given to the main entrance but internal service and personnel access points are poorly thought out.
Structural loading, in all aspects, is critical. Many times centers are designed to handle the equipment loads but cabling and others systems are ignored to a point where it can impact the amount of data transacting and storage equipment that can be installed.
Separation of critical functions. Electrical, mechanical, and communications infrastructure should be divided into at least two parts and should be separated within the facility as far a possible from each other.
In all instances earthquake bracing should be considered to some degree.

Critical Systems: (See Topics: Redundant Infrastructure Schematic)

These are the systems that allow the Data Center to function under abnormal conditions.

Communications:

Incoming critical data and voice circuits should be redundant; fiber looped based; and should be provided by more than one provider.

· They should be dual redundant fiber rings originating from different directions and underground if possible.

· Dual sets of head-in equipment in separate sections of the building.

· These should feed to distribution equipment that is, again, kept separate from each other.

· Cabling should travel separate paths to the final piece of equipment (server, storage unit or etc.).

· All head-in and distribution equipment should have self diagnostic capabilities and should be constantly monitored.

· All fiber cables entering the building with steel strength members must have the steel member properly tied to the facility grounding system

· All copper cables entering the building require surge protection devices and also must be properly tied to the grounding system.

Electrical Service and Distribution:

Reliable power service feeds and distribution is essential to a properly functioning Data Center. All of the components must be designed to where they complement each other and flow through multiple switching paths to the final piece of equipment. A very large portion of the power failures that occur are internal to the building and have nothing to do with incoming service, so careful attention must be paid to its design.

Feeds should terminate on two separate power company transformers, located on different sides of the building, fed from separate substations, and both services should be on loop feeds. Transformers should be within secure enclosures to protect from damage and malicious acts.
There should be two onsite standby generators located on different sides of the building. Standby generators should be within secure enclosures to protect from damage and malicious acts. The generator monitoring system should be tied to the Building Management System.
Incoming switchgear should be located in separate areas within the facility with ties between each unit interfacing at a tie switch kept separate from either switchgear unit. This gear should be capable of synchronization between the Power Company and generators. All of the switchboard functions should be monitored by the Building Management System.
Distribution from the switchboards should be designed and constructed so that approximately half of the building’s general power and mechanical systems are fed from each system. Monitoring points tied to the Building Management System should be installed at various points in this portion of the electrical distribution system.
A battery operated backup emergency lighting system should be installed as a second line of safety protection for personnel.
Each distribution switchboard should feed to a UPS system of some style. There are alternate solutions besides UPS systems available that can be considered. Again these systems should be installed in separate areas of the building.
Each of the UPS systems should have sufficient capacity to allow an organized shutdown of the facility in the event of a catastrophic failure. Both systems should have maintenance by-pass breakers to allow normal power flow during maintenance windows. Full diagnostic functions should be available to the Building Management System.
Power from the UPS units must be routed through isolation transformers that can eliminate most circuit noise and line spikes. Monitoring points to the Building Management System should be installed on both sides of the transformers.
Distribution from this point to the final piece of equipment can be designed in various ways, as long as the two paths are kept separate. Monitoring points tied to the Building Management System should be installed at various points in this portion of the electrical distribution system.
Surge suppression must be installed at various points in the system and tied to the Building Management System.
The total system must be (and maintained) load balanced, harmonics eliminated, and a complete fault current study performed.

Mechanical Systems:

Temperature and humidity control in a Data Center, along with good air distribution, are essential to its proper operation. It does an owner no good to execute all of the items listed above in the electrical system if the mechanical system doesn’t have single points of failure identified and remedied in the design.

· To meet the special needs of Data Centers, certain manufactures have developed HVAC units that can maintain very close tolerances for both temperature and humidity. These are a necessity.

· Air distribution is very crucial. It must be delivered in a way that it can efficiently flow through all the cabinets housing equipment. In most cases, it is best delivered from a raised floor system. The floor system should be constructed so that all air outlets have dampers for the regulation of the air flow to eliminate hot/cold ares.

· If central systems are used to produce cooling, the piping systems should be redundant and readily accessible throughout their paths. If water piping is run to the evaporator units inside the Data Center they should be encased to control leaks.

· Fresh air intakes should be located so they cannot be tampered with and should be equipped with dampers so that they can be closed in case of an external hazardous release into the atmosphere.

· Drains to discharge condensate water should be properly trapped.

· Water sensing devices should be strategically place throughout the Data Center (especially under the raised floor area), Electrical Rooms and Communication Areas. These must be connected to the Building Management System for alarms.

· Temperature, humidity, air flow plus all major functions of all mechanical equipment must feed information and alarms back to the Building Management System.

· A convenient way to shut off the domestic water system should be installed for emergency use.

Grounding and Lightning Protection:

Quite often these systems are ignored or not fully understood as to their critical contribution to a properly functioning Data Center. Most unexplained anomalies that shut down single systems can be traced back to improper grounding. The elimination of power surges and current flow on neutrals and grounds go a long way in eliminating such anomalies.

· Grounding should be designed to the specifications found in the Bell Core Standards for Telecommunications Facilities.

· Data Room equipment and cabinets grounds should be routed to a secondary ground buss located in the Data Center area. This cabling should be designed and constructed so that it flows in a smooth path to the secondary ground buss. All bends shall be radius type and all should be installed so that all bends lead toward the ground buss.

· The raised floor system and any other metallic item in the center should have a separate ground path to the secondary ground buss.

· The secondary ground bus should be installed so that the feed to the main ground buss connects to the center of the secondary buss with the equipment grounds connected on one side and all other connections should be on the other side.

· All telecommunications system grounding ties shall be collected and brought back to the main ground buss separately.

· The main ground buss shall be segregated with the telecommunications and data equipment connections on one end of the main buss and all electrical type connections on the other end with the ties to steel, water piping and exterior ground plain in the center.

· There should be a ground test ability that ties back to the building management system. Also, there should be a point outside of the building where periodic test can be performed on the ground system.

· The only place that the lightning protection system connects to the building ground system is underground, outside of the building past the test point.

Fire Protection:

This is an area requires close attention to not only codes and local Fire Marshals’ specifications but also to your insurance carrier. Everyone must have a full understanding of how you are planning to protect your center and the procedures that take place to escalate a potential fire problem to the next level.

· In most instances codes will require fire sprinkler systems. These should be ‘pre-activated’ (Dry pipe system) if at all possible.

· If the operation is manned on a 7-24/365 basis, early warning detections should be the first line of defense. This will allow personnel to ascertain what the problem is and possibly take safe corrective action before involving the local fire department.

· Chemical systems work well in under-floor areas and places that you do not have ready access too. The design of these systems must be carefully looked at because an improper installation will cause as much, or more, damage than a fire.

· Placement and type of fire extinguishers is very critical. Placement should be review by the local Fire Marshal.

· All fire system should be monitored with the Building Management System

Physical Security:

In many cases this is more to keep untrained personnel isolated from critical systems as it is to handle intentional malicious intrusions. Traffic patterns for entry into a center should be carefully designed.

· Shipping and receiving areas can be a troublesome problem if they are not carefully designed and properly secured.

· Vendor and employee entrances, as well as the shipping/receiving entrance, should be man-trapped to control piggybacking.

· Some method of positive tracking of those individuals that enter and exit the Data Center should be utilized.

· A CCTV system should be implemented that has adequate coverage of all ingress and egress points, and in the Data Center itself.

· Excellent record keeping methods should be part of the system.

· The Facility’s design should provide protection for the employees manning building entrances.

· Your security office should be placed where it is as secure as the Data Center itself and should be manned at all times by well trained personnel.

Building Management System (BMS):

The BMS is the nervous system for the facility. Properly designed, it should be able to recognize many potential problems before they become systems threatening. A good BMS is one of the best tools management has to predict and control system failures.

· It should monitor all functions; electrical, grounding, and mechanical.

· Good communications should be established between Building Management, Security, and Operations so that when a problem evolves all are kept informed and ready to take action.

· This system should have incorporated in it, or linked to, an asset management system. Records of part or total systems failures should be complete and set up for both trouble and total system reviews.

· Using all of the information transmitted to it, the BMS will be able make the infrastructure systems operate at their most efficient level thus minimizing power consumption.

Documentation and Procedures:

All of what has been outlined above means nothing if you do not have a complete, and well maintained, set of documentation for the facility and comprehensive and clearly written procedures to follow in its operation.

· A complete operational procedures manual that outlines all of the procedures and escalation procedures and contact information.

· A complete and comprehensive preventive maintenance program that outlines all of the systems within the facility and when and how they must be maintained.

· Disaster planning and recovery planning. This is a major document that covers all areas involved in handling a disaster and the recovery requirements for the facility.

· Records necessary to meet all governmental regulations such as OSHA and EPA on the federal level to local Fire Marshall.

Summary:

The above discussion points just touch on the high points of designing a Critical Level Data Center. Most Data Centers are not require to be designed to the level described herein though many of the points covered should be considered. The closer a Data Center adheres to the above concepts, the closer it gets to meeting the ‘Five Nines’ of reliability that is looked for in all Data Centers.