Power & Cooling

Power and cooling: getting a bigger bang for your buck

Power and cooling are the top two issues in any data center. The answer lies in using advanced power conditioning solutions such as precision power and air control. A modicum of social responsibility does not hurt either says Faiz Askari.

Administrators and IT managers are always under pressure to get the most from their data centers. Doing so entails fine tuning many factors of which the most important are power and cooling.

Today, data centers are evolving at a faster rate due to which customers have to modify or redesign their data centers every five years. Customers can also look for solutions that adapt to the changing needs of the data center without needing additional investment.

One thing that each and every data center manager agrees upon is that power and cooling are the two important factors required for the smooth functioning of a data center. Sandeep Nair, Managing Director, Emerson Network Power (India) Pvt Ltd said, “Even slight variation [in these two factors] can affect the working of a data center, which, in turn, can have a drastic impact on a company’s business. At Emerson, we do not sell products; we provide solutions. Whenever we visit a client, the first thing that our experts do is analyze the data center scope, study the requirements and along with the application engineering team and project management team we design an optimum solution as per the customer’s needs.”

While highlighting the importance and criticality of power and cooling, Deepak Sharma, Managing Director, Eaton Power said, “As data centers increasingly deploy high density, small form factor computing platforms such as blade server technology, power density, energy efficiency and cooling are becoming critical requirements.”

When it comes to power one of the major concerns is Harmonic Pollution especially in data centers where a number of UPS, high density DC power supplies etc. are in use. This seriously affects the supply transformers’ and DG sets’ performance and determinates the lifespan of cables and other equipment used. Acknowledging the power and cooling crisis in data center environments, Pradeep Pimpley, Vice President, DB Power Electronics added, “There are two major things—firstly, you have to conduct power audits. Secondly, you must get proper consultation to reduce power consumption. As power and cooling both ultimately [have an impact upon] electricity consumption, this consumption pattern needs to be checked on a priority basis.” As a developing nation we do not have abundant electricity, and in the near future the situation is likely to worsen. “So the bottom-line for all of us is to use available power more effectively,” he added.

A window of opportunity

The existing scenario for data centers includes reviewing installed power sources and finding any technical solutions that can reduce the energy demand. For data centers that are in the design stage, it is vital to provision for such devices, or to use the latest power conditioning equipment. One should not go only by the specifications; it is a good idea to measure the power output from a sample device and monitor it. A deep study on the efficiency of the devices being used can prove helpful. Even a one or two percent drop in power consumption can result in substantial cost savings in the long run. Rajesh Dhar, Country Manager, Industry Standard Servers, HP India said, “Data center administrator are grabbing every possibility of reducing power consumption without sacrificing on the quality and continuity of power.” Dhar believes that a direct corollary of this trend is that it provides an opportunity for vendors like HP to deliver better solutions.

Originally if a data center had some devices that were power guzzlers, administrators would scatter them around the data center and not worry overmuch about them. Calvin Nicholson, Director Product Marketing at Server Technology said, “Densities inside the data center have increased and facilities with high-density applications put all of the equipment in one location so that they can understand and deal with the power and cooling challenges knowing that there are efficiency advantages in doing this. There are a lot of solutions that are being looked at to solve these problems. Virtualization is probably the biggest and most popular and also the most successful of these.”

Optimizing power and cooling in the data center

Today’s data centers face critical energy issues—power and cooling limitations, high-energy demands and costs, and even outages from overburdened power grids. A few years back, heat load per data centre used to be 24 kilo watt hour (kWh), today heat load per rack is 24 kWh. This is a huge problem for a data center manager.

Describing the criticality of cooling in a data center environment, Anupam Trivedi, Business Head-IT Solutions, Rittal India stated, “Running and expansion costs are a major problem with conventional cooling. Normally a data centre is planned for 10 years and even if the organization starts the data center with 25 percent of the total capacity, it has to install the HVAC for 100 percent capacity, which will be used after at least five years. This results in a huge operational cost. Also the conventional ‘Hot Aisle, Cold Aisle’ cooling concept is unable to handle the increasing heat load and demands of a modern data center.”

Data center power and cooling go hand-in-hand. To optimize and reduce energy consumption, the focus should be upon adjusting airflow to eliminate hotspots. Air conditioning units run efficiently when operating at approximately 80 percent capacity and when they are fed the hottest air. Nair added, “Introducing additional cooling equipment without first trying to improve the efficiency of the existing setup is a bad way to design a data center.”

Power problems in the modern data center are similar to the problem of cooling. The initial investment, running costs, energy efficiency, and maximum availability in the data centre are major concerns. Adding to this, Dhar said, “Conventional power solutions are as rigid as conventional cooling solution and are unable to handle the demands of today’s data centers.”

Social responsibility

Industry experts believe that there is a need for a solution with a holistic approach. The parameters for selecting such solutions must consider all factors as well as possible upgradations, scalability etc. “We should use available power resources more effectively and efficiently. IT is getting power on priority, for that somebody in the interior of the nation is living in the shadow of load shedding so saving power itself can be a form of corporate social responsibility (CSR),” explained Pimpley.

Nicholson added, “The average cabinet consumed three to four kWh a few years ago whereas the average amount of power being used in a cabinet today is at least six to eight kWh. Our customers with blades and other high density applications are preparing to support cabinets that require 35 kWh as they move forward and we are working on solutions for them.”

Nair points out some critical issues that weigh heavily on the minds of data center managers. He said, “The core demands of a data center administrator included reliability, flexibility to change and TCO in terms of capital expenditure and operational expenses.”

Future proofing the data center

Businesses should reevaluate support system design in light of increasing pressure on data center space. There are several steps that can be taken to reduce the impact of power systems on the data center, while new approaches to cooling enable existing space to accommodate a greater number of high density racks.

Increasing pressure on the data center

Technology compaction is enabling equipment manufacturers to deliver more processing power in less space. The resulting ‘high density’ systems consume increasing amounts of power and generate correspondingly high amounts of heat. The impact of this trend on data center space utilization has taken some organizations by surprise. After all, shouldn’t smaller, more compact systems consume less space?

The answer is a resounding ‘no’. Smaller footprints do not necessarily translate into reduced space requirements because performance increases have primarily been achieved by packing more transistors operating at faster speeds into smaller processors. The upshot is that more free-space is needed to remove the concentrated heat that is generated by these extra-dense processor designs.

For example, the Intel Pentium 3 processor, introduced in 2000, had 28,100,000 transistors and a 1 GHz clock speed, with a maximum power consumption of 26 Watts. The Pentium 4, released just two years later, had almost twice as many transistors and three times the clock speed (55,000,000 transistors with a speed of 3 GHz), and a maximum power consumption of 83 Watts. So, while a Pentium 4-based system may have the same footprint as a Pentium 3-based system, it consumes significantly more power and generates more heat.

In parallel with advances in processor technology, server package sizes have been shrinking substantially. Much of this compression is enabled by smaller disk drives, power supplies and memory formats. New blade server packages are further condensing the computing package, creating even higher density racks.

This puts more pressure on power and cooling systems. High-density systems may even generate so much heat that they create hot spots in the data center, where the temperatures directly above the equipment is hotter than the rest of the room. One of the ways in which data center managers are dealing with this situation is to increase rack spacing, essentially distributing the heat from the equipment over a larger space.

The performance potential of high density systems can only be realized if the corresponding rise in heat density, and its implications on data center space, are successfully addressed.

Data center economics

The data center is a unique environment within most organizations. Generally, it requires precise environmental control, enhanced power protection and tighter security than other space within a facility. Consequently, the cost per square foot is much higher than is the case with general office space. This means that the increased pressure on data center space, if not dealt with effectively, can have a significant economic impact.

Consider a typical 10,000-square-foot data center. Assuming average rack power densities of 1 kilo watt, approximately 35 percent of data center space is used for racks. The remaining 65 percent of space is required for aisles and support systems. Since a typical rack consumes about seven square feet of floor space, this facility can support a maximum of 500 racks of one kWh each.

If average power density increases to 10 kWh per rack, with no other changes in the data center infrastructure, increased rack spacing is required to spread the greater heat load across the room. Now only 3.5 percent of the space in the room is available for racks. The remainder is required for aisles and support systems. As a result, the facility can support only 50 racks.

This illustrates the potential impact of increasing compute densities on data center space. In reality, this transformation is happening gradually and incrementally. However, it is happening. Unless an alternative cooling system that enables closer spacing of high density racks is deployed, it will be necessary to expand current facilities to support high density systems.

Cooling systems and data center space

Traditional precision air conditioning units have provided effective cooling in thousands of data centers around the world; however, as system densities increase, they are being stretched to the limits of their practical capacity.

The key limitations involve the number of precision air conditioners that can be installed in a given room and the amount of air that can be pushed through perforated floor tiles in a raised floor.

Floor-mounted precision air systems take up data center floor space, limiting the number of systems that can be installed in a facility. In addition, there is a physical limitation as to how much air can be efficiently distributed through the raised floor. Trying to push too much air under the floor can create negative pressures that can actually draw cool air back down through the perforated tiles, rather than forcing it up into the room. In addition, the floor tiles themselves have physical limits as to how much air can actually pass through the perforations. Consequently, increasing cooling capacity will not necessarily result in a corresponding increase in cooling across a room.

There are several steps that can be taken to optimize the efficiency of a raised floor system. The first is an examination of the cabling running under the floor to ensure it is not obstructing air flow. Floor height also plays a role. Doubling the floor height has been shown to increase capacity by as much as 50 percent. Data center managers planning new facilities should consider floors higher than the traditional 18-inch height. However, replacing the floor is usually not an option for existing data centers because of the disruption in operations it entails.

The hot aisle/cold aisle concept can be employed to increase cooling system efficiency. It involves arranging racks in a way that separates the cool air coming up from the floor from the hot air being discharged by equipment. Racks are placed face to-face and floor tiles are positioned so that cool air is distributed into this ‘cold’ aisle, where it can be drawn into the rack. Heated air is then exhausted through the back of the equipment into the ‘hot’ aisle. By supplying the cooling system with a smaller volume of hot air than a larger volume of mid-temperature air, more of the cooling system’s capacity is utilized.

Rack spacing can, of course, also be used to dissipate heat from high density racks, if data center space allows. In field tests, raised floor cooling systems have shown a practical cooling capacity of two to three kWh of heat per rack. This means that a 10 kWh/rack system would require cold aisle widths of 10 feet to ensure adequate heat dissipation. Clearly, this will not prove to be a long-term solution as rack densities rise to 20 kWh and beyond. A more effective long-term solution must be developed to support the continual deployment of new systems.

Air control technology

Data centers are sensitive as far as temperature is concerned and work in particular ranges of temperature. A minor fluctuation of even one degree can affect the smooth functioning of a data center; they need precision air conditioning. Since data center equipment generates a lot of heat, it is important to constantly monitor the cooling process. A study from E&Y says that cooling and air flow account for nearly 40 percent of the energy cost in a data center, so cooling needs to be precise as well as energy efficient. In light of this trend, Nair said, “We do thermal management of the data center using state-of-the-art Computerized Fluid Dynamics (CFD) to analyze where the cooling is needed. Instead of installing high capacity cooling machines, we provide room cooling along with supplementary cooling up to the rack level which, in turn, saves on cooling expenditure and provides cooling where it is required. This solution can be scaled as the data center grows.”

While emphasizing the thrust of this technology in the industry, Nicholson shared his experience, and added, “Data center solutions that cool cabinets with air up to about 25 kWh are available.” Core demands from users or administrator are high availability or low downtime, short Mean Time To Repair (MTTR), redundancy, less time and money spent on maintenance, physical security, online monitoring & control of access, power, temperature, smoke etc at the rack level and the early detection of errors to avoid downtime.

Liquid Cooling

There are other solutions such as liquid cooled cabinets, controlling the air flow coming out the tiles, venting the hot air in a chimney as it comes out of the back of the cabinet and many more.  Nicholson said, “For us it is about creating solutions that meet our customers’ requirement like new Modular 3-Phase units that can be mounted in the sides of the cabinet. They don’t block air flow and provide a power distribution solution in the cabinet itself.”

Power systems in the data center

The location and footprint of power systems also has an impact on data center space, and these will vary based upon whether a room- or rack-based protection strategy is being utilized.

The room-based strategy centers on a UPS system sized to provide backup power and conditioning for the entire room. Often, this approach benefits from the cost advantages that come with choosing a larger UPS system at the onset, rather than piecing together a system over time. It also provides added application flexibility by allowing the UPS system to be located outside the data center in a lower cost-per square-foot area of the building.

A rack-based approach provides protection on a rack-by-rack basis, as individual UPS are purchased and installed with each addition of network equipment. This approach is often adopted in smaller facilities that are not expected to grow and in cases where future capacities are impossible to project. While a rack-based approach may seem cost-effective, it is important to evaluate the implications of this approach in terms of both space and dollars.

First, this approach typically does not provide the option of placing power protection equipment outside the data center. This means that UPS take up valuable floor space and add to the heat load in the data center.

Room-level UPS may also be located inside the data center, but when they are, they consume less floor space and generate less heat than highly modular systems. That’s because the larger the UPS, the higher its efficiency rating. This puts highly modular systems at a disadvantage because multiple, distinct UPS units are required to achieve a certain capacity. For example, it might take three modular systems to provide 120 kVA of protection, each operating at a lower efficiency than a single 120 kVA system would. This difference in efficiency translates directly into increased heat dump from the UPS.

A similar scenario holds true in terms of footprint. A highly modular system will typically have a larger footprint than a fixed capacity UPS system and the footprint differential increases with system capacity. A 40 kVA modular UPS requires 53 square feet of floor space, allowing for service clearances, while a 40 kVA fixed capacity system requires just 40 square feet of space—a 32 percent difference. At capacities of 120/130 kVA, the footprint of the modular system grows to 159 square feet, while the traditional system consumes just 75 square feet.

More significantly, the fixed capacity system will deliver greater reliability and availability than a highly modular system because:

• The modular system utilizes more parts than the fixed capacity system, increasing the UPS hardware failure rate and the risk of a critical AC output bus failure;
• The fixed capacity system includes module-level redundancy to enable concurrent maintenance while the highly modular system does not;
• The fixed capacity system provides longer runtimes using in-cabinet batteries than the highly modular system, which typically requires external battery cabinets to achieve desired runtimes.

Modular power protection systems may prove suitable for some applications, but facilities that expect to experience growth should consider the long-term impact on data center space and UPS costs before embarking on a protection strategy based on this approach.

And the solution is...

For a vendor, it is more important to be able to study the data center and provide the right solution. The user is always worried about the uninterrupted working of the data center, which is where a proper cooling, and power solution comes to the fore. Nair said, “When we visit a data center, our aim is to understand its functioning and then suggest the right solution accordingly.”

Nair also said that his company had a unique solution called Liebert Adaptive Architecture which covers all the three concerns with its range of power, cooling, enclosure, monitoring & services solutions. However, in terms of offering a scalable solution, Emerson Network Power being a global leader identified this need long back and pioneered the concept of Liebert Adaptive Architecture. This involves the study of a data center from futuristic point of view, and accordingly designs the solution that can be modified as and when required. Nair also added, “This approach helps in drastically cutting the cost for a customer.”

As utility rates continue to climb, it is important to produce the best efficiency possible. Using an efficient UPS helps lower IT and the facility managers’ energy costs while delivering the most scalable and flexible power protection architecture for data centers and IT environments. “The UPS also needs to feature a scalable design for simple reconfiguration of power systems to meet changing demands in the data center,” said Sharma.

Nicholson said, “With increased densities we work with the customer to provide the proper amount of power going into the cabinet while still understanding that redundancy and the need to add devices to the cabinet are key considerations. Today we offer many high density power distribution units that accept three-phase power at various current levels such as 32 and 60 amps and distribute single phase power to all of the devices in a cabinet.”

Business trends

While analyzing the current market positioning, precision power is considered vital for a data center. Given the current power scenario in the country wherein power fluctuations and cuts are the norm, Nair said, “A precision power solution helps bridge this gap for the company. Even if there is a minor power fluctuation, the system automatically switches over to the backup power supply that helps maintain the balance.”

The solution does not only demand a good UPS, the entire power architecture needs to be taken care of, which involves earthing and bonding, transfer switches, surge protection, power distribution units and cabling and a good UPS. The total design and how power travels with-in the data center from source to load helps handle power challenges in the data center.

Strategies for cutting data center energy costs

Businesses can use different strategies for cutting data center energy costs through enhanced cooling efficiency.

Proper sealing of the data center environment Cooling losses through floors, walls and ceilings, or the introduction of humidity from outside the critical facility, reduce cooling system efficiency. Therefore, the data center should be isolated from the general building and outside environment to the extent possible.

Keep doors closed at all times and use a vapor seal to isolate the data center atmosphere. The vapor seal is one of the least expensive and most important methods of controlling a data center environment and is particularly important in maintaining proper humidity levels.

If humidity is too high in the data center, conductive anodic failures (CAF), hygroscopic dust failures (HDF), tape media errors and excessive wear and corrosion can occur. These risks increase exponentially as relative humidity increases above 55 percent.

If humidity is too low, the magnitude and propensity for electrostatic discharge increases, which can damage equipment or adversely affect operations. Also, tape products and media may have excessive errors when exposed to excessively dry conditions.

This is the first step in any plan to increase efficiency. If the room is not properly sealed, all other measures for improving efficiency will be less effective. A data center assessment, available through various consulting engineering firms or your cooling system supplier, can help identify areas where outside air is entering the controlled environment and recommend strategies for proper sealing.

Optimizing air flow

Once a room is sealed, the next step is to ensure efficient air movement. The goal is to move the maximum amount of heat away from the equipment while utilizing a minimum amount of energy. Optimizing air flow requires evaluation and optimization of rack configuration, air conditioner placement and cable management.

Rack arrangement: Most of the equipment manufactured today is designed to draw in air through the front and exhaust it from the rear. This allows equipment racks to be arranged to create hot aisles and cold aisles. This approach positions racks so that rows of racks face each other, with the front of each opposing row of racks drawing cold air from the same aisle (the ‘cold’ aisle). Hot air from two rows is exhausted into a ‘hot’ aisle, raising the temperature of the air returning to the Computer Room Air Conditioning (CRAC) allowing it to operate more efficiently.

This approach is most effective when cold and hot air do not mix. Therefore, perforated floor tiles should be removed from hot aisles and used only in cold aisles. Blanking panels should be used to fill open spaces in racks to prevent hot air from being drawn back through the rack. Additional steps such as using a return ceiling plenum to draw air back to the CRAC and physical curtains at the ends of cold aisles have also proved to be effective in minimizing the mixing of hot and cold air.

CRAC Placement: When using the hot-aisle/cold-aisle approach, CRAC units should always be placed perpendicular to the hot aisle to reduce air travel and prevent hot air from being pulled down into the cold aisles as it returns to the air conditioner. A return ceiling plenum can be effective in minimizing the mixing of hot and cold air.

Cable Management: The proliferation of servers in data centers has created cable management challenges in many facilities. If left unmanaged, cables can obstruct air flow through perforated floor tiles and prevent air from being exhausted from the rear end of a rack. Check the under-floor plenum to determine if cabling or piping is obstructing air flow. Overhead cabling is becoming popular, eliminating a potential source of obstruction. Deeper racks are now available to allow for increased airflow. Sometimes existing racks can be equipped with expansion channels to add depth for cables and airflow. Be cautious when using cable management ‘swing arms’ as they are not compatible with all IT equipment airflow patterns.

Finally, but perhaps most significantly, investigate bringing high-voltage three-phase power as close as possible to the IT equipment and increasing the voltage of said equipment. These steps will minimize the number and size of the power cable feeds under the floor. This can sometimes be accomplished by using high-voltage three-phase managed power strips within a rack, but may also require the use of multiple-pole distribution panels located within a row of IT equipment racks. Additionally fans can be added to the rear of racks to draw hot air out of them, but be aware that these fans consume energy and generate additional heat that must be removed from the room.

Increase the efficiency of room air conditioners

Three factors are critical when it comes to optimizing the efficiency of CRAC units:

• How efficient the units are while operating at partial load.
• How efficient they are at removing sensible heat as compared to latent heat.
• How well multiple units work together.

Data centers are designed with some level of cooling system redundancy. Plus, the actual capacity of a direct expansion or air-cooled CRAC unit increases as the outdoor ambient temperature decreases below the peak design condition. This means equipment is operating at less than 100 percent load all the time, creating the opportunity to design systems that operate more efficiently during normal operating conditions.

Traditional modulation technologies (cycling units on and off to match load conditions) often consume close to full-load energy regardless of the required capacity. In a system designed for high reliability, the compressors do not just turn on and off. There is a turn-on delay period and a turn-off pump-down period where the compressor is actually running, ensuring proper oil lubrication of compressor bearings, before power is removed.

IT equipment generates sensible (dry) heat. Latent heat comes from people and outdoor humidity infiltration (that can be minimized through the vapor seal discussed previously). As server density or capacity increases, it creates a corresponding increase in the sensible heat load. The latent heat load is unaffected. Thus, using cooling solutions that can operate at a 100 percent sensible capacity, except when dehumidification is required, will result in reduced energy consumption. Operating a variable capacity compressor at a lower capacity raises the temperature of the evaporator coil. This means less latent cooling takes place. Under the vast majority of load conditions, the evaporator coil temperature will be high enough to achieve 100 percent sensible cooling. No energy will be required to add humidity that was inadvertently removed.

Improving co-ordination across multiple units—the data center environment has become more diverse as newer high-density servers are deployed alongside older systems. As a result, without proper coordination between room cooling units, air conditioners may be operating in different modes of temperature and humidity control. For example, a unit on the north side of the room may be sensing low relative humidity conditions and adding humidity, while a unit on the south side of the room is sensing high relative humidity and removing moisture from the air. The actual moisture in the air is equal, but because the measurement is a relative one, the higher the temperature, the lower the relative humidity. Advanced control systems can be deployed across all the CRAC units in a room to enable the units to communicate and coordinate their operation, preventing the ‘fighting mode’ described above from occurring.

Supplemental cooling

This is a relatively new approach to data center cooling that was pioneered by Emerson Network Power. Introduced in 2002, this approach gained rapid acceptance as data center managers seek solutions to help them:

• Overcome cooling capacity limitations of raised floor systems in high heat density applications.
• Increase cooling system efficiency and flexibility.

Raised-floor cooling has proved to be an effective approach to data center environmental management; however, as rack densities exceed five kWh, and load diversity across the room increases, supplemental cooling should be evaluated for its impact on cooling system performance and efficiency.

At higher densities, equipment in the bottom of the rack may consume so much cold air that remaining quantities of cold air are insufficient to cool equipment at the top of the rack. The height of the raised floor creates a physical limitation on the volume of air that can be distributed into the room, so adding additional room air conditioners may not solve the problem.

Two factors contribute to improved energy efficiency: the location of the cooling modules and the refrigerant used.

Higher density applications require fluid-based cooling to effectively remove high concentrations of heat being generated. From an efficiency perspective, refrigerant performs better than water for high-density cooling.

Optimizing the overall power efficiency and cooling of the data center requires a comprehensive approach that focuses on technologies and strategies to minimize power consumption and maximize power efficiency at every level within the data center. The same holds true for cooling. Businesses should first evaluate their current required and their future requirement and address the individual problem areas.

Making decisions that take into consideration all contributing factors, from the risk associated with heat spikes to placement of power systems to data center shell costs, can ensure that increasing rack densities do not drive the need for expanded or new data center facilities in the future. Careful decisions can also ensure that business needs—not support system limitations—drive a company’s adoption of new technologies.

faiz.askari@expressindia.com