Coping with Higher Hyperscale Rack-Power Density
According to the Cisco Global Cloud Index 2015-2020/Synergy Research, by 2020, 47 percent of all servers sold are expected to go to hyperscalers, which deploy distributed computing environments ranging up to tens of thousands of servers. For legacy hardware providers this trend portends a narrower customer base with both a higher technical acumen and closer attention to the bottom line. Server sales to hyperscalers are increasingly captured by “no name” white-box vendors that give them exactly what they want.
New Server Configurations
During the past 20 years, the data center industry has seen rack-power density go up commensurate with compute and storage densities. More servers and hard drives are placed into a single rack than ever before — in a scale-out (more servers) approach rather than a scale-up (mainframe) approach. With more devices packed into racks comes more power consumption. In fact, whereas per-rack consumption used to be between one and three kilowatts, consumption has now risen to 20 and even 40 kilowatts in a cabinet. The U.S. National Renewable Energy Laboratory has confirmed this power usage rise, reporting that 30-kilowatt racks are not uncommon today.
To support greater rack densities, the devices governing power to the servers and drives must also evolve. No longer are the large centralized battery banks and distributed UPS (uninterrupted power supply) systems located along the walls of the data center or at the end of a row of IT cabinets sufficient. Hyperscale data center operators paved the way to eliminate the larger UPS systems in modern data center construction. Current rack power planning finds UPS functionality being distributed across the data center, within the individual IT rack.
Big Racks; Chill Out
Having a tall column full of IT gear generating new levels of heat in hyperscale facilities leads to cooling challenges. Data center architects must commit to either air or water as the cooling medium of choice and adhere to design basics, such as:
- Alternating the direction of equipment to create naturally formed hot aisles and cold aisles.
- Using blanking panels in empty rack spaces to ensure no leakage of cold air directly into the hot aisle.
- Practicing either hot or cold aisle containment.
After the basic cooling designs are considered, there are a number of cooling options to contend with, including:
- Natural convection can be used to move air through a well-designed facility. Cold air falls to the floor and warm air rises to the ceiling. Overhead chilled air service to the cold aisles and either raised ceilings or chimneys atop the back of the racks can be used to push/pull air through the IT systems.
- Adiabatic cooling relies on reducing heat through a change in air pressure caused by volume expansion. Adiabatic processes facilitate “free cooling” methods that make efficient use of water and electricity.
- Liquid cooling is suited to applications where the IT cabinet power and thermal density exceed the cooling capacity of air flowing at a reasonable velocity (up to a few hundred CFM).
- Immersion is the most recent liquid cooling innovation. By placing the IT hardware directly into a liquid cooling medium, the maximum thermal transfer rate can take place, and hence the power density can be maximized.
Ultimately, the decision on cooling will be driven by the data center’s operation parameters.
Customized Power Options for Hyperscale Facilities
There are numerous proposals regarding power infrastructures best suited to hyperscale data centers. Early on, Facebook went with 480V/277V AC power to the rack and 12VDC power to the IT loads within the rack. In Google’s recent contributions to Open Compute, they have proposed 48V DC to power the servers, with direct conversion point-of-load power supplies running from 48V to the working voltages and amperages needed on the motherboard to run the CPU and memory.
Basic power distribution units (PDUs) offer simplicity, reliability and support for high operating temperatures. They are suited to applications where it is “OK” to lose a locked up or failed server that may have been running a dozen virtual machines or several dozen containerized applications. However, basic PDUs lack remote power consumption monitoring and reporting capability, necessitating either upstream or downstream gear to support gathering information for Power Usage Effectiveness (PUE) calculations. With the availability of “build-your-own” PDUs, hyperscale facility designers get to balance off these factors by selecting:
- Input power
- Input cord location
- Outlet count by type (C13, C19, Cx)
- Whether to use alternating phase power
These PDU solution can be tailored to the unique requirements of the data center. Designers can also take advantage of outlet versatility – including outlets that support both C14 and C20 plugs.
There are also PDU solutions that integrate intelligent inline power monitoring and remote monitoring through SNMP, good for racks where the loads are neither static nor equally distributed and there is a diversity of hardware requiring multiple protocols/software interfaces to extract power information from the devices. In addition, environmental monitoring is an integral feature, with support for multiple temperatures and humidity probes deployed inside and outside the rack. In fact, some PDUs enable alarms to be set for user selectable predefined power and temperature thresholds.
Conclusion
As workloads increase, so too will rack densities and power consumption. It’s a perpetual stream of data that will continue to impact floor designs, cooling and the lifeblood on any data center — power. Those who configure and maintain hyperscale facilities have many options and configurations for PUE optimization. There is no singular template to mold hyperscale facilities to achieve the lowest PUE — so consider racks, cooling and power options carefully and seek new, but proven, methods to ensure optimal design.
Marc Cram is Director of Sales for Server Technology (@ServerTechInc), a brand of Legrand (@Legrand).
