UF supercomputer supersizing research
Published: Sunday, May 5, 2013 at 5:36 p.m.
Last Modified: Sunday, May 5, 2013 at 5:36 p.m.
If not for the total absence of windows, the new University of Florida Data Center at the Eastside Campus on Waldo Road would look like many of the red brick buildings around the main campus.
But it's what's on the inside of this $14 million, 25,000-square-foot building that really sets it apart.
The Data Center is the life support system for the $3.4 million HiPerGator research computer inside. The fastest supercomputer in the state, it can process 150 trillion calculations per second — greatly accelerating the development of drugs that fight cancer and HIV — calculate climate change 50 years out, and help research the origins of the universe.
“If each calculation were a word, the supercomputer could read the millions of volumes on UF library shelves several hundred times in one day,” said Chris Moran, director of communications for university relations.
For a researcher like David Ostrov, an associate professor in pathology, immunology and laboratory medicine at the UF College of Medicine, the computational power greatly reduces his research time, which saves money and gets lifesaving drugs to clinical trial more quickly.
Ostrov runs calculations to determine what drugs best match with human proteins to create the best effect with the least damaging side effects.
He matches 140,000 different drugs and drug candidates to specific protein points on the human genome, calculating which combinations are most effective and which ones are most harmful.
A trial-and-error approach would take more than $500,000 and several years, he said. To screen 140,000 compounds takes eight hours on the supercomputer. That enables him to run thousands of combinations in a matter of weeks.
“It's so fast!” Ostrov said. “I'll watch sums come spilling out, and it's incredible how fast it's computing.”
The Food and Drug Administration just approved a drug to fight diabetes that represents the first time Ostrov developed a drug straight from the computer to the clinic.
He also received notice that a $1.75 million grant had been approved that will further his research using the supercomputer.
The new supercomputer, built by Dell, will support $161 million in annual research grants that fund the work of 160 teams totaling 500 researchers, Moran said.
It is connected to the Internet2 Innovation Platform, which further increases the university's data capacity and links UF researchers to scientific data around the globe.
Inside the ‘data hall'
To see the new supercomputer, one must pass through a series of steel doors with electronic security locks. David Burdette, associate director of Data Center Operations, has to swipe a fob and enter a numerical code to get in.
Cameras record people as they enter and walk down a wide hallway past several sets of doors.
Burdette comes to a door at the end of the hallway, swipes his fob again, punches in some numbers on a keypad again and opens the door to a 5,000-square-foot “data hall” with two clusters of computer racks standing on clean white tiles.
The HiPerGator cluster is just the start, Burdette says. The room can hold about 150 racks, and each rack holds 150 petabytes of data. One petabyte equals 1 million gigabytes.
Cold air blows up through floor gratings and gets sucked into the front of the computers to keep their feverish electronic brains from overheating.
The computers blow sauna-like air out the back end, where it is sucked out of the room through ceiling vents to be recirculated through giant chillers.
It takes 200 kilowatts of power to power these computers, Burdette says. Eventually, the data center will require as much as 1.5 megawatts to power all of the computers planned for the data hall.
An electrical room houses the switchboards that receive power from Gaineville Regional Utilities. Cabinets hold the uninterruptible power supply, dozens of interconnected batteries that look like car batteries — enough to run the data center for five minutes. There is room for more cabinets, and more batteries.
In another area stands a back-up generator, a 3,600-horsepower V-20 diesel that can crank out 2.5 million watts of power, enough to run the computers for 72 hours.
Next to the generator are two 2,000-ton Trane chillers, with room for two more. A pump room with two pumps and space for two more sends thousands of gallons of water through the building.
The system is designed to run while workers repair leaks or other maintenance.
“It would have been short-sighted to build what we need today,” Burdette says.
He projects that the building will be full and at full capacity in 10 years but that it should continue to provide computer brain power to the university for the next 40 years.
After all, the original data center was built 40 years ago and is still operating, with no end in sight.
“I don't know why this place wouldn't be operating 40 years from now,” Burdette says.
Planning for high performance
The Data Center took four years of planning and redesign and a year to build, Burdette said. It will be a lights-out facility, designed to operate on its own without human assistance.
“Nobody will be stationed here,” he said, explaining that it will be operated remotely from the operations center on the main campus.
One tiny office is the only acknowledgement of human interdependence and will be used to store documents and manuals.
“The university takes its IT very seriously,” Burdette said. “It needs a facility that can provide support at a mission-critical level. If our computers are down, nothing's happening. This facility exists to make sure they stay up.”
Planning for the HiPerGator computer began in 2004, when UF started a task force to find out the high-performance computing needs for UF, said Erik Deumens, the director for computing research at UF.
Implementation was gradual at first, he said, and accelerated when Elias Eldayrie became vice president and chief information officer for UF in 2010.
In 2011, Eldayrie and Deumens crafted an agreement with the provost and vice president for research “that put us on track for what we have today.” UF President Bernie Machen came on board, Deumens said, “and now we have a first-class infrastructure.”
The HiPerGator is the crowning achievement, the tip of the pyramid of informational technology at UF, Deumens said.
There are plenty of faster computers. Indiana University's Big Red II, which went online in April, is the fastest supercomputer funded and owned by a university.
It's capable of crunching a quadrillion pieces of data per second. That's a thousand teraflops (a teraflop is a trillion calculations), or seven times faster than UF's HiPerGator's 150 teraflops per second.
But this will put UF “on the map as a world-class research institute,” said Tim Carroll, director of Dell's research computing group. Dell has built supercomputing clusters around the world. “In our world you measure things by teraflops — and 150 teraflops is a world-class machine.”
UF is touting more than its newly supercharged brain power. It's boasting its boosted connectivity as well. A $1 million Internet upgrade funded by UF and the National Science Foundation boosted UF's connection from 10 gigabits per second to 100 gigabits per second via the Internet2 Innovation Platform.
The Internet2 Innovation Platform is a collaboration connecting 100 research universities.
“The time researchers sit in a corner and fiddle with something on a piece of paper is gone. It's all collaboration,” Deumens said.
UF researchers now will be able to move large amounts of data around the world to colleagues doing similar research, and it enhances UF's participation in the Higgs Boson research going on at the CERN supercollider under the Alps in Switzerland.
“The chance of having all scientists in one place is not possible,” he said. “That's why you need a fast connection to the Internet. We can produce data here, make it available to other scientists, and draw conclusions nobody could draw before.”