Facebook Parent Company Meta Builds Massive AI Supercomputer 

Fresh off its rebrand last October, Meta (née Facebook) is putting muscle behind its vision of a metaversal future with a massive new AI supercomputer called the AI Research SuperCluster (RSC). Meta says that, at full capacity, RSC will deliver nearly 5 exaflops of mixed-precision AI performance to help build new AI models, develop augmented reality tools, seamlessly analyze multimedia data and more. The supercomputer’s first phase is already operational, and it is scheduled for full build-out by mid-year.

Image courtesy: Meta

About the system

RSC’s first phase, already built-out and operational, consists of 760 Nvidia DGX A100 compute nodes, totaling some 6,080 Nvidia A100 GPUs, all networked with Nvidia’s Quantum 200Gb/s InfiniBand. For storage, the system is equipped with 175PB of Pure Storage FlashArray, 10PB of Pure Storage FlashBlade and 46PB of cache storage housed in Penguin Computing Altus servers. Meta says that, with just this first phase, they “believe [RSC] is among the fastest AI supercomputers running today[.]”

With the completion of the second phase around July, Meta says, RSC will contain a total of 16,000 GPUs (presumably through an additional 1,240 DGX A100 nodes, which Nvidia believes will make it the largest customer installation of DGX A100 systems) and a full exabyte of storage with the capacity to accommodate 16TB/s of training data. Meta indicated 16,000 GPUs will be the maximum configuration of the system. “This is due to the network configuration to reduce the number of hops, to ensure we provide a 1:1 oversubscription,” a Meta spokesperson told us.

Meta says this second phase will increase RSC’s AI training performance by more than 2.5× (tracking with the 2.63× increase in GPUs), cementing it as the single fastest AI supercomputer in the world.

Unlike preceding systems, RSC is intended for use with not just open-source/public datasets, but with real-world, internal production data from Meta. To that end, Meta says, they designed the system to be isolated from the internet, with all connections passing through Meta’s own datacenters. User-generated data—checked for anonymization—is encrypted from the storage systems to the GPUs and only decrypted in-memory immediately prior to its use in model training.

Meta also developed a storage service (called AI Research Store, or AIRStore) to handle the growing bandwidth and capacity requirements of RSC. AIRStore preprocesses training data for AI models and is designed to optimize transfer speeds.

In its announcement of RSC, Meta also quietly detailed the first generation of its AI research supercomputing hardware, launched in 2017. The unnamed cluster, Meta says, has 22,000 Nvidia V100 GPUs and performs 35,000 training jobs per day. Meta says that compared to this previous system, RSC’s early benchmarks show a 20× improvement on computer vision workflows and a 3× improvement in large-scale NLP model training (which, Meta says, translates to weeks of saved time).

So far, Meta has worked with a consistent roster of partners across these systems: Penguin Computing for architecture and managed services; Nvidia for systems, GPUs, networking, and software stack components; and Pure Storage for most of the storage functionality.

RSC as currently built. Image courtesy of Meta.

The fastest AI supercomputer(s)

In terms of flops, Meta estimates that RSC will deliver nearly five exaflops of mixed-precision AI compute power. Using Nvidia’s Selene supercomputer (also comprised of eight-GPU Nvidia DGX A100 nodes) as a benchmark, HPCwire estimates that (were Meta to run the HPL benchmark) the full iteration of RSC might deliver around 227 Linpack petaflops of compute power (up from perhaps 86 petaflops right now), though further optimizations made by Nvidia in the interim may make those numbers underestimates.

That is certainly a powerful system—the first phase of RSC would likely place fourth on November’s Top500 list, and its full form would likely place second—but the race for “fastest AI supercomputer” is crowded. While RSC will almost certainly best current comparable competitors like Selene (63.4 Linpack petaflops) and the similarly A100-based Perlmutter system at NERSC (70.9 Linpack petaflops), the near future boasts much stronger challengers.

The most like-to-like comparison might be EuroHPC’s forthcoming Leonardo system, a pre-exascale Atos-built supercomputer that will also be powered by Nvidia A100s (around 14,000 of them, compared to RSC’s planned 16,000). CINECA, which is slated to launch Leonardo’s GPU-powered booster module this month, expects that module alone to deliver 240.5 Linpack petaflops, and Nvidia has billed the forthcoming system as—you guessed it—the “world’s fastest AI supercomputer” (with an estimated ten exaflops of FP16 AI performance).

Tesla, too, is publicly building an enormous AI supercomputer called Dojo, targeting that system at model training for autonomous vehicle development. Currently, it has an A100-based precursor system that HPCwire previously estimated at around 82 Linpack petaflops, but Dojo itself will be powered by Tesla’s proprietary “D1” chip. Owing to the nontraditional hardware and other uncertainties, it is harder to estimate Dojo’s future Linpack performance, but Tesla says that when Dojo launches (as-yet unspecified) it will be “the fastest AI training computer.”

Two notes: first, HPCwire also estimates that RSC’s V100-based precursor system likely delivers around 135 Linpack petaflops and would probably place third on the current Top500, well above the competition from AI systems like Selene and Perlmutter. This would—at least in terms of the Top500—make it the world’s fastest AI supercomputer. Second: Meta (under the name Facebook) previously submitted a 3.3-Linpack petaflops system to the Top500 in early 2017 (it currently ranks 139th). While that system uses Penguin servers, the specs mention Nvidia Tesla P100s and Quadro GP100s rather than V100s, so it may not be part of the precursor system.

Only time (and benchmarks) will tell who comes out on top.

Image courtesy of Meta.

Into the metaverse

The first phase of RSC is already being used for applications like large-model training for natural language processing (NLP) and computer vision. But the long-term target is the metaverse, the nebulously defined virtual world that Meta (named for the metaverse) clearly believes will constitute a new digital revolution.

Meta has an ambitious vision of RSC for the metaverse, highlighting, as an example, how RSC could train models for real-time voice translation among large groups of people, enabling individuals speaking different languages to collaborate on work or gameplay without a language barrier.

“The experiences we’re building for the metaverse require enormous compute power (quintillions of operations/second!) and RSC will enable new AI models that can learn from trillions of examples, understand hundreds of languages, and more,” said Mark Zuckerberg, CEO of Meta.

Building RSC during a pandemic

Meta ties the ideas behind RSC all the way back to the founding of the Facebook AI Research lab in 2013, but says the real inception of the project dated back to early 2020, when they decided a new system was necessary to take advantage of advances in GPU and network fabric technologies. The headline goal: a system capable of training models with more than a trillion parameters on datasets as large as an exabyte.

Rack delivery for RSC. Image courtesy of Meta.

Covid, of course, impeded the development of such a system. Meta says that RSC started as a completely remote project, and the supply chain challenges that emerged later in the pandemic threw even more roadblocks into the path. Meta explained that supply chain disruptions made it difficult to obtain components from chips to GPUs.

“One does not simply buy and power on a supercomputer,” said George Niznik, sourcing manager for Meta. “RSC was designed and executed under extremely compressed timelines without the benefit of a traditional product release cycle. Additionally, the pandemic and a major industry chip supply shortage hit at precisely the wrong moment in time. We had to fully utilize of all our collective skills and experiences to solve these difficult constraints.”

Nevertheless, a year and a half later, the team had delivered a functioning cluster. Meta told HPCwire that the team had been able to mitigate supply chain issues for phase one and that the phased build is continuing according to plan.

“I think what I’m most proud of is doing this with the team completely remotely,” said Shubho Sengupta, an AI researcher at Meta. “I mean, it is insane that you can do this without ever meeting anybody.”

This article first appeared on sister website HPCwire.