Fresh off its rebrand last October, Meta (née Facebook) is putting muscle behind its vision of a metaversal future with a massive new AI supercomputer called the AI Research SuperCluster (RSC). Meta says that, at full capacity, RSC will deliver nearly 5 exaflops of mixed-precision AI performance to help build new AI models, develop augmented reality tools, seamlessly analyze multimedia data and more. The supercomputer’s first phase is already operational, and it is scheduled for full build-out by mid-year. About the system RSC’s first phase, already built-out and operational, consists of 760 Nvidia DGX A100 compute nodes, totaling some 6,080 Nvidia A100 GPUs, all networked with Nvidia’s Quantum 200Gb/s InfiniBand. For storage, the system is equipped with 175PB of Pure Storage FlashArray, 10PB of Pure Storage FlashBlade and 46PB of cache storage housed in Penguin Computing Altus servers. Meta says that, with just this first phase, they “believe [RSC] is among the fastest AI supercomputers running today[.]” With the completion of the second phase around July, Meta says, RSC will contain a total of 16,000 GPUs (presumably through an additional 1,240 DGX A100 nodes, which Nvidia believes will make it the largest customer installation of DGX A100 systems) and a full exabyte of storage with the capacity to accommodate 16TB/s of training data. Meta indicated 16,000 GPUs will be the maximum configuration of the system. “This is due to the network configuration to reduce the number of hops, to ensure we provide a 1:1 oversubscription,” a Meta spokesperson told us. Meta says this second phase will increase RSC’s AI training performance by more than 2.5× (tracking with the 2.63× increase in GPUs), cementing it as the single fastest AI supercomputer in the world. Unlike preceding systems, RSC is intended for use with not just open-source/public datasets, but with real-world, internal production data from Meta. To that end, Meta says, they designed the system to be isolated from the internet, with all connections passing through Meta’s own datacenters. User-generated data—checked for anonymization—is encrypted from the storage systems to the GPUs and only decrypted in-memory immediately prior to its use in model training. Meta also developed a storage service (called AI Research Store, or AIRStore) to handle the growing bandwidth and capacity requirements of RSC. AIRStore preprocesses training data for AI models and is designed to optimize transfer speeds. In its announcement of RSC, Meta also quietly detailed the first generation of its AI research supercomputing hardware, launched in 2017. The unnamed cluster, Meta says, has 22,000 Nvidia V100 GPUs and performs 35,000 training jobs per day. Meta says that compared to this previous system, RSC’s early benchmarks show a 20× improvement on computer vision workflows and a 3× improvement in large-scale NLP model training (which, Meta says, translates to weeks of saved time). So far, Meta has worked with a consistent roster of partners across these systems: Penguin Computing for architecture and managed services; Nvidia for systems, GPUs, networking, and software stack components; and Pure Storage for most of the storage functionality. The fastest AI supercomputer(s) In terms of flops, Meta estimates that RSC will deliver nearly five exaflops of mixed-precision AI compute power. Using Nvidia’s Selene supercomputer (also comprised of eight-GPU Nvidia DGX A100 nodes) as a benchmark, HPCwire estimates that (were Meta to run the HPL benchmark) the full iteration of RSC might deliver around 227 Linpack petaflops of compute power (up from perhaps 86 petaflops right now), though further optimizations made by Nvidia in the interim may make those numbers underestimates. That is certainly a powerful system—the first phase of RSC would likely place fourth on November’s Top500 list, and its full form would likely place second—but the race for “fastest AI supercomputer” is crowded. While RSC will almost certainly best current comparable competitors like Selene (63.4 Linpack petaflops) and the similarly A100-based Perlmutter system at NERSC (70.9 Linpack petaflops), the near future boasts much stronger challengers. The most like-to-like comparison might be EuroHPC’s forthcoming Leonardo system, a pre-exascale Atos-built supercomputer that will also be powered by Nvidia A100s (around 14,000 of them, compared to RSC’s planned 16,000). CINECA, which is slated to launch Leonardo’s GPU-powered booster module this month, expects that module alone to deliver 240.5 Linpack petaflops, and Nvidia has billed the forthcoming system as—you guessed it—the “world’s fastest AI supercomputer” (with an estimated ten exaflops of FP16 AI performance). Tesla, too, is publicly building an enormous AI supercomputer called Dojo, targeting that system at model training for autonomous vehicle development. Currently, it has an A100-based precursor system that HPCwire previously estimated at around 82 Linpack petaflops, but Dojo itself will be powered by Tesla’s proprietary “D1” chip. Owing to the nontraditional hardware and other uncertainties, it is harder to estimate Dojo’s future Linpack performance, but Tesla says that when Dojo launches (as-yet unspecified) it will be “the fastest AI training computer.” Two notes: first, HPCwire also estimates that RSC’s V100-based precursor system likely delivers around 135 Linpack petaflops and would probably place third on the current Top500, well above the competition from AI systems like Selene and Perlmutter. This would—at least in terms of the Top500—make it the world’s fastest AI supercomputer. Second: Meta (under the name Facebook) previously submitted a 3.3-Linpack petaflops system to the Top500 in early 2017 (it currently ranks 139th). While that system uses Penguin servers, the specs mention Nvidia Tesla P100s and Quadro GP100s rather than V100s, so it may not be part of the precursor system. Only time (and benchmarks) will tell who … Continue reading Facebook Parent Company Meta Builds Massive AI Supercomputer