Cerebras Introduces Its 2-Exaflop AI Supercomputer

“Generative AI is consuming the sector.”

That’s how Andrew Feldman, CEO of Silicon Valley AI laptop maker Cerebras, starts his creation to his corporate’s newest fulfillment: An AI supercomputer in a position to 2 billion billion operations in keeping with 2nd (2 exaflops). The device, referred to as Condor Galaxy 1, is heading in the right direction to double in measurement inside 12 weeks. In early 2024, it’ll be joined by means of two extra methods of double that measurement. The Silicon Valley corporate plans to stay including Condor Galaxy installations subsequent 12 months till it’s operating a community of 9 supercomputers in a position to 36 exaflops in overall.

If large-language fashions and different generative AI are consuming the sector, Cerebras’s plan is to lend a hand them digest it. And the Sunnyvale, Calif., corporate isn’t by myself. Different makers of AI-focused computer systems are construction huge methods round both their very own specialised processors or Nvidia’s newest GPU, the H100. Whilst it’s tricky to pass judgement on the dimensions and functions of these kind of methods, Feldman claims Condor Galaxy 1 is already some of the greatest.

Condor Galaxy 1—assembled and began up in simply 10 days—is made up of 32 Cerebras CS-2 computer systems and is about to extend to 64. The following two methods, to be inbuilt Austin, Texas, and Ashville, N.C., may also space 64 CS-2s every.

The center of every CS-2 is the Waferscale Engine-2, an AI-specific processor with 2.6 trillion transistors and 850,000 AI cores created from a complete wafer of silicon. The chip is so huge that the size of reminiscence, bandwidth, compute assets, and different stuff within the new supercomputers briefly will get just a little ridiculous, as the next graphic presentations.

When you didn’t in finding those numbers overwhelming sufficient, right here’s any other: There are a minimum of 166 trillion transistors within the Condor Galaxy 1.Cerebras

Certainly one of Cerebras’s greatest benefits in construction large AI supercomputers is its skill to scale up assets merely, says Feldman. As an example, a 40 billion–parameter community can also be skilled in about the similar time as a 1 billion–parameter community in the event you commit 40-fold extra {hardware} assets to it. Importantly, one of these scale-up doesn’t require further traces of code. Demonstrating linear scaling has traditionally been very difficult as a result of the trouble of dividing up large neural networks in order that they function successfully. “We scale linearly from 1 to 32 [CS-2s] with a keystroke,” he says.

The Condor Galaxy sequence is owned by means of Abu Dhabi–founded G42, a preserving corporate with 9 AI-based companies together with G42 Cloud, one of the vital greatest cloud-computing suppliers within the Center East. Alternatively, Cerebras will function the supercomputers and will hire assets G42 isn’t the use of for interior paintings.

Call for for coaching huge neural networks has shot up, consistent with Feldman. The choice of corporations coaching neural-network fashions with 50 billion or extra parameters went from 2 in 2021 to greater than 100 this 12 months, he says.

Clearly, Cerebras isn’t the one one going after companies that wish to teach truly huge neural networks. Large gamers equivalent to Amazon, Google, Meta, and Microsoft have their very own choices. Pc clusters constructed round Nvidia GPUs dominate a lot of this industry, however a few of these corporations have evolved their very own silicon for AI, equivalent to Google’s TPU sequence and Amazon’s Trainium. There also are startup competition to Cerebras, making their very own AI accelerators and computer systems together with Habana (now a part of Intel), Graphcore, and Samba Nova.

Meta, as an example, constructed its AI Analysis SuperCluster the use of greater than 6,000 Nvidia A100 GPUs. A deliberate 2nd section would push the cluster to five exaflops. Google built a device containing 4,096 of its TPU v4 accelerators for a complete of one.1 exaflops. That device ripped in the course of the BERT herbal language processor neural community, which is way smaller than as of late’s LLMs, in simply over 10 seconds. Google additionally runs Compute Engine A3, which is constructed round Nvidia H100 GPUs and a customized infrastructure processing unit made with Intel. Cloud supplier CoreWeave, in partnership with Nvidia, examined a device of three,584 H100 GPUs that skilled a benchmark representing the huge language fashion GPT-3 in simply over 10 mins. In 2024, Graphcore plans to construct a 10-exaflop device referred to as the Excellent Pc made up of greater than 8,000 of its Bow processors.

You’ll get right of entry to Condor Galaxy right here.