5 times bigger than ChatGPT! Intel officially announced a 1 trillion parameter AI large model, which is planned to be completed in 2024

Organize | Zheng Liyuan

Listing | CSDN (ID: CSDNnews)

Seeing that ChatGPT has continued to explode for several months, during which technology giants such as Microsoft, Google, and Meta have entered the game one after another, now Intel has finally officially announced its “competition”.

Over the weekend, at the International Supercomputing Conference (ISC) High Performance Conference (HPC) in Hamburg, Germany, Intel not only demonstrated its leadership in HPC and AI workloads, but also announced a surprising plan: Tribute National Laboratory joined hands to develop a generative AI model Aurora genAI with the Aurora supercomputer, and the number of parameters will reach 1 trillion!

Be aware that the parameter size of ChatGPT is only 175 billion, that is, the Aurora genAI model will be at least 5 times larger than it.

(picture from Intel official website)

AI model will be powered by Aurora supercomputing

It is understood that the Intel Aurora genAI model will be based on two frameworks: NVIDIA’s Megatron and Microsoft’s DeepSpeed.

▶ Megatron: An architecture for distributed training of large-scale language models, optimized specifically for Transformer, not only supports data parallelism in traditional distributed training, but also supports model parallelism.

▶ DeepSpeed: Focus on optimizing the training of large-scale deep learning models. By improving the scale, speed, cost and availability, it releases the ability to train 100 billion parameter models and greatly promotes the training of large-scale models.

In addition to these two frameworks, the Aurora genAI model will also be powered by the Aurora supercomputer—the supercomputer Intel designed for Argonne National Laboratory, which has finally taken shape after various delays.

According to the current public information, the Aurora supercomputer is powered by Intel Xeon CPU Max and Xeon GPU Max series chips, with a total of 10,624 nodes, 63,744 Ponte Vecchio GPUs, 21,248 Sapphire Rapids Xeon CPUs, and 1,024 distributed asynchronous Object Storage (DAOS) storage nodes and 10.9 PB of DDR5 Optane persistent memory.

In addition, Intel also revealed the early performance results of the Aurora supercomputer: “Aurora supercomputing has leading performance on scientific and engineering workloads, 2 times the performance of AMD MI250 GPU, and improved QMCPACK quantum mechanics applications compared to H100.” 20% and nearly linear scaling to hundreds of nodes.”

It is worth mentioning that, compared with the original goal of 1 Exaflop, it is expected that when the Aurora supercomputer is launched this year, it will provide double-precision floating-point computing performance of more than 2 Exaflops—more than Frontier, which has repeatedly ranked first in the global supercomputing Top500 list. Supercomputers (1.194 Exaflop/s) are even higher.

Science-Focused Generative AI Model

With the powerful Aurora supercomputing foundation, it is destined that the scale of the Aurora genAI model will not be small. According to Intel’s official introduction, Argonne National Laboratory is leading an international collaboration for the Aurora genAI model.

“This project aims to harness the full potential of the Aurora supercomputer to generate a resource that can be used in downstream science at DOE laboratories and in collaboration with other agencies,” said Rick Stevens, Argonne deputy laboratory director.

Overall, Aurora genAI is a science-focused generative AI model, so it will be trained on generic text, code, scientific text, and scientific data from biology, chemistry, materials science, physics, medicine, etc.

The resulting AI models, with up to 1 trillion parameters, ranging from the design of molecules and materials to the combined knowledge of millions of sources, can be used in a variety of scientific applications: systems biology, cancer research, climate science , cosmology research, polymer chemistry and materials, etc. Beyond science, Aurora genAI models could potentially be used in other fields, such as financial modeling, natural language processing, machine translation, image recognition, and speech recognition, among others.

Planned to be completed in 2024

In addition, more information about the Aurora genAI model Intel has not yet spoiled, but according to foreign media reports, Intel plans to develop and complete the Aurora genAI model in 2024-if it goes well, maybe we will not wait too long Long.

The release of this news has attracted the attention of many people, and Intel’s entry into the AI model announced the start of 1 trillion parameters, which makes people look forward to the future development of competing products such as GPT-4:

▶ "The trillion parameter should be a special limit, but you can also be skeptical and say it’s just a huge integer number to care about. There is no doubt that if this model is similar to GPT-4, this will add a data point. But with companies announcing this and announcing that, I wonder if we’ll be peaking in June.”

▶ “People are building new systems with the H100, and there are already significantly better AI GPUs out there, and if this continues, NVIDIA may need to announce a new card sooner to stay ahead.”

▶ “I guess GPT-4 will not continue to maintain SOTA (state of the art, refers to the best method or model in a specific task) in many benchmark tests soon, and maybe it will also be in the world in the future The fastest supercomputer for training. For reference, the OpenAI supercomputer has about 10,000 GPUs, while Aurora has 63,744 GPUs.”

Reference link: