Interesting write up and visual of the exponential growth in AI computation & compute trends - across 3 eras of machine learning - courtesy of the Visual Capitalist.
Here's a visual representation of compute trends across 3 eras of machine learning - courtesy of the Visual Capitalist.
The data comes from this paper - "Compute Trends Across Three Eras of Machine Learning", written by Jaime Sevilla, Lennart Heim, Anson Ho,Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos,
The authors determined that the training compute has grown by a factor of 10 billion since 2010, with a doubling rate of around 5-6 months. They identify three eras of compute in ML:
- Pre-Deep Learning Era (1950-2010): During this era, the training compute grew in line with Moore's law, doubling roughly every 20months.
- Deep Learning Era (2010-2015): With the advent of deep learning, the scaling of training compute accelerated, doubling approximately every 6 months.
- Large-Scale Era (2015-present): In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-foldlarger requirements in training compute.
The paper shows that before 2010, training compute grew in line with Moore’s law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute.
Worth noting that the authors' dataset is limited to a subset of 123 milestone ML systems and findings may not be generalizable to all ML systems. However, their work provides a valuable contribution to our understanding of compute trends in ML.
