In this post
This post summarizes the Y Combinator video How Scaling Laws Will Determine AI’s Future | YC Decoded (January 23, 2025). Where the video makes a claim, the post says so and links the primary paper where one exists.
In recent years, AI labs have scaled up large language models (LLMs) by adding parameters, data, and compute, and models have kept improving. The AI community is now debating whether this approach has reached its limits or whether a new paradigm is coming.
Key Takeaways #
- Scaling laws describe how model loss falls as parameters, data, and compute grow.
- Larger models need more data and compute.
- DeepMind’s 2022 Chinchilla paper found that large models of the time were undertrained.
- Some researchers expect gains from scaling compute at inference time instead.
The Rise of Large Language Models #
In November 2019, OpenAI released GPT-2 with 1.5 billion parameters. The following summer came GPT-3, with 175 billion parameters, more than 100 times larger. The video presents this as the point where the era of scaling laws arrived.
Understanding Scaling Laws #
Think of training AI models like cooking. You need three main ingredients:
- The Model: The architecture and parameters.
- The Data: The information used for training.
- Compute Power: The hardware that runs the training.
In January 2020, Kaplan et al. at OpenAI reported that language model loss scales as a power law with model size, dataset size, and training compute, with trends spanning more than seven orders of magnitude. They also found that architecture details such as network width or depth have minimal effects within a wide range.
The Chinchilla Breakthrough #
In 2022, Google DeepMind published Training Compute-Optimal Large Language Models (Hoffmann et al.). They trained over 400 language models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. They found that current large language models were significantly undertrained, and that model size and training tokens should scale equally: for every doubling of model size, the number of training tokens should also double.
To test this, they trained Chinchilla, with 70 billion parameters, using the same compute budget as Gopher (280 billion parameters) but 4 times more data. Chinchilla outperformed Gopher, GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream tasks. The shift was from making models bigger to also giving them enough training data.
Are We Hitting a Wall? #
The video says some people argue that as the latest models have grown bigger and more expensive, capability gains have started to plateau. It also mentions rumors of failed training runs and diminishing returns at major labs, and a worry about running out of high-quality training data. The video does not confirm the rumors, and neither do we.
A New Paradigm for AI #
The video points to OpenAI’s reasoning models, o1 and its successor o3, which work through a problem with a chain of thought. It says OpenAI researchers found that the longer o1 could think, the better it performed, and that o3 beat earlier results on benchmarks in software engineering, math, and science. We did not verify those benchmark results here.
The idea the video draws from this is that researchers may shift from scaling model size during training to scaling compute at inference time, called test time compute. That is a prediction, not an established result.
The Future of AI Scaling #
The video says the principles of scaling appear to hold for other kinds of models too, including image diffusion, protein folding, and robotics world models, and that scaling for those is at an earlier stage than for LLMs. Whether that holds is an open question.
