Scaling Laws for Neural Language Models
Found that loss falls smoothly, as a power law, with parameters, data and compute.
Power lawrelation between loss and scale
Why it matters
Turned "just make it bigger" from a gamble into a calculable investment, and underwrote the bet on GPT-3.
Costs and controversy
Underestimated the importance of data; corrected two years later by Chinchilla.
Led to
Source: arxiv.org ↗