Scaling Laws and the Compute Budget
Held-out loss falls as a smooth power of parameters, tokens and compute, which turns a training decision into arithmetic. The same curve shows where a budget is being wasted, usually on a model larger than the data it will ever see.
What a learner can do afterwards
- Read a scaling curve on log axes and predict the loss at a larger budget
- Split a fixed compute budget between model size and token count, and defend the split
- Name two things the curve says nothing about
1 · Read
Held-out loss falls as a fitted power of parameters, tokens and compute over the range studied: each tenfold rise in compute multiplies loss by a fixed ratio, so on log-log axes the curve is a straight line. Extend that line to estimate loss at a larger budget, and training choice becomes arithmetic.
Suppose each tenfold compute step multiplies loss by 0.75: loss 4.0 at one budget, 3.0 at ten times the compute, 2.25 at a hundred times. The same curve exposes waste: a model larger than its data can ever feed sits above the line it should be on.
Split a fixed budget between model size and token count, and defend the split from both curves. If one side has flattened, move budget to the other. These estimates hold only while the data mix and training recipe stay the same. And remember two things the curve never says: what inference will cost, and whether new abilities will appear.
State what your extrapolation assumes: the same data mix, the same training recipe, and no cliff at the new scale.
Extend the straight line to predict loss, balance size with data, and name what the curve omits.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.