---
title: "Scaling Laws and the Compute Budget"
description: "Held-out loss falls as a smooth power of parameters, tokens and compute, which turns a training decision into arithmetic. The same curve shows where a budget is being wasted, usually on a model larger"
canonical: https://lightmysky.com/learn/computing/scaling-laws-and-the-compute-budget-mt_tzl0sQblIM
source: https://lightmysky.com/learn/computing/scaling-laws-and-the-compute-budget-mt_tzl0sQblIM.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Scaling Laws and the Compute Budget

Held-out loss falls as a smooth power of parameters, tokens and compute, which turns a training decision into arithmetic. The same curve shows where a budget is being wasted, usually on a model larger than the data it will ever see.

Subject: Computing · Area: Machine Learning · Ages 22 to 23
Page: https://lightmysky.com/learn/computing/scaling-laws-and-the-compute-budget-mt_tzl0sQblIM

## Ready when they can

- Read a scaling curve on log axes and predict the loss at a larger budget
- Split a fixed compute budget between model size and token count, and defend the split
- Name two things the curve says nothing about

## Lesson: Reading the scaling curve

Held-out loss falls as a fitted power of parameters, tokens and compute over the range studied: each tenfold rise in compute multiplies loss by a fixed ratio, so on log-log axes the curve is a straight line. Extend that line to estimate loss at a larger budget, and training choice becomes arithmetic.

**Example.** Suppose each tenfold compute step multiplies loss by 0.75: loss 4.0 at one budget, 3.0 at ten times the compute, 2.25 at a hundred times. The same curve exposes waste: a model larger than its data can ever feed sits above the line it should be on.

Split a fixed budget between model size and token count, and defend the split from both curves. If one side has flattened, move budget to the other. These estimates hold only while the data mix and training recipe stay the same. And remember two things the curve never says: what inference will cost, and whether new abilities will appear.

**Tip.** State what your extrapolation assumes: the same data mix, the same training recipe, and no cliff at the new scale.

**Recap.** Extend the straight line to predict loss, balance size with data, and name what the curve omits.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Pretraining by Predicting the Next Token](https://lightmysky.com/learn/computing/pretraining-by-predicting-the-next-token-mt_D8b2phZJfp)
- [Cross-Validation and Honest Model Selection](https://lightmysky.com/learn/computing/cross-validation-and-honest-model-selection-mt_YG7X3HYCrT)

## Opens up

- [Adapting a Pretrained Model: Fine-Tuning and Low-Rank Updates](https://lightmysky.com/learn/computing/adapting-a-pretrained-model-fine-tuning-and-low-rank-updates-mt_NaA2iGtMPc)