Adapting a Pretrained Model: Fine-Tuning and Low-Rank Updates
Most work starts from weights someone else paid for. A full fine-tune moves every parameter and costs a whole copy to store, while a low-rank update trains a small factored correction and leaves the base weights alone.
What a learner can do afterwards
- Compare parameter count and storage for a full fine-tune against a low-rank adapter
- Say when adapting beats prompting for a task, and when it does not
- Describe catastrophic forgetting and one measure that keeps it in check
1 · Read
Most work starts from weights someone else paid for. A full fine-tune moves every parameter and costs a whole copy to store. A low-rank update instead trains a small factored correction and leaves the base weights alone: the update is the product of two thin matrices, B times A.
Count the storage before you choose. A full n by n update stores n squared numbers, while the factored pair stores 2 times n times the rank. The adapter wins whenever the rank sits below half the size. A 4 by 4 layer with rank 2 stores 16 numbers, the same as full, while a 100 by 100 layer with rank 4 stores 800 against 10000, and rank 60 stores 12000, worse than full. If a matrix has rank 1, every row is a multiple of the first: with rows [2, 4] and [3, x], 3 is 1.5 times 2, so x is 1.5 times 4, which is 6.
Pick prompting when the need is one-off or the budget is thin, since no training runs at all. Pick adapting when the task repeats often enough to repay training and prompting keeps missing. Between adapters, pick a low rank when storage and forgetting matter, and pay for fuller moves only when the task demands capacity the thin update cannot hold.
Full moves risk catastrophic forgetting: new training washes out old skills. Keep old-task checks running as you train, freeze more and train less where you can, and stop when the old scores start to slip.
Train a thin factored correction when it suffices, count 2 n r against n squared, and watch old tasks while you adapt.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.