Adapting a Pretrained Model: Fine-Tuning and Low-Rank Updates
Most work starts from weights someone else paid for. A full fine-tune moves every parameter and costs a whole copy to store, while a low-rank update trains a small factored correction and leaves the base weights alone.
What a learner can do afterwards
- Compare parameter count and storage for a full fine-tune against a low-rank adapter
- Say when adapting beats prompting for a task, and when it does not
- Describe catastrophic forgetting and one measure that keeps it in check
1 · Read
The scaling curve says what a fresh training run would cost, and most teams cannot pay it, so most work starts from weights someone else paid for. A full fine-tune moves every parameter and costs a whole copy to store. A low-rank update instead trains a small factored correction and leaves the base weights alone: the update is the product of two thin matrices, B times A.
Count the storage before you choose. A full n by n update stores n squared numbers, while the factored pair stores 2 times n times the rank. The adapter wins whenever the rank sits below half the size. A 4 by 4 layer with rank 2 stores 16 numbers, the same as full, while a 100 by 100 layer with rank 4 stores 800 against 10000, and rank 60 stores 12000, worse than full.
Count the numbers stored before you choose: 2nr against n squared.
A rank 1 matrix is one column times one row, so each row is that row times a number, and some of those numbers can be zero: the rows [0, 0] and [1, 2] make a rank 1 matrix whose first row is zero. With rows [2, 4] and [3, x], the first row is not zero, so the second row must be a multiple of it: 3 is 1.5 times 2, so x is 1.5 times 4, which is 6. A rank r update stacks r such column-times-row products, which is why B has r columns and A has r rows.
Pick prompting when the need is one-off or the budget is thin, since no training runs at all. Pick adapting when the task repeats often enough to repay training and prompting keeps missing. Between adapters, pick a low rank when storage and forgetting matter, and pay for fuller moves only when the task demands capacity the thin update cannot hold.
Full moves risk catastrophic forgetting: new training washes out old skills. Keep old-task checks running as you train, freeze more and train less where you can, and stop when the old scores start to slip.
Train a thin factored correction when it suffices, count 2 n r against n squared, and watch old tasks while you adapt.
2 · Watch
به انگلیسیTake it off screen
Where it sits
12 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.