Small updates on big weights

Computing · Machine Learning · ages 22-24

Name ______________________ Date ____________

What moves in a full fine-tune?
- Every parameter, costing a whole copy to store
- Only the input layer, leaving the rest frozen
- Nothing; the base weights train themselves
A rank r update to an n by n matrix stores 2 times n times r numbers, cheaper than n squared whenever r sits below half of n.
Circle one: True False
What is catastrophic forgetting?
- Forgetting to save the adapter after training
- New training washing out skills the model once held
- The base weights freezing by accident
True or false: a rank r update to an n by n weight matrix can be stored as two thin matrices holding 2 times n times r numbers in total, which is cheaper than storing all n squared numbers whenever r is smaller than n divided by 2.
Circle one: True False
A task repeats daily and prompting keeps missing. The budget covers a small adapter but not a full run. What fits?
- Keep prompting and accept the misses
- Pay for a full fine-tune anyway
- Train a low-rank adapter and watch old-task checks
A full fine-tune of one layer updates a 4 by 4 weight matrix, which holds 16 numbers. A low-rank adapter instead learns two small matrices, B with shape 4 by 2 and A with shape 2 by 4, and their product BA is the update. How many numbers does the adapter store in total?
Answer: ______________
For the matrix with rows [2, 4] and [3, x] to have rank 1, the second row must be a multiple of the first. What value of x makes that true?
Answer: ______________
A matrix with rows [2, 4] and [3, x] has rank 1, so the second row is a multiple of the first. What is x?
Answer: ______________
Old-task scores start slipping midway through adapting. What is the right response?
- Train longer until the new task is perfect
- Stop or lighten training: forgetting has started
- Delete the old-task checks as outdated
A layer has a 100 by 100 weight matrix. Which storage plan for an update uses the fewest numbers?
- A full fine-tune storing the whole updated matrix
- A rank 4 adapter stored as two thin matrices
- A rank 60 adapter stored as two thin matrices
- Storing every changed entry of the matrix separately

Answer key

For grown-ups. Fold this page away before handing over the rest.

Small updates on big weights W1-mt_NaA2iGtMPc-s1

Every parameter, costing a whole copy to store · Full means full: all weights move and all must be kept.
True · Two thin factors hold n r plus r n against the full square.
New training washing out skills the model once held · Moving everything can overwrite what earlier training built.
True · The two factors hold n times r plus r times n, which is 2nr numbers. That beats n squared exactly when 2r is less than n, and real adapters use r like 8 or 16 against n in the thousands.
Train a low-rank adapter and watch old-task checks · Repeating failure repays training, and the thin update fits the budget.
16 · B has 4 times 2 equals 8 numbers and A has 2 times 4 equals 8 numbers, so the adapter stores 16 numbers, the same as the full matrix here because the rank 2 is half of 4.
6 · Rank 1 means every row is a scaled copy of one row. The second row starts with 3, which is 1.5 times 2, so x must be 1.5 times 4, which is 6.
6 · Three is 1.5 times 2, so x is 1.5 times 4.
Stop or lighten training: forgetting has started · Slipping old scores are the alarm; the checks exist to ring it.
A rank 4 adapter stored as two thin matrices · A full fine-tune stores 10000 numbers. A rank 4 adapter stores 2 times 100 times 4 equals 800. A rank 60 adapter stores 12000, which is worse than full. Rank 4 wins easily.