---
title: "Adapting a Pretrained Model: Fine-Tuning and Low-Rank Updates"
description: "Most work starts from weights someone else paid for. A full fine-tune moves every parameter and costs a whole copy to store, while a low-rank update trains a small factored correction and leaves the b"
canonical: https://lightmysky.com/learn/computing/adapting-a-pretrained-model-fine-tuning-and-low-rank-updates-mt_NaA2iGtMPc
source: https://lightmysky.com/learn/computing/adapting-a-pretrained-model-fine-tuning-and-low-rank-updates-mt_NaA2iGtMPc.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Adapting a Pretrained Model: Fine-Tuning and Low-Rank Updates

Most work starts from weights someone else paid for. A full fine-tune moves every parameter and costs a whole copy to store, while a low-rank update trains a small factored correction and leaves the base weights alone.

Subject: Computing · Area: Machine Learning · Ages 22 to 24
Page: https://lightmysky.com/learn/computing/adapting-a-pretrained-model-fine-tuning-and-low-rank-updates-mt_NaA2iGtMPc

## Ready when they can

- Compare parameter count and storage for a full fine-tune against a low-rank adapter
- Say when adapting beats prompting for a task, and when it does not
- Describe catastrophic forgetting and one measure that keeps it in check

## Lesson: Small updates on big weights

Most work starts from weights someone else paid for. A full fine-tune moves every parameter and costs a whole copy to store. A low-rank update instead trains a small factored correction and leaves the base weights alone: the update is the product of two thin matrices, B times A.

Count the storage before you choose. A full n by n update stores n squared numbers, while the factored pair stores 2 times n times the rank. The adapter wins whenever the rank sits below half the size. A 4 by 4 layer with rank 2 stores 16 numbers, the same as full, while a 100 by 100 layer with rank 4 stores 800 against 10000, and rank 60 stores 12000, worse than full. If a matrix has rank 1, every row is a multiple of the first: with rows [2, 4] and [3, x], 3 is 1.5 times 2, so x is 1.5 times 4, which is 6.

**Example.** Pick prompting when the need is one-off or the budget is thin, since no training runs at all. Pick adapting when the task repeats often enough to repay training and prompting keeps missing. Between adapters, pick a low rank when storage and forgetting matter, and pay for fuller moves only when the task demands capacity the thin update cannot hold.

**Tip.** Full moves risk catastrophic forgetting: new training washes out old skills. Keep old-task checks running as you train, freeze more and train less where you can, and stop when the old scores start to slip.

**Recap.** Train a thin factored correction when it suffices, count 2 n r against n squared, and watch old tasks while you adapt.

## Practice

12 questions on this page, each with its working shown.

## Needs first

- [Regularisation: Charging for Complexity](https://lightmysky.com/learn/computing/regularisation-charging-for-complexity-mt_RVgsDP0XzP)
- [Scaling Laws and the Compute Budget](https://lightmysky.com/learn/computing/scaling-laws-and-the-compute-budget-mt_tzl0sQblIM)

## Opens up

- [Generative Models: Learning a Distribution You Can Sample](https://lightmysky.com/learn/computing/generative-models-learning-a-distribution-you-can-sample-mt_3CHsmpWVX5)
- [Learning from Human Preferences: Reward Models and Policy Optimisation](https://lightmysky.com/learn/computing/learning-from-human-preferences-reward-models-and-policy-optimisation-mt_yA1ud53Oy_)