LLM Lab

Fine-tuning

Pretraining teaches a model language. Fine-tuning teaches it your task, for a fraction of the cost.

Where this sits in the life of an LLM
Pretraining at scale→Supervised fine-tuning→Preference tuning (RLHF, DPO)→Serving (KV cache, quantisation)

Continue training on your data

A pretrained model already knows a lot. Fine-tuning keeps training it on a small dataset of examples for your task: classifying support tickets, following instructions, answering in a house style. The loss on that data falls quickly because the model is not starting from scratch.

The price: forgetting

The same updates that help your task can damage what the model knew. Push too far and loss on general text climbs. Parameter-efficient methods such as LoRA freeze the base weights and train a small add-on, which limits the damage and cuts memory.

The lab uses made-up numbers with the right shape. Drag the sliders and watch the two losses trade off.

A worked example

A 7-billion-parameter model has about 4,096 × 4,096 = 16.8 million numbers in each attention projection. Full fine-tuning updates all of them, plus the optimiser keeps extra state for each. LoRA with rank 8 adds two matrices of 4,096 × 8 and 8 × 4,096, 65,536 numbers, and trains only those. The update B·A still has the full 4,096 × 4,096 shape, but it can only change the weights along 8 directions.

full fine-tune16,777,216 numbers (one 4096 × 4096 matrix)
LoRA, r = 865,536 numbers (0.39%)
Trainable numbers for one 4096 × 4096 weight matrix. LoRA trains two thin matrices, 4096 × 8 and 8 × 4096, and leaves the original weights frozen.

In practice

Supervised fine-tuning on instruction and answer pairs is what turns a base model that continues text into a chat model that follows requests. A few thousand high-quality examples can change behaviour noticeably. Keeping some general data in the mix, using a small learning rate and stopping early all limit forgetting.

Easy to confuse

Check yourself · 1 of 3

What does fine-tuning start from?

Check yourself · 2 of 3

After many steps of fine-tuning at a high learning rate, the model gets worse at general tasks. What is this called?

Check yourself · 3 of 3

What does LoRA train?

Read more in the books

  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 6, “Finetuning for Text Classification” and Ch. 7, “Finetuning to Follow Instructions”.
  • AI Engineering, Chip Huyen: Ch. 7, “Finetuning”.
  • Build a Large Language Model (From Scratch), Sebastian Raschka: Appendix E, “Parameter-efficient Finetuning with LoRA”.
All books
Labloss on new data
How to read this
  • The chart shows two losses during fine-tuning: on your task (teal) and on general text (orange). The numbers are illustrative, but the shape is real.
  • Learning rate, steps and method change the curves; the readout gives start and end values and flags forgetting.
  • Below: LoRA’s shapes. The faded grid is the frozen weight W; B and A are the small trainable matrices. The rank slider changes their width.
your taskgeneral text
Method