Full fine-tuning updates every parameter in the model. All 7 billion, 13 billion, or 70 billion of them.
Advantages:
- Maximum capacity for learning new behavior
- No architectural constraints
Disadvantages:
- Massive memory requirements
- Risk of catastrophic forgetting
- Need to store full model per task
Full fine-tuning is powerful but often overkill. Consider parameter-efficient methods first.